Spark RnD

-

The following stack of tools need to be configured once you downloaded and setup java 8 and Scala on your laptop. Remember to have a laptop with a higher RAM above 8 GB to set up this Big Data cluster on your machine.
  1. Configure Hadoop
  2. Configure Hbase
  3. Configure Kafka with Zookeeper
  4. Download Spark version which prebuilt with your hadoop version
  5. extract it and set %SPARK_HOME%\bin to Path variable
  6. create spark-env.cmd file in %SPARK_HOME%\conf folder
  7. Take the classpaths of your hadoop system by executing command on your windows command prompt [%HADOOP_HOME%\bin\hadoop classpath]
  8. copy the output and add that path to [set SPARK_DIST_CLASSPATH=] section of spark-env.cmd file
  9. Once you set with these all you are ready to execute spark-shell command


Comments

Post a Comment

Popular posts from this blog

Let's try to build scrum masters/project managers/software architects/even a company with training AI models

-
The concept: The basic concept is to build trained AI model for each role separately in scalable fashion within a private cloud. As an example to train a scrum master as an AI component, we have to probably feed good knowledge of day to day tasks of a scrum master. Apart from that the model need to aware how scrum master manage the SCRUM board in the relevant tool stack eg: JIRA. One pod within the Kubernetes cluster can be considered as a one scrum master per a team. Likewise we have some sort of ability to maintain a good knowledge base as a source of training data. Apart from these steps, we need to think about how to simulate speaking and handling, commanding and escalating in the meetings, The end goal of this conceptual project is to fully automate the software life cycle and fine grain the each responsibilities of scrum masters, project managers, software architects, CTO and CEO. however the each role should be backed by a single human being. The ultimate goal of this project to
Read more

TCP Ports list

-
TCP Ports list (3498 ports in list) 1 tcpmux TCP Port Service Multiplexer 2 compressnet Management Utility 3 compressnet Compression Process 5 rje Remote Job Entry 7 echo Echo 9 discard Discard 11 systat Active Users 13 daytime Daytime (RFC 867) 15 netstat netstat 15/tcp 17 qotd Quote of the Day 18 msp Message Send Protocol 19 chargen Character Generator 20 ftp-data File Transfer [Default Data] 21 ftp File Transfer [Control] 22 ssh SSH Remote Login Protocol 23 telnet Telnet 24 priv-mail any private mail system 25 smtp Simple Mail Transfer 27 nsw-fe NSW User System FE 29 msg-icp MSG ICP 31 msg-auth MSG Authentication 33 dsp Display Support Protocol 35 priv-print any private printer server 37 time Time 38 rap Route Access Protocol 39 rlp Resource Location Protocol 41 graphics Graphics 42 name Host Name Server 43 nicname Who Is 44 mpm-flags MPM FLAGS Protocol 45 mpm Message Processing Module [recv] 46 mpm-snd MPM [default send] 47 ni-ftp NI FTP 48 audi
Read more

Climate Change News from all over the world through API

-
Image
 Free Public API - Climate Change News Hello developers, here is a simple API to include in the product that supports you to retrieve climate change news around the globe within a few seconds. Rapid API - climate change news This is a really simple to use API which helps you to find out the latest climate change news around the globe within seconds. it provides two endpoints to retrieve the climate change news: Retrieve all the climate change news at once - eg: http://{rapid-api-host}/api/news Retrieve all the climate change news from individual sources - eg: http://{rapid-api-host}/api/news/guardian for now, we are supporting the following individual sources - I. guardian II. thetimes III. telegraph IV. nasa V. dailymail VI. nypost VII. cnn Remember to include the required headers by rapid API in order to make everything smooth. Thank you for consuming the climate change news API for your valuable product.
Read more