Some important points consider when developing an Apache Storm topology

-

  1. Kafka Consumer lag ---kafka tool to monitor
  2. Consumer group configuration of flux weaving YAML
  3. Storm UI metrics such as capacity < 1
  4. Spout parallelism should be equal to partition count of relevant topic
  5. Bolt parallelism should be increased according to the following equation
  6.                      **Throughput = Executor count * 1000/(process latency) * Capacity**
  7. Topology workers and supervisor server configurations accordingly
  8. Hbase database connection bottlenecks in Kerberized environment.
  9. Other Storm-related configuration parameters to be set in flux topology YAML or Ambari
  10. Maintain Tuple anchoring to guarantee the message processing through the downstream


References,

http://storm.apache.org/releases/1.0.6/Guaranteeing-message-processing.html
http://storm.apache.org/releases/1.0.6/Understanding-the-parallelism-of-a-Storm-topology.html

Comments

Post a Comment

Popular posts from this blog

Let's try to build scrum masters/project managers/software architects/even a company with training AI models

-
The concept: The basic concept is to build trained AI model for each role separately in scalable fashion within a private cloud. As an example to train a scrum master as an AI component, we have to probably feed good knowledge of day to day tasks of a scrum master. Apart from that the model need to aware how scrum master manage the SCRUM board in the relevant tool stack eg: JIRA. One pod within the Kubernetes cluster can be considered as a one scrum master per a team. Likewise we have some sort of ability to maintain a good knowledge base as a source of training data. Apart from these steps, we need to think about how to simulate speaking and handling, commanding and escalating in the meetings, The end goal of this conceptual project is to fully automate the software life cycle and fine grain the each responsibilities of scrum masters, project managers, software architects, CTO and CEO. however the each role should be backed by a single human being. The ultimate goal of this project to...
Read more

TCP Ports list

-
TCP Ports list (3498 ports in list) 1 tcpmux TCP Port Service Multiplexer 2 compressnet Management Utility 3 compressnet Compression Process 5 rje Remote Job Entry 7 echo Echo 9 discard Discard 11 systat Active Users 13 daytime Daytime (RFC 867) 15 netstat netstat 15/tcp 17 qotd Quote of the Day 18 msp Message Send Protocol 19 chargen Character Generator 20 ftp-data File Transfer [Default Data] 21 ftp File Transfer [Control] 22 ssh SSH Remote Login Protocol 23 telnet Telnet 24 priv-mail any private mail system 25 smtp Simple Mail Transfer 27 nsw-fe NSW User System FE 29 msg-icp MSG ICP 31 msg-auth MSG Authentication 33 dsp Display Support Protocol 35 priv-print any private printer server 37 time Time 38 rap Route Access Protocol 39 rlp Resource Location Protocol 41 graphics Graphics 42 name Host Name Server 43 nicname Who Is 44 mpm-flags MPM FLAGS Protocol 45 mpm Message Processing Module [recv] 46 mpm-snd MPM [default send] 47 ni-ftp NI FTP 48 audi...
Read more

Problem Solving: Allotment calculator

-
 Problem Statement: Suppose the Government plans to issue up to $10,000 of Savings Bonds. Four individuals applied for a total of $18,000 of Savings Bonds: A ($2,000), B ($4,000) C ($5,500) and D ($6,500). The available bonds will be spread out among as many investors as possible in the following manner: Applications are filled in denominations of $500 upwards. After Round 4, $8,000 of Savings Bonds have been allotted, and Investor A's application has been fully met. $2,000 of Savings Bonds are left. In Round 5, $1,500 of Savings Bonds are allotted. In Round 6, the remaining $500 is insufficient to fill all applications. One person among Investor B, Investor C and Investor D is randomly allotted the remaining $500. In this case, Investor C gets it. The cutoff amount in this case is $2,500. In the final allotment: Investor A is allotted $2,000. Investor B and Investor D get $2,500 each. Investor C gets $3,000. limitations: 1. The minimum individual investment amount is $500 2. The m...
Read more