Sunday, January 27, 2019

Some important points consider when developing an Apache Storm topology


  1. Kafka Consumer lag ---kafka tool to monitor
  2. Consumer group configuration of flux weaving YAML
  3. Storm UI metrics such as capacity < 1
  4. Spout parallelism should be equal to partition count of relevant topic
  5. Bolt parallelism should be increased according to the following equation
  6.                      **Throughput = Executor count * 1000/(process latency) * Capacity**
  7. Topology workers and supervisor server configurations accordingly
  8. Hbase database connection bottlenecks in Kerberized environment.
  9. Other Storm-related configuration parameters to be set in flux topology YAML or Ambari
  10. Maintain Tuple anchoring to guarantee the message processing through the downstream


References,

http://storm.apache.org/releases/1.0.6/Guaranteeing-message-processing.html
http://storm.apache.org/releases/1.0.6/Understanding-the-parallelism-of-a-Storm-topology.html

No comments:

Post a Comment

Let's try to build scrum masters/project managers/software architects/even a company with training AI models

The concept: The basic concept is to build trained AI model for each role separately in scalable fashion within a private cloud. As an examp...