(STORM-1674) Idle KafkaSpout consumes more bandwidth than needed

[moesol]

• Allows minBytes in fetch request to be configured
  from KafkaConfig.fetchMinBytes.
• Defaults new configuration KafkaConfig.fetchMinBytes to 1.
• Defaults fetchMaxWait to 100ms instead of 10000ms.

Discovered 30 megabits of traffic flowing between a set of KafkaSpouts
and our kafka servers even though no Kafka messages were moving.
Using the wireshark kafka dissector, we were able to see that
each FetchRequest had maxWait set to 10000
and minBytes set to 0. When binBytes is set to 0 the kafka server
responds immediately when there are no messages. In turn the KafkaSpout
polls without any delay causing a constant stream of FetchRequest/
FetchResponse messages. Using a non-KafkaSpout client had a similar
traffic pattern with two key differences

1. minBytes was 1
2. maxWait was 100
   With these FetchRequest parameters and no messages flowing,
   the kafka server delays the FetchResponse by 100 ms. This reduces
   the network traffic from megabits to the low kilobits. It also
   reduced the CPU utilization of our kafka server from 140% to 2%.

Hopefully the risk of this change is low because
the old behavior can be restored using the API by setting
KafkaConfig.fetchMaxWait to 10000
KafkaConfig.fetchMinBytes to 0

[harshach]

@moesol there is patch that merged into 1.0 release https://github.com/apache/storm/pull/1309/files . Does that changes work for you. Any reason for the MaxWait to be at 100

[moesol]

Without the MaxWait being set to 100, it is set to 10000, which means that the bolt will waiting for 10 seconds when no messages are available. On the 1.x branch, I was going to use:

• public int fetchMaxWait = FetchRequest.DefaultMaxWait();
• public int fetchMinBytes = FetchRequest.DefaultMinBytes();

to get Kafka default behavior of 100/1.