Uh oh!
There was an error while loading. Please reload this page.
Disruptor Queue Batching (DON'T MERGE YET) - #704
Conversation
revans2
commented
Sep 18, 2015
I have done a lot of testing on this, and we need batching, but this is not the correct way to do it. I have a proof of concept with some micro benchmarks, but I am going to probably do a pull request for a fixed version shortly. |
mjsax
commented
Sep 21, 2015
"but this is not the correct way to do it" Do you refer to batching I work on or you own changes? |
revans2
commented
Sep 21, 2015
@mjsax Sorry about the confusion. I have been playing around with batching and I was not referring to your code changes. Disruptor offers two different ways of batching. One is on the read side, which this patch is about. That did not work in improving the efficiency/throughput of the queue. Instead the batching has to happen on the enqueue side to reduce the load it is placing on the OS in signaling/waking up the worker thread. I have some micro benchmarks that show this, and how much of a performance improvement we can potentially expect to see. I have been able to do over 2,000,000 word count sentences per second on my Mac Book Pro laptop, where as storm is much lower. I have not finished collecting numbers, and I am working on some code to do this in storm itself so we can see the impact it can have. The main reason why I like this approach over batching at the spout is that the code is isolated to just the queue itself, and should not have an impact of the rest of storm, except potentially to remove some of the other code changes that were put into storm initially to try and improve the throughput. But I really want to have a working prototype with performance numbers before I try to compare the two approaches. |
This is mostly a proof of concept, as a comparison to #694. They may complement each other. Of there may be a lot of overlap between them. I want to run some more tests. Currently I have a few benchmarks based on this branch showing the CPU savings, and increased performance by doing some batching.
The following were all using the modified version of wordcount on my branch. They were run on my Mac Book Pro with 8 cores. The first number is the max spout pending. The second number is the batch size, and the third number is the disruptor queue timeout.
Also before merging anything I want to explore reducing the number of threads and queues. Because if context switching is killing us having less threads should make it a lot better, and may reduce the need for this a lot.
wc-1-1-100 8 cores System - 30% User - 60% Idle - 10%
wc-1-100-100 8 cores System - 4% User - 5% Idle - 91%
wc-10-1-100 8 cores System - 30% User - 65% Idle - 6%
wc-10-2-100 8 cores System - 30% User - 65% Idle - 6%
wc-10-3-100 8 cores System - 30% User - 65% Idle - 6%
wc-200-1-100 8 cores System - 50% User - 50% Idle - 0%
wc-200-100-100 8 cores System - 20% User - 30-40% Idle - 40-50%
wc-400-1-100 8 cores System - 50% User - 50% Idle - 0%
wc-400-100-100 8 cores System - 25% User - 40-50% Idle - 20-30%
wc-1000-200-100 8 cores System - 28% User - 68% Idle - 3%
wc-1000-1-100 8 cores System - 50% User - 50% Idle - 0%