Uh oh!
There was an error while loading. Please reload this page.
HBASE-22634 : Improve performance of BufferedMutator - #343
Conversation
busbey
commented
Jun 26, 2019
Is there something specific to branch-2.1 for this that makes it irrelevant or substantively different for the master branch? |
sbarnoud
commented
Jun 26, 2019
via email
If i didn’t mistake BufferedMutatorImpl doesn’t exist anymore on master.
I build this on my cluster on a 2.1.4 ... … Le 26 juin 2019 à 20:00, Sean Busbey ***@***.***> a écrit :
Is there something specific to branch-2.1 for this that makes it irrelevant or substantively different for the master branch?
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub, or mute the thread. |
Apache-HBase
commented
Jun 26, 2019
💔 -1 overall
This message was automatically generated. |
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Apache9
commented
Jun 27, 2019
Yes, it is gone on master. We reimplement the sync client based on async client on master so the BufferedMutator and AsyncProcess are both removed. |
Apache-HBase
commented
Jun 27, 2019
💔 -1 overall
This message was automatically generated. |
Apache-HBase
commented
Jun 27, 2019
💔 -1 overall
This message was automatically generated. |
Apache-HBase
commented
Jun 27, 2019
💔 -1 overall
This message was automatically generated. |
Apache-HBase
commented
Jun 27, 2019
💔 -1 overall
This message was automatically generated. |
Apache-HBase
commented
Jun 27, 2019
💔 -1 overall
This message was automatically generated. |
Apache-HBase
commented
Jun 27, 2019
💔 -1 overall
This message was automatically generated. |
Apache-HBase
commented
Jun 27, 2019
💔 -1 overall
This message was automatically generated. |
Apache-HBase
commented
Jun 28, 2019
💔 -1 overall
This message was automatically generated. |
Apache-HBase
commented
Jun 28, 2019
💔 -1 overall
This message was automatically generated. |
Apache-HBase
commented
Jun 28, 2019
💔 -1 overall
This message was automatically generated. |
Apache-HBase
commented
Jun 28, 2019
💔 -1 overall
This message was automatically generated. |
Apache-HBase
commented
Jun 28, 2019
💔 -1 overall
This message was automatically generated. |
Apache-HBase
commented
Jun 28, 2019
💔 -1 overall
This message was automatically generated. |
Apache-HBase
commented
Jun 28, 2019
🎊 +1 overall
This message was automatically generated. |
Apache-HBase
commented
Jun 28, 2019
🎊 +1 overall
This message was automatically generated. |
sbarnoud
commented
Jul 1, 2019
A spark job with 7 executors, 4 cores each, has an average throughput of more than 500000 insert/s with optimal tuning of:
Where the input dataset is well balanced on the rowkey,on a table presplitted on 26 region servers, with a tiny average record size (500 bytes average). The source of the Spark job is as simple as: |
Apache-HBase
commented
Jul 12, 2019
💔 -1 overall
This message was automatically generated. |
Apache-HBase
commented
Jul 15, 2019
💔 -1 overall
This message was automatically generated. |
Apache-HBase
commented
Jul 16, 2019
💔 -1 overall
This message was automatically generated. |
Apache-HBase
commented
Aug 7, 2019
🎊 +1 overall
This message was automatically generated. |
| disableWriteBufferPeriodicFlush(); | ||
| ap.waitAllSlot(); | ||
| try { | ||
| // Let time to the periodic flush thread to exit (task are finished, but not the code after) |
There was a problem hiding this comment.
What is the problem if the code after is not finished? I'm a bit nervous that we just set a magic 5ms sleep time here...
| private final boolean cleanupPoolOnClose; | ||
| private volatile boolean closed = false; | ||
| private final AsyncProcess ap; | ||
| private List<AsyncRequestFuture> asfList; |
There was a problem hiding this comment.
Does this field need to be a class member? It seems that we only use it in the doFlush method.
| if (flushAll || writeBufferSize == 0) { | ||
| // if we have setWriteBufferPeriodicFlushTimeoutMs we may have concurrent update | ||
| List<AsyncRequestFuture> waitList; | ||
| synchronized(asfList) { |
There was a problem hiding this comment.
Oh, I see, BufferedMutator is thread safe, so multiple threads can enter this method at the same time and we want them to share the safe asfList, so it should be class member.
| } | ||
| if (nbRemoved == 0) { | ||
| try { | ||
| Thread.sleep(1); |
There was a problem hiding this comment.
So this means we will do a busy waiting here? This is not always the best choice, maybe we should provide a configurable way to wait here, the default one should be the typical wait/notify, and if do not care wasting the CPU cycles but only want the maximum throughput, you can use busy waiting.
| } else { | ||
| // Do some cleanup in asfList to decrease memory | ||
| int nbRemoved = 0; | ||
| while (asfList.size() >= maxThreads*4) { |
There was a problem hiding this comment.
Here we get size out of synchornized?
Apache-HBase
commented
Aug 7, 2019
🎊 +1 overall
This message was automatically generated. |
Apache-HBase
commented
Aug 7, 2019
🎊 +1 overall
This message was automatically generated. |
Apache-HBase
commented
Aug 7, 2019
🎊 +1 overall
This message was automatically generated. |
Apache-HBase
commented
Aug 8, 2019
🎊 +1 overall
This message was automatically generated. |
Apache-HBase
commented
Aug 8, 2019
🎊 +1 overall
This message was automatically generated. |
Apache-HBase
commented
Aug 8, 2019
🎊 +1 overall
This message was automatically generated. |
Apache-HBase
commented
Aug 9, 2019
🎊 +1 overall
This message was automatically generated. |
Apache-HBase
commented
Aug 9, 2019
🎊 +1 overall
This message was automatically generated. |
Apache-HBase
commented
Aug 11, 2019
🎊 +1 overall
This message was automatically generated. |
Apache-HBase
commented
Aug 12, 2019
💔 -1 overall
This message was automatically generated. |
Apache-HBase
commented
Aug 12, 2019
🎊 +1 overall
This message was automatically generated. |
Apache-HBase
commented
Aug 13, 2019
🎊 +1 overall
This message was automatically generated. |
Apache-HBase
commented
Aug 13, 2019
🎊 +1 overall
This message was automatically generated. |
Apache-HBase
commented
Aug 13, 2019
🎊 +1 overall
This message was automatically generated. |
Apache-HBase
commented
Aug 14, 2019
🎊 +1 overall
This message was automatically generated. |
Apache-HBase
commented
Aug 14, 2019
🎊 +1 overall
This message was automatically generated. |
Apache-HBase
commented
Aug 14, 2019
🎊 +1 overall
This message was automatically generated. |
Apache9
commented
Aug 14, 2019
Close since no big progress here. Can open a new PR which targets to branch-2 if you have time @sbarnoud . Or I will try to port the patch if I have time. |
sbarnoud
commented
Aug 14, 2019
via email
I’m on holiday.
Anyway, i’m just surprised you close a performance patch without doing any performance test.
So, i will keep the improvement for me ;-) … Le 14 août 2019 à 08:24, Duo Zhang ***@***.***> a écrit :
Close since no big progress here. Can open a new PR which targets to branch-2 if you have time @sbarnoud . Or I will try to port the patch if I have time.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread. |
Apache9
commented
Aug 14, 2019
Performance is not everything. To be honest, the patch is still not qualified enough to be merged. Please see my last comments, doing busy wait( sleep(1) ) on a critical path is usually a no no, especially that you are waiting for external I/O to complete rather than an in memory operation... Anyway, I got your point that the current implementation of BufferMutator could be optimized to increase the concurrency. Will take a look if I have time. Thanks. |
sbarnoud
commented
Aug 14, 2019
via email
The sleep is on a « backpressure » path. Of course, we can replace it with a notify.
But, when a system is overloaded it is often better to let it some time to unbuffer and not to overload it as soon as possible.
This kind of decision can only be taken after a benchmark and not just after a code review … Le 14 août 2019 à 10:59, Duo Zhang ***@***.***> a écrit :
Performance is not everything. To be honest, the patch is still not qualified enough to be merged. Please see my last comments, doing busy wait( sleep(1) ) on a critical path is usually a no no, especially that you are waiting for external I/O to complete rather than an in memory operation...
Anyway, I got your point that the current implementation of BufferMutator could be optimized to increase the concurrency. Will take a look if I have time.
Thanks.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread. |
Apache9
commented
Aug 14, 2019
Not sure why you keep saying 'benchmark' since I never challenge the performance improvement of your patch? |
sbarnoud
commented
Aug 14, 2019
via email
What i’m saying is that the sleep may provide better performance than a notify because it is called when the server latency increases (probably because it is overloaded).
In that case, it may be better to give some time to the server to dequeue queries instead of submitting as soon as possible new one.
Only a benchmark could differentiate the 2 approach, not a code review … Le 14 août 2019 à 11:42, Duo Zhang ***@***.***> a écrit :
Not sure why you keep saying 'benchmark' since I never challenge the performance improvement of your patch?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread. |
Apache9
commented
Aug 14, 2019
Please read my comment carefully
I'm not telling that sleep will have a worse performance right? Just make it configurable. Maximum throughput is not always the best choice. User may have a mixed workload and do not want to waste too many CPU cycles here.
I can not get the point here, we must wait until the server dequeue queries right? Otherwise we do not need to introduce the complicated slot logic... Thanks. |
sbarnoud
commented
Aug 29, 2019
No 'while/sleep' nor 'wait/notify' needed anymore. |
As requested in the Jira HBASE-22634