improved speed & parallelism when reading WPTs

[yakra]

Closesyakra#111. This picks up where #280 left off, improving multi-threaded performance Reading waypoints for all routes by moving unique location counting & colocation detection (parts of these improvements also found their way into siteupdate.py, removing some redundant iteration down thru quadtree nodes) to where it's not necessary to keep the quadtree locked, and then finally reworking WaypointQuadtree::mtx to lock individual nodes, allowing different branches of the quadtree to be refined in parallel.

Graphs were produced by grepping the beginning and ending timestamps for a given task in siteupdate.log files (3000 in total!) produced by the different binaries, averaging the times of 10 different runs each.

• The blue line shows time reading waypoints for all routes before no-mutex NMPs #280, back when NMP detection was still included in this task.
• The 3 red-orange lines represent the changes in no-mutex NMPs #280, when NMP detection was carved out to be done separately. The solid line, the sum of the two dotted/dashed lines, is the time spent performing both tasks, and thus our basis of comparison against the blue line.
• The remaining (yellow, green, purple) lines are for reading WPTs only, sans NMPs, and should be compared to the upper dashed line, "readWPT".

---

IMO our sanest & nicest-looking graph overall. Lab2's 20MB L3 cache helps it scale up to a higher number of threads, making it just barely faster at this task than the faster-clocked lab1.

---

Here we see our bottleneck(s) as a big factor in 30d84fd, and the effects of removing it.
I would assume that if our only bottleneck were the mutex, with nothing else at play, our time would make an asymptotic approach to the time required to do the effectively single-threaded bits, and then stay relatively flat. What we see is efficiency peaking early, and things quickly getting worse. My suspicion = memory; lab3 has 15MB L3 cache (edit: per CPU), and 1333 MHz RAM vice lab2's 1600. When we have to start going (earlier) to slower main memory for our data, it's all over.
Though lab3 is a dual CPU machine with a RAM bank for each CPU socket, this does not seem to double its memory bandwidth. Lab3 has the highest raw GHz*#cores, but that can't overcome its other limitations.
By contrast, 10f9c92 calms things down quite a bit. Once NMPs are out of the equation, we're working with a smaller data set at once, and can keep more of what we're using in cache.

---

Raw clock speed gives lab1 an initial edge, but it starts to feel the slowdown a bit earlier. 8MB L3 cache here.

---

Wow. Worst of both worlds on lab1.5: 8MB L3 cache + 1333 MHz RAM clock = ouch!
What we don't see here: While expectedly slower at this particular task, lab1.5 runs siteupdate faster overall than lab1 (speed records are 42.5s & 48.5s respectively). This despite a Xeon E3-1230 CPU (vice an E3-1230 v2) with slightly slower clock speed and 1333 MHz RAM instead of 1600. I've not yet checked out what tasks account for the time savings.
Potential factors: Different OS. Different chipset?

---

And now last, but not to be the least, BiggaTomato, my daily driver desktop. The 30d84fd line looks a bit funky here, but it doesn't appear to be an artifact of low sample size; the data checks out. Weird.

---

If you'd like, I'll give an extra run thru the stats, logs and graphs and make sure everything checks out compared to what's in the master branch now.

[jteresco]

I'm willing to give the python version a shot with tonight's update. Reasonable?

[yakra]

Reasonable, yes, though I'd hate for any unintended glitches to sneak into the "official" commit history.
I was thinking more of the Python version when I made that comment.

[yakra]

Gimme maybe 30 minutes? ;)

[jteresco]

I'm going to give it a shot.

[jteresco]

Sure. Comment here or email when you're ready for me to launch a site update.

[yakra]

Will comment when ready. Running both versions of siteupdate.py.
I have a shell script that automates comparing stats, logs, nmp_merged files, and graphs. Makes that all a lot easier.

[yakra]

LOL nevermind! ;)
Noticed I was running siteupdate.py with one thread as per my SOP, decided that the "full abundance of caution route" would be to use 4 threads, and considered starting over...

[yakra]

Ready!

[jteresco]

And launched, thanks.

[yakra]

Python DIFFs are good.

[yakra]

C++ DIFFs good too.

Edited the OP to add:

Though lab3 is a dual CPU machine with a RAM bank for each CPU socket, this does not seem to double its memory bandwidth.

[yakra]

ping @seaweedfactory@FL0gnap -- pretty pictures!