held: 0.186 (0.191, 0.187, 0.186)
wait: 0.038 (0.046, 0.041, 0.039)
<140287451305792>
held: 0.015 (0.029, 0.017, 0.015)
wait: 0.002 (0.002, 0.002, 0.002)
<140284185433856>
held: 0.065 (0.061, 0.064, 0.065)
wait: 0.016 (0.015, 0.017, 0.016)
<140284540389120>
held: 0.0 (0.0, 0.0, 0.0)
wait: 0.0 (0.0, 0.0, 0.0)
<140284590728960>
held: 0.006 (0.006, 0.006, 0.006)
wait: 0.002 (0.002, 0.002, 0.002)
<140284599121664>
held: 0.006 (0.006, 0.006, 0.006)
wait: 0.002 (0.002, 0.002, 0.001)
<140284759570176>
held: 0.006 (0.008, 0.007, 0.007)
wait: 0.002 (0.001, 0.001, 0.002)
<140284751177472>
held: 0.006 (0.006, 0.006, 0.006)
wait: 0.002 (0.001, 0.001, 0.002)
<140283956950784>
held: 0.006 (0.006, 0.006, 0.006)
wait: 0.001 (0.001, 0.001, 0.001)
<140283948558080>
held: 0.006 (0.006, 0.006, 0.006)
wait: 0.001 (0.001, 0.001, 0.001)
<140283940165376>
held: 0.006 (0.006, 0.006, 0.006)
wait: 0.002 (0.002, 0.002, 0.002)
<140283931772672>
held: 0.006 (0.006, 0.006, 0.006)
wait: 0.002 (0.001, 0.002, 0.002)
<140283923379968>
held: 0.006 (0.006, 0.006, 0.006)
wait: 0.002 (0.001, 0.002, 0.002)
<140283914987264>
held: 0.006 (0.007, 0.007, 0.007)
wait: 0.002 (0.001, 0.002, 0.002)
<140283295561472>
held: 0.006 (0.006, 0.006, 0.006)
wait: 0.002 (0.002, 0.002, 0.002)
<140283287168768>
held: 0.006 (0.006, 0.006, 0.006)
wait: 0.002 (0.002, 0.002, 0.002)
<140283278776064>
held: 0.006 (0.006, 0.007, 0.006)
wait: 0.002 (0.002, 0.002, 0.002)
<140283270383360>
held: 0.006 (0.006, 0.006, 0.006)
wait: 0.001 (0.001, 0.001, 0.001)
<140283261990656>
held: 0.006 (0.006, 0.006, 0.006)
wait: 0.001 (0.002, 0.002, 0.001)
<140283253597952>
held: 0.006 (0.006, 0.006, 0.006)
wait: 0.002 (0.001, 0.001, 0.001)
<140283245205248>
held: 0.0 (0.0, 0.0, 0.0)
wait: 0.0 (0.0, 0.0, 0.0)
<140282691581696>
held: 0.001 (0.0, 0.001, 0.001)
wait: 0.001 (0.001, 0.001, 0.001)
<140282683188992>
held: 0.002 (0.002, 0.002, 0.002)
wait: 0.001 (0.001, 0.001, 0.001)
<140282674796288>
held: 0.001 (0.001, 0.001, 0.001)
wait: 0.003 (0.012, 0.004, 0.003)
This is a dump of some of the performance experiments. It's part of a larger issue of performance setup and best practices for dask/sgkit and genetic data. The goal is to share the findings and continue the discussion.
Where not otherwise stated, the test machine is a GCE VM, 16 cores and 64GB of memory, 400 SPD. Dask cluster is a single node process based. If the data is read from GCS, the bucket is in the same region as the VM:
Specs/libs
The issue with suboptimal saturation was originally reported for this code:
With local input, performance graph:
It's pretty clear the cores are well saturated. I also measure GIL, GIL was held for 13% of time and waited on for 2.1%, with each worker thread (16 threads) holding it for 0.7% and waiting for 0.1% of time.
For GCS input (via fsspec):
GIL summary: GIL was held for 18% of time and waited on for 3.8%, with each worker thread (16 threads) holding it for 0.6% and waiting for 0.2% of time, with one thread holding GIL for 6.5% and waiting 1.6% time.
Details
It's clear that the CPU usage is lower, and not fully saturated, GIL wait time is a bit up (with a concerning spike in one thread). With remote/fsspec input, we have the overhead of data decryption and potential network IO overhead (tho it doesn't seem like we hit network limits).