Skip to content

mark FPs while writing subgraph files #379

Description

@yakra

Single-threaded datacheck FP matching is already pretty efficient:

  • Currently roughly 2/3 of datacheck entries are FPs.
  • datacheckerrors->entries is sorted asciibetically, and datacheckfps.csv is sorted pretty darn close to asciibetically.
  • Thus when an FP is matched, it's found & erased right off the beginning of the FP list, very little searching necessary.
  • When there's no FP entry to match, we do search the whole list, but at least it's continually shrinking.

Still, it's about 5% of total execution time on lab2.
It doesn't lend itself well to multithreading:

  • How to avoid data races when reading & erasing from the datacheckfps list?
  • We could use a mutex, but the performance penalty would probably be a big one.
  • Attempting to avoid this by splitting datacheckerrors->entries and datacheckfps into n chunks to feed to n threads hurts our ability to "cross off" items right away.

A better approach?
Some datacheck errors are flagged during HighwayGraph construction, but after the structure is built out, we're all set.

  • Rather than lose efficiency with any of the above approaches, we can process datacheck FPs in the background as we begin to write subgraphs.
  • Spawn one less SubgraphThread to start, and have our datacheck FP thread spawn one when it finishes up, just like MasterTmgThread does.
  • This will require getting a bit more clever when threading is enabled but -t 1 is specified.

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions