📦 Crateification

[oeb25]

This is an experiment to see how splitting core into multiple crates will affect compile times. Whether or not we want this structure is very TBD.

Currently contains the commits of #111.

flowchart TB
alice --> stdx
alice --> stract_config
alice --> stract_llm
collector --> schema
collector --> simhash
collector --> stdx
collector --> stract_config
crawler --> distributed
crawler --> hyperloglog
crawler --> kv
crawler --> sonic
crawler --> stdx
crawler --> stract_config
crawler --> warc
crawler --> webgraph
crawler --> webpage
entity_index --> imager
entity_index --> kv
entity_index --> stdx
entity_index --> tokenizer
imager --> distributed
imager --> kv
imager --> stdx
mapreduce --> distributed
mapreduce --> sonic
naive_bayes --> stdx
schema --> stdx
schema --> tokenizer
simhash --> tokenizer
spell --> schema
spell --> stdx
spell --> stract_query
stract_cli --> stract_config
stract_cli --> stract_core
stract_cli --> webgraph
stract_config --> distributed
stract_core --> alice
stract_core --> collector
stract_core --> crawler
stract_core --> distributed
stract_core --> entity_index
stract_core --> executor
stract_core --> hyperloglog
stract_core --> imager
stract_core --> kuchiki
stract_core --> kv
stract_core --> mapreduce
stract_core --> naive_bayes
stract_core --> optics
stract_core --> schema
stract_core --> simhash
stract_core --> sonic
stract_core --> spell
stract_core --> stdx
stract_core --> stract_config
stract_core --> stract_llm
stract_core --> stract_query
stract_core --> tokenizer
stract_core --> warc
stract_core --> webgraph
stract_core --> webpage
stract_llm --> stdx
stract_query --> optics
stract_query --> schema
stract_query --> stdx
tokenizer --> stdx
webgraph --> executor
webgraph --> hyperloglog
webgraph --> kv
webgraph --> stdx
webpage --> kuchiki
webpage --> naive_bayes
webpage --> schema
webpage --> simhash
webpage --> stdx
webpage --> tokenizer
webpage --> webgraph

Loading

[oeb25]

Current cargo check-incremental improvements:

Command	Mean [s]	Min [s]	Max [s]	Relative
update-deps	2.414 ± 0.020	2.389	2.454	1.56 ± 0.03
crateification	1.543 ± 0.021	1.506	1.587	1.00

And for cargo build-incremental:

Command	Mean [s]	Min [s]	Max [s]	Relative
update-deps	8.001 ± 0.030	7.971	8.058	1.20 ± 0.02
crateification	6.655 ± 0.135	6.389	6.825	1.00

[oeb25]

Current cargo build-incremental:

Command	Mean [s]	Min [s]	Max [s]	Relative
main	7.504 ± 0.049	7.439	7.586	1.30 ± 0.02
crateification	5.771 ± 0.069	5.699	5.936	1.00