Add comptime hashmap

[Vexu]

It's a hashmap constructed at compile time that can be used as a lookup table.

As a small benchmark here is zig fmt lib/std

Benchmark #1: before
Time (mean ± σ): 3.589 s ± 0.073 s [User: 3.459 s, System: 0.093 s]
Range (min … max): 3.495 s … 3.750 s 10 runs
Benchmark #2 after
Time (mean ± σ): 3.427 s ± 0.066 s [User: 3.300 s, System: 0.092 s]
Range (min … max): 3.323 s … 3.542 s 10 runs
Summary
after ran 1.05 ± 0.03 times faster than before

[Vexu]

Few questions regarding the API:

• Should there be a way to set things? e.g. fn set(key: K, val: V) void
• Should get return a pointer to the value, the value or a pointer to a KV struct.
• Should there be an iterator? It wouldn't be very useful without a way to set the values.

[squeek502]

Out of curiosity, could you run your benchmark using the previous Keyword lookup implementation, but with the hashing removed (i.e. just doing mem.eql in a loop)? In something I was doing a while back that pretty much copied the Zig tokenizer's implementation, I found that removing the hashing part of the lookup actually made it faster, although I had a smaller set of tokens and my benchmarking didn't seem ideal. See squeek502/zua@ce11708 for the relevant commit/changes.

[Vexu]

Sure, there would have to be a lot of similar keys for the hash version to be faster than the plain comparison so it was not really appropriate here. The differences are however so small that I value the simplicity this provides more than the speed.

Benchmark #1: ./hash fmt lib/std
Time (mean ± σ): 3.295 s ± 0.036 s [User: 3.181 s, System: 0.084 s]
Range (min … max): 3.242 s … 3.363 s 10 runs
Benchmark #2: ./none fmt lib/std
Time (mean ± σ): 3.253 s ± 0.027 s [User: 3.142 s, System: 0.086 s]
Range (min … max): 3.213 s … 3.285 s 10 runs
Benchmark #3: ./comptime-map fmt lib/std
Time (mean ± σ): 3.214 s ± 0.048 s [User: 3.100 s, System: 0.085 s]
Range (min … max): 3.131 s … 3.281 s 10 runs
Summary
'./comptime-map fmt lib/std' ran
1.01 ± 0.02 times faster than './none fmt lib/std'
1.03 ± 0.02 times faster than './hash fmt lib/std'

[squeek502]

Cool, thanks for satisfying my curiosity. 👍

[daurnimator]

* Should there be a way to set things? e.g. `fn set(key: K, val: V) void`

yes.... but how would/could that work with current comptime semantics?

[daurnimator]

See also #3863: a comptime hashmap (indexable at runtime) is able to use perfect hashes.

[frmdstryr]

A more generic solution would be to just have a comptime allocator #1291.

[squeek502]

Re #3863: I wrote a working version of a comptime minimal perfect hashing algorithm based on this Go implementation a while back but stopped after running into #4055 while testing a perfect hashing stringToEnum implementation with larger enums.

(I've also been meaning to put the mph code in a public repo for a while but never got around to it; will try to do that and link to it in #3863)

Anyway, I've tried it with the Zig tokenizer keyword lookup here to get an idea of what the difference might be if perfect hashing was used.

Using release builds of the current master branch and the modified version that uses perfect hashing:

$ hyperfine --min-runs 50 './build/zig-master fmt tmp/lib/std''./build/zig-perfect fmt tmp/lib/std'Benchmark #1: ./build/zig-master fmt tmp/lib/std Time (mean ± σ): 657.4 ms ± 3.0 ms [User: 595.0 ms, System: 61.0 ms] Range (min … max): 652.5 ms … 671.9 ms 50 runsBenchmark #2: ./build/zig-perfect fmt tmp/lib/std Time (mean ± σ): 638.0 ms ± 2.7 ms [User: 574.9 ms, System: 61.9 ms] Range (min … max): 633.5 ms … 646.7 ms 50 runsSummary './build/zig-perfect fmt tmp/lib/std' ran 1.03 ± 0.01 times faster than './build/zig-master fmt tmp/lib/std'

So seemingly no real performance gain to be had from perfect hashing here, as the gain is the same as this comptime hashmap (although there is potential for caveats if my perfect hashing implementation is bad, which is certainly possible).

---

EDIT: Actually adding this branch into the comparison, my perfect hashing actually performs worse:

$ hyperfine --min-runs 50 './build/zig-master fmt tmp/lib/std''./build/zig-vexu fmt tmp/lib/std''./build/zig-perfect fmt tmp/lib/std'Benchmark #1: ./build/zig-master fmt tmp/lib/std Time (mean ± σ): 656.4 ms ± 2.4 ms [User: 594.8 ms, System: 59.9 ms] Range (min … max): 653.3 ms … 666.8 ms 50 runsBenchmark #2: ./build/zig-vexu fmt tmp/lib/std Time (mean ± σ): 624.5 ms ± 2.2 ms [User: 565.3 ms, System: 57.8 ms] Range (min … max): 620.6 ms … 632.3 ms 50 runsBenchmark #3: ./build/zig-perfect fmt tmp/lib/std Time (mean ± σ): 637.0 ms ± 2.0 ms [User: 579.3 ms, System: 56.4 ms] Range (min … max): 633.9 ms … 643.1 ms 50 runsSummary './build/zig-vexu fmt tmp/lib/std' ran 1.02 ± 0.00 times faster than './build/zig-perfect fmt tmp/lib/std' 1.05 ± 0.01 times faster than './build/zig-master fmt tmp/lib/std'

---

EDIT#2: In thinking about it more, the mph performing worse makes sense. In the ComptimeHashMap implementation, lookup should be as fast as a normal HashMap, but it will waste some memory storing all the unused indexes. A MPH implementation will take up less memory, but will always have to do some amount of extra work on the lookup in order to allow that compression. In my implementation, the index and indexChecked (which is actually what's used in this case) lookup functions look like this:

fnindex(str: []constu8) usize {
consthash: u64=Wyhash.hash(0, str);
consti: u64=hash & (num_buckets-1);
constseed: i32=seeds[i];
if (seed<0) {
returnoriginal_indexes[@intCast(usize, -seed)-1];
}
constfinal_i=xorshiftMult64(@intCast(usize, seed) +hash) & (num_buckets-1);
returnoriginal_indexes[final_i];
}
fnindexChecked(str: []constu8) ?usize {
consti=index(str);
returnif (std.mem.eql(u8, str, strs[i])) ielsenull;
}

where xorshiftMult64 is

fnxorshiftMult64(x_: u64) u64 {
varx=x_;
x^=x>>12;
x^=x<<25;
x^=x>>27;
returnx*%2685821657736338717;
}

So, for small sets like keyword lookup, this tradeoff (size of the computed map vs lookup speed) is mostly irrelevant (size of computed map won't be too big to matter). It'd become more relevant when creating a ComptimeHashMap for something like all English words in the dictionary or something, though.

[joachimschmidt557]

Would ComptimeAutoHashMap maybe be a better name?

[daurnimator]

Will this work with keys of type type? That would be the most common usecase IMO.

[Vexu]

type cannot be hashed in userland so there would be little reason to use it in a hash map. You can however use the ComptimeStringHashMap with @typeName.

EDIT: I guess you could also make it use @typeInfo but that still wouldn't make sense over just using inline for on an array.

[andrewrk]

The changes to the tokenizer and translate-c look really nice!

I think this can be merged as is, but here's an idea for the future: given that we have all the values up front, it would make sense to not have to provide a hash function, since the hash function could be derived from the values. We could probably make it a lot faster and save memory too, because we don't need the 60% full handicap if we have a better hashing strategy.

For example, we can look at all the values and come up with a hash function, and then test it on all the values, and find out if there are any collisions. If there are, we can try one of two things:

• re-roll the hash function, see if a different RNG seed can give us better luck (fewer collisions)
• increase the list size

Iterate until the number of collisions is within some upper bound, and then we're done!

It would be fun to explore this, but I'm guessing we will probably want to wait until we have self-hosted readily available because I could definitely see this blowing up memory usage and compilation times in the stage1 compiler.

Anyway, feel free to merge when you feel it is ready @Vexu, it looks like a nice change.

[andrewrk]

It would be interesting to make an API specifically for small lists of strings like this, where it would compute the minimal function needed for unique index lookup. For example:

conststrings= .{
"one",
"two",
"three",
"four",
"five",
"X",
};
// sort by lengthconststrings= .{
"X",
"one",
"two",
"four",
"five",
"three",
};
// generate this functionfnparse(input: []constu8) ?Tag {
switch (input.len) {
1=>returncheck("X", .X),
3=>switch (input[0]) {
'o'=>returncheck("one", .one),
't'=>returncheck("two", .two),
else=>returnnull,
},
4=>switch (input[1]) {
'o'=>returncheck("four", .four),
'i'=>returncheck("five", .five),
else=>returnnull,
},
5=>returncheck("three", .three),
else=>returnnull,
}
}
fncheck(input: []constu8, expected: []constu8, tag: Tag) ?Tag {
if (mem.eql(u8, input, expected)) {
returntag;
} else {
returnnull;
}
}

I suspect this would have improved performance characteristics, specifically because it quickly finds the cases where an identifier is not a keyword, which is really common in zig code. Many identifiers will have a len that does not match any keyword len, and would go straight from switch (input.len) to else => return null.

[andrewrk]

OK I think this turned out to be a pretty good idea, see #5442

[squeek502]

From messing around with making the implementation in #5442 re-usable (#5442 (comment)), I found that using [0] and [1] here restricts kv to being an anonymous list literal (or an array, presumably, but that would require K and V to be the same type). This limits the use cases since (AFAIK) it's not possible to generate arrays/slices of anonymous list literals at compile time, which would be needed for using ComptimeHashMap in something like a std.meta.stringToEnum implementation.

Using kv.@"0" and kv.@"1" might be better (since that will work with both anonymous list literals and struct types with fields specifically named @"0" and @"1"), or maybe make it conditional on std.meta.trait.isIndexable if arrays are intended to be supported as well?

[Vexu]

I can't think of any non string related use case for this. If someone does eventually come up with one then this should be easy to get working again.

[squeek502]

I think it's still worthwhile. One use case I've come across is creating a Enum -> string lookup table (i.e. the inverse of the string -> Enum lookup use case). Specifically, I could use it to address the FIXME here: https://github.com/squeek502/zua/blob/5831fdbe0cd2675124fa671512bd21c99fb2806c/src/lex.zig#L106-L116

[Vexu]

Enums are just fancy integers so it would be faster and likely simpler to just use an array or a switch.

Only situation where this could be nice would be if the keys were structs but I can't come up with any case where the structs would be comptime known.

[s5bug]

I can't think of any non string related use case for this. If someone does eventually come up with one then this should be easy to get working again.

I'd like to use this in a situation where I need a sparse 2d array with "groups". My exact use-case is I'm trying to represent Puyo-Puyo dropsets:

In the base game, there's never more than 4 colors in a 2x2 area in one drop. However, I felt like

// null for empty, 0 for color group 0, 1 for color group 1, etcpubconstDrop= [4]?u2;
pubconstDropSet= [16]Drop;

was a bad way to represent Drop. I'd like to be able to expand it past a 2x2 area and past 4 colors in the future. Additionally, representing specific drops was clunky; for example, representing the I drop, where both Puyo are able to be in a different color group:

was done like so:

.i= [4]?u2{ 0, null, 1, null }

which additionally won't scale well past 4 elements.

Commonly, to represent groups like this, you combine a Map[Position, GroupId] and a Map[GroupId, Set[Position]]. Because all of the dropsets are known at compile time, and will never change during the program's runtime, I've been looking for a compile time HashMap+HashSet to implement a compile time Grouping over. For example, it makes much more sense to represent the I drop as:

.i=ComptimeGrouping(
[2]u8,
.{
.{ [2]u8{ 0, 0 } },
.{ [2]u8{ 0, 1 } }
}
)

or the vertical L drop:

as:

.lv=ComptimeGrouping(
[2]u8,
.{
.{ [2]u8{ 0, 0 }, [2]u8{ 0, 1 } },
.{ [2]u8{ 1, 0 } }
}
)

and then am able to index via .lv.getGroup({ 1, 0 }) or some other mechanism.

I understand that ComptimeStringMap exists now, but I need an arbitrary key (in this case, [2]u8).

[Vexu]

I doubt this is the best solution for your use case, but I guess could be a good enough reason to add this at least until the std lib audit #1629.

[Vexu]

@sorenbug I extracted this into its own repo.
Using it should be quite easy despite the lack of a package manager:

// build.zigyour_exe.addPackagePath("chm", "path/to/comptime_hash_map.zig");
// in your project@import("chm").AutoComptimeHashMap(Type, .{values});
// or just@import("path/to/comptime_hash_map.zig").AutoComptimeHashMap(Type, .{values});