Improve performance of token compilation

[estensen]

📝 Summary

• Grow builder before using it to avoid unnecessary memory allocation and copy.
• Use hex.EncodeToString to convert bytes to hex

📚 References

strings.Builder is backed by a slice, so Grow() pre-allocates this slice
https://cs.opensource.google/go/go/+/refs/tags/go1.19.3:src/strings/builder.go;l=15

TODO: Add benchmark

---

• I have seen and agree to CONTRIBUTING.md

[Ruteri]

Is this also the case upstream @estensen ? Maybe it'd make sense to contribute there too

[Ruteri]

Correct me if I'm wrong
bin.WriteString(fmt.Sprintf("%x", v)) can write more than a byte, so the string is not guaranteed to not reallocate right?
Do you have any data as to how often it's the case either way?

[estensen]

Yeah, planning to upstream this as well :)

As far as I understand, it doesn't matter if you add a byte (converted to string) or a longer string. The string builder will append one item per iteration and it's the number of items that causes re-allocation of slices and not the length of the items.

[estensen]

I'm getting my feet wet here. How many tokens are typically compiled?

Running benchmarks locally growing the string builder is faster from around 25 items.
https://github.com/estensen/benchmarks/blob/master/strings/concatenation.go
https://github.com/estensen/benchmarks/blob/master/strings/concatenation_test.go

// 10
BenchmarkConcatString/concat_string-12 3155166 368.5 ns/op 600 B/op 9 allocs/op
BenchmarkConcatString/concat_buffer-12 7609831 157.1 ns/op 304 B/op 3 allocs/op
BenchmarkConcatString/concat_builder-12 8009695 150.8 ns/op 240 B/op 4 allocs/op
BenchmarkConcatString/concat_grown_builder-12 6620098 182.3 ns/op 376 B/op 5 allocs/op

// 25
BenchmarkConcatString/concat_string-12 994122 1190 ns/op 3400 B/op 24 allocs/op
BenchmarkConcatString/concat_buffer-12 3759690 316.4 ns/op 704 B/op 4 allocs/op
BenchmarkConcatString/concat_builder-12 4358062 272.9 ns/op 496 B/op 5 allocs/op
BenchmarkConcatString/concat_grown_builder-12 5165503 232.3 ns/op 480 B/op 4 allocs/op

// 100
BenchmarkConcatString/concat_string-12 113984 10431 ns/op 53480 B/op 99 allocs/op
BenchmarkConcatString/concat_buffer-12 1000000 1097 ns/op 3008 B/op 6 allocs/op
BenchmarkConcatString/concat_builder-12 1000000 1048 ns/op 3312 B/op 8 allocs/op
BenchmarkConcatString/concat_grown_builder-12 1405492 859.5 ns/op 2656 B/op 5 allocs/op

[estensen]

hex.EncodeToString was faster than fmt.Sprintf for all the benchmarks I did

https://github.com/estensen/benchmarks/blob/master/encoding/encoding.go
https://github.com/estensen/benchmarks/blob/master/encoding/encoding_test.go

BenchmarkByteToHex/encode_bytes_to_hex_string_with_fmt.Sprintf_for_1_byte-12 12061374 86.88 ns/op 26 B/op 2 allocs/op
BenchmarkByteToHex/encode_bytes_to_hex_string_with_hex.EncodeToString_for_1_byte-12 84560934 14.46 ns/op 2 B/op 1 allocs/op
BenchmarkByteToHex/encode_bytes_to_hex_string_with_fmt.Sprintf_for_5_bytes-12 11170132 106.2 ns/op 40 B/op 2 allocs/op
BenchmarkByteToHex/encode_bytes_to_hex_string_with_hex.EncodeToString_for_5_bytes-12 42864435 27.07 ns/op 16 B/op 1 allocs/op
BenchmarkByteToHex/encode_bytes_to_hex_string_with_fmt.Sprintf_for_10_bytes-12 10061223 115.1 ns/op 48 B/op 2 allocs/op
BenchmarkByteToHex/encode_bytes_to_hex_string_with_hex.EncodeToString_for_10_bytes-12 33870447 34.29 ns/op 24 B/op 1 allocs/op
BenchmarkByteToHex/encode_bytes_to_hex_string_with_fmt.Sprintf_for_50_bytes-12 5790674 204.7 ns/op 136 B/op 2 allocs/op
BenchmarkByteToHex/encode_bytes_to_hex_string_with_hex.EncodeToString_for_50_bytes-12 11064771 108.3 ns/op 224 B/op 2 allocs/op

[Ruteri]

Sounds good.
Thanks for doing detailed benchmarks and your contribution!
I'll merge once tested.