test(replays): jemalloc and serde struct deserialization

[JoshFerge]

• Re-introduce the custom deserializer introduced in ref(replay): More efficient deserialization #1782
• change the memory allocator used in relay-server to jemalloc using https://github.com/tikv/jemallocator (tried previously here a while back ref: Use jemalloc instead of system allocator #344)
• Configure jemalloc's conf with background_thread:true, mentioned as a solution to memory heap growth here physical memory goes high every several hours jemalloc/jemalloc#1128 (comment). I was able to observe stable memory usage with this change in my testing environment, as opposed to unbounded growth using system malloc, and jemalloc without background_thread:true (Centos Linux VM)
• A sentry-cli error was causing the Docker build to fail, upgrading seems to have fixed it.

[jan-auer]

Thank you for researching jemalloc! The changes you're listing here would be useful to run separately to see their individual impact, particularly:

1. Swap the system memory allocator with jemalloc in default
2. Then enable the background_thread flag to check for differences. The case mentioned in the linked issue where their application goes idle doesn't directly apply to us, but there could be a benefit. For that we'd need to understand better where the "leak" is coming from in the first place.
3. Finally, re-introduce struct deserialization side-by-side with the current transcoder.

The jemalloc experiments can still make sense independently, but I would wait with 3 until we know that the transcoding approach does not work.

[jan-auer]

We'll eventually re-attempt jemalloc in isolation. Thanks for the work on this!

[JoshFerge]

• Re-introduce the custom deserializer introduced in ref(replay): More efficient deserialization #1782
• change the memory allocator used in relay-server to jemalloc using https://github.com/tikv/jemallocator (tried previously here a while back ref: Use jemalloc instead of system allocator #344)
• Configure jemalloc's conf with background_thread:true, mentioned as a solution to memory heap growth here physical memory goes high every several hours jemalloc/jemalloc#1128 (comment). I was able to observe stable memory usage with this change in my testing environment, as opposed to unbounded growth using system malloc, and jemalloc without background_thread:true (Centos Linux VM)
• A sentry-cli error was causing the Docker build to fail, upgrading seems to have fixed it.

[jan-auer]

Thank you for researching jemalloc! The changes you're listing here would be useful to run separately to see their individual impact, particularly:

1. Swap the system memory allocator with jemalloc in default
2. Then enable the background_thread flag to check for differences. The case mentioned in the linked issue where their application goes idle doesn't directly apply to us, but there could be a benefit. For that we'd need to understand better where the "leak" is coming from in the first place.
3. Finally, re-introduce struct deserialization side-by-side with the current transcoder.

The jemalloc experiments can still make sense independently, but I would wait with 3 until we know that the transcoding approach does not work.

[jan-auer]

We'll eventually re-attempt jemalloc in isolation. Thanks for the work on this!

[JoshFerge]

• Re-introduce the custom deserializer introduced in ref(replay): More efficient deserialization #1782
• change the memory allocator used in relay-server to jemalloc using https://github.com/tikv/jemallocator (tried previously here a while back ref: Use jemalloc instead of system allocator #344)
• Configure jemalloc's conf with background_thread:true, mentioned as a solution to memory heap growth here physical memory goes high every several hours jemalloc/jemalloc#1128 (comment). I was able to observe stable memory usage with this change in my testing environment, as opposed to unbounded growth using system malloc, and jemalloc without background_thread:true (Centos Linux VM)
• A sentry-cli error was causing the Docker build to fail, upgrading seems to have fixed it.

[jan-auer]

Thank you for researching jemalloc! The changes you're listing here would be useful to run separately to see their individual impact, particularly:

1. Swap the system memory allocator with jemalloc in default
2. Then enable the background_thread flag to check for differences. The case mentioned in the linked issue where their application goes idle doesn't directly apply to us, but there could be a benefit. For that we'd need to understand better where the "leak" is coming from in the first place.
3. Finally, re-introduce struct deserialization side-by-side with the current transcoder.

The jemalloc experiments can still make sense independently, but I would wait with 3 until we know that the transcoding approach does not work.

[jan-auer]

We'll eventually re-attempt jemalloc in isolation. Thanks for the work on this!

[JoshFerge]

• Re-introduce the custom deserializer introduced in ref(replay): More efficient deserialization #1782
• change the memory allocator used in relay-server to jemalloc using https://github.com/tikv/jemallocator (tried previously here a while back ref: Use jemalloc instead of system allocator #344)
• Configure jemalloc's conf with background_thread:true, mentioned as a solution to memory heap growth here physical memory goes high every several hours jemalloc/jemalloc#1128 (comment). I was able to observe stable memory usage with this change in my testing environment, as opposed to unbounded growth using system malloc, and jemalloc without background_thread:true (Centos Linux VM)
• A sentry-cli error was causing the Docker build to fail, upgrading seems to have fixed it.

[jan-auer]

Thank you for researching jemalloc! The changes you're listing here would be useful to run separately to see their individual impact, particularly:

1. Swap the system memory allocator with jemalloc in default
2. Then enable the background_thread flag to check for differences. The case mentioned in the linked issue where their application goes idle doesn't directly apply to us, but there could be a benefit. For that we'd need to understand better where the "leak" is coming from in the first place.
3. Finally, re-introduce struct deserialization side-by-side with the current transcoder.

The jemalloc experiments can still make sense independently, but I would wait with 3 until we know that the transcoding approach does not work.

[jan-auer]

We'll eventually re-attempt jemalloc in isolation. Thanks for the work on this!

[JoshFerge]

• Re-introduce the custom deserializer introduced in ref(replay): More efficient deserialization #1782
• change the memory allocator used in relay-server to jemalloc using https://github.com/tikv/jemallocator (tried previously here a while back ref: Use jemalloc instead of system allocator #344)
• Configure jemalloc's conf with background_thread:true, mentioned as a solution to memory heap growth here physical memory goes high every several hours jemalloc/jemalloc#1128 (comment). I was able to observe stable memory usage with this change in my testing environment, as opposed to unbounded growth using system malloc, and jemalloc without background_thread:true (Centos Linux VM)
• A sentry-cli error was causing the Docker build to fail, upgrading seems to have fixed it.

[jan-auer]

Thank you for researching jemalloc! The changes you're listing here would be useful to run separately to see their individual impact, particularly:

1. Swap the system memory allocator with jemalloc in default
2. Then enable the background_thread flag to check for differences. The case mentioned in the linked issue where their application goes idle doesn't directly apply to us, but there could be a benefit. For that we'd need to understand better where the "leak" is coming from in the first place.
3. Finally, re-introduce struct deserialization side-by-side with the current transcoder.

The jemalloc experiments can still make sense independently, but I would wait with 3 until we know that the transcoding approach does not work.

[jan-auer]

We'll eventually re-attempt jemalloc in isolation. Thanks for the work on this!

[JoshFerge]

• Re-introduce the custom deserializer introduced in ref(replay): More efficient deserialization #1782
• change the memory allocator used in relay-server to jemalloc using https://github.com/tikv/jemallocator (tried previously here a while back ref: Use jemalloc instead of system allocator #344)
• Configure jemalloc's conf with background_thread:true, mentioned as a solution to memory heap growth here physical memory goes high every several hours jemalloc/jemalloc#1128 (comment). I was able to observe stable memory usage with this change in my testing environment, as opposed to unbounded growth using system malloc, and jemalloc without background_thread:true (Centos Linux VM)
• A sentry-cli error was causing the Docker build to fail, upgrading seems to have fixed it.

[jan-auer]

Thank you for researching jemalloc! The changes you're listing here would be useful to run separately to see their individual impact, particularly:

1. Swap the system memory allocator with jemalloc in default
2. Then enable the background_thread flag to check for differences. The case mentioned in the linked issue where their application goes idle doesn't directly apply to us, but there could be a benefit. For that we'd need to understand better where the "leak" is coming from in the first place.
3. Finally, re-introduce struct deserialization side-by-side with the current transcoder.

The jemalloc experiments can still make sense independently, but I would wait with 3 until we know that the transcoding approach does not work.

[jan-auer]

We'll eventually re-attempt jemalloc in isolation. Thanks for the work on this!

[JoshFerge]

• Re-introduce the custom deserializer introduced in ref(replay): More efficient deserialization #1782
• change the memory allocator used in relay-server to jemalloc using https://github.com/tikv/jemallocator (tried previously here a while back ref: Use jemalloc instead of system allocator #344)
• Configure jemalloc's conf with background_thread:true, mentioned as a solution to memory heap growth here physical memory goes high every several hours jemalloc/jemalloc#1128 (comment). I was able to observe stable memory usage with this change in my testing environment, as opposed to unbounded growth using system malloc, and jemalloc without background_thread:true (Centos Linux VM)
• A sentry-cli error was causing the Docker build to fail, upgrading seems to have fixed it.

[jan-auer]

Thank you for researching jemalloc! The changes you're listing here would be useful to run separately to see their individual impact, particularly:

1. Swap the system memory allocator with jemalloc in default
2. Then enable the background_thread flag to check for differences. The case mentioned in the linked issue where their application goes idle doesn't directly apply to us, but there could be a benefit. For that we'd need to understand better where the "leak" is coming from in the first place.
3. Finally, re-introduce struct deserialization side-by-side with the current transcoder.

The jemalloc experiments can still make sense independently, but I would wait with 3 until we know that the transcoding approach does not work.

[jan-auer]

We'll eventually re-attempt jemalloc in isolation. Thanks for the work on this!

[JoshFerge]

• Re-introduce the custom deserializer introduced in ref(replay): More efficient deserialization #1782
• change the memory allocator used in relay-server to jemalloc using https://github.com/tikv/jemallocator (tried previously here a while back ref: Use jemalloc instead of system allocator #344)
• Configure jemalloc's conf with background_thread:true, mentioned as a solution to memory heap growth here physical memory goes high every several hours jemalloc/jemalloc#1128 (comment). I was able to observe stable memory usage with this change in my testing environment, as opposed to unbounded growth using system malloc, and jemalloc without background_thread:true (Centos Linux VM)
• A sentry-cli error was causing the Docker build to fail, upgrading seems to have fixed it.

[jan-auer]

Thank you for researching jemalloc! The changes you're listing here would be useful to run separately to see their individual impact, particularly:

1. Swap the system memory allocator with jemalloc in default
2. Then enable the background_thread flag to check for differences. The case mentioned in the linked issue where their application goes idle doesn't directly apply to us, but there could be a benefit. For that we'd need to understand better where the "leak" is coming from in the first place.
3. Finally, re-introduce struct deserialization side-by-side with the current transcoder.

The jemalloc experiments can still make sense independently, but I would wait with 3 until we know that the transcoding approach does not work.

[jan-auer]

We'll eventually re-attempt jemalloc in isolation. Thanks for the work on this!