Add benchmarks for Sequel with common plugins

[hmistry]

Adds benchmarks for Sequel with no plugins and with as many plugins as applicable. Would like to track JIT performance for the Module plugin architecture used in Sequel and Roda where base methods are overridden in plugins. Having both benchmarks allows us to see any impacts due to plugins.

Please advise how you'd like me to setup the 2 different benchmarks - base Sequel and Sequel + plugins. Do you prefer separate folders or 2 files is ok?

[hmistry]

CLA signed. Need help rerunning CI 🙏

[noahgibbs]

Currently there's no easy way to return two different measurements from a single benchmark, so "sequel no plugins" and "sequel with plugins" would need to be two different benchmarks.

[noahgibbs]

It would be possible to keep your current setup, which gives the "sequel" benchmark, and then add a file called something like "benchmarks/sequel_with_plugins.rb" that changes into the sequel directory before use_gemfile. Basically, make one of them a single-file benchmark while the other is a directory benchmark, but otherwise keep basically the same setup.

@maximecb often has opinions on how many similar benchmarks we want, though, and may or may not be up for two different Sequel benchmarks.

I believe she's away for the holidays, maybe until 5th Jan? So our side may take a bit to reply back here.

With that said, this code looks fine on quick inspection, and I think a Sequel benchmark is a good idea in general.

[hmistry]

@noahgibbs Thanks for the quick review. I made the recommended changes. If @maximecb feels there's no value to having 2 different sequel benchmarks, it's fine with me - happy to change as required. My main goal is to ensure the Module plugin architecture also benefits from YJIT optimizations.

[maximecb]

Hi there. Thanks for putting this together @hmistry. Database access is something we don't really have in our benchmarks and so it seems like a good idea to benchmark it.

I think I would prefer to have just one benchmark for this package. What do you think is the most realistic/typical use case for this gem? Do people usually have many plugins?

[hmistry]

Hi @maximecb. Ok, I'll make this a single benchmark after knowing whether you prefer to be consistent with ActiveRecord benchmark or have a realistic case.

Regarding plugins, yes, a typical application will use several plugins but not all. I'm not sure if you're familiar with this plugin architecture but it's a very minimal vanilla core and then all the functionality is composed using plugins that override the core methods. That way you have the best of both worlds - performance and just the right functionality with nothing extra.

[noahgibbs]

Database access is something we don't really have in our benchmarks and so it seems like a good idea to benchmark it.

Sequel is probably best thought of as an ActiveRecord competitor, which is presumably why he followed the structure of that benchmark. There are significant differences, but overall it's used for very similar things.

[maximecb]

Hi @maximecb. Ok, I'll make this a single benchmark after knowing whether you prefer to be consistent with ActiveRecord benchmark or have a realistic case.

Regarding plugins, yes, a typical application will use several plugins but not all. I'm not sure if you're familiar with this plugin architecture but it's a very minimal vanilla core and then all the functionality is composed using plugins that override the core methods. That way you have the best of both worlds - performance and just the right functionality with nothing extra.

Can you make a judgment call as to which plugins are very commonly used and only include those? 😅

[hmistry]

Can you make a judgment call as to which plugins are very commonly used and only include those? 😅

Yes absolutely. I changed to only use a common set of plugins. Hope that'll be enough to determine any YJIT performance optimizations for a chain of method overrides used by the plugin architecture.

Also reduced it to one benchmark per your request. Thank you all for your feedback, please review the changes.

[maximecb]

LGTM. Will give @noahgibbs a chance to comment on Monday as well 👍

[noahgibbs]

LGTM too!

[maximecb]

There's just one last detail missing, which is that we should add this benchmark to the list, in the headline category:
https://github.com/Shopify/yjit-bench/blob/main/benchmarks.yml

[hmistry]

There's just one last detail missing, which is that we should add this benchmark to the list, in the headline category:
https://github.com/Shopify/yjit-bench/blob/main/benchmarks.yml

Done 🙂

[maximecb]

Well done @hmistry, and thanks for your patience in getting this merged 🙏 :)

[hmistry]

@maximecb You're welcome! I've been following your work along with MJIT and MIR. I really look forward to seeing YJIT bring significant performance increases to Ruby. Thank you for leading YJIT development! 🙏

[eregon]

FYI this is actually an inefficient way to query a record in Sequel, Post[i] or Post.first(id: i) are much faster: #207

[hmistry]

Adds benchmarks for Sequel with no plugins and with as many plugins as applicable. Would like to track JIT performance for the Module plugin architecture used in Sequel and Roda where base methods are overridden in plugins. Having both benchmarks allows us to see any impacts due to plugins.

Please advise how you'd like me to setup the 2 different benchmarks - base Sequel and Sequel + plugins. Do you prefer separate folders or 2 files is ok?

[hmistry]

CLA signed. Need help rerunning CI 🙏

[noahgibbs]

Currently there's no easy way to return two different measurements from a single benchmark, so "sequel no plugins" and "sequel with plugins" would need to be two different benchmarks.

[noahgibbs]

It would be possible to keep your current setup, which gives the "sequel" benchmark, and then add a file called something like "benchmarks/sequel_with_plugins.rb" that changes into the sequel directory before use_gemfile. Basically, make one of them a single-file benchmark while the other is a directory benchmark, but otherwise keep basically the same setup.

@maximecb often has opinions on how many similar benchmarks we want, though, and may or may not be up for two different Sequel benchmarks.

I believe she's away for the holidays, maybe until 5th Jan? So our side may take a bit to reply back here.

With that said, this code looks fine on quick inspection, and I think a Sequel benchmark is a good idea in general.

[hmistry]

@noahgibbs Thanks for the quick review. I made the recommended changes. If @maximecb feels there's no value to having 2 different sequel benchmarks, it's fine with me - happy to change as required. My main goal is to ensure the Module plugin architecture also benefits from YJIT optimizations.

[maximecb]

Hi there. Thanks for putting this together @hmistry. Database access is something we don't really have in our benchmarks and so it seems like a good idea to benchmark it.

I think I would prefer to have just one benchmark for this package. What do you think is the most realistic/typical use case for this gem? Do people usually have many plugins?

[hmistry]

Hi @maximecb. Ok, I'll make this a single benchmark after knowing whether you prefer to be consistent with ActiveRecord benchmark or have a realistic case.

Regarding plugins, yes, a typical application will use several plugins but not all. I'm not sure if you're familiar with this plugin architecture but it's a very minimal vanilla core and then all the functionality is composed using plugins that override the core methods. That way you have the best of both worlds - performance and just the right functionality with nothing extra.

[noahgibbs]

Database access is something we don't really have in our benchmarks and so it seems like a good idea to benchmark it.

Sequel is probably best thought of as an ActiveRecord competitor, which is presumably why he followed the structure of that benchmark. There are significant differences, but overall it's used for very similar things.

[maximecb]

Hi @maximecb. Ok, I'll make this a single benchmark after knowing whether you prefer to be consistent with ActiveRecord benchmark or have a realistic case.

Regarding plugins, yes, a typical application will use several plugins but not all. I'm not sure if you're familiar with this plugin architecture but it's a very minimal vanilla core and then all the functionality is composed using plugins that override the core methods. That way you have the best of both worlds - performance and just the right functionality with nothing extra.

Can you make a judgment call as to which plugins are very commonly used and only include those? 😅

[hmistry]

Can you make a judgment call as to which plugins are very commonly used and only include those? 😅

Yes absolutely. I changed to only use a common set of plugins. Hope that'll be enough to determine any YJIT performance optimizations for a chain of method overrides used by the plugin architecture.

Also reduced it to one benchmark per your request. Thank you all for your feedback, please review the changes.

[maximecb]

LGTM. Will give @noahgibbs a chance to comment on Monday as well 👍

[noahgibbs]

LGTM too!

[maximecb]

There's just one last detail missing, which is that we should add this benchmark to the list, in the headline category:
https://github.com/Shopify/yjit-bench/blob/main/benchmarks.yml

[hmistry]

There's just one last detail missing, which is that we should add this benchmark to the list, in the headline category:
https://github.com/Shopify/yjit-bench/blob/main/benchmarks.yml

Done 🙂

[maximecb]

Well done @hmistry, and thanks for your patience in getting this merged 🙏 :)

[hmistry]

@maximecb You're welcome! I've been following your work along with MJIT and MIR. I really look forward to seeing YJIT bring significant performance increases to Ruby. Thank you for leading YJIT development! 🙏

[eregon]

FYI this is actually an inefficient way to query a record in Sequel, Post[i] or Post.first(id: i) are much faster: #207

[hmistry]

Adds benchmarks for Sequel with no plugins and with as many plugins as applicable. Would like to track JIT performance for the Module plugin architecture used in Sequel and Roda where base methods are overridden in plugins. Having both benchmarks allows us to see any impacts due to plugins.

Please advise how you'd like me to setup the 2 different benchmarks - base Sequel and Sequel + plugins. Do you prefer separate folders or 2 files is ok?

[hmistry]

CLA signed. Need help rerunning CI 🙏

[noahgibbs]

Currently there's no easy way to return two different measurements from a single benchmark, so "sequel no plugins" and "sequel with plugins" would need to be two different benchmarks.

[noahgibbs]

It would be possible to keep your current setup, which gives the "sequel" benchmark, and then add a file called something like "benchmarks/sequel_with_plugins.rb" that changes into the sequel directory before use_gemfile. Basically, make one of them a single-file benchmark while the other is a directory benchmark, but otherwise keep basically the same setup.

@maximecb often has opinions on how many similar benchmarks we want, though, and may or may not be up for two different Sequel benchmarks.

I believe she's away for the holidays, maybe until 5th Jan? So our side may take a bit to reply back here.

With that said, this code looks fine on quick inspection, and I think a Sequel benchmark is a good idea in general.

[hmistry]

@noahgibbs Thanks for the quick review. I made the recommended changes. If @maximecb feels there's no value to having 2 different sequel benchmarks, it's fine with me - happy to change as required. My main goal is to ensure the Module plugin architecture also benefits from YJIT optimizations.

[maximecb]

Hi there. Thanks for putting this together @hmistry. Database access is something we don't really have in our benchmarks and so it seems like a good idea to benchmark it.

I think I would prefer to have just one benchmark for this package. What do you think is the most realistic/typical use case for this gem? Do people usually have many plugins?

[hmistry]

Hi @maximecb. Ok, I'll make this a single benchmark after knowing whether you prefer to be consistent with ActiveRecord benchmark or have a realistic case.

Regarding plugins, yes, a typical application will use several plugins but not all. I'm not sure if you're familiar with this plugin architecture but it's a very minimal vanilla core and then all the functionality is composed using plugins that override the core methods. That way you have the best of both worlds - performance and just the right functionality with nothing extra.

[noahgibbs]

Database access is something we don't really have in our benchmarks and so it seems like a good idea to benchmark it.

Sequel is probably best thought of as an ActiveRecord competitor, which is presumably why he followed the structure of that benchmark. There are significant differences, but overall it's used for very similar things.

[maximecb]

Hi @maximecb. Ok, I'll make this a single benchmark after knowing whether you prefer to be consistent with ActiveRecord benchmark or have a realistic case.

Regarding plugins, yes, a typical application will use several plugins but not all. I'm not sure if you're familiar with this plugin architecture but it's a very minimal vanilla core and then all the functionality is composed using plugins that override the core methods. That way you have the best of both worlds - performance and just the right functionality with nothing extra.

Can you make a judgment call as to which plugins are very commonly used and only include those? 😅

[hmistry]

Can you make a judgment call as to which plugins are very commonly used and only include those? 😅

Yes absolutely. I changed to only use a common set of plugins. Hope that'll be enough to determine any YJIT performance optimizations for a chain of method overrides used by the plugin architecture.

Also reduced it to one benchmark per your request. Thank you all for your feedback, please review the changes.

[maximecb]

LGTM. Will give @noahgibbs a chance to comment on Monday as well 👍

[noahgibbs]

LGTM too!

[maximecb]

There's just one last detail missing, which is that we should add this benchmark to the list, in the headline category:
https://github.com/Shopify/yjit-bench/blob/main/benchmarks.yml

[hmistry]

There's just one last detail missing, which is that we should add this benchmark to the list, in the headline category:
https://github.com/Shopify/yjit-bench/blob/main/benchmarks.yml

Done 🙂

[maximecb]

Well done @hmistry, and thanks for your patience in getting this merged 🙏 :)

[hmistry]

@maximecb You're welcome! I've been following your work along with MJIT and MIR. I really look forward to seeing YJIT bring significant performance increases to Ruby. Thank you for leading YJIT development! 🙏

[eregon]

FYI this is actually an inefficient way to query a record in Sequel, Post[i] or Post.first(id: i) are much faster: #207

[hmistry]

Adds benchmarks for Sequel with no plugins and with as many plugins as applicable. Would like to track JIT performance for the Module plugin architecture used in Sequel and Roda where base methods are overridden in plugins. Having both benchmarks allows us to see any impacts due to plugins.

Please advise how you'd like me to setup the 2 different benchmarks - base Sequel and Sequel + plugins. Do you prefer separate folders or 2 files is ok?

[hmistry]

CLA signed. Need help rerunning CI 🙏

[noahgibbs]

Currently there's no easy way to return two different measurements from a single benchmark, so "sequel no plugins" and "sequel with plugins" would need to be two different benchmarks.

[noahgibbs]

It would be possible to keep your current setup, which gives the "sequel" benchmark, and then add a file called something like "benchmarks/sequel_with_plugins.rb" that changes into the sequel directory before use_gemfile. Basically, make one of them a single-file benchmark while the other is a directory benchmark, but otherwise keep basically the same setup.

@maximecb often has opinions on how many similar benchmarks we want, though, and may or may not be up for two different Sequel benchmarks.

I believe she's away for the holidays, maybe until 5th Jan? So our side may take a bit to reply back here.

With that said, this code looks fine on quick inspection, and I think a Sequel benchmark is a good idea in general.

[hmistry]

@noahgibbs Thanks for the quick review. I made the recommended changes. If @maximecb feels there's no value to having 2 different sequel benchmarks, it's fine with me - happy to change as required. My main goal is to ensure the Module plugin architecture also benefits from YJIT optimizations.

[maximecb]

Hi there. Thanks for putting this together @hmistry. Database access is something we don't really have in our benchmarks and so it seems like a good idea to benchmark it.

I think I would prefer to have just one benchmark for this package. What do you think is the most realistic/typical use case for this gem? Do people usually have many plugins?

[hmistry]

Hi @maximecb. Ok, I'll make this a single benchmark after knowing whether you prefer to be consistent with ActiveRecord benchmark or have a realistic case.

Regarding plugins, yes, a typical application will use several plugins but not all. I'm not sure if you're familiar with this plugin architecture but it's a very minimal vanilla core and then all the functionality is composed using plugins that override the core methods. That way you have the best of both worlds - performance and just the right functionality with nothing extra.

[noahgibbs]

Database access is something we don't really have in our benchmarks and so it seems like a good idea to benchmark it.

Sequel is probably best thought of as an ActiveRecord competitor, which is presumably why he followed the structure of that benchmark. There are significant differences, but overall it's used for very similar things.

[maximecb]

Hi @maximecb. Ok, I'll make this a single benchmark after knowing whether you prefer to be consistent with ActiveRecord benchmark or have a realistic case.

Regarding plugins, yes, a typical application will use several plugins but not all. I'm not sure if you're familiar with this plugin architecture but it's a very minimal vanilla core and then all the functionality is composed using plugins that override the core methods. That way you have the best of both worlds - performance and just the right functionality with nothing extra.

Can you make a judgment call as to which plugins are very commonly used and only include those? 😅

[hmistry]

Can you make a judgment call as to which plugins are very commonly used and only include those? 😅

Yes absolutely. I changed to only use a common set of plugins. Hope that'll be enough to determine any YJIT performance optimizations for a chain of method overrides used by the plugin architecture.

Also reduced it to one benchmark per your request. Thank you all for your feedback, please review the changes.

[maximecb]

LGTM. Will give @noahgibbs a chance to comment on Monday as well 👍

[noahgibbs]

LGTM too!

[maximecb]

There's just one last detail missing, which is that we should add this benchmark to the list, in the headline category:
https://github.com/Shopify/yjit-bench/blob/main/benchmarks.yml

[hmistry]

There's just one last detail missing, which is that we should add this benchmark to the list, in the headline category:
https://github.com/Shopify/yjit-bench/blob/main/benchmarks.yml

Done 🙂

[maximecb]

Well done @hmistry, and thanks for your patience in getting this merged 🙏 :)

[hmistry]

@maximecb You're welcome! I've been following your work along with MJIT and MIR. I really look forward to seeing YJIT bring significant performance increases to Ruby. Thank you for leading YJIT development! 🙏

[eregon]

FYI this is actually an inefficient way to query a record in Sequel, Post[i] or Post.first(id: i) are much faster: #207

[hmistry]

Adds benchmarks for Sequel with no plugins and with as many plugins as applicable. Would like to track JIT performance for the Module plugin architecture used in Sequel and Roda where base methods are overridden in plugins. Having both benchmarks allows us to see any impacts due to plugins.

Please advise how you'd like me to setup the 2 different benchmarks - base Sequel and Sequel + plugins. Do you prefer separate folders or 2 files is ok?

[hmistry]

CLA signed. Need help rerunning CI 🙏

[noahgibbs]

Currently there's no easy way to return two different measurements from a single benchmark, so "sequel no plugins" and "sequel with plugins" would need to be two different benchmarks.

[noahgibbs]

It would be possible to keep your current setup, which gives the "sequel" benchmark, and then add a file called something like "benchmarks/sequel_with_plugins.rb" that changes into the sequel directory before use_gemfile. Basically, make one of them a single-file benchmark while the other is a directory benchmark, but otherwise keep basically the same setup.

@maximecb often has opinions on how many similar benchmarks we want, though, and may or may not be up for two different Sequel benchmarks.

I believe she's away for the holidays, maybe until 5th Jan? So our side may take a bit to reply back here.

With that said, this code looks fine on quick inspection, and I think a Sequel benchmark is a good idea in general.

[hmistry]

@noahgibbs Thanks for the quick review. I made the recommended changes. If @maximecb feels there's no value to having 2 different sequel benchmarks, it's fine with me - happy to change as required. My main goal is to ensure the Module plugin architecture also benefits from YJIT optimizations.

[maximecb]

Hi there. Thanks for putting this together @hmistry. Database access is something we don't really have in our benchmarks and so it seems like a good idea to benchmark it.

I think I would prefer to have just one benchmark for this package. What do you think is the most realistic/typical use case for this gem? Do people usually have many plugins?

[hmistry]

Hi @maximecb. Ok, I'll make this a single benchmark after knowing whether you prefer to be consistent with ActiveRecord benchmark or have a realistic case.

Regarding plugins, yes, a typical application will use several plugins but not all. I'm not sure if you're familiar with this plugin architecture but it's a very minimal vanilla core and then all the functionality is composed using plugins that override the core methods. That way you have the best of both worlds - performance and just the right functionality with nothing extra.

[noahgibbs]

Database access is something we don't really have in our benchmarks and so it seems like a good idea to benchmark it.

Sequel is probably best thought of as an ActiveRecord competitor, which is presumably why he followed the structure of that benchmark. There are significant differences, but overall it's used for very similar things.

[maximecb]

Hi @maximecb. Ok, I'll make this a single benchmark after knowing whether you prefer to be consistent with ActiveRecord benchmark or have a realistic case.

Regarding plugins, yes, a typical application will use several plugins but not all. I'm not sure if you're familiar with this plugin architecture but it's a very minimal vanilla core and then all the functionality is composed using plugins that override the core methods. That way you have the best of both worlds - performance and just the right functionality with nothing extra.

Can you make a judgment call as to which plugins are very commonly used and only include those? 😅

[hmistry]

Can you make a judgment call as to which plugins are very commonly used and only include those? 😅

Yes absolutely. I changed to only use a common set of plugins. Hope that'll be enough to determine any YJIT performance optimizations for a chain of method overrides used by the plugin architecture.

Also reduced it to one benchmark per your request. Thank you all for your feedback, please review the changes.

[maximecb]

LGTM. Will give @noahgibbs a chance to comment on Monday as well 👍

[noahgibbs]

LGTM too!

[maximecb]

There's just one last detail missing, which is that we should add this benchmark to the list, in the headline category:
https://github.com/Shopify/yjit-bench/blob/main/benchmarks.yml

[hmistry]

There's just one last detail missing, which is that we should add this benchmark to the list, in the headline category:
https://github.com/Shopify/yjit-bench/blob/main/benchmarks.yml

Done 🙂

[maximecb]

Well done @hmistry, and thanks for your patience in getting this merged 🙏 :)

[hmistry]

@maximecb You're welcome! I've been following your work along with MJIT and MIR. I really look forward to seeing YJIT bring significant performance increases to Ruby. Thank you for leading YJIT development! 🙏

[eregon]

FYI this is actually an inefficient way to query a record in Sequel, Post[i] or Post.first(id: i) are much faster: #207

[hmistry]

Adds benchmarks for Sequel with no plugins and with as many plugins as applicable. Would like to track JIT performance for the Module plugin architecture used in Sequel and Roda where base methods are overridden in plugins. Having both benchmarks allows us to see any impacts due to plugins.

Please advise how you'd like me to setup the 2 different benchmarks - base Sequel and Sequel + plugins. Do you prefer separate folders or 2 files is ok?

[hmistry]

CLA signed. Need help rerunning CI 🙏

[noahgibbs]

Currently there's no easy way to return two different measurements from a single benchmark, so "sequel no plugins" and "sequel with plugins" would need to be two different benchmarks.

[noahgibbs]

It would be possible to keep your current setup, which gives the "sequel" benchmark, and then add a file called something like "benchmarks/sequel_with_plugins.rb" that changes into the sequel directory before use_gemfile. Basically, make one of them a single-file benchmark while the other is a directory benchmark, but otherwise keep basically the same setup.

@maximecb often has opinions on how many similar benchmarks we want, though, and may or may not be up for two different Sequel benchmarks.

I believe she's away for the holidays, maybe until 5th Jan? So our side may take a bit to reply back here.

With that said, this code looks fine on quick inspection, and I think a Sequel benchmark is a good idea in general.

[hmistry]

@noahgibbs Thanks for the quick review. I made the recommended changes. If @maximecb feels there's no value to having 2 different sequel benchmarks, it's fine with me - happy to change as required. My main goal is to ensure the Module plugin architecture also benefits from YJIT optimizations.

[maximecb]

Hi there. Thanks for putting this together @hmistry. Database access is something we don't really have in our benchmarks and so it seems like a good idea to benchmark it.

I think I would prefer to have just one benchmark for this package. What do you think is the most realistic/typical use case for this gem? Do people usually have many plugins?

[hmistry]

Hi @maximecb. Ok, I'll make this a single benchmark after knowing whether you prefer to be consistent with ActiveRecord benchmark or have a realistic case.

Regarding plugins, yes, a typical application will use several plugins but not all. I'm not sure if you're familiar with this plugin architecture but it's a very minimal vanilla core and then all the functionality is composed using plugins that override the core methods. That way you have the best of both worlds - performance and just the right functionality with nothing extra.

[noahgibbs]

Database access is something we don't really have in our benchmarks and so it seems like a good idea to benchmark it.

Sequel is probably best thought of as an ActiveRecord competitor, which is presumably why he followed the structure of that benchmark. There are significant differences, but overall it's used for very similar things.

[maximecb]

Hi @maximecb. Ok, I'll make this a single benchmark after knowing whether you prefer to be consistent with ActiveRecord benchmark or have a realistic case.

Regarding plugins, yes, a typical application will use several plugins but not all. I'm not sure if you're familiar with this plugin architecture but it's a very minimal vanilla core and then all the functionality is composed using plugins that override the core methods. That way you have the best of both worlds - performance and just the right functionality with nothing extra.

Can you make a judgment call as to which plugins are very commonly used and only include those? 😅

[hmistry]

Can you make a judgment call as to which plugins are very commonly used and only include those? 😅

Yes absolutely. I changed to only use a common set of plugins. Hope that'll be enough to determine any YJIT performance optimizations for a chain of method overrides used by the plugin architecture.

Also reduced it to one benchmark per your request. Thank you all for your feedback, please review the changes.

[maximecb]

LGTM. Will give @noahgibbs a chance to comment on Monday as well 👍

[noahgibbs]

LGTM too!

[maximecb]

There's just one last detail missing, which is that we should add this benchmark to the list, in the headline category:
https://github.com/Shopify/yjit-bench/blob/main/benchmarks.yml

[hmistry]

There's just one last detail missing, which is that we should add this benchmark to the list, in the headline category:
https://github.com/Shopify/yjit-bench/blob/main/benchmarks.yml

Done 🙂

[maximecb]

Well done @hmistry, and thanks for your patience in getting this merged 🙏 :)

[hmistry]

@maximecb You're welcome! I've been following your work along with MJIT and MIR. I really look forward to seeing YJIT bring significant performance increases to Ruby. Thank you for leading YJIT development! 🙏

[eregon]

FYI this is actually an inefficient way to query a record in Sequel, Post[i] or Post.first(id: i) are much faster: #207

[hmistry]

Adds benchmarks for Sequel with no plugins and with as many plugins as applicable. Would like to track JIT performance for the Module plugin architecture used in Sequel and Roda where base methods are overridden in plugins. Having both benchmarks allows us to see any impacts due to plugins.

Please advise how you'd like me to setup the 2 different benchmarks - base Sequel and Sequel + plugins. Do you prefer separate folders or 2 files is ok?

[hmistry]

CLA signed. Need help rerunning CI 🙏

[noahgibbs]

Currently there's no easy way to return two different measurements from a single benchmark, so "sequel no plugins" and "sequel with plugins" would need to be two different benchmarks.

[noahgibbs]

It would be possible to keep your current setup, which gives the "sequel" benchmark, and then add a file called something like "benchmarks/sequel_with_plugins.rb" that changes into the sequel directory before use_gemfile. Basically, make one of them a single-file benchmark while the other is a directory benchmark, but otherwise keep basically the same setup.

@maximecb often has opinions on how many similar benchmarks we want, though, and may or may not be up for two different Sequel benchmarks.

I believe she's away for the holidays, maybe until 5th Jan? So our side may take a bit to reply back here.

With that said, this code looks fine on quick inspection, and I think a Sequel benchmark is a good idea in general.

[hmistry]

@noahgibbs Thanks for the quick review. I made the recommended changes. If @maximecb feels there's no value to having 2 different sequel benchmarks, it's fine with me - happy to change as required. My main goal is to ensure the Module plugin architecture also benefits from YJIT optimizations.

[maximecb]

Hi there. Thanks for putting this together @hmistry. Database access is something we don't really have in our benchmarks and so it seems like a good idea to benchmark it.

I think I would prefer to have just one benchmark for this package. What do you think is the most realistic/typical use case for this gem? Do people usually have many plugins?

[hmistry]

Hi @maximecb. Ok, I'll make this a single benchmark after knowing whether you prefer to be consistent with ActiveRecord benchmark or have a realistic case.

Regarding plugins, yes, a typical application will use several plugins but not all. I'm not sure if you're familiar with this plugin architecture but it's a very minimal vanilla core and then all the functionality is composed using plugins that override the core methods. That way you have the best of both worlds - performance and just the right functionality with nothing extra.

[noahgibbs]

Database access is something we don't really have in our benchmarks and so it seems like a good idea to benchmark it.

Sequel is probably best thought of as an ActiveRecord competitor, which is presumably why he followed the structure of that benchmark. There are significant differences, but overall it's used for very similar things.

[maximecb]

Hi @maximecb. Ok, I'll make this a single benchmark after knowing whether you prefer to be consistent with ActiveRecord benchmark or have a realistic case.

Regarding plugins, yes, a typical application will use several plugins but not all. I'm not sure if you're familiar with this plugin architecture but it's a very minimal vanilla core and then all the functionality is composed using plugins that override the core methods. That way you have the best of both worlds - performance and just the right functionality with nothing extra.

Can you make a judgment call as to which plugins are very commonly used and only include those? 😅

[hmistry]

Can you make a judgment call as to which plugins are very commonly used and only include those? 😅

Yes absolutely. I changed to only use a common set of plugins. Hope that'll be enough to determine any YJIT performance optimizations for a chain of method overrides used by the plugin architecture.

Also reduced it to one benchmark per your request. Thank you all for your feedback, please review the changes.

[maximecb]

LGTM. Will give @noahgibbs a chance to comment on Monday as well 👍

[noahgibbs]

LGTM too!

[maximecb]

There's just one last detail missing, which is that we should add this benchmark to the list, in the headline category:
https://github.com/Shopify/yjit-bench/blob/main/benchmarks.yml

[hmistry]

There's just one last detail missing, which is that we should add this benchmark to the list, in the headline category:
https://github.com/Shopify/yjit-bench/blob/main/benchmarks.yml

Done 🙂

[maximecb]

Well done @hmistry, and thanks for your patience in getting this merged 🙏 :)

[hmistry]

@maximecb You're welcome! I've been following your work along with MJIT and MIR. I really look forward to seeing YJIT bring significant performance increases to Ruby. Thank you for leading YJIT development! 🙏

[eregon]

FYI this is actually an inefficient way to query a record in Sequel, Post[i] or Post.first(id: i) are much faster: #207

[hmistry]

Adds benchmarks for Sequel with no plugins and with as many plugins as applicable. Would like to track JIT performance for the Module plugin architecture used in Sequel and Roda where base methods are overridden in plugins. Having both benchmarks allows us to see any impacts due to plugins.

Please advise how you'd like me to setup the 2 different benchmarks - base Sequel and Sequel + plugins. Do you prefer separate folders or 2 files is ok?

[hmistry]

CLA signed. Need help rerunning CI 🙏

[noahgibbs]

Currently there's no easy way to return two different measurements from a single benchmark, so "sequel no plugins" and "sequel with plugins" would need to be two different benchmarks.

[noahgibbs]

It would be possible to keep your current setup, which gives the "sequel" benchmark, and then add a file called something like "benchmarks/sequel_with_plugins.rb" that changes into the sequel directory before use_gemfile. Basically, make one of them a single-file benchmark while the other is a directory benchmark, but otherwise keep basically the same setup.

@maximecb often has opinions on how many similar benchmarks we want, though, and may or may not be up for two different Sequel benchmarks.

I believe she's away for the holidays, maybe until 5th Jan? So our side may take a bit to reply back here.

With that said, this code looks fine on quick inspection, and I think a Sequel benchmark is a good idea in general.

[hmistry]

@noahgibbs Thanks for the quick review. I made the recommended changes. If @maximecb feels there's no value to having 2 different sequel benchmarks, it's fine with me - happy to change as required. My main goal is to ensure the Module plugin architecture also benefits from YJIT optimizations.

[maximecb]

Hi there. Thanks for putting this together @hmistry. Database access is something we don't really have in our benchmarks and so it seems like a good idea to benchmark it.

I think I would prefer to have just one benchmark for this package. What do you think is the most realistic/typical use case for this gem? Do people usually have many plugins?

[hmistry]

Hi @maximecb. Ok, I'll make this a single benchmark after knowing whether you prefer to be consistent with ActiveRecord benchmark or have a realistic case.

Regarding plugins, yes, a typical application will use several plugins but not all. I'm not sure if you're familiar with this plugin architecture but it's a very minimal vanilla core and then all the functionality is composed using plugins that override the core methods. That way you have the best of both worlds - performance and just the right functionality with nothing extra.

[noahgibbs]

Database access is something we don't really have in our benchmarks and so it seems like a good idea to benchmark it.

Sequel is probably best thought of as an ActiveRecord competitor, which is presumably why he followed the structure of that benchmark. There are significant differences, but overall it's used for very similar things.

[maximecb]

Hi @maximecb. Ok, I'll make this a single benchmark after knowing whether you prefer to be consistent with ActiveRecord benchmark or have a realistic case.

Regarding plugins, yes, a typical application will use several plugins but not all. I'm not sure if you're familiar with this plugin architecture but it's a very minimal vanilla core and then all the functionality is composed using plugins that override the core methods. That way you have the best of both worlds - performance and just the right functionality with nothing extra.

Can you make a judgment call as to which plugins are very commonly used and only include those? 😅

[hmistry]

Can you make a judgment call as to which plugins are very commonly used and only include those? 😅

Yes absolutely. I changed to only use a common set of plugins. Hope that'll be enough to determine any YJIT performance optimizations for a chain of method overrides used by the plugin architecture.

Also reduced it to one benchmark per your request. Thank you all for your feedback, please review the changes.

[maximecb]

LGTM. Will give @noahgibbs a chance to comment on Monday as well 👍

[noahgibbs]

LGTM too!

[maximecb]

There's just one last detail missing, which is that we should add this benchmark to the list, in the headline category:
https://github.com/Shopify/yjit-bench/blob/main/benchmarks.yml

[hmistry]

There's just one last detail missing, which is that we should add this benchmark to the list, in the headline category:
https://github.com/Shopify/yjit-bench/blob/main/benchmarks.yml

Done 🙂

[maximecb]

Well done @hmistry, and thanks for your patience in getting this merged 🙏 :)

[hmistry]

@maximecb You're welcome! I've been following your work along with MJIT and MIR. I really look forward to seeing YJIT bring significant performance increases to Ruby. Thank you for leading YJIT development! 🙏

[eregon]

FYI this is actually an inefficient way to query a record in Sequel, Post[i] or Post.first(id: i) are much faster: #207