Umbrellio Utils

Installation

Add this line to your application's Gemfile:

gem"umbrellio-utils"

And then execute:

$ bundle install

Or install it yourself as:

$ gem install umbrellio-utils

Usage

Quick Start

You can use modules and classes directly by accessing modules and classes under namespace UmbrellioUtils. Or you can include UmbrellioUtils to other module with name you like.

# Direct usingUmbrellioUtils::Constants.get_class!(:object)#=> Object# Aliasing to shorter name.moduleUtilsincludeUmbrellioUtilsendUtils::Constants.get_class!(:object)#=> ObjectUtils::Constants#=> UmbrellioUtils::Constants

Configuration

Some modules and classes are configurable. Here's the full list of settings and what they do:

• store_table_name — table which is used by UmbrellioUtils::Store module. Defaults to :store
• http_client_name — fiber-local variable name for http client instance in UmbrellioUtils::HTTPClient. Defaults to :application_httpclient

You can change config in two ways. Firstly, you can change values by accessing configuration directly. Secondly, you can use UmbrellioUtils::configure method which accepts a block.

# First methodUmbrellioUtils.config.store_table_name=:cool_name# Second methodmoduleUtilsincludeUmbrellioUtilsconfiguredo |config|
config.store_table_name=:cool_nameendend

Keep in mind that the config is common to all modules: if you use multiple modules that include UmbrellioUtils, then all modules will use the same configuration object.

Extension

You can extend module with you own project specific methods via UmbrellioUtils::extend_util!.

moduleUtilsincludeUmbrellioUtilsconfiguredo |config|
config.store_table_name=:cool_nameendextend_util!(:Constants)dodefuseful_method"Just string"endendendUtils::Constants.useful_method#=> "Just string"

Or you can define methods in your module and then extend the desired module.

moduleMyHelpersdefuseful_method"Just string"endendmoduleUtilsincludeUmbrellioUtilsextend_util!(:Constants){extendMyHelpers}endUtils::Constants.useful_method#=> "Just string"

ClickHouse deduplication

Datasets built with CH.from understand ClickHouse's LIMIT n BY:

CH.from(:events).order(Sequel.desc(:version)).limit_by(:user_id,rows: 3)#=> SELECT * FROM "events" ORDER BY "version" DESC LIMIT 3 BY "user_id"

On a ReplacingMergeTree table, #deduplicate uses that to collapse row versions by hand rather than relying on the final setting, which merges the entire table even for a point lookup:

CH.from(:external_operations_distributed).where(order_id: 42)# inside the dedup subquery.deduplicate# boundary.order(:created_at)# outside

The sorting key, version column and is_deleted column are read from system.tables and cached per process; Distributed tables are resolved through to the local table they wrap. Deduplicated datasets are sent with final: 0 automatically, since running FINAL inside the subquery would be both slow and redundant. An explicit final: passed to query / count always wins.

#deduplicate is a boundary, and which side a filter lands on matters:

• Before it — immutable selectors only (primary keys, foreign keys). Filtering a mutable column first can match a superseded row version and resurrect a row that FINAL would have dropped.
• After it — everything else, including any filter on a column that changes over a row's lifetime.

is_deleted is applied after the boundary for exactly that reason, and is omitted when the engine declares no such column.

Two chain methods are handled rather than passed through, because the dedup subquery has to control them: the subquery always projects SELECT * (so the outer query can still filter on is_deleted and on columns you did not select) and any projection you set is re-applied outside it; and the version ordering leads the subquery's ORDER BY, since it decides which row survives, with any ordering you set kept after it as a tiebreaker.

#deduplicate raises on a table it cannot collapse — a non-Replacing engine, a Replacing engine declared without a version argument, or a dataset that is not a single table source (joined, multi-source, or a subquery).

Instrumentation

The gem ships a set of opt-in files for collecting GVL and allocation stats (they are not loaded by default and require the gvltools gem in your app):

# Extends ActiveSupport::Notifications::Event with #gvl_time and #malloc_increase_bytesrequire"umbrellio_utils/patches/active_support_event"# Enriches the "Completed" action controller log entry (rails_semantic_logger)# with GC, GVL and allocation statsrequire"umbrellio_utils/patches/rails_semantic_logger"# Logs Sidekiq job completion with the same stats. Relies on the# "perform.sidekiq_job" notification published by the umbrellio fork of yabeda-sidekiqrequire"umbrellio_utils/semantic_logger/sidekiq_job_metrics"UmbrellioUtils::SemanticLogger::SidekiqJobMetrics.subscribe!

Don't forget to enable the GVL timer as early as possible (e.g. right after Bundler.require in config/application.rb):

GVLTools::LocalTimer.enable

Contributing

Bug reports and pull requests are welcome on GitHub at https://github.com/umbrellio/utils.

License

The gem is available as open source under the terms of the MIT License.

Authors

Created by Umbrellio's Ruby developers