Searchkick

🚀 Intelligent search made easy

Searchkick learns what your users are looking for. As more people search, it gets smarter and the results get better. It’s friendly for developers - and magical for your users.

Searchkick handles:

stemming - tomatoes matches tomato
special characters - jalapeno matches jalapeño
extra whitespace - dishwasher matches dish washer
misspellings - zuchini matches zucchini
custom synonyms - qtip matches cotton swab

Plus:

query like SQL - no need to learn a new query language
reindex without downtime
easily personalize results for each user
autocomplete
“Did you mean” suggestions
works with ActiveRecord, Mongoid, and NoBrainer

💬 Get handcrafted updates for new features

🍊 Battle-tested at Instacart

Getting Started

Install Elasticsearch. For Homebrew, use:

brew install elasticsearch
brew services start elasticsearch

Add this line to your application’s Gemfile:

gem'searchkick'

The latest version works with Elasticsearch 2 and 5. For Elasticsearch 1, use version 1.5.1 and this readme.

Add searchkick to models you want to search.

classProduct < ActiveRecord::Basesearchkickend

Add data to the search index.

Product.reindex

And to query, use:

products=Product.search"apples"products.eachdo |product|
putsproduct.nameend

Searchkick supports the complete Elasticsearch Search API. As your search becomes more advanced, we recommend you use the Elasticsearch DSL for maximum flexibility.

Querying

Query like SQL

Product.search"apples",where: {in_stock: true},limit: 10,offset: 50

Search specific fields

fields: [:name,:brand]

Where

where: {expires_at: {gt: Time.now},# lt, gte, lte also availableorders_count: 1..10,# equivalent to {gte: 1, lte: 10}aisle_id: [25,30],# instore_id: {not: 2},# notaisle_id: {not: [25,30]},# not inuser_ids: {all: [1,3]},# all elements in arraycategory: /frozen .+/,# regexp_or: [{in_stock: true},{backordered: true}]}

Order

order: {_score: :desc}# most relevant first - default

All of these sort options are supported

Limit / offset

limit: 20,offset: 40

Select

select: [:name]

Results

Searches return a Searchkick::Results object. This responds like an array to most methods.

results=Product.search("milk")results.sizeresults.any?results.each{ |result| ... }

By default, ids are fetched from Elasticsearch and records are fetched from your database. To fetch everything from Elasticsearch, use:

Product.search("apples",load: false)

Get total results

results.total_count

Get the time the search took (in milliseconds)

results.took

Get the full response from Elasticsearch

results.response

Boosting

Boost important fields

fields: ["title^10","description"]

Boost by the value of a field (field must be numeric)

boost_by: [:orders_count]# give popular documents a little boostboost_by: {orders_count: {factor: 10}}# default factor is 1

Boost matching documents

boost_where: {user_id: 1}boost_where: {user_id: {value: 1,factor: 100}}# default factor is 1000boost_where: {user_id: [{value: 1,factor: 100},{value: 2,factor: 200}]}

Conversions are also a great way to boost.

Get Everything

Use a * for the query.

Product.search"*"

Pagination

Plays nicely with kaminari and will_paginate.

# controller@products=Product.search"milk",page: params[:page],per_page: 20

View with kaminari

<%= paginate @products %>

View with will_paginate

<%= will_paginate @products %>

Partial Matches

By default, results must match all words in the query.

Product.search"fresh honey"# fresh AND honey

To change this, use:

Product.search"fresh honey",operator: "or"# fresh OR honey

By default, results must match the entire word - back will not match backpack. You can change this behavior with:

classProduct < ActiveRecord::Basesearchkickword_start: [:name]end

And to search (after you reindex):

Product.search"back",fields: [:name],match: :word_start

Available options are:

:word# default:word_start:word_middle:word_end:text_start:text_middle:text_end

Exact Matches

To match a field exactly (case-insensitive), use:

User.searchquery,fields: [{email: :exact},:name]

Phrase Matches

To only match the exact order, use:

User.search"fresh honey",match: :phrase

Language

Searchkick defaults to English for stemming. To change this, use:

classProduct < ActiveRecord::Basesearchkicklanguage: "german"end

See the list of stemmers

Synonyms

classProduct < ActiveRecord::Basesearchkicksynonyms: [["scallion","green onion"],["qtip","cotton swab"]]end

Call Product.reindex after changing synonyms.

To read synonyms from a file, use:

synonyms: ->{CSV.read("/some/path/synonyms.csv")}

For directional synonyms, use:

synonyms: ["lightbulb => halogenlamp"]

Tags and Dynamic Synonyms

The above approach works well when your synonym list is static, but in practice, this is often not the case. When you analyze search conversions, you often want to add new synonyms or tags without a full reindex. You can use a library like ActsAsTaggableOn and do:

classProduct < ActiveRecord::Baseacts_as_taggablescope:search_import,->{includes(:tags)}defsearch_data{name_tagged: "#{name}#{tags.map(&:name).join(" ")}"}endend

Search with:

Product.searchquery,fields: [:name_tagged]

WordNet

Prepopulate English synonyms with the WordNet database.

Download WordNet 3.0 to each Elasticsearch server and move wn_s.pl to the /var/lib directory.

cd /tmp
curl -o wordnet.tar.gz http://wordnetcode.princeton.edu/3.0/WNprolog-3.0.tar.gz
tar -zxvf wordnet.tar.gz
mv prolog/wn_s.pl /var/lib

Tell each model to use it:

classProduct < ActiveRecord::Basesearchkickwordnet: trueend

Misspellings

By default, Searchkick handles misspelled queries by returning results with an edit distance of one.

You can change this with:

Product.search"zucini",misspellings: {edit_distance: 2}# zucchini

To prevent poor precision and improve performance for correctly spelled queries (which should be a majority for most applications), Searchkick can first perform a search without misspellings, and if there are too few results, perform another with them.

Product.search"zuchini",misspellings: {below: 5}

If there are fewer than 5 results, a 2nd search is performed with misspellings enabled. The result of this query is returned.

Turn off misspellings with:

Product.search"zuchini",misspellings: false# no zucchini

Bad Matches

If a user searches butter, they may also get results for peanut butter. To prevent this, use:

Product.search"butter",exclude: ["peanut butter"]

You can map queries and terms to exclude with:

exclude_queries={"butter"=>["peanut butter"],"cream"=>["ice cream","whipped cream"]}Product.searchquery,exclude: exclude_queries[query]

Emoji

Search 🍨🍰 and get ice cream cake!

Add this line to your application’s Gemfile:

gem'gemoji-parser'

And use:

Product.search"🍨🍰",emoji: true

Indexing

Control what data is indexed with the search_data method. Call Product.reindex after changing this method.

classProduct < ActiveRecord::Basebelongs_to:departmentdefsearch_data{name: name,department_name: department.name,on_sale: sale_price.present?}endend

Searchkick uses find_in_batches to import documents. To eager load associations, use the search_import scope.

classProduct < ActiveRecord::Basescope:search_import,->{includes(:department)}end

By default, all records are indexed. To control which records are indexed, use the should_index? method together with the search_import scope.

classProduct < ActiveRecord::Basescope:search_import,->{where(active: true)}defshould_index?active# only index active recordsendend

If a reindex is interrupted, you can resume it with:

Product.reindex(resume: true)

For large data sets, try parallel reindexing.

To Reindex, or Not to Reindex

Reindex

when you install or upgrade searchkick
change the search_data method
change the searchkick method

No need to reindex

app starts

Stay Synced

There are four strategies for keeping the index synced with your database.

Immediate (default)

Anytime a record is inserted, updated, or deleted

Asynchronous

Use background jobs for better performance

classProduct < ActiveRecord::Basesearchkickcallbacks: :asyncend

And install Active Job for Rails 4.1 and below. Jobs are added to a queue named searchkick.

Queuing

Push ids of records that need updated to a queue and reindex in the background in batches. This is more performant than the asynchronous method, which updates records individually. See how to set up.

Manual

Turn off automatic syncing

classProduct < ActiveRecord::Basesearchkickcallbacks: falseend

You can also do bulk updates.

Searchkick.callbacks(:bulk)doUser.find_each(&:update_fields)end

Or temporarily skip updates.

Searchkick.callbacks(false)doUser.find_each(&:update_fields)end

Associations

Data is not automatically synced when an association is updated. If this is desired, add a callback to reindex:

classImage < ActiveRecord::Basebelongs_to:productafter_commit:reindex_productdefreindex_productproduct.reindex# or reindex_asyncendend

Analytics

The best starting point to improve your search by far is to track searches and conversions.

Searchjoy makes it easy.

Product.search"apple",track: {user_id: current_user.id}

See the docs for how to install and use.

Focus on:

top searches with low conversions
top searches with no results

Keep Getting Better

Searchkick can use conversion data to learn what users are looking for. If a user searches for “ice cream” and adds Ben & Jerry’s Chunky Monkey to the cart (our conversion metric at Instacart), that item gets a little more weight for similar searches.

The first step is to define your conversion metric and start tracking conversions. The database works well for low volume, but feel free to use Redis or another datastore.

You do not need to clean up the search queries. Searchkick automatically treats apple and APPLES the same.

Next, add conversions to the index.

classProduct < ActiveRecord::Basehas_many:searches,class_name: "Searchjoy::Search",as: :convertablesearchkickconversions: ["conversions"]# name of fielddefsearch_data{name: name,conversions: searches.group(:query).uniq.count(:user_id)# {"ice cream" => 234, "chocolate" => 67, "cream" => 2}}endend

Reindex and set up a cron job to add new conversions daily.

rakesearchkick:reindexCLASS=Product

Note: For a more performant (but more advanced) approach, check out performant conversions.

Personalized Results

Order results differently for each user. For example, show a user’s previously purchased products before other results.

classProduct < ActiveRecord::Basedefsearch_data{name: name,orderer_ids: orders.pluck(:user_id)# boost this product for these users}endend

Reindex and search with:

Product.search"milk",boost_where: {orderer_ids: current_user.id}

Instant Search / Autocomplete

Autocomplete predicts what a user will type, making the search experience faster and easier.

Note: To autocomplete on general categories (like cereal rather than product names), check out Autosuggest.

Note 2: If you only have a few thousand records, don’t use Searchkick for autocomplete. It’s much faster to load all records into JavaScript and autocomplete there (eliminates network requests).

First, specify which fields use this feature. This is necessary since autocomplete can increase the index size significantly, but don’t worry - this gives you blazing faster queries.

classMovie < ActiveRecord::Basesearchkickword_start: [:title,:director]end

Reindex and search with:

Movie.search"jurassic pa",fields: [:title],match: :word_start

Typically, you want to use a JavaScript library like typeahead.js or jQuery UI.

Here’s how to make it work with Rails

First, add a route and controller action.

classMoviesController < ApplicationControllerdefautocompleterenderjson: Movie.search(params[:query],{fields: ["title^5","director"],match: :word_start,limit: 10,load: false,misspellings: {below: 5}}).map(&:title)endend

Note: Use load: false and misspellings: {below: n} (or misspellings: false) for best performance.

Then add the search box and JavaScript code to a view.

<inputtype="text" id="query" name="query" /><scriptsrc="jquery.js"></script><scriptsrc="typeahead.bundle.js"></script><script>varmovies=newBloodhound({datumTokenizer: Bloodhound.tokenizers.whitespace,queryTokenizer: Bloodhound.tokenizers.whitespace,remote: {url: '/movies/autocomplete?query=%QUERY',wildcard: '%QUERY'}});$('#query').typeahead(null,{source: movies});</script>

Suggestions

classProduct < ActiveRecord::Basesearchkicksuggest: [:name]# fields to generate suggestionsend

Reindex and search with:

products=Product.search"peantu butta",suggest: trueproducts.suggestions# ["peanut butter"]

Aggregations

Aggregations provide aggregated search data.

products=Product.search"chuck taylor",aggs: [:product_type,:gender,:brand]products.aggs

By default, where conditions apply to aggregations.

Product.search"wingtips",where: {color: "brandy"},aggs: [:size]# aggregations for brandy wingtips are returned

Change this with:

Product.search"wingtips",where: {color: "brandy"},aggs: [:size],smart_aggs: false# aggregations for all wingtips are returned

Set where conditions for each aggregation separately with:

Product.search"wingtips",aggs: {size: {where: {color: "brandy"}}}

Limit

Product.search"apples",aggs: {store_id: {limit: 10}}

Order

Product.search"wingtips",aggs: {color: {order: {"_term"=>"asc"}}}# alphabetically

All of these options are supported

Ranges

price_ranges=[{to: 20},{from: 20,to: 50},{from: 50}]Product.search"*",aggs: {price: {ranges: price_ranges}}

Minimum document count

Product.search"apples",aggs: {store_id: {min_doc_count: 2}}

Date histogram

Product.search"pear",aggs: {products_per_year: {date_histogram: {field: :created_at,interval: :year}}}

Moving From Facets

Replace facets with aggs in searches. Note: Stats facets are not supported at this time.

products=Product.search"chuck taylor",facets: [:brand]# toproducts=Product.search"chuck taylor",aggs: [:brand]

Replace the facets method with aggs for results.

products.facets# toproducts.aggs

The keys in results differ slightly. Instead of:

{
"_type":"terms",
"missing":0,
"total":45,
"other":34,
"terms":[
{"term":14.0,"count":11}
]
}

You get:

{
"doc_count":45,
"doc_count_error_upper_bound":0,
"sum_other_doc_count":34,
"buckets":[
{"key":14.0,"doc_count":11}
]
}

Update your application to handle this.

By default, where conditions apply to aggregations. This is equivalent to smart_facets: true. If you have smart_facets: true, you can remove it. If this is not desired, set smart_aggs: false.
If you have any range facets with dates, change the key from ranges to date_ranges.

facets: {date_field: {ranges: date_ranges}}# toaggs: {date_field: {date_ranges: date_ranges}}

Highlight

Specify which fields to index with highlighting.

classProduct < ActiveRecord::Basesearchkickhighlight: [:name]end

Highlight the search query in the results.

bands=Band.search"cinema",fields: [:name],highlight: true

Note: The fields option is required, unless highlight options are given - see below.

View the highlighted fields with:

bands.eachdo |band|
band.search_highlights[:name]# "Two Door <em>Cinema</em> Club"end

To change the tag, use:

Band.search"cinema",fields: [:name],highlight: {tag: "<strong>"}

To highlight and search different fields, use:

Band.search"cinema",fields: [:name],highlight: {fields: [:description]}

Additional options, including fragment size, can be specified for each field:

Band.search"cinema",fields: [:name],highlight: {fields: {name: {fragment_size: 200}}}

You can find available highlight options in the Elasticsearch reference.

Similar Items

Find similar items.

product=Product.firstproduct.similar(fields: [:name],where: {size: "12 oz"})

Geospatial Searches

classRestaurant < ActiveRecord::Basesearchkicklocations: [:location]defsearch_dataattributes.mergelocation: {lat: latitude,lon: longitude}endend

Reindex and search with:

Restaurant.search"pizza",where: {location: {near: {lat: 37,lon: -114},within: "100mi"}}# or 160km

Bounded by a box

Restaurant.search"sushi",where: {location: {top_left: {lat: 38,lon: -123},bottom_right: {lat: 37,lon: -122}}}

Bounded by a polygon

Restaurant.search"dessert",where: {location: {geo_polygon: {points: [{lat: 38,lon: -123},{lat: 39,lon: -123},{lat: 37,lon: 122}]}}}

Boost By Distance

Boost results by distance - closer results are boosted more

Restaurant.search"noodles",boost_by_distance: {location: {origin: {lat: 37,lon: -122}}}

Also supports additional options

Restaurant.search"wings",boost_by_distance: {location: {origin: {lat: 37,lon: -122},function: "linear",scale: "30mi",decay: 0.5}}

Geo Shapes

You can also index and search geo shapes.

classRestaurant < ActiveRecord::Basesearchkickgeo_shape: {bounds: {tree: "geohash",precision: "1km"}}defsearch_dataattributes.merge(bounds: {type: "envelope",coordinates: [{lat: 4,lon: 1},{lat: 2,lon: 3}]})endend

See the Elasticsearch documentation for details.

Find shapes intersecting with the query shape

Restaurant.search"soup",where: {bounds: {geo_shape: {type: "polygon",coordinates: [[{lat: 38,lon: -123}, ...]]}}}

Falling entirely within the query shape

Restaurant.search"salad",where: {bounds: {geo_shape: {type: "circle",relation: "within",coordinates: [{lat: 38,lon: -123}],radius: "1km"}}}

Not touching the query shape

Restaurant.search"burger",where: {bounds: {geo_shape: {type: "envelope",relation: "disjoint",coordinates: [{lat: 38,lon: -123},{lat: 37,lon: -122}]}}}

Containing the query shape (Elasticsearch 2.2+)

Restaurant.search"fries",where: {bounds: {geo_shape: {type: "envelope",relation: "contains",coordinates: [{lat: 38,lon: -123},{lat: 37,lon: -122}]}}}

Inheritance

Searchkick supports single table inheritance.

classDog < Animalend

The parent and child model can both reindex.

Animal.reindexDog.reindex# equivalent

And to search, use:

Animal.search"*"# all animalsDog.search"*"# just dogsAnimal.search"*",type: [Dog,Cat]# just cats and dogs

Note: The suggest option retrieves suggestions from the parent at the moment.

Dog.search"airbudd",suggest: true# suggestions for all animals

Debugging Queries

To help with debugging queries, you can use:

Product.search("soap",debug: true)

This prints useful info to stdout.

See how Elasticsearch scores your queries with:

Product.search("soap",explain: true).response

See how Elasticsearch tokenizes your queries with:

Product.search_index.tokens("Dish Washer Soap",analyzer: "searchkick_index")# ["dish", "dishwash", "washer", "washersoap", "soap"]Product.search_index.tokens("dishwasher soap",analyzer: "searchkick_search")# ["dishwashersoap"] - no matchProduct.search_index.tokens("dishwasher soap",analyzer: "searchkick_search2")# ["dishwash", "soap"] - match!!

Partial matches

Product.search_index.tokens("San Diego",analyzer: "searchkick_word_start_index")# ["s", "sa", "san", "d", "di", "die", "dieg", "diego"]Product.search_index.tokens("dieg",analyzer: "searchkick_word_search")# ["dieg"] - match!!

See the complete list of analyzers.

Deployment

Searchkick uses ENV["ELASTICSEARCH_URL"] for the Elasticsearch server. This defaults to http://localhost:9200.

Heroku

Choose an add-on: SearchBox, Bonsai, or Elastic Cloud.

# SearchBox
heroku addons:create searchbox:starter
heroku config:set ELASTICSEARCH_URL=`heroku config:get SEARCHBOX_URL`# Bonsai
heroku addons:create bonsai
heroku config:set ELASTICSEARCH_URL=`heroku config:get BONSAI_URL`# Found
heroku addons:create foundelasticsearch
heroku config:set ELASTICSEARCH_URL=`heroku config:get FOUNDELASTICSEARCH_URL`

Then deploy and reindex:

heroku run rake searchkick:reindex CLASS=Product

Amazon Elasticsearch Service

Include elasticsearch 1.0.15 or greater in your Gemfile.

gem'elasticsearch','>= 1.0.15'

Create an initializer config/initializers/elasticsearch.rb with:

ENV["ELASTICSEARCH_URL"]="https://es-domain-1234.us-east-1.es.amazonaws.com"

To use signed request, include in your Gemfile:

gem'faraday_middleware-aws-signers-v4'

and add to your initializer:

Searchkick.aws_credentials={access_key_id: ENV["AWS_ACCESS_KEY_ID"],secret_access_key: ENV["AWS_SECRET_ACCESS_KEY"],region: "us-east-1"}

Then deploy and reindex:

rake searchkick:reindex CLASS=Product

Other

Create an initializer config/initializers/elasticsearch.rb with:

ENV["ELASTICSEARCH_URL"]="http://username:password@api.searchbox.io"

Then deploy and reindex:

rake searchkick:reindex CLASS=Product

Automatic Failover

Create an initializer config/initializers/elasticsearch.rb with multiple hosts:

ENV["ELASTICSEARCH_URL"]="http://localhost:9200,http://localhost:9201"Searchkick.client_options={retry_on_failure: true}

See elasticsearch-transport for a complete list of options.

Lograge

Add the following to config/environments/production.rb:

config.lograge.custom_options=lambdado |event|
options={}options[:search]=event.payload[:searchkick_runtime]ifevent.payload[:searchkick_runtime].to_f > 0optionsend

See Production Rails for other good practices.

Performance

JSON Generation

Significantly increase performance with faster JSON generation. Add Oj to your Gemfile.

gem'oj'

This speeds up all JSON generation and parsing in your application (automatically!)

Persistent HTTP Connections

Significantly increase performance with persistent HTTP connections. Add Typhoeus to your Gemfile and it’ll automatically be used.

gem'typhoeus'

To reduce log noise, create an initializer with:

Ethon.logger=Logger.new("/dev/null")

If you run into issues on Windows, check out this post.

Searchable Fields

By default, all string fields are searchable (can be used in fields option). Speed up indexing and reduce index size by only making some fields searchable. This disables the _all field unless it’s listed.

classProduct < ActiveRecord::Basesearchkicksearchable: [:name]end

Filterable Fields

By default, all string fields are filterable (can be used in where option). Speed up indexing and reduce index size by only making some fields filterable.

classProduct < ActiveRecord::Basesearchkickfilterable: [:brand]end

Note: Non-string fields will always be filterable and should not be passed to this option.

Parallel Reindexing

For large data sets, you can use background jobs to parallelize reindexing.

Product.reindex(async: true)# {index_name: "products_production_20170111210018065"}

Once the jobs complete, promote the new index with:

Product.search_index.promote(index_name)

You can optionally track the status with Redis:

Searchkick.redis=Redis.new

And use:

Searchkick.reindex_status(index_name)

You can also have Searchkick wait for reindexing to complete [master]

Searchkick.reindex(async: {wait: true})

You can use ActiveJob::TrafficControl to control concurrency. Install the gem:

gem'activejob-traffic_control','>= 0.1.3'

And create an initializer with:

ActiveJob::TrafficControl.client=Searchkick.redisclassSearchkick::BulkReindexJobconcurrency3end

This will allow only 3 jobs to run at once.

Refresh Interval

You can specify a longer refresh interval while reindexing to increase performance.

Product.reindex(async: true,refresh_interval: "30s")

Note: This only makes a noticable difference with parallel reindexing.

When promoting, have it restored to the value in your mapping (defaults to 1s).

Product.search_index.promote(index_name,update_refresh_interval: true)

Queuing

Push ids of records needing reindexed to a queue and reindex in bulk for better performance. First, set up Redis in an initializer. We recommend using connection_pool.

Searchkick.redis=ConnectionPool.new{Redis.new}

And ask your models to queue updates.

classProduct < ActiveRecord::Basesearchkickcallbacks: :queueend

Then, set up a background job to run.

Searchkick::ProcessQueueJob.perform_later(class_name: "Product")

You can check the queue length with:

Product.search_index.reindex_queue.length

For more tips, check out Keeping Elasticsearch in Sync.

Routing

Searchkick supports Elasticsearch’s routing feature, which can significantly speed up searches.

classBusiness < ActiveRecord::Basesearchkickrouting: truedefsearch_routingcity_idendend

Reindex and search with:

Business.search"ice cream",routing: params[:city_id]

Partial Reindexing

Reindex a subset of attributes to reduce time spent generating search data and cut down on network traffic.

classProduct < ActiveRecord::Basedefsearch_data{name: name}.merge(search_prices)enddefsearch_prices{price: price,sale_price: sale_price}endend

And use:

Product.reindex(:search_prices)

Performant Conversions

Split out conversions into a separate method so you can use partial reindexing, and cache conversions to prevent N+1 queries. Be sure to use a centralized cache store like Memcached or Redis.

classProduct < ActiveRecord::Basedefsearch_data{name: name}.merge(search_conversions)enddefsearch_conversions{conversions: Rails.cache.read("search_conversions:#{self.class.name}:#{id}") || {}}endend

Create a job to update the cache and reindex records with new conversions.

classReindexConversionsJob < ActiveJob::Basedefperform(class_name)# get records that have a recent conversionrecently_converted_ids=Searchjoy::Search.where("convertable_type = ? AND converted_at > ?",class_name,1.day.ago).order(:convertable_id).uniq.pluck(:convertable_id)# split into groupsrecently_converted_ids.in_groups_of(1000,false)do |ids|
# fetch conversionsconversions=Searchjoy::Search.where(convertable_id: ids,convertable_type: class_name).group(:convertable_id,:query).uniq.count(:user_id)# group conversions by recordconversions_by_record={}conversions.eachdo |(id,query),count|
(conversions_by_record[id] ||= {})[query]=countend# write to cacheconversions_by_record.eachdo |id,conversions|
Rails.cache.write("search_conversions:#{class_name}:#{id}",conversions)end# partial reindexclass_name.constantize.where(id: ids).reindex(:search_conversions)endendend

Run the job with:

ReindexConversionsJob.perform_later("Product")

Advanced

Searchkick makes it easy to use the Elasticsearch DSL on its own.

Advanced Mapping

Create a custom mapping:

classProduct < ActiveRecord::Basesearchkickmappings: {product: {properties: {name: {type: "string",analyzer: "keyword"}}}}end

Note: If you use a custom mapping, you'll need to use custom searching as well.

To keep the mappings and settings generated by Searchkick, use:

classProduct < ActiveRecord::Basesearchkickmerge_mappings: true,mappings: {...}end

Advanced Search

And use the body option to search:

products=Product.searchbody: {match: {name: "milk"}}

Note: This replaces the entire body, so other options are ignored.

View the response with:

products.response

To modify the query generated by Searchkick, use:

products=Product.search"milk",body_options: {min_score: 1}

or

products=Product.search"apples"do |body|
body[:min_score]=1end

Elasticsearch Gem

Searchkick is built on top of the elasticsearch gem. To access the client directly, use:

Searchkick.client

Multi Search

To batch search requests for performance, use:

fresh_products=Product.search("fresh",execute: false)frozen_products=Product.search("frozen",execute: false)Searchkick.multi_search([fresh_products,frozen_products])

Then use fresh_products and frozen_products as typical results.

Note: Errors are not raised as with single requests. Use the error method on each query to check for errors. Also, if you use the below option for misspellings, misspellings will be disabled.

Multiple Indices

Search across multiple indices with:

Searchkick.search"milk",index_name: [Product,Category]

Boost specific indices with:

indices_boost: {Category=>2,Product=>1}

Nested Data

To query nested data, use dot notation.

User.search"san",fields: ["address.city"],where: {"address.zip_code"=>12345}

Search Concepts

Precision and Recall

Precision and recall are two key concepts in search (also known as information retrieval). To help illustrate, let’s walk through an example.

You have a store with 16 types of apples. A user searches for apples gets 10 results. 8 of the results are for apples, and 2 are for apple juice.

Precision is the fraction of documents in the results that are relevant. There are 10 results and 8 are relevant, so precision is 80%.

Recall is the fraction of relevant documents in the results out of all relevant documents. There are 16 apples and only 8 in the results, so recall is 50%.

There’s typically a trade-off between the two. As you tweak your search to increase precision (not return irrelevant documents), there’s are greater chance a relevant document also isn’t returned, which decreases recall. The opposite also applies. As you try to increase recall (return a higher number of relevent documents), there’s a greater chance you also return an irrelevant document, decreasing precision.

Reference

Reindex one record

product=Product.find(1)product.reindex# or to reindex in the backgroundproduct.reindex_async

Reindex multiple records

Product.where(store_id: 1).reindex

Reindex associations

store.products.reindex

Remove old indices

Product.search_index.clean_indices

Use custom settings

classProduct < ActiveRecord::Basesearchkicksettings: {number_of_shards: 3}end

Use a different index name

classProduct < ActiveRecord::Basesearchkickindex_name: "products_v2"end

Use a dynamic index name

classProduct < ActiveRecord::Basesearchkickindex_name: ->{"#{name.tableize}-#{I18n.locale}"}end

Prefix the index name

classProduct < ActiveRecord::Basesearchkickindex_prefix: "datakick"end

Use a different term for boosting by conversions

Product.search("banana",conversions_term: "organic banana")

Multiple conversion fields

classProduct < ActiveRecord::Basehas_many:searches,class_name: "Searchjoy::Search"# searchkick also supports multiple "conversions" fieldssearchkickconversions: ["unique_user_conversions","total_conversions"]defsearch_data{name: name,unique_user_conversions: searches.group(:query).uniq.count(:user_id),# {"ice cream" => 234, "chocolate" => 67, "cream" => 2}total_conversions: searches.group(:query).count# {"ice cream" => 412, "chocolate" => 117, "cream" => 6}}endend

and during query time:

Product.search("banana")# boost by both fields (default)Product.search("banana",conversions: "total_conversions")# only boost by total_conversionsProduct.search("banana",conversions: false)# no conversion boosting

Change timeout

Searchkick.timeout=15# defaults to 10

Set a lower timeout for searches

Searchkick.search_timeout=3

Change the search method name

Searchkick.search_method_name=:lookup

Change search queue name

Searchkick.queue_name=:search_reindex

Eager load associations

Product.search"milk",includes: [:brand,:stores]

Turn off special characters

classProduct < ActiveRecord::Base# A will not match Äsearchkickspecial_characters: falseend

Use a different similarity algorithm for scoring

classProduct < ActiveRecord::Basesearchkicksimilarity: "classic"end

Change import batch size

classProduct < ActiveRecord::Basesearchkickbatch_size: 200# defaults to 1000end

Create index without importing

Product.reindex(import: false)

Lazy searching

products=Product.search("carrots",execute: false)products.each{ ... }# search not executed until here

Add request parameters, like search_type and query_cache

Product.search("carrots",request_params: {search_type: "dfs_query_then_fetch"})

Reindex conditionally

classProduct < ActiveRecord::Basesearchkickcallbacks: false# add the callbacks manuallyafter_commit:reindex,if: ->(model){model.previous_changes.key?("name")}# use your own conditionend

Reindex all models - Rails only

rake searchkick:reindex:all

Turn on misspellings after a certain number of characters

Product.search"api",misspellings: {prefix_length: 2}# api, apt, no ahi

Note: With this option, if the query length is the same as prefix_length, misspellings are turned off

Product.search"ah",misspellings: {prefix_length: 2}# ah, no aha

Testing

For performance, only enable Searchkick callbacks for the tests that need it.

Minitest

Add to your test/test_helper.rb:

# reindex modelsProduct.reindex# and disable callbacksSearchkick.disable_callbacks

And use:

classProductTest < Minitest::TestdefsetupSearchkick.enable_callbacksenddefteardownSearchkick.disable_callbacksenddeftest_searchProduct.create!(name: "Apple")Product.search_index.refreshassert_equal["Apple"],Product.search("apple").map(&:name)endend

RSpec

Add to your spec/spec_helper.rb:

RSpec.configuredo |config|
config.before(:suite)do# reindex modelsProduct.reindex# and disable callbacksSearchkick.disable_callbacksendconfig.around(:each,search: true)do |example|
Searchkick.enable_callbacksexample.runSearchkick.disable_callbacksendend

And use:

describeProduct,search: truedoit"searches"doProduct.create!(name: "Apple")Product.search_index.refreshassert_equal["Apple"],Product.search("apple").map(&:name)endend

Factory Girl

Use a trait and an after create hook for each indexed model:

FactoryGirl.definedofactory:productdo# ...# Note: This should be the last trait in the list so `reindex` is called# after all the other callbacks complete.trait:reindexdoafter(:create)do |product,_evaluator|
product.reindex(refresh: true)endendendend# use itFactoryGirl.create(:product,:some_trait,:reindex,some_attribute: "foo")

Parallel Tests

Set:

Searchkick.index_suffix=ENV["TEST_ENV_NUMBER"]

Multi-Tenancy

Check out this great post on the Apartment gem. Follow a similar pattern if you use another gem.

Upgrading

View the changelog.

Important notes are listed below.

2.0.0

Added support for reindex on associations

Breaking Changes

Removed support for Elasticsearch 1 as it reaches end of life
Removed facets, legacy options, and legacy methods
Invalid options now throw an ArgumentError
The query and json options have been removed in favor of body
The include option has been removed in favor of includes
The personalize option has been removed in favor of boost_where
The partial option has been removed in favor of operator
Renamed select_v2 to select (legacy select no longer available)
The _all field is disabled if searchable option is used (for performance)
The partial_reindex(:method_name) method has been replaced with reindex(:method_name)
The unsearchable and only_analyzed options have been removed in favor of searchable and filterable
load: false no longer returns an array in Elasticsearch 2

1.0.0

Added support for Elasticsearch 2.0
Facets are deprecated in favor of aggregations - see how to upgrade

Breaking Changes

ActiveRecord 4.1+ and Mongoid 3+: Attempting to reindex with a scope now throws a Searchkick::DangerousOperation error to keep your from accidentally recreating your index with only a few records.
```
Product.where(color: "brandy").reindex# error!
```
If this is what you intend to do, use:
```
Product.where(color: "brandy").reindex(accept_danger: true)
```
Misspellings are enabled by default for partial matches. Use misspellings: false to disable.
Transpositions are enabled by default for misspellings. Use misspellings: {transpositions: false} to disable.

0.6.0 and 0.7.0

If running Searchkick 0.6.0 or 0.7.0 and Elasticsearch 0.90, we recommend upgrading to Searchkick 0.6.1 or 0.7.1 to fix an issue that causes downtime when reindexing.

0.3.0

Before 0.3.0, locations were indexed incorrectly. When upgrading, be sure to reindex immediately.

Elasticsearch Gotchas

Consistency

Elasticsearch is eventually consistent, meaning it can take up to a second for a change to reflect in search. You can use the refresh method to have it show up immediately.

product.save!Product.search_index.refresh

Inconsistent Scores

Due to the distributed nature of Elasticsearch, you can get incorrect results when the number of documents in the index is low. You can read more about it here. To fix this, do:

classProduct < ActiveRecord::Basesearchkicksettings: {number_of_shards: 1}end

For convenience, this is set by default in the test environment.

Thanks

Thanks to Karel Minarik for Elasticsearch Ruby and Tire, Jaroslav Kalistsuk for zero downtime reindexing, and Alex Leschenko for Elasticsearch autocomplete.

Roadmap

Reindex API
Incorporate human eval

Contributing

Everyone is encouraged to help improve this project. Here are a few ways you can help:

Report bugs
Fix bugs and submit pull requests
Write, clarify, or fix documentation
Suggest or add new features

If you’re looking for ideas, try here.

To get started with development and testing:

git clone https://github.com/ankane/searchkick.git
cd searchkick
bundle install
rake test

Name		Name	Last commit message	Last commit date
benchmark		benchmark
lib		lib
test		test
.gitignore		.gitignore
.travis.yml		.travis.yml
CHANGELOG.md		CHANGELOG.md
Gemfile		Gemfile
LICENSE.txt		LICENSE.txt
README.md		README.md
Rakefile		Rakefile
searchkick.gemspec		searchkick.gemspec

Latest commit

History

Folders and files

Repository files navigation

Searchkick

Contents

Getting Started

Querying

Results

Boosting

Get Everything

Pagination

Partial Matches

Exact Matches

Phrase Matches

Language

Synonyms

Tags and Dynamic Synonyms

WordNet

Misspellings

Bad Matches

Emoji

Indexing

To Reindex, or Not to Reindex

Reindex

No need to reindex

Stay Synced

Associations

Analytics

Keep Getting Better

Personalized Results

Instant Search / Autocomplete

Here’s how to make it work with Rails

Suggestions

Aggregations

Moving From Facets

Highlight

Similar Items

Geospatial Searches

Boost By Distance

Geo Shapes

Inheritance

Debugging Queries

Deployment

Heroku

Amazon Elasticsearch Service

Other

Automatic Failover

Lograge

Performance

JSON Generation

Persistent HTTP Connections

Searchable Fields

Filterable Fields

Parallel Reindexing

Refresh Interval

Queuing

Routing

Partial Reindexing

Performant Conversions

Advanced

Advanced Mapping

Advanced Search

Elasticsearch Gem

Multi Search

Multiple Indices

Nested Data

Search Concepts

Precision and Recall

Reference

Testing

Minitest

RSpec

Factory Girl

Parallel Tests

Multi-Tenancy

Upgrading

2.0.0

Breaking Changes

1.0.0