Sunspot

Sunspot is a Ruby library for expressive, powerful interaction with the Solr search engine. Sunspot is built on top of the RSolr library, which provides a low-level interface for Solr interaction; Sunspot provides a simple, intuitive, expressive DSL backed by powerful features for indexing objects and searching for them.

Sunspot is designed to be easily plugged in to any ORM, or even non-database-backed objects such as the filesystem.

This README provides a high level overview; class-by-class and method-by-method documentation is available in the API reference.

For questions about how to use Sunspot in your app, please use the Sunspot Mailing List or search Stack Overflow.

Quickstart with Rails 3 / 4

Add to Gemfile:

gem'sunspot_rails'gem'sunspot_solr'# optional pre-packaged Solr distribution for use in development

Bundle it!

bundle install

Generate a default configuration file:

rails generate sunspot_rails:install

If sunspot_solr was installed, start the packaged Solr distribution with:

bundle exec rake sunspot:solr:start # or sunspot:solr:run to start in foreground

Setting Up Objects

Add a searchable block to the objects you wish to index.

classPost < ActiveRecord::Basesearchabledotext:title,:bodytext:commentsdocomments.map{ |comment| comment.body}endboolean:featuredinteger:blog_idinteger:author_idinteger:category_ids,:multiple=>truedouble:average_ratingtime:published_attime:expired_atstring:sort_titledotitle.downcase.gsub(/^(an?|the)/,'')endendend

text fields will be full-text searchable. Other fields (e.g., integer and string) can be used to scope queries.

Searching Objects

Post.searchdofulltext'best pizza'with:blog_id,1with(:published_at).less_thanTime.noworder_by:published_at,:descpaginate:page=>2,:per_page=>15facet:category_ids,:author_idend

Search In Depth

Given an object Post setup in earlier steps ...

Full Text

# All posts with a `text` field (:title, :body, or :comments) containing 'pizza'Post.search{fulltext'pizza'}# Posts with pizza, scored higher if pizza appears in the titlePost.searchdofulltext'pizza'doboost_fields:title=>2.0endend# Posts with pizza, scored higher if featuredPost.searchdofulltext'pizza'doboost(2.0){with(:featured,true)}endend# Posts with pizza *only* in the titlePost.searchdofulltext'pizza'dofields(:title)endend# Posts with pizza in the title (boosted) or in the body (not boosted)Post.searchdofulltext'pizza'dofields(:body,:title=>2.0)endend

Phrases

Solr allows searching for phrases: search terms that are close together.

In the default query parser used by Sunspot (edismax), phrase searches are represented as a double quoted group of words.

# Posts with the exact phrase "great pizza"Post.searchdofulltext'"great pizza"'end

If specified, query_phrase_slop sets the number of words that may appear between the words in a phrase.

# One word can appear between the words in the phrase, so "great big pizza"# also matches, in addition to "great pizza"Post.searchdofulltext'"great pizza"'doquery_phrase_slop1endend

Phrase Boosts

Phrase boosts add boost to terms that appear in close proximity; the terms do not have to appear in a phrase, but if they do, the document will score more highly.

# Matches documents with great and pizza, and scores documents more# highly if the terms appear in a phrase in the title fieldPost.searchdofulltext'great pizza'dophrase_fields:title=>2.0endend# Matches documents with great and pizza, and scores documents more# highly if the terms appear in a phrase (or with one word between them)# in the title fieldPost.searchdofulltext'great pizza'dophrase_fields:title=>2.0phrase_slop1endend

Scoping (Scalar Fields)

Fields not defined as text (e.g., integer, boolean, time, etc...) can be used to scope (restrict) queries before full-text matching is performed.

Positive Restrictions

# Posts with a blog_id of 1Post.searchdowith(:blog_id,1)end# Posts with an average rating between 3.0 and 5.0Post.searchdowith(:average_rating,3.0..5.0)end# Posts with a category of 1, 3, or 5Post.searchdowith(:category_ids,[1,3,5])end# Posts published since a week agoPost.searchdowith(:published_at).greater_than(1.week.ago)end

Negative Restrictions

# Posts not in category 1 or 3Post.searchdowithout(:category_ids,[1,3])end# All examples in "positive" also work negated using `without`

Empty Restrictions

# Passing an empty array is equivalent to a no-op, allowing you to replace this...Post.searchdowith(:category_ids,id_list)ifid_list.present?end# ...with thisPost.searchdowith(:category_ids,id_list)end

Disjunctions and Conjunctions

# Posts that do not have an expired time or have not yet expiredPost.searchdoany_ofdowith(:expired_at).greater_than(Time.now)with(:expired_at,nil)endend

# Posts with blog_id 1 and author_id 2Post.searchdoall_ofdowith(:blog_id,1)with(:author_id,2)endend

Disjunctions and conjunctions may be nested

Post.searchdoany_ofdowith(:blog_id,1)all_ofdowith(:blog_id,2)with(:category_ids,3)endendend

Combined with Full-Text

Scopes/restrictions can be combined with full-text searching. The scope/restriction pares down the objects that are searched for the full-text term.

# Posts with blog_id 1 and 'pizza' in the titlePost.searchdowith(:blog_id,1)fulltext("pizza")end

Pagination

All results from Solr are paginated

The results array that is returned has methods mixed in that allow it to operate seamlessly with common pagination libraries like will_paginate and kaminari.

By default, Sunspot requests the first 30 results from Solr.

search=Post.searchdofulltext"pizza"end# Imagine there are 60 *total* results (at 30 results/page, that is two pages)results=search.results# => Array with 30 Post elementssearch.total# => 60results.total_pages# => 2results.first_page?# => trueresults.last_page?# => falseresults.previous_page# => nilresults.next_page# => 2results.out_of_bounds?# => falseresults.offset# => 0

To retrieve the next page of results, recreate the search and use the paginate method.

search=Post.searchdofulltext"pizza"paginate:page=>2end# Again, imagine there are 60 total results; this is the second pageresults=search.results# => Array with 30 Post elementssearch.total# => 60results.total_pages# => 2results.first_page?# => falseresults.last_page?# => trueresults.previous_page# => 1results.next_page# => nilresults.out_of_bounds?# => falseresults.offset# => 30

A custom number of results per page can be specified with the :per_page option to paginate:

search=Post.searchdofulltext"pizza"paginate:page=>1,:per_page=>50end

Faceting

Faceting is a feature of Solr that determines the number of documents that match a given search and an additional criterion. This allows you to build powerful drill-down interfaces for search.

Each facet returns zero or more rows, each of which represents a particular criterion conjoined with the actual query being performed. For field facets, each row represents a particular value for a given field. For query facets, each row represents an arbitrary scope; the facet itself is just a means of logically grouping the scopes.

By default Sunspot will only return the first 100 facet values. You can increase this limit, or force it to return all facets by setting limit to -1.

Field Facets

# Posts that match 'pizza' returning counts for each :author_idsearch=Post.searchdofulltext"pizza"facet:author_idendsearch.facet(:author_id).rows.eachdo |facet|
puts"Author #{facet.value} has #{facet.count} pizza posts!"end

If you are searching by a specific field and you still want to see all the options available in that field you can exclude it in the faceting.

# Posts that match 'pizza' and author with id 42# Returning counts for each :author_id (even those not in the search result)search=Post.searchdofulltext"pizza"author_filter=with(:author_id,42)facet:author_id,exclude: [author_filter]endsearch.facet(:author_id).rows.eachdo |facet|
puts"Author #{facet.value} has #{facet.count} pizza posts!"end

Query Facets

# Posts faceted by ranges of average ratingssearch=Post.searchdofacet(:average_rating)dorow(1.0..2.0)dowith(:average_rating,1.0..2.0)endrow(2.0..3.0)dowith(:average_rating,2.0..3.0)endrow(3.0..4.0)dowith(:average_rating,3.0..4.0)endrow(4.0..5.0)dowith(:average_rating,4.0..5.0)endendend# e.g.,# Number of posts with rating within 1.0..2.0: 2# Number of posts with rating within 2.0..3.0: 1search.facet(:average_rating).rows.eachdo |facet|
puts"Number of posts with rating within #{facet.value}: #{facet.count}"end

Range Facets

# Posts faceted by range of average ratingsSunspot.search(Post)dofacet:average_rating,:range=>1..5,:range_interval=>1end

Ordering

By default, Sunspot orders results by "score": the Solr-determined relevancy metric. Sorting can be customized with the order_by method:

# Order by average rating, descendingPost.searchdofulltext("pizza")order_by(:average_rating,:desc)end# Order by relevancy score and in the case of a tie, average ratingPost.searchdofulltext("pizza")order_by(:score,:desc)order_by(:average_rating,:desc)end# Randomized orderingPost.searchdofulltext("pizza")order_by(:random)end

Solr 3.1 and above

Solr supports sorting on multiple fields using custom functions. Supported operators and more details are available on the Solr Wiki

To sort results by a custom function use the order_by_function method. Functions are defined with prefix notation:

# Order by sum of two example fields: rating1 + rating2Post.searchdofulltext("pizza")order_by_function(:sum,:rating1,:rating2,:desc)end# Order by nested functions: rating1 + (rating2*rating3)Post.searchdofulltext("pizza")order_by_function(:sum,:rating1,[:product,:rating2,:rating3],:desc)end# Order by fields and constants: rating1 + (rating2 * 5)Post.searchdofulltext("pizza")order_by_function(:sum,:rating1,[:product,:rating2,'5'],:desc)end# Order by average of three fields: (rating1 + rating2 + rating3) / 3Post.searchdofulltext("pizza")order_by_function(:div,[:sum,:rating1,:rating2,:rating3],'3',:desc)end

Grouping

Solr 3.3 and above

Solr supports grouping documents, similar to an SQL GROUP BY. More information about result grouping/field collapsing is available on the Solr Wiki.

Grouping is only supported on string fields that are not multivalued. To group on a field of a different type (e.g., integer), add a denormalized string type

classPost < ActiveRecord::Basesearchabledo# Denormalized `string` field because grouping can only be performed# on string fieldsstring(:blog_id_str){ |p| p.blog_id.to_s}endend# Returns only the top scoring document per blog_idsearch=Post.searchdogroup:blog_id_strendsearch.group(:blog_id_str).matches# Total number of matches to the querysearch.group(:blog_id_str).groups.eachdo |group|
putsgroup.value# blog_id of the each document in the group# By default, there is only one document per group (the highest# scoring one); if `limit` is specified (see below), multiple# documents can be returned per groupgroup.results.eachdo |result|
# ...endend

Additional options are supported by the DSL:

# Returns the top 3 scoring documents per blog_idPost.searchdogroup:blog_id_strdolimit3endend# Returns document ordered within each group by published_at (by# default, the ordering is score)Post.searchdogroup:blog_id_strdoorder_by(:average_rating,:desc)endend# Facet count is based on the most relevant document of each group# matching the query (>= Solr 3.4)Post.searchdogroup:blog_id_strdotruncateendfacet:blog_id_str,:extra=>:anyend

Geospatial

Sunspot 2.0 only

Sunspot 2.0 supports geospatial features of Solr 3.1 and above.

Geospatial features require a field defined with latlon:

classPost < ActiveRecord::Basesearchabledo# ...latlon(:location){Sunspot::Util::Coordinates.new(lat,lon)}endend

Filter By Radius

# Searches posts within 100 kilometers of (32, -68)Post.searchdowith(:location).in_radius(32, -68,100)end

Filter By Radius (inexact with bbox)

# Searches posts within 100 kilometers of (32, -68) with `bbox`. This is# an approximation so searches run quicker, but it may include other# points that are slightly outside of the required distancePost.searchdowith(:location).in_radius(32, -68,100,:bbox=>true)end

Filter By Bounding Box

# Searches posts within the bounding box defined by the corners (45,# -94) to (46, -93)Post.searchdowith(:location).in_bounding_box([45, -94],[46, -93])end

Sort By Distance

# Orders documents by closeness to (32, -68)Post.searchdoorder_by_geodist(:location,32, -68)end

Joins

Solr 4 and above

Solr joins allow you to filter objects by joining on additional documents. More information can be found on the Solr Wiki.

classPhoto < ActiveRecord::Basesearchabledotext:caption,:default_boost=>1.5time:created_atinteger:photo_container_idendendclassPhotoContainer < ActiveRecord::Basesearchabledotext:namejoin(:caption,:type=>:string,:join_string=>'from=photo_container_id to=id')join(:photos_created,:type=>:time,:join_string=>'from=photo_container_id to=id',:as=>'created_at_d')endendPhotoContainer.searchdowith(:caption,'blah')with(:photos_created).between(Date.new(2011,3,1),Date.new(2011,4,1))end

Highlighting

Highlighting allows you to display snippets of the part of the document that matched the query.

The fields you wish to highlight must be stored.

classPost < ActiveRecord::Basesearchabledo# ...text:body,:stored=>trueendend

Highlighting matches on the body field, for instance, can be achieved like:

search=Post.searchdofulltext"pizza"dohighlight:bodyendend# Will output something similar to:# Post #1# I really love *pizza*# *Pizza* is my favorite thing# Post #2# Pepperoni *pizza* is delicioussearch.hits.eachdo |hit|
puts"Post ##{hit.primary_key}"hit.highlights(:body).eachdo |highlight|
puts" " + highlight.format{ |word| "*#{word}*"}endend

Stats

Solr can return some statistics on indexed numeric fields. Fetching statistics for average_rating:

search=Post.searchdostats:average_ratingendputs"Minimum average rating: #{search.stats(:average_rating).min}"puts"Maximum average rating: #{search.stats(:average_rating).max}"

Stats on multiple fields

search=Post.searchdostats:average_rating,:blog_idend

Faceting on stats

It's possible to facet field stats on another field:

search=Post.searchdostats:average_ratingdofacet:featuredendendsearch.stats(:average_rating).facet(:featured).rowsdo |row|
puts"Minimum average rating for featured=#{row.value}: #{row.min}"end

Take care when requesting facets on a stats field, since all facet results are returned by Solr!

Multiple stats and selective faceting

search=Post.searchdostats:average_ratingdofacet:featuredendstats:blog_iddofacet:average_ratingendend

Functions

TODO

More Like This

Sunspot can extract related items using more_like_this. When searching for similar items, you can pass a block with the following options:

• fields :field_1[, :field_2, ...]
• minimum_term_frequency ##
• minimum_document_frequency ##
• minimum_word_length ##
• maximum_word_length ##
• maximum_query_terms ##
• boost_by_relevance true/false

classPost < ActiveRecord::Basesearchabledo# The :more_like_this option must be set to truetext:body,:more_like_this=>trueendendpost=Post.firstresults=Sunspot.more_like_this(post)dofields:bodyminimum_term_frequency5end

Indexes In Depth

TODO

Index-Time Boosts

To specify that a field should be boosted in relation to other fields for all queries, you can specify the boost at index time:

classPost < ActiveRecord::Basesearchabledotext:title,:boost=>5.0text:bodyendend

Stored Fields

Stored fields keep an original (untokenized/unanalyzed) version of their contents in Solr.

Stored fields allow data to be retrieved without also hitting the underlying database (usually an SQL server). They are also required for highlighting and more like this queries.

Stored fields come at some performance cost in the Solr index, so use them wisely.

classPost < ActiveRecord::Basesearchabledotext:body,:stored=>trueendend# Retrieving stored contents without hitting the databasePost.search.hits.eachdo |hit|
putshit.stored(:body)end

Hits vs. Results

Sunspot simply stores the type and primary key of objects in Solr. When results are retrieved, those primary keys are used to load the actual object (usually from an SQL database).

# Using #results pulls in the records from the object-relational# mapper (e.g., ActiveRecord + a SQL server)Post.search.results.eachdo |result|
putsresult.bodyend

To access information about the results without querying the underlying database, use hits:

# Using #hits gives back all information requested from Solr, but does# not load the object from the object-relational mapperPost.search.hits.eachdo |hit|
putshit.stored(:body)end

If you need both the result (ORM-loaded object) and Hit (e.g., for faceting, highlighting, etc...), you can use the convenience method each_hit_with_result:

Post.search.each_hit_with_resultdo |hit,result|
# ...end

Reindexing Objects

If you are using Rails, objects are automatically indexed to Solr as a part of the save callbacks.

There are a number of ways to index manually within Ruby:

# On a class itselfPerson.reindexSunspot.commit# On mixed objectsSunspot.index[post1,item2]Sunspot.indexperson3Sunspot.commit# With autocommitSunspot.index![post1,item2,person3]

If you make a change to the object's "schema" (code in the searchable block), you must reindex all objects so the changes are reflected in Solr:

bundle exec rake sunspot:solr:reindex
# or, to be specific to a certain model with a certain batch size:
bundle exec rake sunspot:solr:reindex[500,Post] # some shells will require escaping [ with \[ and ] with \]# to skip the prompt asking you if you want to proceed with the reindexing:
bundle exec rake sunspot:solr:reindex[,,true] # some shells will require escaping [ with \[ and ] with \]

Use Without Rails

TODO

Threading

The default Sunspot Session is not thread-safe. If used in a multi-threaded environment (such as sidekiq), you should configure Sunspot to use the ThreadLocalSessionProxy:

Sunspot.session=Sunspot::SessionProxy::ThreadLocalSessionProxy.new

Manually Adjusting Solr Parameters

To add or modify parameters sent to Solr, use adjust_solr_params:

Post.searchdoadjust_solr_paramsdo |params|
params[:q] += " AND something_s:more"endend

Session Proxies

TODO

Type Reference

TODO

Configuration

Configure Sunspot by creating a config/sunspot.yml file or by setting a SOLR_URL or a WEBSOLR_URL environment variable. The defaults are as follows.

development:
solr:
hostname: localhostport: 8982log_level: INFOtest:
solr:
hostname: localhostport: 8981log_level: WARNING

You may want to use SSL for production environments with a username and password. For example, set SOLR_URL to https://username:password@production.solr.example.com/solr.

You can examine the value of Sunspot::Rails.configuration at runtime.

Development

Running Tests

sunspot

Install the required gem dependencies:

cd /path/to/sunspot/sunspot
bundle install

Start a Solr instance on port 8983:

bundle exec sunspot-solr start -p 8983
# or `bundle exec sunspot-solr run -p 8983` to run in foreground

Run the tests:

bundle exec rake spec

If desired, stop the Solr instance:

bundle exec sunspot-solr stop

sunspot_rails

Install the gem dependencies for sunspot:

cd /path/to/sunspot/sunspot
bundle install

Start a Solr instance on port 8983:

bundle exec sunspot-solr start -p 8983
# or `bundle exec sunspot-solr run -p 8983` to run in foreground

Navigate to the sunspot_rails directory:

cd ../sunspot_rails

Run the tests:

rake spec # all Rails versions
rake spec RAILS=3.1.1 # specific Rails version only

If desired, stop the Solr instance:

cd ../sunspot
bundle exec sunspot-solr stop

Generating Documentation

Install the yard and redcarpet gems:

$ gem install yard redcarpet

Uninstall the rdiscount gem, if installed:

$ gem uninstall rdiscount

Generate the documentation from topmost directory:

$ yardoc -o docs */lib/**/*.rb - README.md

Tutorials and Articles

• Using Sunspot, Websolr, and Solr on Heroku (mrdanadams)
• Full Text Searching with Solr and Sunspot (Collective Idea)
• Full-text search in Rails with Sunspot (Tropical Software Observations)
• Sunspot Full-text Search for Rails/Ruby (The Rail World)
• A Few Sunspot Tips (spiral_code)
• Sunspot: A Solr-Powered Search Engine for Ruby (Linux Magazine)
• Sunspot Showed Me the Light (ben koonse)
• RubyGems.org — A case study in upgrading to full-text search (Websolr)
• How to Implement Spatial Search with Sunspot and Solr (Code Quest)
• Sunspot 1.2 with Spatial Solr Plugin 2.0 (joelmats)
• rails3 + heroku + sunspot : madness (anhaminha)
• heroku + websolr + sunspot (Onemorecloud)
• How to get full text search working with Sunspot (Hobo Cookbook)
• Full text search with Sunspot in Rails (hemju)
• Using Sunspot for Free-Text Search with Redis (While I Pondered...)
• Fuzzy searching in SOLR with Sunspot (pipe :to => /dev/null)
• Default scope with Sunspot (Cloudspace)
• Index External Models with Sunspot/Solr (Medihack)
• Testing with Sunspot and Cucumber (Collective Idea)
• Cucumber and Sunspot (opensoul.org)
• Testing Sunspot with Cucumber (spiral_code)
• Running cucumber features with sunspot_rails (Kabisa Blog)
• Testing Sunspot with Test::Unit (Type Slowly)
• Sunspot Quickstart (WebSolr)
• Solr, and Sunspot (YT!)
• The Saga of the Switch (mrb -- includes comparison of Sunspot and Ultrasphinx)
• Conditional Indexing with Sunspot (mikepack)
• Introduction to Full Text Search for Rails Developers (Valve's)

License

Sunspot is distributed under the MIT License, copyright (c) 2008-2013 Mat Brown