Run any code in parallel Processes(> use all CPUs) or Threads(> speedup blocking operations).
Best suited for map-reduce or e.g. parallel downloads/uploads.

Install

gem install parallel

Usage

# 2 CPUs -> work in 2 processes (a,b + c)results=Parallel.map(['a','b','c'])do |one_letter|
expensive_calculation(one_letter)end# 3 Processes -> finished after 1 runresults=Parallel.map(['a','b','c'],:in_processes=>3){|one_letter| ... }# 3 Threads -> finished after 1 runresults=Parallel.map(['a','b','c'],:in_threads=>3){|one_letter| ... }

Same can be done with each

Parallel.each(['a','b','c']){|one_letter| ... }

or each_with_index or map_with_index

Produce one item at a time with lambda (anything that responds to .call) or Queue.

items=[1,2,3]Parallel.each(lambda{items.pop || Parallel::Stop}){|number| ... }

Processes/Threads are workers, they grab the next piece of work when they finish.

Processes

• Speedup through multiple CPUs
• Speedup for blocking operations
• Protects global data
• Extra memory used ( very low on REE through copy_on_write_friendly )
• Child processes are killed when your main process is killed through Ctrl+c or kill -2

Threads

• Speedup for blocking operations
• Global data can be modified
• No extra memory used

ActiveRecord

Try any of those to get working parallel AR

# reproducibly fixes things (spec/cases/map_with_ar.rb)Parallel.each(User.all,:in_processes=>8)do |user|
user.update_attribute(:some_attribute,some_value)endUser.connection.reconnect!# maybe helps: explicitly use connection poolParallel.each(User.all,:in_threads=>8)do |user|
ActiveRecord::Base.connection_pool.with_connectiondouser.update_attribute(:some_attribute,some_value)endend# maybe helps: reconnect once inside every forkParallel.each(User.all,:in_processes=>8)do |user|
@reconnected ||= User.connection.reconnect! || trueuser.update_attribute(:some_attribute,some_value)end

Break

Parallel.map(User.all)do |user|
raiseParallel::Break# -> stops after all current items are finishedend

Kill

Only use if whatever is executing in the sub-command is safe to kill at any point

Parallel.map([1,2,3]) do |x|
raise Parallel::Kill if x == 1# -> stop all sub-processes, killing them instantly
sleep 100
end

Progress / ETA

# gem install ruby-progressbarParallel.map(1..50,:progress=>"Doing stuff"){sleep1}# Doing stuff | ETA: 00:00:02 | ==================== | Time: 00:00:10

Use :finish or :start hook to get progress information.

• :start has item and index
• :finish has item, index, result

They are called on the main process and protected with a mutex.

Parallel.map(1..100,:finish=>lambda{ |item,i,result| ... dosomething ... }){sleep1}

Tips

• [Benchmark/Test] Disable threading/forking with :in_threads => 0 or :in_processes => 0, great to test performance or to debug parallel issues

TODO

• Replace Signal trapping with simple rescue Interrupt handler

Authors

Contributors

• Przemyslaw Wroblewski
• TJ Holowaychuk
• Masatomo Nakano
• Fred Wu
• mikezter
• Jeremy Durham
• Nick Gauthier
• Andrew Bowerman
• Byron Bowerman
• Mikko Kokkonen
• brian p o'rourke
• [Norio Sato]
• Neal Stewart
• Jurriaan Pruis
• Rob Worley
• Tasveer Singh
• Joachim
• yaoguai
• Bartosz Dziewoński
• yaoguai
• Guillaume Hain
• Adam Wróbel
• Matthew Brennan
• Brendan Dougherty

Michael Grosser
michael@grosser.it
License: MIT