[11.0][ADD] queue_job_batch

[etobella]

When you split a single process in small processes managed with queue you may have problems in order to know that everything finished. For example, if you try to execute 1000 contracts, you may want to split them by customer and execute a job for each customer. But you are unable to know that everything finished.
This PR tries to solve this issue creating Queue Job Groups.
It allows to show your jobs in a grouped form in order to know better the
results.

from odoo import models, fields, api
from odoo.addons.queue_job.job import job
class MyModel(models.Model):
_name = 'my.model'
@api.multi
@job
def my_method(self, a, k=None):
_logger.info('executed with a: %s and k: %s', a, k)
class MyOtherModel(models.Model):
_name = 'my.other.model'
@api.multi
def button_do_stuff(self):
group = self.env['queue.job.group'].get_new_group('Group')
for i in range(1, 100):
self.env['my.model'].with_context(
job_group=group
).with_delay().my_method('a', k=i)
group.enqueue()

In the snippet of code above, when we call button_do_stuff, 100 jobs
capturing the method and arguments will be postponed. It will be executed as
soon as the Jobrunner has a free bucket, which can be instantaneous if no other
job is running.

Once all the jobs have finished, the grouper will be marked as finished.

You can manage your group jobs from the Systray. A new button will be shown
with your currently executing job groups and the recently finished job groups.

A testing module is installed in order to test that everything works as expected.

[etobella]

ping @jbeficent @jarroyomorales

[jarroyomorales]

Maybe you could add this style to the view all button, I think it will fit more

.o_view_all_group_jobs {
.btn-link;
padding: 5px;
}

[jarroyomorales]

LGTM, and very useful module 👍

[guewen]

Looks interesting, I'll take some time for reviewing asap.
Out of curiosity, did you see these specs and had you any reason to follow another way? #129

[etobella]

Dependencies should be used on a graph of jobs, this tries to solve when you need to know when one has finished (order is not important). It could be a special case of your implementation, but it is easier to solve this use case. I don't know if this happens in other cases, but we could use it in a lot of places (validation of multiple invoices, generation of contract invoices...)

[guewen]

Dependencies should be used on a graph of jobs, this tries to solve when you need to know when one has finished (order is not important).

If I rephrase, the groups are used to show the group's state to users, but they are not used to delay dependent jobs, do I understand correctly?

It could be a special case of your implementation, but it is easier to solve this use case.

Yes probably, I think that with my implementation proposal, we could propose a view of the graph.

Anyway, my proposal is only what it is now: a proposal with no time yet to implement it, when you have an implementation for your use case. I only want to avoid 2 solutions to grow concurrently in different directions, but I recon that should not be the case here.

[etobella]

If I rephrase, the groups are used to show the group's state to users, but they are not used to delay dependent jobs, do I understand correctly?

You are right, the group is only for information.

I made a simple video showing it:

[sbidoul]

Nice feature!

A quick question: is the new "waiting" state on queue jobs absolutely necessary?

[etobella]

Probably not, but I think that you need to wait to enqueue all the jobs. For that reason, I created this state.

Do you think that it is better to create the job without checking the state?

[guewen]

As group.enqueue() is meant to be called in the same transaction, jobs will not be picked up by the jobrunner until the transaction is committed, so I tend to say that the state is not needed.

[guewen]

`` is meant to be called in the same transaction

Or maybe it isn't?
As #129 mentions a similar state (wait_dependency there), we could maybe share a single state for both use cases?

Regarding the naming, would you accept to rename Group to Batch to prevent a confusion with the Group notion of #129? (I'd like to keep the Group term in the dependencies if possible to keep the same terms than celery)

[etobella]

sure! I will change it batch.
Can I change the state behavior or do you think it is better to wait?

[etobella]

@guewen Changed to batch 😄

[etobella]

I think that this PR should wait for #154.

Or maybe we can post here the creation of a new state for waiting. What do you think @guewen?
Should we include this approach to the dependencies system?

[JordiBForgeFlow]

Tested with #155 and works very well!

[OCA-git-bot]

This PR has the approved label and has been created more than 5 days ago. It should therefore be ready to merge by a maintainer (or a PSC member if the concerned addon has no declared maintainer). 🤖

[etobella]

@jbeficent Comments attended 😄

[sbidoul]

I'm still not convinced a new job state is needed for this, even if jobs are created in several transactions.

Would it be a problem if jobs starts while being in a draft batch?

[etobella]

Well, it is not a problem for me to remove the state. @jbeficent What do you think?

[etobella]

@sbidoul I removed the state. I think this solutions is a better approach for you comments

[guewen]

Thanks for the changes, this is great :)

[JordiBForgeFlow]

/ocabot merge

[OCA-git-bot]

What a great day to merge this nice PR. Let's do it!
Rebased to 11.0-ocabot-merge-pr-151-by-jbeficent-bump-no, awaiting test results.

[OCA-git-bot]

Congratulations, your PR was merged at 0e4aba3. Thanks a lot for contributing to OCA. ❤️

PS: Don't worry if GitHub says there are unmerged commits: it is due to a rebase before merge. All commits of this PR have been merged into 11.0.