Skip to content

[13.0] Optimize queue.job creation - #305

Merged
OCA-git-bot merged 1 commit into
OCA:13.0from
guewen:13.0-optimize-job-create
Feb 4, 2021
Merged

[13.0] Optimize queue.job creation#305
OCA-git-bot merged 1 commit into
OCA:13.0from
guewen:13.0-optimize-job-create

Conversation

@guewen

@guewenguewen commented Feb 2, 2021

Copy link
Copy Markdown
Member

Several fields on queue.job are initialized using computed fields,
then never changed again. On creation of a queue.job record, we'll have
an initial INSERT + at least one following UPDATE for the computed
fields.

Replace all the stored computed fields by a raw initialization of the
values in Job.store() when the job is created, so we have only a single
INSERT.

Highlights:

  • as channel is no longer a compute/inverse field, override_channel is
    useless, I dropped it (actually the same value was stored in both
    channel and override_channel as the channel field was stored)
  • one functional diff is that now, when changing a channel on a
    job.function, the channel is no longer synchronized on existing jobs,
    it will be applied only on new jobs: actually this is an improvement,
    because it was impossible to change the channel of a job function
    in a large queue_job table as it meant writing on all the done/started
    jobs
  • searching the queue.job.function is now cached, as each job using the
    same will run a query on queue_job_function

Rough benchmark when generating jobs for ddmrp buffers (it includes some base
time to fetch the ddmrp buffers too). 9191 jobs were created:

executed withduration
no optimizations2m18s
create optimization (current PR)1m41s
mail.message/follower optimization (#302)1m11s
both optimizations35s

@guewen
guewenforce-pushed the 13.0-optimize-job-create branch 2 times, most recently from 75f9b70 to ea57079CompareFebruary 2, 2021 07:32
Comment threadqueue_job/models/queue_job.py Outdated
Several fields on queue.job are initialized using computed fields,
then never changed again. On creation of a queue.job record, we'll have
an initial INSERT + at least one following UPDATE for the computed
fields.
Replace all the stored computed fields by a raw initialization of the
values in `Job.store()` when the job is created, so we have only a
single INSERT.
Highlights:
* as channel is no longer a compute/inverse field, override_channel is
useless, I dropped it (actually the same value was stored in both
channel and override_channel as the channel field was stored)
* one functional diff is that now, when changing a channel on a
job.function, the channel is no longer synchronized on existing jobs,
it will be applied only on new jobs: actually this is an improvement,
because it was impossible to change the channel of a job function
in a large queue_job table as it meant writing on all the done/started
jobs
* searching the queue.job.function is now cached, as each job using the
same will run a query on queue_job_function
@guewen

Copy link
Copy Markdown
MemberAuthor

Added a small benchmark in the description

TDu
TDu approved these changes Feb 3, 2021
@simahawk

Copy link
Copy Markdown
Contributor

/ocabot merge minor

@OCA-git-bot

Copy link
Copy Markdown
Contributor

This PR looks fantastic, let's merge it!
Prepared branch 13.0-ocabot-merge-pr-305-by-simahawk-bump-minor, awaiting test results.

@OCA-git-bot

Copy link
Copy Markdown
Contributor

Congratulations, your PR was merged at 15dc56f. Thanks a lot for contributing to OCA. ❤️

Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants

@guewen@simahawk@OCA-git-bot@TDu