[16.0][IMP] queue_job: remove dead jobs requeuer cron and automatically requeue dead jobs

[AnizR]

Goal

Automatically re-queue jobs that have been started but whose worker have been killed.
And rely on Odoo's limit_time_cpu and limit_time_real for job execution.

Technical explanation

Everything relies on a new table queue_job_locks which contains ids of jobs that have been started.
When a job is executed, its row queue_job_locks is locked.

If a row is in queue_job_locks with a state='started' but not locked, it is either:

• a job killed
• a job not yet started (happens in a very small time frame)

Using this information, we can re-queue these jobs.

Why not lock directly in the `queue_job' table?

This was tried in #423 but it wasn't working when a job was raising an error.
It seems that the row was locked and it tried to write on the row to set as failed before committing.

Improve current behavior

Re-queue jobs that have been killed but increment their 'retries' to avoid having a job that is always get killed in infinite re-queuing.

[OCA-git-bot]

Hi @guewen,
some modules you are maintaining are being modified, check this out!

[sbidoul]

This looks very promising!

A few thoughts:

• There is a Caveat comment in runner.py that needs updating.
• This should handle the enqueued state too, which is the state when the runner has decided that a jobs needs to run but then /queue_job/runjob controller has not set it started yet. This state normally exists for a very short time, but I have seen situations where the workers are overloaded and take time to accept /queue_job/runjob requests then die, leaving jobs in enqueued state forever.
• Since there are two race conditions between enqueued and started, and between started and the time the job transaction actually starts with the lock, I wonder if we should not introduce a small elapsed time condition (10s?) in reset_dead_jobs. May be based on date_enqueued.

[AnizR]

This looks very promising!

A few thoughts:

* There is a Caveat comment in `runner.py` that needs updating.
* This should handle the `enqueued` state too, which is the state when the runner has decided that a jobs needs to run but then `/queue_job/runjob` controller has not set it started yet. This state normally exists for a very short time, but I have seen situations where the workers are overloaded and take time to accept `/queue_job/runjob` requests then die, leaving jobs in `enqueued` state forever.
* Since there are two race conditions between `enqueued` and `started`, and between `started` and the time the job transaction actually starts with the lock, I wonder if we should not introduce a small elapsed time condition (10s?) in `reset_dead_jobs`. May be based on `date_enqueued`.

Thank you for your suggestions. I have implemented the necessary corrections.
While I have not encountered a case where a job is left in the 'enqueued' state, I have made the appropriate correction to ensure the code is more robust.

[sbidoul]

A few more minor comments.

[sbidoul]

Why adding this? This sounds unrelated.

[AnizR]

The check that max_retries is greater than retry was done only when a RetryableJobError was raised during the execution of the job.

Since I am increasing the retry when re-queuing a job that was killed, I need to do the check before peforming the job, not only when raising an exception.

Moreover, I think that it doesn't make sense to perform the check only when an exception has been raised 🤔

[guewen]

Thanks for tackling this old issue, I like this elegant solution. It should be quite optimized also. Congrats for this work

[amh-mw]

Should this become

<deleteid="ir_cron_queue_job_garbage_collector"model="ir.cron"/>

to clean up upgrades / avoid filling cron logs with errors since requeue_stuck_jobs method is gone?

[AnizR]

I have archived the cron in the pre-migration.py. Therefore, there won't be any error.
I always prefer to archive (set active to false) rather than deleting.

[sbidoul]

@AnizR I think we can also remove the set_job_pending function, as the new mechanism will take care of that part.

[AnizR]

@AnizR I think we can also remove the set_job_pending function, as the new mechanism will take care of that part.

Yes, jobs that have not been started will be re-queued by my new mechanism.
Removing this part will provide a more "uniform" process of re-queuing jobs.

[lmignon]

Thank you @AnizR for all your hard work. For the last 3 weeks, this PR has been working perfectly in production. We no longer encounter dead jobs and the retry error handling works even better than before.

[AnizR]

Thank you @AnizR for all your hard work. For the last 3 weeks, this PR has been working perfectly in production. We no longer encounter dead jobs and the retry error handling works even better than before.

I am happy to contribute and see that it works and is used 👍

I have also used this PR on a production environment without any issue.
I think that it can be merged

[sbidoul]

@amh-mw have your concerns been addressed?

[amh-mw]

LGTM

[sbidoul]

Let's go.

/ocabot merge minor

[OCA-git-bot]

What a great day to merge this nice PR. Let's do it!
Prepared branch 16.0-ocabot-merge-pr-716-by-sbidoul-bump-minor, awaiting test results.

[OCA-git-bot]

This PR has the approved label and has been created more than 5 days ago. It should therefore be ready to merge by a maintainer (or a PSC member if the concerned addon has no declared maintainer). 🤖

[OCA-git-bot]

Congratulations, your PR was merged at a28339b. Thanks a lot for contributing to OCA. ❤️

[sbidoul]

FW port to 17.0 in #748