Fix OpenLineage DeprecationWarning about fork() on Python 3.12+

[br413]

Summary

• Remove \os.fork()-based task event emission in the OpenLineage listener
• Extract operator metadata in the parent process; emit lifecycle events via picklable _emit_task_instance_event()\ and \ProcessPoolExecutor\ with \orkserver\ context
• Use \orkserver\ multiprocessing context for scheduler-side DAG-run and manual state-change emissions

Problem

On Python 3.12+, the OpenLineage listener emits:

\
DeprecationWarning: This process is multi-threaded, use of fork() may lead to deadlocks in the child.
\\

This came from _fork_execute()\ calling \os.fork()\ during task lifecycle events, and from \ProcessPoolExecutor\ defaulting to the \ork\ start method on Linux.

Approach

Task lifecycle handlers (\on_running, \on_success, \on_failure, \on_skipped) follow the same two-phase pattern as DAG-run listeners:

1. Parent process: metadata extraction while ORM/task objects are available
2. Pool worker (forkserver): adapter emission + metrics via \submit_callable(_emit_task_instance_event, ...)\

This preserves process isolation for OL transport code without \os.fork()\ in multi-threaded worker/scheduler processes.

Test plan

• \pytest providers/openlineage/tests/unit/openlineage/plugins/test_listener.py\
• CI provider tests for \openlineage\ pass
• \prek run --all-files\

Closes#47160

[eladkal]

This change looks odd to me.
Did you test it?

[br413]

Thanks for the review, @eladkal — fair question.

The first version removed \os.fork()\ by calling lifecycle handlers inline, which dropped the process isolation the fork path provided. I've updated the approach to match what we already do for DAG-run listeners and manual state-change emissions:

1. Metadata extraction stays in the parent process (where ORM / task objects are available).
2. Adapter emission runs via \submit_callable\ → \ProcessPoolExecutor\ with \forkserver\ context, using a new picklable module-level _emit_task_instance_event()\ (same pattern as _emit_manual_state_change_event\ / _run_adapter_method).

This removes the Python 3.12 \DeprecationWarning\ without running OL transport code synchronously in the multi-threaded worker/scheduler process.

Rebased onto latest \main\ and pushed. Happy to address any further feedback.

[mobuchowski]

#68708 - should fix this issue rather than using forkserver

[eladkal]

#68708 - should fix this issue rather than using forkserver

Great so closing this one as solved by #68708

[br413]

Summary

• Remove \os.fork()-based task event emission in the OpenLineage listener
• Extract operator metadata in the parent process; emit lifecycle events via picklable _emit_task_instance_event()\ and \ProcessPoolExecutor\ with \orkserver\ context
• Use \orkserver\ multiprocessing context for scheduler-side DAG-run and manual state-change emissions

Problem

On Python 3.12+, the OpenLineage listener emits:

\
DeprecationWarning: This process is multi-threaded, use of fork() may lead to deadlocks in the child.
\\

This came from _fork_execute()\ calling \os.fork()\ during task lifecycle events, and from \ProcessPoolExecutor\ defaulting to the \ork\ start method on Linux.

Approach

Task lifecycle handlers (\on_running, \on_success, \on_failure, \on_skipped) follow the same two-phase pattern as DAG-run listeners:

1. Parent process: metadata extraction while ORM/task objects are available
2. Pool worker (forkserver): adapter emission + metrics via \submit_callable(_emit_task_instance_event, ...)\

This preserves process isolation for OL transport code without \os.fork()\ in multi-threaded worker/scheduler processes.

Test plan

• \pytest providers/openlineage/tests/unit/openlineage/plugins/test_listener.py\
• CI provider tests for \openlineage\ pass
• \prek run --all-files\

Closes#47160

[eladkal]

This change looks odd to me.
Did you test it?

[br413]

Thanks for the review, @eladkal — fair question.

The first version removed \os.fork()\ by calling lifecycle handlers inline, which dropped the process isolation the fork path provided. I've updated the approach to match what we already do for DAG-run listeners and manual state-change emissions:

1. Metadata extraction stays in the parent process (where ORM / task objects are available).
2. Adapter emission runs via \submit_callable\ → \ProcessPoolExecutor\ with \forkserver\ context, using a new picklable module-level _emit_task_instance_event()\ (same pattern as _emit_manual_state_change_event\ / _run_adapter_method).

This removes the Python 3.12 \DeprecationWarning\ without running OL transport code synchronously in the multi-threaded worker/scheduler process.

Rebased onto latest \main\ and pushed. Happy to address any further feedback.

[mobuchowski]

#68708 - should fix this issue rather than using forkserver

[eladkal]

#68708 - should fix this issue rather than using forkserver

Great so closing this one as solved by #68708

[br413]

Summary

• Remove \os.fork()-based task event emission in the OpenLineage listener
• Extract operator metadata in the parent process; emit lifecycle events via picklable _emit_task_instance_event()\ and \ProcessPoolExecutor\ with \orkserver\ context
• Use \orkserver\ multiprocessing context for scheduler-side DAG-run and manual state-change emissions

Problem

On Python 3.12+, the OpenLineage listener emits:

\
DeprecationWarning: This process is multi-threaded, use of fork() may lead to deadlocks in the child.
\\

This came from _fork_execute()\ calling \os.fork()\ during task lifecycle events, and from \ProcessPoolExecutor\ defaulting to the \ork\ start method on Linux.

Approach

Task lifecycle handlers (\on_running, \on_success, \on_failure, \on_skipped) follow the same two-phase pattern as DAG-run listeners:

1. Parent process: metadata extraction while ORM/task objects are available
2. Pool worker (forkserver): adapter emission + metrics via \submit_callable(_emit_task_instance_event, ...)\

This preserves process isolation for OL transport code without \os.fork()\ in multi-threaded worker/scheduler processes.

Test plan

• \pytest providers/openlineage/tests/unit/openlineage/plugins/test_listener.py\
• CI provider tests for \openlineage\ pass
• \prek run --all-files\

Closes#47160

[eladkal]

This change looks odd to me.
Did you test it?

[br413]

Thanks for the review, @eladkal — fair question.

The first version removed \os.fork()\ by calling lifecycle handlers inline, which dropped the process isolation the fork path provided. I've updated the approach to match what we already do for DAG-run listeners and manual state-change emissions:

1. Metadata extraction stays in the parent process (where ORM / task objects are available).
2. Adapter emission runs via \submit_callable\ → \ProcessPoolExecutor\ with \forkserver\ context, using a new picklable module-level _emit_task_instance_event()\ (same pattern as _emit_manual_state_change_event\ / _run_adapter_method).

This removes the Python 3.12 \DeprecationWarning\ without running OL transport code synchronously in the multi-threaded worker/scheduler process.

Rebased onto latest \main\ and pushed. Happy to address any further feedback.

[mobuchowski]

#68708 - should fix this issue rather than using forkserver

[eladkal]

#68708 - should fix this issue rather than using forkserver

Great so closing this one as solved by #68708

[br413]

Summary

• Remove \os.fork()-based task event emission in the OpenLineage listener
• Extract operator metadata in the parent process; emit lifecycle events via picklable _emit_task_instance_event()\ and \ProcessPoolExecutor\ with \orkserver\ context
• Use \orkserver\ multiprocessing context for scheduler-side DAG-run and manual state-change emissions

Problem

On Python 3.12+, the OpenLineage listener emits:

\
DeprecationWarning: This process is multi-threaded, use of fork() may lead to deadlocks in the child.
\\

This came from _fork_execute()\ calling \os.fork()\ during task lifecycle events, and from \ProcessPoolExecutor\ defaulting to the \ork\ start method on Linux.

Approach

Task lifecycle handlers (\on_running, \on_success, \on_failure, \on_skipped) follow the same two-phase pattern as DAG-run listeners:

1. Parent process: metadata extraction while ORM/task objects are available
2. Pool worker (forkserver): adapter emission + metrics via \submit_callable(_emit_task_instance_event, ...)\

This preserves process isolation for OL transport code without \os.fork()\ in multi-threaded worker/scheduler processes.

Test plan

• \pytest providers/openlineage/tests/unit/openlineage/plugins/test_listener.py\
• CI provider tests for \openlineage\ pass
• \prek run --all-files\

Closes#47160

[eladkal]

This change looks odd to me.
Did you test it?

[br413]

Thanks for the review, @eladkal — fair question.

The first version removed \os.fork()\ by calling lifecycle handlers inline, which dropped the process isolation the fork path provided. I've updated the approach to match what we already do for DAG-run listeners and manual state-change emissions:

1. Metadata extraction stays in the parent process (where ORM / task objects are available).
2. Adapter emission runs via \submit_callable\ → \ProcessPoolExecutor\ with \forkserver\ context, using a new picklable module-level _emit_task_instance_event()\ (same pattern as _emit_manual_state_change_event\ / _run_adapter_method).

This removes the Python 3.12 \DeprecationWarning\ without running OL transport code synchronously in the multi-threaded worker/scheduler process.

Rebased onto latest \main\ and pushed. Happy to address any further feedback.

[mobuchowski]

#68708 - should fix this issue rather than using forkserver

[eladkal]

#68708 - should fix this issue rather than using forkserver

Great so closing this one as solved by #68708

[br413]

Summary

• Remove \os.fork()-based task event emission in the OpenLineage listener
• Extract operator metadata in the parent process; emit lifecycle events via picklable _emit_task_instance_event()\ and \ProcessPoolExecutor\ with \orkserver\ context
• Use \orkserver\ multiprocessing context for scheduler-side DAG-run and manual state-change emissions

Problem

On Python 3.12+, the OpenLineage listener emits:

\
DeprecationWarning: This process is multi-threaded, use of fork() may lead to deadlocks in the child.
\\

This came from _fork_execute()\ calling \os.fork()\ during task lifecycle events, and from \ProcessPoolExecutor\ defaulting to the \ork\ start method on Linux.

Approach

Task lifecycle handlers (\on_running, \on_success, \on_failure, \on_skipped) follow the same two-phase pattern as DAG-run listeners:

1. Parent process: metadata extraction while ORM/task objects are available
2. Pool worker (forkserver): adapter emission + metrics via \submit_callable(_emit_task_instance_event, ...)\

This preserves process isolation for OL transport code without \os.fork()\ in multi-threaded worker/scheduler processes.

Test plan

• \pytest providers/openlineage/tests/unit/openlineage/plugins/test_listener.py\
• CI provider tests for \openlineage\ pass
• \prek run --all-files\

Closes#47160

[eladkal]

This change looks odd to me.
Did you test it?

[br413]

Thanks for the review, @eladkal — fair question.

The first version removed \os.fork()\ by calling lifecycle handlers inline, which dropped the process isolation the fork path provided. I've updated the approach to match what we already do for DAG-run listeners and manual state-change emissions:

1. Metadata extraction stays in the parent process (where ORM / task objects are available).
2. Adapter emission runs via \submit_callable\ → \ProcessPoolExecutor\ with \forkserver\ context, using a new picklable module-level _emit_task_instance_event()\ (same pattern as _emit_manual_state_change_event\ / _run_adapter_method).

This removes the Python 3.12 \DeprecationWarning\ without running OL transport code synchronously in the multi-threaded worker/scheduler process.

Rebased onto latest \main\ and pushed. Happy to address any further feedback.

[mobuchowski]

#68708 - should fix this issue rather than using forkserver

[eladkal]

#68708 - should fix this issue rather than using forkserver

Great so closing this one as solved by #68708

[br413]

Summary

• Remove \os.fork()-based task event emission in the OpenLineage listener
• Extract operator metadata in the parent process; emit lifecycle events via picklable _emit_task_instance_event()\ and \ProcessPoolExecutor\ with \orkserver\ context
• Use \orkserver\ multiprocessing context for scheduler-side DAG-run and manual state-change emissions

Problem

On Python 3.12+, the OpenLineage listener emits:

\
DeprecationWarning: This process is multi-threaded, use of fork() may lead to deadlocks in the child.
\\

This came from _fork_execute()\ calling \os.fork()\ during task lifecycle events, and from \ProcessPoolExecutor\ defaulting to the \ork\ start method on Linux.

Approach

Task lifecycle handlers (\on_running, \on_success, \on_failure, \on_skipped) follow the same two-phase pattern as DAG-run listeners:

1. Parent process: metadata extraction while ORM/task objects are available
2. Pool worker (forkserver): adapter emission + metrics via \submit_callable(_emit_task_instance_event, ...)\

This preserves process isolation for OL transport code without \os.fork()\ in multi-threaded worker/scheduler processes.

Test plan

• \pytest providers/openlineage/tests/unit/openlineage/plugins/test_listener.py\
• CI provider tests for \openlineage\ pass
• \prek run --all-files\

Closes#47160

[eladkal]

This change looks odd to me.
Did you test it?

[br413]

Thanks for the review, @eladkal — fair question.

The first version removed \os.fork()\ by calling lifecycle handlers inline, which dropped the process isolation the fork path provided. I've updated the approach to match what we already do for DAG-run listeners and manual state-change emissions:

1. Metadata extraction stays in the parent process (where ORM / task objects are available).
2. Adapter emission runs via \submit_callable\ → \ProcessPoolExecutor\ with \forkserver\ context, using a new picklable module-level _emit_task_instance_event()\ (same pattern as _emit_manual_state_change_event\ / _run_adapter_method).

This removes the Python 3.12 \DeprecationWarning\ without running OL transport code synchronously in the multi-threaded worker/scheduler process.

Rebased onto latest \main\ and pushed. Happy to address any further feedback.

[mobuchowski]

#68708 - should fix this issue rather than using forkserver

[eladkal]

#68708 - should fix this issue rather than using forkserver

Great so closing this one as solved by #68708

[br413]

Summary

• Remove \os.fork()-based task event emission in the OpenLineage listener
• Extract operator metadata in the parent process; emit lifecycle events via picklable _emit_task_instance_event()\ and \ProcessPoolExecutor\ with \orkserver\ context
• Use \orkserver\ multiprocessing context for scheduler-side DAG-run and manual state-change emissions

Problem

On Python 3.12+, the OpenLineage listener emits:

\
DeprecationWarning: This process is multi-threaded, use of fork() may lead to deadlocks in the child.
\\

This came from _fork_execute()\ calling \os.fork()\ during task lifecycle events, and from \ProcessPoolExecutor\ defaulting to the \ork\ start method on Linux.

Approach

Task lifecycle handlers (\on_running, \on_success, \on_failure, \on_skipped) follow the same two-phase pattern as DAG-run listeners:

1. Parent process: metadata extraction while ORM/task objects are available
2. Pool worker (forkserver): adapter emission + metrics via \submit_callable(_emit_task_instance_event, ...)\

This preserves process isolation for OL transport code without \os.fork()\ in multi-threaded worker/scheduler processes.

Test plan

• \pytest providers/openlineage/tests/unit/openlineage/plugins/test_listener.py\
• CI provider tests for \openlineage\ pass
• \prek run --all-files\

Closes#47160

[eladkal]

This change looks odd to me.
Did you test it?

[br413]

Thanks for the review, @eladkal — fair question.

The first version removed \os.fork()\ by calling lifecycle handlers inline, which dropped the process isolation the fork path provided. I've updated the approach to match what we already do for DAG-run listeners and manual state-change emissions:

1. Metadata extraction stays in the parent process (where ORM / task objects are available).
2. Adapter emission runs via \submit_callable\ → \ProcessPoolExecutor\ with \forkserver\ context, using a new picklable module-level _emit_task_instance_event()\ (same pattern as _emit_manual_state_change_event\ / _run_adapter_method).

This removes the Python 3.12 \DeprecationWarning\ without running OL transport code synchronously in the multi-threaded worker/scheduler process.

Rebased onto latest \main\ and pushed. Happy to address any further feedback.

[mobuchowski]

#68708 - should fix this issue rather than using forkserver

[eladkal]

#68708 - should fix this issue rather than using forkserver

Great so closing this one as solved by #68708

[br413]

Summary

• Remove \os.fork()-based task event emission in the OpenLineage listener
• Extract operator metadata in the parent process; emit lifecycle events via picklable _emit_task_instance_event()\ and \ProcessPoolExecutor\ with \orkserver\ context
• Use \orkserver\ multiprocessing context for scheduler-side DAG-run and manual state-change emissions

Problem

On Python 3.12+, the OpenLineage listener emits:

\
DeprecationWarning: This process is multi-threaded, use of fork() may lead to deadlocks in the child.
\\

This came from _fork_execute()\ calling \os.fork()\ during task lifecycle events, and from \ProcessPoolExecutor\ defaulting to the \ork\ start method on Linux.

Approach

Task lifecycle handlers (\on_running, \on_success, \on_failure, \on_skipped) follow the same two-phase pattern as DAG-run listeners:

1. Parent process: metadata extraction while ORM/task objects are available
2. Pool worker (forkserver): adapter emission + metrics via \submit_callable(_emit_task_instance_event, ...)\

This preserves process isolation for OL transport code without \os.fork()\ in multi-threaded worker/scheduler processes.

Test plan

• \pytest providers/openlineage/tests/unit/openlineage/plugins/test_listener.py\
• CI provider tests for \openlineage\ pass
• \prek run --all-files\

Closes#47160

[eladkal]

This change looks odd to me.
Did you test it?

[br413]

Thanks for the review, @eladkal — fair question.

The first version removed \os.fork()\ by calling lifecycle handlers inline, which dropped the process isolation the fork path provided. I've updated the approach to match what we already do for DAG-run listeners and manual state-change emissions:

1. Metadata extraction stays in the parent process (where ORM / task objects are available).
2. Adapter emission runs via \submit_callable\ → \ProcessPoolExecutor\ with \forkserver\ context, using a new picklable module-level _emit_task_instance_event()\ (same pattern as _emit_manual_state_change_event\ / _run_adapter_method).

This removes the Python 3.12 \DeprecationWarning\ without running OL transport code synchronously in the multi-threaded worker/scheduler process.

Rebased onto latest \main\ and pushed. Happy to address any further feedback.

[mobuchowski]

#68708 - should fix this issue rather than using forkserver

[eladkal]

#68708 - should fix this issue rather than using forkserver

Great so closing this one as solved by #68708