Skip to content

Fix flaky sqlite tests with test_xcom_map_nest hopefully - #33145

Merged
potiuk merged 1 commit into
apache:mainfrom
potiuk:fix-sqlite-flaky-tests
Aug 5, 2023
Merged

Fix flaky sqlite tests with test_xcom_map_nest hopefully#33145
potiuk merged 1 commit into
apache:mainfrom
potiuk:fix-sqlite-flaky-tests

Conversation

@potiuk

@potiukpotiuk commented Aug 5, 2023•
edited
Loading

Copy link
Copy Markdown
Member

Recently sqlite started to fail randomly during teardown of test_xcom_map_nest or test_xcom_map_zip_nest.
It looks very strange:

sqlite3.ProgrammingError: SQLite objects created in a thread can only be
used in that same thread

By analysing possible reasons it seems that it was a side effect of the existing test_xcom_map_nest that allocated a new session in the run method of task instance rather than pass the sesion that is created and torrn down in the pytest fixture.

The hypothesis is that the session created in the test_xcom_map_nest were being reclaimed and closed while the test_xcom_map_zip_nest test was already starting in a different thread started by Pytest.

The fix is to pass the session object to run method of the taskinstance in the test_xcom_map_nest test.

^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

@potiuk
Fix flaky sqlite tests with test_xcom_map_nest hopefully
6d2257e
Recently sqlite started to fail randomly during teardown of
`test_xcom_map_nest` or `test_xcom_map_zip_nest`. This happpened
after adding `test_xcom_map_zip_nest` . It looked very strange:
```
sqlite3.ProgrammingError: SQLite objects created in a thread can only be
used in that same thread
```
By analysing possible reasons it seems that it was a side effect
of the existing `test_xcom_map_nest` that allocated a new
session in the run method of task instance rather than pass
the sesion that is created and torrn down in the pytest fixture.
The hypothesis is that the session created in the ``test_xcom_map_nest``
were being reclaimed and closed while the `test_xcom_map_zip_nest` test
was already starting in a different thread started by Pytest.
The fix is to pass the session object to run method of the taskinstance
in the ``test_xcom_map_nest`` test.
@potiuk
potiuk requested a review from uranusjrAugust 5, 2023 20:51
@potiuk

potiuk commented Aug 5, 2023

Copy link
Copy Markdown
MemberAuthor

Hey @uranusjr -> I think I found the reason of the test_xcom_map_nest behaving flaky. Not sure why it started to happen now - but I have a hypothesis explaining the side-effect of one "map_nest" on the "zip" one.

@potiuk

potiuk commented Aug 5, 2023

Copy link
Copy Markdown
MemberAuthor

Example flaky tests:

---------------------------- Captured log teardown -----------------------------
ERROR sqlalchemy.pool.impl.NullPool:base.py:791 Exception during reset or similar
Traceback (most recent call last):
File "/usr/local/lib/python3.11/site-packages/sqlalchemy/pool/base.py", line 763, in _finalize_fairy
fairy._reset(pool, transaction_was_reset)
File "/usr/local/lib/python3.11/site-packages/sqlalchemy/pool/base.py", line 1038, in _reset
pool._dialect.do_rollback(self)
File "/usr/local/lib/python3.11/site-packages/sqlalchemy/engine/default.py", line 683, in do_rollback
dbapi_connection.rollback()
sqlite3.ProgrammingError: SQLite objects created in a thread can only be used in that same thread. The object was created in thread id 140220884317952 and this is thread id 140222344125312.
ERROR sqlalchemy.pool.impl.NullPool:base.py:264 Exception closing connection <sqlite3.Connection object at 0x7f879f914c70>
Traceback (most recent call last):
File "/usr/local/lib/python3.11/site-packages/sqlalchemy/pool/base.py", line 763, in _finalize_fairy
fairy._reset(pool, transaction_was_reset)
File "/usr/local/lib/python3.11/site-packages/sqlalchemy/pool/base.py", line 1038, in _reset
pool._dialect.do_rollback(self)
File "/usr/local/lib/python3.11/site-packages/sqlalchemy/engine/default.py", line 683, in do_rollback
dbapi_connection.rollback()
sqlite3.ProgrammingError: SQLite objects created in a thread can only be used in that same thread. The object was created in thread id 140220884317952 and this is thread id 140222344125312.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.11/site-packages/sqlalchemy/pool/base.py", line 260, in _close_connection
self._dialect.do_terminate(connection)
File "/usr/local/lib/python3.11/site-packages/sqlalchemy/engine/default.py", line 689, in do_terminate
self.do_close(dbapi_connection)
File "/usr/local/lib/python3.11/site-packages/sqlalchemy/engine/default.py", line 692, in do_close
dbapi_connection.close()
sqlite3.ProgrammingError: SQLite objects created in a thread can only be used in that same thread. The object was created in thread id 140220884317952 and this is thread id 140222344125312.
=================================== FAILURES ===================================

hussein-awala
hussein-awala approved these changes Aug 5, 2023
View reviewed changes

@hussein-awalahussein-awala left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice catch!

@potiuk
potiuk merged commit d1d6fc9 into apache:mainAug 5, 2023
@potiuk
potiuk deleted the fix-sqlite-flaky-tests branch August 5, 2023 21:15
@potiuk

potiuk commented Aug 5, 2023

Copy link
Copy Markdown
MemberAuthor

Nice catch!

Yeah.... Let's see if it helps, but looks plausible.

@potiukpotiuk added this to the Airflow 2.7.0 milestone Aug 5, 2023
@potiuk

potiuk commented Aug 5, 2023

Copy link
Copy Markdown
MemberAuthor

Also add it to 2.7.0 in the hopes it will stabilize the tests there.

potiuk added a commit to potiuk/airflow that referenced this pull request Aug 6, 2023
@potiuk
Attempt to stabilise tests for xcom_arg_map
aa6cb34
Similarly to apache#33145 - this is an attempt to stabilise flaky tests
for the test_xcom_arg_map.
Even if the mechanism is not entirely clear (provide_session should
also close the connection) seems like using pytest-fixture provided
session works better than relying on a new session created in run()
methods.
@potiukpotiuk mentioned this pull request Aug 6, 2023
Merged
potiuk added a commit that referenced this pull request Aug 6, 2023
@potiuk
Attempt to stabilise tests for xcom_arg_map (#33150)
3dd0c99
Similarly to #33145 - this is an attempt to stabilise flaky tests
for the test_xcom_arg_map.
Even if the mechanism is not entirely clear (provide_session should
also close the connection) seems like using pytest-fixture provided
session works better than relying on a new session created in run()
methods.
@potiukpotiuk mentioned this pull request Aug 7, 2023
Closed
1 task
ephraimbuddy pushed a commit that referenced this pull request Aug 8, 2023
@potiuk@ephraimbuddy
Attempt to stabilise tests for xcom_arg_map (#33150)
c80d91b
Similarly to #33145 - this is an attempt to stabilise flaky tests
for the test_xcom_arg_map.
Even if the mechanism is not entirely clear (provide_session should
also close the connection) seems like using pytest-fixture provided
session works better than relying on a new session created in run()
methods.
(cherry picked from commit 3dd0c99)
ephraimbuddy pushed a commit that referenced this pull request Aug 9, 2023
@potiuk@ephraimbuddy
Fix flaky sqlite tests withtest_xcom_map_nesthopefully (#33145)
4e817c4
Recently sqlite started to fail randomly during teardown of
`test_xcom_map_nest` or `test_xcom_map_zip_nest`. This happpened
after adding `test_xcom_map_zip_nest` . It looked very strange:
```
sqlite3.ProgrammingError: SQLite objects created in a thread can only be
used in that same thread
```
By analysing possible reasons it seems that it was a side effect
of the existing `test_xcom_map_nest` that allocated a new
session in the run method of task instance rather than pass
the sesion that is created and torrn down in the pytest fixture.
The hypothesis is that the session created in the ``test_xcom_map_nest``
were being reclaimed and closed while the `test_xcom_map_zip_nest` test
was already starting in a different thread started by Pytest.
The fix is to pass the session object to run method of the taskinstance
in the ``test_xcom_map_nest`` test.
(cherry picked from commit d1d6fc9)
@ephraimbuddyephraimbuddy mentioned this pull request Aug 15, 2023
Merged
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@hussein-awalahussein-awalahussein-awala approved these changes
@uranusjruranusjrAwaiting requested review from uranusjr
@ephraimbuddyephraimbuddyAwaiting requested review from ephraimbuddy
@dstandishdstandishAwaiting requested review from dstandish
@eladkaleladkalAwaiting requested review from eladkal

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

Airflow 2.7.0

Development

Successfully merging this pull request may close these issues.

2 participants

@potiuk@hussein-awala
, 'i'); if (__m === '*' || __re.test(location.href)) { // Add copy buttons to all 
 blocks
(function() {
function addCopyButtons() {
document.querySelectorAll('pre code').forEach(function(codeBlock) {
if (codeBlock.parentElement.hasAttribute('data-copy-added')) return;
codeBlock.parentElement.setAttribute('data-copy-added', 'true');
var btn = document.createElement('button');
btn.textContent = 'Copy';
btn.style.cssText = 'position:absolute;top:4px;right:4px;padding:2px 8px;font-size:11px;background:#4ecdc4;border:none;border-radius:4px;color:#1a1a2e;cursor:pointer;opacity:0.7;transition:opacity 0.2s;';
btn.onmouseover = function() { this.style.opacity = '1'; };
btn.onmouseout = function() { this.style.opacity = '0.7'; };
btn.onclick = function() {
navigator.clipboard.writeText(codeBlock.textContent).then(function() {
btn.textContent = 'Copied!';
setTimeout(function() { btn.textContent = 'Copy'; }, 1500);
});
};
codeBlock.parentElement.style.position = 'relative';
codeBlock.parentElement.appendChild(btn);
});
}
addCopyButtons();
// Re-run on dynamic content
var observer = new MutationObserver(addCopyButtons);
observer.observe(document.body, { childList: true, subtree: true });
})();
}
} catch(__e) { console.warn('[Userscript:Add Copy Buttons to Code Blocks]', __e); }
})();
(function(){
try {
var __m = "github.com";
var __re = new RegExp('^' + "github\\.com" + '
Fix flaky sqlite tests with `test_xcom_map_nest` hopefully by potiuk · Pull Request #33145 · apache/airflow · GitHub
Skip to content
Fix flaky sqlite tests with test_xcom_map_nest hopefully - #33145
Merged
potiuk merged 1 commit into
apache:mainfrom 
potiuk:fix-sqlite-flaky-tests
Aug 5, 2023
Merged
Fix flaky sqlite tests with test_xcom_map_nest hopefully#33145
potiuk merged 1 commit into
apache:mainfrom 
potiuk:fix-sqlite-flaky-tests
Conversation
@potiuk
@potiukpotiuk
commented
Aug 5, 2023•

edited

Loading

Copy link


Copy Markdown

Member
Recently sqlite started to fail randomly during teardown of test_xcom_map_nest or test_xcom_map_zip_nest.

It looks very strange:
sqlite3.ProgrammingError: SQLite objects created in a thread can only be
used in that same thread

By analysing possible reasons it seems that it was a side effect of the existing test_xcom_map_nest that allocated a new session in the run method of task instance rather than pass the sesion that is created and torrn down in the pytest fixture.
The hypothesis is that the session created in the test_xcom_map_nest were being reclaimed and closed while the test_xcom_map_zip_nest test was already starting in a different thread started by Pytest.
The fix is to pass the session object to run method of the taskinstance in the test_xcom_map_nest test.
^ Add meaningful description above

Read the Pull Request Guidelines for more information.

In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.

In case of a new dependency, check compliance with the ASF 3rd Party License Policy.

In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.
@potiuk
Fix flaky sqlite tests with test_xcom_map_nest hopefully
6d2257e
Recently sqlite started to fail randomly during teardown of
`test_xcom_map_nest` or `test_xcom_map_zip_nest`. This happpened
after adding `test_xcom_map_zip_nest` . It looked very strange:
```
sqlite3.ProgrammingError: SQLite objects created in a thread can only be
used in that same thread
```
By analysing possible reasons it seems that it was a side effect
of the existing `test_xcom_map_nest` that allocated a new
session in the run method of task instance rather than pass
the sesion that is created and torrn down in the pytest fixture.
The hypothesis is that the session created in the ``test_xcom_map_nest``
were being reclaimed and closed while the `test_xcom_map_zip_nest` test
was already starting in a different thread started by Pytest.
The fix is to pass the session object to run method of the taskinstance
in the ``test_xcom_map_nest`` test.
@potiuk
potiuk
requested a review
from uranusjrAugust 5, 2023 20:51
@potiuk
potiuk
commented
Aug 5, 2023

Copy link


Copy Markdown

MemberAuthor
Hey @uranusjr -> I think I found the reason of the test_xcom_map_nest behaving flaky. Not sure why it started to happen now - but I have a hypothesis explaining the side-effect of one "map_nest" on the "zip" one.
@potiuk
potiuk
commented
Aug 5, 2023

Copy link


Copy Markdown

MemberAuthor
Example flaky tests:
---------------------------- Captured log teardown -----------------------------
ERROR sqlalchemy.pool.impl.NullPool:base.py:791 Exception during reset or similar
Traceback (most recent call last):
File "/usr/local/lib/python3.11/site-packages/sqlalchemy/pool/base.py", line 763, in _finalize_fairy
fairy._reset(pool, transaction_was_reset)
File "/usr/local/lib/python3.11/site-packages/sqlalchemy/pool/base.py", line 1038, in _reset
pool._dialect.do_rollback(self)
File "/usr/local/lib/python3.11/site-packages/sqlalchemy/engine/default.py", line 683, in do_rollback
dbapi_connection.rollback()
sqlite3.ProgrammingError: SQLite objects created in a thread can only be used in that same thread. The object was created in thread id 140220884317952 and this is thread id 140222344125312.
ERROR sqlalchemy.pool.impl.NullPool:base.py:264 Exception closing connection <sqlite3.Connection object at 0x7f879f914c70>
Traceback (most recent call last):
File "/usr/local/lib/python3.11/site-packages/sqlalchemy/pool/base.py", line 763, in _finalize_fairy
fairy._reset(pool, transaction_was_reset)
File "/usr/local/lib/python3.11/site-packages/sqlalchemy/pool/base.py", line 1038, in _reset
pool._dialect.do_rollback(self)
File "/usr/local/lib/python3.11/site-packages/sqlalchemy/engine/default.py", line 683, in do_rollback
dbapi_connection.rollback()
sqlite3.ProgrammingError: SQLite objects created in a thread can only be used in that same thread. The object was created in thread id 140220884317952 and this is thread id 140222344125312.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.11/site-packages/sqlalchemy/pool/base.py", line 260, in _close_connection
self._dialect.do_terminate(connection)
File "/usr/local/lib/python3.11/site-packages/sqlalchemy/engine/default.py", line 689, in do_terminate
self.do_close(dbapi_connection)
File "/usr/local/lib/python3.11/site-packages/sqlalchemy/engine/default.py", line 692, in do_close
dbapi_connection.close()
sqlite3.ProgrammingError: SQLite objects created in a thread can only be used in that same thread. The object was created in thread id 140220884317952 and this is thread id 140222344125312.
=================================== FAILURES ===================================

hussein-awala
hussein-awala
approved these changes
Aug 5, 2023
View reviewed changes
@hussein-awalahussein-awala
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice catch!
@potiuk
potiuk
merged commit d1d6fc9
into
apache:mainAug 5, 2023
@potiuk
potiuk
deleted the

fix-sqlite-flaky-tests

branch
August 5, 2023 21:15
@potiuk
potiuk
commented
Aug 5, 2023

Copy link


Copy Markdown

MemberAuthor
Nice catch!
Yeah.... Let's see if it helps, but looks plausible.
@potiukpotiuk
added this to the Airflow 2.7.0 milestone
Aug 5, 2023
@potiuk
potiuk
commented
Aug 5, 2023

Copy link


Copy Markdown

MemberAuthor
Also add it to 2.7.0 in the hopes it will stabilize the tests there.
potiuk added a commit
to potiuk/airflow
that referenced
this pull request
Aug 6, 2023
@potiuk
Attempt to stabilise tests for xcom_arg_map
aa6cb34
Similarly to apache#33145 - this is an attempt to stabilise flaky tests
for the test_xcom_arg_map.
Even if the mechanism is not entirely clear (provide_session should
also close the connection) seems like using pytest-fixture provided
session works better than relying on a new session created in run()
methods.
@potiukpotiuk
mentioned this pull request
Aug 6, 2023
 Merged

potiuk added a commit
that referenced
this pull request
Aug 6, 2023
@potiuk
Attempt to stabilise tests for xcom_arg_map (#33150)
3dd0c99
Similarly to #33145 - this is an attempt to stabilise flaky tests
for the test_xcom_arg_map.
Even if the mechanism is not entirely clear (provide_session should
also close the connection) seems like using pytest-fixture provided
session works better than relying on a new session created in run()
methods.
@potiukpotiuk
mentioned this pull request
Aug 7, 2023

Closed

1 task
ephraimbuddy pushed a commit
that referenced
this pull request
Aug 8, 2023
@potiuk@ephraimbuddy
Attempt to stabilise tests for xcom_arg_map (#33150)
c80d91b
Similarly to #33145 - this is an attempt to stabilise flaky tests
for the test_xcom_arg_map.
Even if the mechanism is not entirely clear (provide_session should
also close the connection) seems like using pytest-fixture provided
session works better than relying on a new session created in run()
methods.
(cherry picked from commit 3dd0c99)
ephraimbuddy pushed a commit
that referenced
this pull request
Aug 9, 2023
@potiuk@ephraimbuddy
Fix flaky sqlite tests withtest_xcom_map_nesthopefully (#33145)
4e817c4
Recently sqlite started to fail randomly during teardown of
`test_xcom_map_nest` or `test_xcom_map_zip_nest`. This happpened
after adding `test_xcom_map_zip_nest` . It looked very strange:
```
sqlite3.ProgrammingError: SQLite objects created in a thread can only be
used in that same thread
```
By analysing possible reasons it seems that it was a side effect
of the existing `test_xcom_map_nest` that allocated a new
session in the run method of task instance rather than pass
the sesion that is created and torrn down in the pytest fixture.
The hypothesis is that the session created in the ``test_xcom_map_nest``
were being reclaimed and closed while the `test_xcom_map_zip_nest` test
was already starting in a different thread started by Pytest.
The fix is to pass the session object to run method of the taskinstance
in the ``test_xcom_map_nest`` test.
(cherry picked from commit d1d6fc9)
@ephraimbuddyephraimbuddy
mentioned this pull request
Aug 15, 2023
 Merged

Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@hussein-awalahussein-awalahussein-awala approved these changes
@uranusjruranusjrAwaiting requested review from uranusjr
@ephraimbuddyephraimbuddyAwaiting requested review from ephraimbuddy
@dstandishdstandishAwaiting requested review from dstandish
@eladkaleladkalAwaiting requested review from eladkal

Assignees


No one assigned


Labels


None yet


Projects


None yet


Milestone

Airflow 2.7.0

Development

Successfully merging this pull request may close these issues.

2 participants

@potiuk@hussein-awala

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Force GitHub README to respect dark mode
(function() {
var style = document.createElement('style');
style.textContent = '
.markdown-body {
color-scheme: dark light;
}
.markdown-body pre { background: #161b22 !important; }
.markdown-body code { background: rgba(110, 118, 129, 0.4) !important; }
.markdown-body table th, .markdown-body table td { border-color: #30363d !important; }
.markdown-body img { background: #0d1117; }
.markdown-body blockquote { border-left-color: #8b949e; }
.markdown-body hr { border-color: #30363d; }
';
document.head.appendChild(style);
})();
}
} catch(__e) { console.warn('[Userscript:GitHub Dark Mode README Fix]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '
Fix flaky sqlite tests with `test_xcom_map_nest` hopefully by potiuk · Pull Request #33145 · apache/airflow · GitHub
Skip to content
Fix flaky sqlite tests with test_xcom_map_nest hopefully - #33145
Merged
potiuk merged 1 commit into
apache:mainfrom 
potiuk:fix-sqlite-flaky-tests
Aug 5, 2023
Merged
Fix flaky sqlite tests with test_xcom_map_nest hopefully#33145
potiuk merged 1 commit into
apache:mainfrom 
potiuk:fix-sqlite-flaky-tests
Conversation
@potiuk
@potiukpotiuk
commented
Aug 5, 2023•

edited

Loading

Copy link


Copy Markdown

Member
Recently sqlite started to fail randomly during teardown of test_xcom_map_nest or test_xcom_map_zip_nest.

It looks very strange:
sqlite3.ProgrammingError: SQLite objects created in a thread can only be
used in that same thread

By analysing possible reasons it seems that it was a side effect of the existing test_xcom_map_nest that allocated a new session in the run method of task instance rather than pass the sesion that is created and torrn down in the pytest fixture.
The hypothesis is that the session created in the test_xcom_map_nest were being reclaimed and closed while the test_xcom_map_zip_nest test was already starting in a different thread started by Pytest.
The fix is to pass the session object to run method of the taskinstance in the test_xcom_map_nest test.
^ Add meaningful description above

Read the Pull Request Guidelines for more information.

In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.

In case of a new dependency, check compliance with the ASF 3rd Party License Policy.

In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.
@potiuk
Fix flaky sqlite tests with test_xcom_map_nest hopefully
6d2257e
Recently sqlite started to fail randomly during teardown of
`test_xcom_map_nest` or `test_xcom_map_zip_nest`. This happpened
after adding `test_xcom_map_zip_nest` . It looked very strange:
```
sqlite3.ProgrammingError: SQLite objects created in a thread can only be
used in that same thread
```
By analysing possible reasons it seems that it was a side effect
of the existing `test_xcom_map_nest` that allocated a new
session in the run method of task instance rather than pass
the sesion that is created and torrn down in the pytest fixture.
The hypothesis is that the session created in the ``test_xcom_map_nest``
were being reclaimed and closed while the `test_xcom_map_zip_nest` test
was already starting in a different thread started by Pytest.
The fix is to pass the session object to run method of the taskinstance
in the ``test_xcom_map_nest`` test.
@potiuk
potiuk
requested a review
from uranusjrAugust 5, 2023 20:51
@potiuk
potiuk
commented
Aug 5, 2023

Copy link


Copy Markdown

MemberAuthor
Hey @uranusjr -> I think I found the reason of the test_xcom_map_nest behaving flaky. Not sure why it started to happen now - but I have a hypothesis explaining the side-effect of one "map_nest" on the "zip" one.
@potiuk
potiuk
commented
Aug 5, 2023

Copy link


Copy Markdown

MemberAuthor
Example flaky tests:
---------------------------- Captured log teardown -----------------------------
ERROR sqlalchemy.pool.impl.NullPool:base.py:791 Exception during reset or similar
Traceback (most recent call last):
File "/usr/local/lib/python3.11/site-packages/sqlalchemy/pool/base.py", line 763, in _finalize_fairy
fairy._reset(pool, transaction_was_reset)
File "/usr/local/lib/python3.11/site-packages/sqlalchemy/pool/base.py", line 1038, in _reset
pool._dialect.do_rollback(self)
File "/usr/local/lib/python3.11/site-packages/sqlalchemy/engine/default.py", line 683, in do_rollback
dbapi_connection.rollback()
sqlite3.ProgrammingError: SQLite objects created in a thread can only be used in that same thread. The object was created in thread id 140220884317952 and this is thread id 140222344125312.
ERROR sqlalchemy.pool.impl.NullPool:base.py:264 Exception closing connection <sqlite3.Connection object at 0x7f879f914c70>
Traceback (most recent call last):
File "/usr/local/lib/python3.11/site-packages/sqlalchemy/pool/base.py", line 763, in _finalize_fairy
fairy._reset(pool, transaction_was_reset)
File "/usr/local/lib/python3.11/site-packages/sqlalchemy/pool/base.py", line 1038, in _reset
pool._dialect.do_rollback(self)
File "/usr/local/lib/python3.11/site-packages/sqlalchemy/engine/default.py", line 683, in do_rollback
dbapi_connection.rollback()
sqlite3.ProgrammingError: SQLite objects created in a thread can only be used in that same thread. The object was created in thread id 140220884317952 and this is thread id 140222344125312.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.11/site-packages/sqlalchemy/pool/base.py", line 260, in _close_connection
self._dialect.do_terminate(connection)
File "/usr/local/lib/python3.11/site-packages/sqlalchemy/engine/default.py", line 689, in do_terminate
self.do_close(dbapi_connection)
File "/usr/local/lib/python3.11/site-packages/sqlalchemy/engine/default.py", line 692, in do_close
dbapi_connection.close()
sqlite3.ProgrammingError: SQLite objects created in a thread can only be used in that same thread. The object was created in thread id 140220884317952 and this is thread id 140222344125312.
=================================== FAILURES ===================================

hussein-awala
hussein-awala
approved these changes
Aug 5, 2023
View reviewed changes
@hussein-awalahussein-awala
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice catch!
@potiuk
potiuk
merged commit d1d6fc9
into
apache:mainAug 5, 2023
@potiuk
potiuk
deleted the

fix-sqlite-flaky-tests

branch
August 5, 2023 21:15
@potiuk
potiuk
commented
Aug 5, 2023

Copy link


Copy Markdown

MemberAuthor
Nice catch!
Yeah.... Let's see if it helps, but looks plausible.
@potiukpotiuk
added this to the Airflow 2.7.0 milestone
Aug 5, 2023
@potiuk
potiuk
commented
Aug 5, 2023

Copy link


Copy Markdown

MemberAuthor
Also add it to 2.7.0 in the hopes it will stabilize the tests there.
potiuk added a commit
to potiuk/airflow
that referenced
this pull request
Aug 6, 2023
@potiuk
Attempt to stabilise tests for xcom_arg_map
aa6cb34
Similarly to apache#33145 - this is an attempt to stabilise flaky tests
for the test_xcom_arg_map.
Even if the mechanism is not entirely clear (provide_session should
also close the connection) seems like using pytest-fixture provided
session works better than relying on a new session created in run()
methods.
@potiukpotiuk
mentioned this pull request
Aug 6, 2023
 Merged

potiuk added a commit
that referenced
this pull request
Aug 6, 2023
@potiuk
Attempt to stabilise tests for xcom_arg_map (#33150)
3dd0c99
Similarly to #33145 - this is an attempt to stabilise flaky tests
for the test_xcom_arg_map.
Even if the mechanism is not entirely clear (provide_session should
also close the connection) seems like using pytest-fixture provided
session works better than relying on a new session created in run()
methods.
@potiukpotiuk
mentioned this pull request
Aug 7, 2023

Closed

1 task
ephraimbuddy pushed a commit
that referenced
this pull request
Aug 8, 2023
@potiuk@ephraimbuddy
Attempt to stabilise tests for xcom_arg_map (#33150)
c80d91b
Similarly to #33145 - this is an attempt to stabilise flaky tests
for the test_xcom_arg_map.
Even if the mechanism is not entirely clear (provide_session should
also close the connection) seems like using pytest-fixture provided
session works better than relying on a new session created in run()
methods.
(cherry picked from commit 3dd0c99)
ephraimbuddy pushed a commit
that referenced
this pull request
Aug 9, 2023
@potiuk@ephraimbuddy
Fix flaky sqlite tests withtest_xcom_map_nesthopefully (#33145)
4e817c4
Recently sqlite started to fail randomly during teardown of
`test_xcom_map_nest` or `test_xcom_map_zip_nest`. This happpened
after adding `test_xcom_map_zip_nest` . It looked very strange:
```
sqlite3.ProgrammingError: SQLite objects created in a thread can only be
used in that same thread
```
By analysing possible reasons it seems that it was a side effect
of the existing `test_xcom_map_nest` that allocated a new
session in the run method of task instance rather than pass
the sesion that is created and torrn down in the pytest fixture.
The hypothesis is that the session created in the ``test_xcom_map_nest``
were being reclaimed and closed while the `test_xcom_map_zip_nest` test
was already starting in a different thread started by Pytest.
The fix is to pass the session object to run method of the taskinstance
in the ``test_xcom_map_nest`` test.
(cherry picked from commit d1d6fc9)
@ephraimbuddyephraimbuddy
mentioned this pull request
Aug 15, 2023
 Merged

Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@hussein-awalahussein-awalahussein-awala approved these changes
@uranusjruranusjrAwaiting requested review from uranusjr
@ephraimbuddyephraimbuddyAwaiting requested review from ephraimbuddy
@dstandishdstandishAwaiting requested review from dstandish
@eladkaleladkalAwaiting requested review from eladkal

Assignees


No one assigned


Labels


None yet


Projects


None yet


Milestone

Airflow 2.7.0

Development

Successfully merging this pull request may close these issues.

2 participants

@potiuk@hussein-awala

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Highlight search terms from Google/DuckDuckGo/Bing referrer
(function() {
var ref = document.referrer;
var terms = [];
if (ref.includes('google.com') || ref.includes('duckduckgo.com') || ref.includes('bing.com')) {
var url = new URL(ref);
var q = url.searchParams.get('q') || url.searchParams.get('p');
if (q) {
terms = q.split(/\s+/).filter(function(t) { return t.length > 2; });
}
}
if (terms.length === 0) return;
var style = document.createElement('style');
style.textContent = '.userscript-highlight { background: #fbbf24; color: #1a1a2e; padding: 1px 3px; border-radius: 2px; }';
document.head.appendChild(style);
function highlight(node) {
if (node.nodeType === 3) { // text node
var text = node.textContent;
var found = false;
terms.forEach(function(term) {
var regex = new RegExp('(' + term.replace(/[.*+?^${}()|[\]\\]/g, '\\') + ')', 'gi');
if (regex.test(text)) {
found = true;
var frag = document.createDocumentFragment();
var parts = text.split(regex);
parts.forEach(function(part, i) {
if (i % 2 === 0) {
frag.appendChild(document.createTextNode(part));
} else {
var span = document.createElement('span');
span.className = 'userscript-highlight';
span.textContent = part;
frag.appendChild(span);
}
});
node.parentNode.replaceChild(frag, node);
}
});
} else if (node.nodeType === 1 && node.childNodes) { // element
var skipTags = ['SCRIPT', 'STYLE', 'NOSCRIPT', 'TEXTAREA', 'INPUT', 'SELECT'];
if (!skipTags.includes(node.tagName)) {
Array.from(node.childNodes).forEach(highlight);
}
}
}
highlight(document.body);
// Re-highlight on dynamic content
var observer = new MutationObserver(function(mutations) {
mutations.forEach(function(m) {
m.addedNodes.forEach(function(node) {
if (node.nodeType === 1 || node.nodeType === 3) highlight(node);
});
});
});
observer.observe(document.body, { childList: true, subtree: true });
})();
}
} catch(__e) { console.warn('[Userscript:Highlight Search Terms]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '
Fix flaky sqlite tests with `test_xcom_map_nest` hopefully by potiuk · Pull Request #33145 · apache/airflow · GitHub
Skip to content
Fix flaky sqlite tests with test_xcom_map_nest hopefully - #33145
Merged
potiuk merged 1 commit into
apache:mainfrom 
potiuk:fix-sqlite-flaky-tests
Aug 5, 2023
Merged
Fix flaky sqlite tests with test_xcom_map_nest hopefully#33145
potiuk merged 1 commit into
apache:mainfrom 
potiuk:fix-sqlite-flaky-tests
Conversation
@potiuk
@potiukpotiuk
commented
Aug 5, 2023•

edited

Loading

Copy link


Copy Markdown

Member
Recently sqlite started to fail randomly during teardown of test_xcom_map_nest or test_xcom_map_zip_nest.

It looks very strange:
sqlite3.ProgrammingError: SQLite objects created in a thread can only be
used in that same thread

By analysing possible reasons it seems that it was a side effect of the existing test_xcom_map_nest that allocated a new session in the run method of task instance rather than pass the sesion that is created and torrn down in the pytest fixture.
The hypothesis is that the session created in the test_xcom_map_nest were being reclaimed and closed while the test_xcom_map_zip_nest test was already starting in a different thread started by Pytest.
The fix is to pass the session object to run method of the taskinstance in the test_xcom_map_nest test.
^ Add meaningful description above

Read the Pull Request Guidelines for more information.

In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.

In case of a new dependency, check compliance with the ASF 3rd Party License Policy.

In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.
@potiuk
Fix flaky sqlite tests with test_xcom_map_nest hopefully
6d2257e
Recently sqlite started to fail randomly during teardown of
`test_xcom_map_nest` or `test_xcom_map_zip_nest`. This happpened
after adding `test_xcom_map_zip_nest` . It looked very strange:
```
sqlite3.ProgrammingError: SQLite objects created in a thread can only be
used in that same thread
```
By analysing possible reasons it seems that it was a side effect
of the existing `test_xcom_map_nest` that allocated a new
session in the run method of task instance rather than pass
the sesion that is created and torrn down in the pytest fixture.
The hypothesis is that the session created in the ``test_xcom_map_nest``
were being reclaimed and closed while the `test_xcom_map_zip_nest` test
was already starting in a different thread started by Pytest.
The fix is to pass the session object to run method of the taskinstance
in the ``test_xcom_map_nest`` test.
@potiuk
potiuk
requested a review
from uranusjrAugust 5, 2023 20:51
@potiuk
potiuk
commented
Aug 5, 2023

Copy link


Copy Markdown

MemberAuthor
Hey @uranusjr -> I think I found the reason of the test_xcom_map_nest behaving flaky. Not sure why it started to happen now - but I have a hypothesis explaining the side-effect of one "map_nest" on the "zip" one.
@potiuk
potiuk
commented
Aug 5, 2023

Copy link


Copy Markdown

MemberAuthor
Example flaky tests:
---------------------------- Captured log teardown -----------------------------
ERROR sqlalchemy.pool.impl.NullPool:base.py:791 Exception during reset or similar
Traceback (most recent call last):
File "/usr/local/lib/python3.11/site-packages/sqlalchemy/pool/base.py", line 763, in _finalize_fairy
fairy._reset(pool, transaction_was_reset)
File "/usr/local/lib/python3.11/site-packages/sqlalchemy/pool/base.py", line 1038, in _reset
pool._dialect.do_rollback(self)
File "/usr/local/lib/python3.11/site-packages/sqlalchemy/engine/default.py", line 683, in do_rollback
dbapi_connection.rollback()
sqlite3.ProgrammingError: SQLite objects created in a thread can only be used in that same thread. The object was created in thread id 140220884317952 and this is thread id 140222344125312.
ERROR sqlalchemy.pool.impl.NullPool:base.py:264 Exception closing connection <sqlite3.Connection object at 0x7f879f914c70>
Traceback (most recent call last):
File "/usr/local/lib/python3.11/site-packages/sqlalchemy/pool/base.py", line 763, in _finalize_fairy
fairy._reset(pool, transaction_was_reset)
File "/usr/local/lib/python3.11/site-packages/sqlalchemy/pool/base.py", line 1038, in _reset
pool._dialect.do_rollback(self)
File "/usr/local/lib/python3.11/site-packages/sqlalchemy/engine/default.py", line 683, in do_rollback
dbapi_connection.rollback()
sqlite3.ProgrammingError: SQLite objects created in a thread can only be used in that same thread. The object was created in thread id 140220884317952 and this is thread id 140222344125312.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.11/site-packages/sqlalchemy/pool/base.py", line 260, in _close_connection
self._dialect.do_terminate(connection)
File "/usr/local/lib/python3.11/site-packages/sqlalchemy/engine/default.py", line 689, in do_terminate
self.do_close(dbapi_connection)
File "/usr/local/lib/python3.11/site-packages/sqlalchemy/engine/default.py", line 692, in do_close
dbapi_connection.close()
sqlite3.ProgrammingError: SQLite objects created in a thread can only be used in that same thread. The object was created in thread id 140220884317952 and this is thread id 140222344125312.
=================================== FAILURES ===================================

hussein-awala
hussein-awala
approved these changes
Aug 5, 2023
View reviewed changes
@hussein-awalahussein-awala
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice catch!
@potiuk
potiuk
merged commit d1d6fc9
into
apache:mainAug 5, 2023
@potiuk
potiuk
deleted the

fix-sqlite-flaky-tests

branch
August 5, 2023 21:15
@potiuk
potiuk
commented
Aug 5, 2023

Copy link


Copy Markdown

MemberAuthor
Nice catch!
Yeah.... Let's see if it helps, but looks plausible.
@potiukpotiuk
added this to the Airflow 2.7.0 milestone
Aug 5, 2023
@potiuk
potiuk
commented
Aug 5, 2023

Copy link


Copy Markdown

MemberAuthor
Also add it to 2.7.0 in the hopes it will stabilize the tests there.
potiuk added a commit
to potiuk/airflow
that referenced
this pull request
Aug 6, 2023
@potiuk
Attempt to stabilise tests for xcom_arg_map
aa6cb34
Similarly to apache#33145 - this is an attempt to stabilise flaky tests
for the test_xcom_arg_map.
Even if the mechanism is not entirely clear (provide_session should
also close the connection) seems like using pytest-fixture provided
session works better than relying on a new session created in run()
methods.
@potiukpotiuk
mentioned this pull request
Aug 6, 2023
 Merged

potiuk added a commit
that referenced
this pull request
Aug 6, 2023
@potiuk
Attempt to stabilise tests for xcom_arg_map (#33150)
3dd0c99
Similarly to #33145 - this is an attempt to stabilise flaky tests
for the test_xcom_arg_map.
Even if the mechanism is not entirely clear (provide_session should
also close the connection) seems like using pytest-fixture provided
session works better than relying on a new session created in run()
methods.
@potiukpotiuk
mentioned this pull request
Aug 7, 2023

Closed

1 task
ephraimbuddy pushed a commit
that referenced
this pull request
Aug 8, 2023
@potiuk@ephraimbuddy
Attempt to stabilise tests for xcom_arg_map (#33150)
c80d91b
Similarly to #33145 - this is an attempt to stabilise flaky tests
for the test_xcom_arg_map.
Even if the mechanism is not entirely clear (provide_session should
also close the connection) seems like using pytest-fixture provided
session works better than relying on a new session created in run()
methods.
(cherry picked from commit 3dd0c99)
ephraimbuddy pushed a commit
that referenced
this pull request
Aug 9, 2023
@potiuk@ephraimbuddy
Fix flaky sqlite tests withtest_xcom_map_nesthopefully (#33145)
4e817c4
Recently sqlite started to fail randomly during teardown of
`test_xcom_map_nest` or `test_xcom_map_zip_nest`. This happpened
after adding `test_xcom_map_zip_nest` . It looked very strange:
```
sqlite3.ProgrammingError: SQLite objects created in a thread can only be
used in that same thread
```
By analysing possible reasons it seems that it was a side effect
of the existing `test_xcom_map_nest` that allocated a new
session in the run method of task instance rather than pass
the sesion that is created and torrn down in the pytest fixture.
The hypothesis is that the session created in the ``test_xcom_map_nest``
were being reclaimed and closed while the `test_xcom_map_zip_nest` test
was already starting in a different thread started by Pytest.
The fix is to pass the session object to run method of the taskinstance
in the ``test_xcom_map_nest`` test.
(cherry picked from commit d1d6fc9)
@ephraimbuddyephraimbuddy
mentioned this pull request
Aug 15, 2023
 Merged

Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@hussein-awalahussein-awalahussein-awala approved these changes
@uranusjruranusjrAwaiting requested review from uranusjr
@ephraimbuddyephraimbuddyAwaiting requested review from ephraimbuddy
@dstandishdstandishAwaiting requested review from dstandish
@eladkaleladkalAwaiting requested review from eladkal

Assignees


No one assigned


Labels


None yet


Projects


None yet


Milestone

Airflow 2.7.0

Development

Successfully merging this pull request may close these issues.

2 participants

@potiuk@hussein-awala

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Strip utm_, fbclid, gclid, etc. from all links on page
(function() {
var trackingParams = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content',
'fbclid', 'gclid', 'dclid', 'msclkid', 'yclid',
'ref', 'ref_src', 'source', 'medium', 'campaign'];
function cleanUrl(url) {
try {
var u = new URL(url, window.location.origin);
var changed = false;
trackingParams.forEach(function(p) {
if (u.searchParams.has(p)) {
u.searchParams.delete(p);
changed = true;
}
});
return changed ? u.toString() : url;
} catch (e) {
return url;
}
}
function cleanLinks() {
document.querySelectorAll('a[href]').forEach(function(a) {
var clean = cleanUrl(a.href);
if (clean !== a.href) a.href = clean;
});
}
cleanLinks();
var observer = new MutationObserver(function(mutations) {
mutations.forEach(function(m) {
m.addedNodes.forEach(function(node) {
if (node.nodeType === 1) {
if (node.tagName === 'A') cleanLinks();
node.querySelectorAll('a[href]').forEach(function(a) {
var clean = cleanUrl(a.href);
if (clean !== a.href) a.href = clean;
});
}
});
});
});
observer.observe(document.body, { childList: true, subtree: true });
})();
}
} catch(__e) { console.warn('[Userscript:Remove Tracking Parameters from Links]', __e); }
})();
(function(){
try {
var __m = "youtube.com";
var __re = new RegExp('^' + "youtube\\.com" + '
Fix flaky sqlite tests with `test_xcom_map_nest` hopefully by potiuk · Pull Request #33145 · apache/airflow · GitHub
Skip to content
Fix flaky sqlite tests with test_xcom_map_nest hopefully - #33145
Merged
potiuk merged 1 commit into
apache:mainfrom 
potiuk:fix-sqlite-flaky-tests
Aug 5, 2023
Merged
Fix flaky sqlite tests with test_xcom_map_nest hopefully#33145
potiuk merged 1 commit into
apache:mainfrom 
potiuk:fix-sqlite-flaky-tests
Conversation
@potiuk
@potiukpotiuk
commented
Aug 5, 2023•

edited

Loading

Copy link


Copy Markdown

Member
Recently sqlite started to fail randomly during teardown of test_xcom_map_nest or test_xcom_map_zip_nest.

It looks very strange:
sqlite3.ProgrammingError: SQLite objects created in a thread can only be
used in that same thread

By analysing possible reasons it seems that it was a side effect of the existing test_xcom_map_nest that allocated a new session in the run method of task instance rather than pass the sesion that is created and torrn down in the pytest fixture.
The hypothesis is that the session created in the test_xcom_map_nest were being reclaimed and closed while the test_xcom_map_zip_nest test was already starting in a different thread started by Pytest.
The fix is to pass the session object to run method of the taskinstance in the test_xcom_map_nest test.
^ Add meaningful description above

Read the Pull Request Guidelines for more information.

In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.

In case of a new dependency, check compliance with the ASF 3rd Party License Policy.

In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.
@potiuk
Fix flaky sqlite tests with test_xcom_map_nest hopefully
6d2257e
Recently sqlite started to fail randomly during teardown of
`test_xcom_map_nest` or `test_xcom_map_zip_nest`. This happpened
after adding `test_xcom_map_zip_nest` . It looked very strange:
```
sqlite3.ProgrammingError: SQLite objects created in a thread can only be
used in that same thread
```
By analysing possible reasons it seems that it was a side effect
of the existing `test_xcom_map_nest` that allocated a new
session in the run method of task instance rather than pass
the sesion that is created and torrn down in the pytest fixture.
The hypothesis is that the session created in the ``test_xcom_map_nest``
were being reclaimed and closed while the `test_xcom_map_zip_nest` test
was already starting in a different thread started by Pytest.
The fix is to pass the session object to run method of the taskinstance
in the ``test_xcom_map_nest`` test.
@potiuk
potiuk
requested a review
from uranusjrAugust 5, 2023 20:51
@potiuk
potiuk
commented
Aug 5, 2023

Copy link


Copy Markdown

MemberAuthor
Hey @uranusjr -> I think I found the reason of the test_xcom_map_nest behaving flaky. Not sure why it started to happen now - but I have a hypothesis explaining the side-effect of one "map_nest" on the "zip" one.
@potiuk
potiuk
commented
Aug 5, 2023

Copy link


Copy Markdown

MemberAuthor
Example flaky tests:
---------------------------- Captured log teardown -----------------------------
ERROR sqlalchemy.pool.impl.NullPool:base.py:791 Exception during reset or similar
Traceback (most recent call last):
File "/usr/local/lib/python3.11/site-packages/sqlalchemy/pool/base.py", line 763, in _finalize_fairy
fairy._reset(pool, transaction_was_reset)
File "/usr/local/lib/python3.11/site-packages/sqlalchemy/pool/base.py", line 1038, in _reset
pool._dialect.do_rollback(self)
File "/usr/local/lib/python3.11/site-packages/sqlalchemy/engine/default.py", line 683, in do_rollback
dbapi_connection.rollback()
sqlite3.ProgrammingError: SQLite objects created in a thread can only be used in that same thread. The object was created in thread id 140220884317952 and this is thread id 140222344125312.
ERROR sqlalchemy.pool.impl.NullPool:base.py:264 Exception closing connection <sqlite3.Connection object at 0x7f879f914c70>
Traceback (most recent call last):
File "/usr/local/lib/python3.11/site-packages/sqlalchemy/pool/base.py", line 763, in _finalize_fairy
fairy._reset(pool, transaction_was_reset)
File "/usr/local/lib/python3.11/site-packages/sqlalchemy/pool/base.py", line 1038, in _reset
pool._dialect.do_rollback(self)
File "/usr/local/lib/python3.11/site-packages/sqlalchemy/engine/default.py", line 683, in do_rollback
dbapi_connection.rollback()
sqlite3.ProgrammingError: SQLite objects created in a thread can only be used in that same thread. The object was created in thread id 140220884317952 and this is thread id 140222344125312.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.11/site-packages/sqlalchemy/pool/base.py", line 260, in _close_connection
self._dialect.do_terminate(connection)
File "/usr/local/lib/python3.11/site-packages/sqlalchemy/engine/default.py", line 689, in do_terminate
self.do_close(dbapi_connection)
File "/usr/local/lib/python3.11/site-packages/sqlalchemy/engine/default.py", line 692, in do_close
dbapi_connection.close()
sqlite3.ProgrammingError: SQLite objects created in a thread can only be used in that same thread. The object was created in thread id 140220884317952 and this is thread id 140222344125312.
=================================== FAILURES ===================================

hussein-awala
hussein-awala
approved these changes
Aug 5, 2023
View reviewed changes
@hussein-awalahussein-awala
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice catch!
@potiuk
potiuk
merged commit d1d6fc9
into
apache:mainAug 5, 2023
@potiuk
potiuk
deleted the

fix-sqlite-flaky-tests

branch
August 5, 2023 21:15
@potiuk
potiuk
commented
Aug 5, 2023

Copy link


Copy Markdown

MemberAuthor
Nice catch!
Yeah.... Let's see if it helps, but looks plausible.
@potiukpotiuk
added this to the Airflow 2.7.0 milestone
Aug 5, 2023
@potiuk
potiuk
commented
Aug 5, 2023

Copy link


Copy Markdown

MemberAuthor
Also add it to 2.7.0 in the hopes it will stabilize the tests there.
potiuk added a commit
to potiuk/airflow
that referenced
this pull request
Aug 6, 2023
@potiuk
Attempt to stabilise tests for xcom_arg_map
aa6cb34
Similarly to apache#33145 - this is an attempt to stabilise flaky tests
for the test_xcom_arg_map.
Even if the mechanism is not entirely clear (provide_session should
also close the connection) seems like using pytest-fixture provided
session works better than relying on a new session created in run()
methods.
@potiukpotiuk
mentioned this pull request
Aug 6, 2023
 Merged

potiuk added a commit
that referenced
this pull request
Aug 6, 2023
@potiuk
Attempt to stabilise tests for xcom_arg_map (#33150)
3dd0c99
Similarly to #33145 - this is an attempt to stabilise flaky tests
for the test_xcom_arg_map.
Even if the mechanism is not entirely clear (provide_session should
also close the connection) seems like using pytest-fixture provided
session works better than relying on a new session created in run()
methods.
@potiukpotiuk
mentioned this pull request
Aug 7, 2023

Closed

1 task
ephraimbuddy pushed a commit
that referenced
this pull request
Aug 8, 2023
@potiuk@ephraimbuddy
Attempt to stabilise tests for xcom_arg_map (#33150)
c80d91b
Similarly to #33145 - this is an attempt to stabilise flaky tests
for the test_xcom_arg_map.
Even if the mechanism is not entirely clear (provide_session should
also close the connection) seems like using pytest-fixture provided
session works better than relying on a new session created in run()
methods.
(cherry picked from commit 3dd0c99)
ephraimbuddy pushed a commit
that referenced
this pull request
Aug 9, 2023
@potiuk@ephraimbuddy
Fix flaky sqlite tests withtest_xcom_map_nesthopefully (#33145)
4e817c4
Recently sqlite started to fail randomly during teardown of
`test_xcom_map_nest` or `test_xcom_map_zip_nest`. This happpened
after adding `test_xcom_map_zip_nest` . It looked very strange:
```
sqlite3.ProgrammingError: SQLite objects created in a thread can only be
used in that same thread
```
By analysing possible reasons it seems that it was a side effect
of the existing `test_xcom_map_nest` that allocated a new
session in the run method of task instance rather than pass
the sesion that is created and torrn down in the pytest fixture.
The hypothesis is that the session created in the ``test_xcom_map_nest``
were being reclaimed and closed while the `test_xcom_map_zip_nest` test
was already starting in a different thread started by Pytest.
The fix is to pass the session object to run method of the taskinstance
in the ``test_xcom_map_nest`` test.
(cherry picked from commit d1d6fc9)
@ephraimbuddyephraimbuddy
mentioned this pull request
Aug 15, 2023
 Merged

Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@hussein-awalahussein-awalahussein-awala approved these changes
@uranusjruranusjrAwaiting requested review from uranusjr
@ephraimbuddyephraimbuddyAwaiting requested review from ephraimbuddy
@dstandishdstandishAwaiting requested review from dstandish
@eladkaleladkalAwaiting requested review from eladkal

Assignees


No one assigned


Labels


None yet


Projects


None yet


Milestone

Airflow 2.7.0

Development

Successfully merging this pull request may close these issues.

2 participants

@potiuk@hussein-awala

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Auto-enable theater mode on YouTube
(function() {
function tryTheater() {
var btn = document.querySelector('button[aria-label="Theater mode"], ytd-player #player button[title="Theater mode"]');
if (btn && !btn.classList.contains('activated')) {
btn.click();
}
}
// Try immediately
tryTheater();
// Try after navigation (SPA)
var lastUrl = location.href;
setInterval(function() {
if (location.href !== lastUrl) {
lastUrl = location.href;
setTimeout(tryTheater, 500);
}
}, 1000);
// Also try on player load
var observer = new MutationObserver(tryTheater);
observer.observe(document.body, { childList: true, subtree: true });
})();
}
} catch(__e) { console.warn('[Userscript:YouTube Theater Mode Default]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '
Fix flaky sqlite tests with `test_xcom_map_nest` hopefully by potiuk · Pull Request #33145 · apache/airflow · GitHub
Skip to content
Fix flaky sqlite tests with test_xcom_map_nest hopefully - #33145
Merged
potiuk merged 1 commit into
apache:mainfrom 
potiuk:fix-sqlite-flaky-tests
Aug 5, 2023
Merged
Fix flaky sqlite tests with test_xcom_map_nest hopefully#33145
potiuk merged 1 commit into
apache:mainfrom 
potiuk:fix-sqlite-flaky-tests
Conversation
@potiuk
@potiukpotiuk
commented
Aug 5, 2023•

edited

Loading

Copy link


Copy Markdown

Member
Recently sqlite started to fail randomly during teardown of test_xcom_map_nest or test_xcom_map_zip_nest.

It looks very strange:
sqlite3.ProgrammingError: SQLite objects created in a thread can only be
used in that same thread

By analysing possible reasons it seems that it was a side effect of the existing test_xcom_map_nest that allocated a new session in the run method of task instance rather than pass the sesion that is created and torrn down in the pytest fixture.
The hypothesis is that the session created in the test_xcom_map_nest were being reclaimed and closed while the test_xcom_map_zip_nest test was already starting in a different thread started by Pytest.
The fix is to pass the session object to run method of the taskinstance in the test_xcom_map_nest test.
^ Add meaningful description above

Read the Pull Request Guidelines for more information.

In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.

In case of a new dependency, check compliance with the ASF 3rd Party License Policy.

In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.
@potiuk
Fix flaky sqlite tests with test_xcom_map_nest hopefully
6d2257e
Recently sqlite started to fail randomly during teardown of
`test_xcom_map_nest` or `test_xcom_map_zip_nest`. This happpened
after adding `test_xcom_map_zip_nest` . It looked very strange:
```
sqlite3.ProgrammingError: SQLite objects created in a thread can only be
used in that same thread
```
By analysing possible reasons it seems that it was a side effect
of the existing `test_xcom_map_nest` that allocated a new
session in the run method of task instance rather than pass
the sesion that is created and torrn down in the pytest fixture.
The hypothesis is that the session created in the ``test_xcom_map_nest``
were being reclaimed and closed while the `test_xcom_map_zip_nest` test
was already starting in a different thread started by Pytest.
The fix is to pass the session object to run method of the taskinstance
in the ``test_xcom_map_nest`` test.
@potiuk
potiuk
requested a review
from uranusjrAugust 5, 2023 20:51
@potiuk
potiuk
commented
Aug 5, 2023

Copy link


Copy Markdown

MemberAuthor
Hey @uranusjr -> I think I found the reason of the test_xcom_map_nest behaving flaky. Not sure why it started to happen now - but I have a hypothesis explaining the side-effect of one "map_nest" on the "zip" one.
@potiuk
potiuk
commented
Aug 5, 2023

Copy link


Copy Markdown

MemberAuthor
Example flaky tests:
---------------------------- Captured log teardown -----------------------------
ERROR sqlalchemy.pool.impl.NullPool:base.py:791 Exception during reset or similar
Traceback (most recent call last):
File "/usr/local/lib/python3.11/site-packages/sqlalchemy/pool/base.py", line 763, in _finalize_fairy
fairy._reset(pool, transaction_was_reset)
File "/usr/local/lib/python3.11/site-packages/sqlalchemy/pool/base.py", line 1038, in _reset
pool._dialect.do_rollback(self)
File "/usr/local/lib/python3.11/site-packages/sqlalchemy/engine/default.py", line 683, in do_rollback
dbapi_connection.rollback()
sqlite3.ProgrammingError: SQLite objects created in a thread can only be used in that same thread. The object was created in thread id 140220884317952 and this is thread id 140222344125312.
ERROR sqlalchemy.pool.impl.NullPool:base.py:264 Exception closing connection <sqlite3.Connection object at 0x7f879f914c70>
Traceback (most recent call last):
File "/usr/local/lib/python3.11/site-packages/sqlalchemy/pool/base.py", line 763, in _finalize_fairy
fairy._reset(pool, transaction_was_reset)
File "/usr/local/lib/python3.11/site-packages/sqlalchemy/pool/base.py", line 1038, in _reset
pool._dialect.do_rollback(self)
File "/usr/local/lib/python3.11/site-packages/sqlalchemy/engine/default.py", line 683, in do_rollback
dbapi_connection.rollback()
sqlite3.ProgrammingError: SQLite objects created in a thread can only be used in that same thread. The object was created in thread id 140220884317952 and this is thread id 140222344125312.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.11/site-packages/sqlalchemy/pool/base.py", line 260, in _close_connection
self._dialect.do_terminate(connection)
File "/usr/local/lib/python3.11/site-packages/sqlalchemy/engine/default.py", line 689, in do_terminate
self.do_close(dbapi_connection)
File "/usr/local/lib/python3.11/site-packages/sqlalchemy/engine/default.py", line 692, in do_close
dbapi_connection.close()
sqlite3.ProgrammingError: SQLite objects created in a thread can only be used in that same thread. The object was created in thread id 140220884317952 and this is thread id 140222344125312.
=================================== FAILURES ===================================

hussein-awala
hussein-awala
approved these changes
Aug 5, 2023
View reviewed changes
@hussein-awalahussein-awala
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice catch!
@potiuk
potiuk
merged commit d1d6fc9
into
apache:mainAug 5, 2023
@potiuk
potiuk
deleted the

fix-sqlite-flaky-tests

branch
August 5, 2023 21:15
@potiuk
potiuk
commented
Aug 5, 2023

Copy link


Copy Markdown

MemberAuthor
Nice catch!
Yeah.... Let's see if it helps, but looks plausible.
@potiukpotiuk
added this to the Airflow 2.7.0 milestone
Aug 5, 2023
@potiuk
potiuk
commented
Aug 5, 2023

Copy link


Copy Markdown

MemberAuthor
Also add it to 2.7.0 in the hopes it will stabilize the tests there.
potiuk added a commit
to potiuk/airflow
that referenced
this pull request
Aug 6, 2023
@potiuk
Attempt to stabilise tests for xcom_arg_map
aa6cb34
Similarly to apache#33145 - this is an attempt to stabilise flaky tests
for the test_xcom_arg_map.
Even if the mechanism is not entirely clear (provide_session should
also close the connection) seems like using pytest-fixture provided
session works better than relying on a new session created in run()
methods.
@potiukpotiuk
mentioned this pull request
Aug 6, 2023
 Merged

potiuk added a commit
that referenced
this pull request
Aug 6, 2023
@potiuk
Attempt to stabilise tests for xcom_arg_map (#33150)
3dd0c99
Similarly to #33145 - this is an attempt to stabilise flaky tests
for the test_xcom_arg_map.
Even if the mechanism is not entirely clear (provide_session should
also close the connection) seems like using pytest-fixture provided
session works better than relying on a new session created in run()
methods.
@potiukpotiuk
mentioned this pull request
Aug 7, 2023

Closed

1 task
ephraimbuddy pushed a commit
that referenced
this pull request
Aug 8, 2023
@potiuk@ephraimbuddy
Attempt to stabilise tests for xcom_arg_map (#33150)
c80d91b
Similarly to #33145 - this is an attempt to stabilise flaky tests
for the test_xcom_arg_map.
Even if the mechanism is not entirely clear (provide_session should
also close the connection) seems like using pytest-fixture provided
session works better than relying on a new session created in run()
methods.
(cherry picked from commit 3dd0c99)
ephraimbuddy pushed a commit
that referenced
this pull request
Aug 9, 2023
@potiuk@ephraimbuddy
Fix flaky sqlite tests withtest_xcom_map_nesthopefully (#33145)
4e817c4
Recently sqlite started to fail randomly during teardown of
`test_xcom_map_nest` or `test_xcom_map_zip_nest`. This happpened
after adding `test_xcom_map_zip_nest` . It looked very strange:
```
sqlite3.ProgrammingError: SQLite objects created in a thread can only be
used in that same thread
```
By analysing possible reasons it seems that it was a side effect
of the existing `test_xcom_map_nest` that allocated a new
session in the run method of task instance rather than pass
the sesion that is created and torrn down in the pytest fixture.
The hypothesis is that the session created in the ``test_xcom_map_nest``
were being reclaimed and closed while the `test_xcom_map_zip_nest` test
was already starting in a different thread started by Pytest.
The fix is to pass the session object to run method of the taskinstance
in the ``test_xcom_map_nest`` test.
(cherry picked from commit d1d6fc9)
@ephraimbuddyephraimbuddy
mentioned this pull request
Aug 15, 2023
 Merged

Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@hussein-awalahussein-awalahussein-awala approved these changes
@uranusjruranusjrAwaiting requested review from uranusjr
@ephraimbuddyephraimbuddyAwaiting requested review from ephraimbuddy
@dstandishdstandishAwaiting requested review from dstandish
@eladkaleladkalAwaiting requested review from eladkal

Assignees


No one assigned


Labels


None yet


Projects


None yet


Milestone

Airflow 2.7.0

Development

Successfully merging this pull request may close these issues.

2 participants

@potiuk@hussein-awala

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Remove or un-stick sticky/fixed headers that block content
(function() {
function unstick() {
document.querySelectorAll('header, nav, [role="banner"], .header, .navbar, .sticky, .fixed-top, [style*="position: fixed"], [style*="position:sticky"]').forEach(function(el) {
if (el.style.position === 'fixed' || el.style.position === 'sticky' || getComputedStyle(el).position === 'fixed' || getComputedStyle(el).position === 'sticky') {
el.style.position = 'static';
el.style.top = 'auto';
el.style.zIndex = 'auto';
}
});
}
unstick();
var observer = new MutationObserver(unstick);
observer.observe(document.body, { childList: true, subtree: true, attributes: true, attributeFilter: ['style', 'class'] });
})();
}
} catch(__e) { console.warn('[Userscript:Kill Sticky Headers]', __e); }
})();
})();
Fix flaky sqlite tests with `test_xcom_map_nest` hopefully by potiuk · Pull Request #33145 · apache/airflow · GitHub
Skip to content
Fix flaky sqlite tests with test_xcom_map_nest hopefully - #33145
Merged
potiuk merged 1 commit into
apache:mainfrom 
potiuk:fix-sqlite-flaky-tests
Aug 5, 2023
Merged
Fix flaky sqlite tests with test_xcom_map_nest hopefully#33145
potiuk merged 1 commit into
apache:mainfrom 
potiuk:fix-sqlite-flaky-tests
Conversation
@potiuk
@potiukpotiuk
commented
Aug 5, 2023•

edited

Loading

Copy link


Copy Markdown

Member
Recently sqlite started to fail randomly during teardown of test_xcom_map_nest or test_xcom_map_zip_nest.

It looks very strange:
sqlite3.ProgrammingError: SQLite objects created in a thread can only be
used in that same thread

By analysing possible reasons it seems that it was a side effect of the existing test_xcom_map_nest that allocated a new session in the run method of task instance rather than pass the sesion that is created and torrn down in the pytest fixture.
The hypothesis is that the session created in the test_xcom_map_nest were being reclaimed and closed while the test_xcom_map_zip_nest test was already starting in a different thread started by Pytest.
The fix is to pass the session object to run method of the taskinstance in the test_xcom_map_nest test.
^ Add meaningful description above

Read the Pull Request Guidelines for more information.

In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.

In case of a new dependency, check compliance with the ASF 3rd Party License Policy.

In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.
@potiuk
Fix flaky sqlite tests with test_xcom_map_nest hopefully
6d2257e
Recently sqlite started to fail randomly during teardown of
`test_xcom_map_nest` or `test_xcom_map_zip_nest`. This happpened
after adding `test_xcom_map_zip_nest` . It looked very strange:
```
sqlite3.ProgrammingError: SQLite objects created in a thread can only be
used in that same thread
```
By analysing possible reasons it seems that it was a side effect
of the existing `test_xcom_map_nest` that allocated a new
session in the run method of task instance rather than pass
the sesion that is created and torrn down in the pytest fixture.
The hypothesis is that the session created in the ``test_xcom_map_nest``
were being reclaimed and closed while the `test_xcom_map_zip_nest` test
was already starting in a different thread started by Pytest.
The fix is to pass the session object to run method of the taskinstance
in the ``test_xcom_map_nest`` test.
@potiuk
potiuk
requested a review
from uranusjrAugust 5, 2023 20:51
@potiuk
potiuk
commented
Aug 5, 2023

Copy link


Copy Markdown

MemberAuthor
Hey @uranusjr -> I think I found the reason of the test_xcom_map_nest behaving flaky. Not sure why it started to happen now - but I have a hypothesis explaining the side-effect of one "map_nest" on the "zip" one.
@potiuk
potiuk
commented
Aug 5, 2023

Copy link


Copy Markdown

MemberAuthor
Example flaky tests:
---------------------------- Captured log teardown -----------------------------
ERROR sqlalchemy.pool.impl.NullPool:base.py:791 Exception during reset or similar
Traceback (most recent call last):
File "/usr/local/lib/python3.11/site-packages/sqlalchemy/pool/base.py", line 763, in _finalize_fairy
fairy._reset(pool, transaction_was_reset)
File "/usr/local/lib/python3.11/site-packages/sqlalchemy/pool/base.py", line 1038, in _reset
pool._dialect.do_rollback(self)
File "/usr/local/lib/python3.11/site-packages/sqlalchemy/engine/default.py", line 683, in do_rollback
dbapi_connection.rollback()
sqlite3.ProgrammingError: SQLite objects created in a thread can only be used in that same thread. The object was created in thread id 140220884317952 and this is thread id 140222344125312.
ERROR sqlalchemy.pool.impl.NullPool:base.py:264 Exception closing connection <sqlite3.Connection object at 0x7f879f914c70>
Traceback (most recent call last):
File "/usr/local/lib/python3.11/site-packages/sqlalchemy/pool/base.py", line 763, in _finalize_fairy
fairy._reset(pool, transaction_was_reset)
File "/usr/local/lib/python3.11/site-packages/sqlalchemy/pool/base.py", line 1038, in _reset
pool._dialect.do_rollback(self)
File "/usr/local/lib/python3.11/site-packages/sqlalchemy/engine/default.py", line 683, in do_rollback
dbapi_connection.rollback()
sqlite3.ProgrammingError: SQLite objects created in a thread can only be used in that same thread. The object was created in thread id 140220884317952 and this is thread id 140222344125312.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.11/site-packages/sqlalchemy/pool/base.py", line 260, in _close_connection
self._dialect.do_terminate(connection)
File "/usr/local/lib/python3.11/site-packages/sqlalchemy/engine/default.py", line 689, in do_terminate
self.do_close(dbapi_connection)
File "/usr/local/lib/python3.11/site-packages/sqlalchemy/engine/default.py", line 692, in do_close
dbapi_connection.close()
sqlite3.ProgrammingError: SQLite objects created in a thread can only be used in that same thread. The object was created in thread id 140220884317952 and this is thread id 140222344125312.
=================================== FAILURES ===================================

hussein-awala
hussein-awala
approved these changes
Aug 5, 2023
View reviewed changes
@hussein-awalahussein-awala
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice catch!
@potiuk
potiuk
merged commit d1d6fc9
into
apache:mainAug 5, 2023
@potiuk
potiuk
deleted the

fix-sqlite-flaky-tests

branch
August 5, 2023 21:15
@potiuk
potiuk
commented
Aug 5, 2023

Copy link


Copy Markdown

MemberAuthor
Nice catch!
Yeah.... Let's see if it helps, but looks plausible.
@potiukpotiuk
added this to the Airflow 2.7.0 milestone
Aug 5, 2023
@potiuk
potiuk
commented
Aug 5, 2023

Copy link


Copy Markdown

MemberAuthor
Also add it to 2.7.0 in the hopes it will stabilize the tests there.
potiuk added a commit
to potiuk/airflow
that referenced
this pull request
Aug 6, 2023
@potiuk
Attempt to stabilise tests for xcom_arg_map
aa6cb34
Similarly to apache#33145 - this is an attempt to stabilise flaky tests
for the test_xcom_arg_map.
Even if the mechanism is not entirely clear (provide_session should
also close the connection) seems like using pytest-fixture provided
session works better than relying on a new session created in run()
methods.
@potiukpotiuk
mentioned this pull request
Aug 6, 2023
 Merged

potiuk added a commit
that referenced
this pull request
Aug 6, 2023
@potiuk
Attempt to stabilise tests for xcom_arg_map (#33150)
3dd0c99
Similarly to #33145 - this is an attempt to stabilise flaky tests
for the test_xcom_arg_map.
Even if the mechanism is not entirely clear (provide_session should
also close the connection) seems like using pytest-fixture provided
session works better than relying on a new session created in run()
methods.
@potiukpotiuk
mentioned this pull request
Aug 7, 2023

Closed

1 task
ephraimbuddy pushed a commit
that referenced
this pull request
Aug 8, 2023
@potiuk@ephraimbuddy
Attempt to stabilise tests for xcom_arg_map (#33150)
c80d91b
Similarly to #33145 - this is an attempt to stabilise flaky tests
for the test_xcom_arg_map.
Even if the mechanism is not entirely clear (provide_session should
also close the connection) seems like using pytest-fixture provided
session works better than relying on a new session created in run()
methods.
(cherry picked from commit 3dd0c99)
ephraimbuddy pushed a commit
that referenced
this pull request
Aug 9, 2023
@potiuk@ephraimbuddy
Fix flaky sqlite tests withtest_xcom_map_nesthopefully (#33145)
4e817c4
Recently sqlite started to fail randomly during teardown of
`test_xcom_map_nest` or `test_xcom_map_zip_nest`. This happpened
after adding `test_xcom_map_zip_nest` . It looked very strange:
```
sqlite3.ProgrammingError: SQLite objects created in a thread can only be
used in that same thread
```
By analysing possible reasons it seems that it was a side effect
of the existing `test_xcom_map_nest` that allocated a new
session in the run method of task instance rather than pass
the sesion that is created and torrn down in the pytest fixture.
The hypothesis is that the session created in the ``test_xcom_map_nest``
were being reclaimed and closed while the `test_xcom_map_zip_nest` test
was already starting in a different thread started by Pytest.
The fix is to pass the session object to run method of the taskinstance
in the ``test_xcom_map_nest`` test.
(cherry picked from commit d1d6fc9)
@ephraimbuddyephraimbuddy
mentioned this pull request
Aug 15, 2023
 Merged

Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@hussein-awalahussein-awalahussein-awala approved these changes
@uranusjruranusjrAwaiting requested review from uranusjr
@ephraimbuddyephraimbuddyAwaiting requested review from ephraimbuddy
@dstandishdstandishAwaiting requested review from dstandish
@eladkaleladkalAwaiting requested review from eladkal

Assignees


No one assigned


Labels


None yet


Projects


None yet


Milestone

Airflow 2.7.0

Development

Successfully merging this pull request may close these issues.

2 participants

@potiuk@hussein-awala