Uh oh!
There was an error while loading. Please reload this page.
Replace sequence concatination by unpacking in Airflow core - #33934
Conversation
hussein-awala
commented
Aug 30, 2023
More readable and fater: $ python -m timeit '[1, 2, 3] + [4] + [5] + [6] + [7]'
2000000 loops, best of 5: 179 nsec per loop
$ python -m timeit '[*[1, 2, 3], 4, 5, 6, 7]'
5000000 loops, best of 5: 79 nsec per loop
|
| # Crafting the right filter for dag_id and task_ids combo | ||
| conditions = [] | ||
| for dag in self.subdags + [self]: | ||
| for dag in [*self.subdags, self]: |
There was a problem hiding this comment.
I wonder if itertools.chain would be better here
There was a problem hiding this comment.
I did some tests on it. It seems performance-wise [*[1, 2, 3], 4, 5, 6, 7] is the best solution
$ python -m timeit '[1, 2, 3] + [4] + [5] + [6] + [7]'
2000000 loops, best of 5: 179 nsec per loop
$ python -m timeit '[*[1, 2, 3], 4, 5, 6, 7]'
5000000 loops, best of 5: 69.2 nsec per loop
$ python -m timeit 'import itertools; itertools.chain([1, 2, 3], [4, 5, 6, 7])'
2000000 loops, best of 5: 177 nsec per loop
$ python -m timeit 'import itertools; [1, 2, 3] + [4] + [5] + [6] + [7]'
1000000 loops, best of 5: 242 nsec per loop
$ python -m timeit 'import itertools; [*[1, 2, 3], 4, 5, 6, 7]'
2000000 loops, best of 5: 131 nsec per loop
$ python -m timeit 'import itertools; list(itertools.chain([1, 2, 3], [4, 5, 6, 7]))'
1000000 loops, best of 5: 305 nsec per loop
There was a problem hiding this comment.
A big chunk of this would come from import itertools though, which would not be relevant in Airflow since the module is already imported in a lot of places. I would not be surprised if * is still best for small lists though, since the itertools version still needs to build an additional list (of a single item).
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
potiuk
commented
Sep 3, 2023
LGTM. I think using itertools for those is indeed over-the-top. I am fine witth the current version :) |
(cherry picked from commit 33e5d03)
^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named
{pr_number}.significant.rstor{issue_number}.significant.rst, in newsfragments.