Uh oh!
There was an error while loading. Please reload this page.
Stop dask state change on Iris import - #2870
Conversation
pelson
commented
Oct 26, 2017
Closes #2739. |
| """ | ||
| dask_opts = {} | ||
| if 'pool' not in dask.context._globals and \ |
There was a problem hiding this comment.
I'd probably guard about accessing dask.context._globals. They could legitimately remove this and leave iris unable to load data... 🤒
There was a problem hiding this comment.
(to be clear, in this context I simply mean getattr(dask.context, '_globals'))
There was a problem hiding this comment.
Given the nature of this API (private), I'd also be tempted to add a test that asserts dask's set_opt behaviour continues to be a valid assumption in dask.context._globals.
DPeterK
commented
Oct 26, 2017
@pelson I had a sneaky extra 10min, so I implemented the changes you suggested 😎 |
pelson
left a comment
There was a problem hiding this comment.
Thanks for updating @dkillick. Good use of a sneaky 10 mins 😄
| _iris_dask_defaults() | ||
| dask_opts = {} | ||
| dask_globals = getattr(dask.context, '_globals') | ||
| if dask_globals is not None: |
There was a problem hiding this comment.
So if dask change their dask.context._globals storage location, we let dask give us the default engine. Is that what is intended?
| # We may need to unset a previously-set default. | ||
| if dask_opts.get('get') is not None: | ||
| dask_opts = {key: value for key, value in dask_opts.items() | ||
| if key != 'get'} |
There was a problem hiding this comment.
I'm trying to understand why the get is not used from the dask options. I think this comes down to my not-understanding the dask globals and what they actually mean...
pelson
commented
Oct 27, 2017
Deferring for now. #2879 removes the default change, and we can re-introduce a different default when we have a "smoking gun"/evidence of need to divert from dask's defaults. |
We have chosen to set default Iris behaviour to use only one thread when processing dask graphs initially. This was done by setting the global dask state using
dask.set_options(). This is far from ideal as it means Iris is changing dask state; a situation that should not be. This PR updates the system for setting dask processing options so that behaviour is maintained without changing dask state.