Skip to content

bpo-34204: Use pickle.DEFAULT_PROTOCOL in shelve - #19639

Merged
miss-islington merged 5 commits into
python:masterfrom
ZackerySpytz:bpo-34204-shelve-pickle-DEFAULT_PROTOCOL
Oct 29, 2020
Merged

bpo-34204: Use pickle.DEFAULT_PROTOCOL in shelve#19639
miss-islington merged 5 commits into
python:masterfrom
ZackerySpytz:bpo-34204-shelve-pickle-DEFAULT_PROTOCOL

Conversation

@ZackerySpytz

@ZackerySpytzZackerySpytz commented Apr 21, 2020•
edited by miss-islington
Loading

Copy link
Copy Markdown
Contributor

Use pickle.DEFAULT_PROTOCOL (currently 5) in shelve instead of a
hardcoded 3.

https://bugs.python.org/issue34204

Automerge-Triggered-By: GH:csabella

@ZackerySpytz
bpo-34204: Use pickle.DEFAULT_PROTOCOL in shelve
1c9214e
Use pickle.DEFAULT_PROTOCOL (currently 4) in shelve instead of a
hardcoded 3.
corona10
corona10 reviewed Apr 22, 2020
View reviewed changes

@corona10corona10 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would you like to add NEWS on what's new?
It looks like noticiable changing for user.

@corona10corona10 reopened this Apr 23, 2020
@corona10
corona10 requested a review from vstinnerApril 23, 2020 13:12
@corona10

corona10 commented Apr 23, 2020

Copy link
Copy Markdown
Member

@vstinner Do you have any ideas?

@vstinner

vstinner commented Apr 23, 2020

Copy link
Copy Markdown
Member

@vstinner Do you have any ideas?

I would prefer to get the approval of @pitrou or @applio on such change. pickle can be used to share data between different Python versions. I'm not sure about the backward compatibility here.

@pitrou

pitrou commented Apr 23, 2020

Copy link
Copy Markdown
Member

I don't think this is ok to change. cc @rhettinger

@vstinner

vstinner commented Apr 23, 2020

Copy link
Copy Markdown
Member

Oh sorry, I misunderstood the PR. I read that it changes the default pickle version in multiprocessing (oops), but it's about the shelve module :-)

@csabella

csabella commented May 27, 2020

Copy link
Copy Markdown
Contributor

@corona10@vstinner@pitrou@applio There are two approvals on this PR, but I'm not sure if there's consensus. Does it need more discussion on the bug tracker or should it be merged?

@corona10

corona10 commented May 27, 2020

Copy link
Copy Markdown
Member

@csabella

Does it need more discussion on the bug tracker

IMHO, we need more discussion.

vstinner
vstinner approved these changes May 27, 2020
View reviewed changes

@vstinnervstinner left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM if the PR is updated for Python 3.10 (right now, it says "new in 3.9").

Sorry, first I read that this PR was for multiprocessing... For shelve, IMO it's reaonsable to use the latest pickle version. The main drawback is that shelve files created by Python 3.10 can no longer be read by Python 3.7 and older: protocol 5 was introduced in Python 3.8.
https://docs.python.org/dev/library/pickle.html#data-stream-format

IMHO it's a reasonable limitation. Also, it's well documented that the pickle protocol can easily controlled by the protocol parameter of shelve.open():
https://docs.python.org/dev/library/shelve.html#shelve.open

@pitrou@corona10: What do you think?

@vstinner

vstinner commented May 27, 2020

Copy link
Copy Markdown
Member

Oh, @serhiy-storchaka created https://bugs.python.org/issue34204 so it would be nice to also hear @serhiy-storchaka on this PR :-)

@serhiy-storchaka

serhiy-storchaka commented May 27, 2020

Copy link
Copy Markdown
Member

We can also use just None instead of pickle.DEFAULT_PROTOCOL.

@vstinner

vstinner commented May 27, 2020

Copy link
Copy Markdown
Member

We can also use just None instead of pickle.DEFAULT_PROTOCOL.

I prefer to have s._protocol = pickle.DEFAULT_PROTOCOL than s._protocol = None, it's more explicit.

@pitrou

pitrou commented May 27, 2020

Copy link
Copy Markdown
Member

Also, it's well documented that the pickle protocol can easily controlled by the protocol parameter of shelve.open()

Yes, so if it's well documented, what's the point of changing the default and breaking compatibility?

@vstinner

vstinner commented May 27, 2020

Copy link
Copy Markdown
Member

Hum, I measured the file disk depending on the pickle protocol. The efficiency depends on values: https://bugs.python.org/issue34204

@marco-c

marco-c commented Oct 19, 2020

Copy link
Copy Markdown

@ZackerySpytz are you going to update the PR?
I have opened a new issue and PR as I didn't notice this, I will close my PR if you are planning on updating yours.

@marco-c

marco-c commented Oct 28, 2020

Copy link
Copy Markdown

My PR to fix the same issue is #22751.

@ZackerySpytz

ZackerySpytz commented Oct 28, 2020

Copy link
Copy Markdown
ContributorAuthor

I have just pushed an update.

@marco-c

marco-c commented Oct 28, 2020

Copy link
Copy Markdown

I have just pushed an update.

Thanks, I'll close mine then!

@miss-islington
miss-islington merged commit df59273 into python:masterOct 29, 2020
shihai1991 added a commit to shihai1991/cpython that referenced this pull request Oct 29, 2020
@shihai1991
Merge remote-tracking branch 'origin/master' into bpo_41073_add_types…
b3f992a
…lots1
* origin/master: (365 commits)
bpo-42029: Remove IRIX code (pythonGH-23023)
bpo-42143: Ensure PyFunction_NewWithQualName() can't fail after creating the func object (pythonGH-22953)
bpo-34204: Use pickle.DEFAULT_PROTOCOL in shelve (pythonGH-19639)
bpo-41805: Documentation for PEP 585 (pythonGH-22615)
bpo-42161: Micro-optimize _collections._count_elements() (pythonGH-23008)
bpo-42161: Remove private _PyLong_Zero and _PyLong_One (pythonGH-23003)
bpo-42099: Fix reference to ob_type in unionobject.c and ceval (pythonGH-22829)
bpo-41659: Disallow curly brace directly after primary (pythonGH-22996)
bpo-6761: Enhance __call__ documentation (pythonGH-7987)
bpo-42161: Modules/ uses _PyLong_GetZero() and _PyLong_GetOne() (pythonGH-22998)
bpo-41474, Makefile: Add dependency on cpython/frameobject.h (pythonGH-22999)
bpo-42157: Rename unicodedata.ucnhash_CAPI (pythonGH-22994)
bpo-42161: Use _PyLong_GetZero() and _PyLong_GetOne() (pythonGH-22995)
bpo-30681: Support invalid date format or value in email Date header (pythonGH-22090)
bpo-42161: Add _PyLong_GetZero() and _PyLong_GetOne() (pythonGH-22993)
bpo-42123: Run the parser two times and only enable invalid rules on the second run (pythonGH-22111)
bpo-42157: Convert unicodedata.UCD to heap type (pythonGH-22991)
bpo-42157: unicodedata avoids references to UCD_Type (pythonGH-22990)
bpo-39101: Fixes BaseException hang in IsolatedAsyncioTestCase. (pythonGH-22654)
bpo-1635741: _PyUnicode_Name_CAPI moves to internal C API (pythonGH-22713)
...
adorilson pushed a commit to adorilson/cpython that referenced this pull request Mar 13, 2021
@ZackerySpytz@adorilson
bpo-34204: Use pickle.DEFAULT_PROTOCOL in shelve (pythonGH-19639)
e3cfa33
Use pickle.DEFAULT_PROTOCOL (currently 5) in shelve instead of a
hardcoded 3.
@serhiy-storchakaserhiy-storchaka mentioned this pull request Apr 10, 2022
Closed
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@corona10corona10corona10 approved these changes
@vstinnervstinnervstinner approved these changes
@rhettingerrhettingerrhettinger approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

11 participants

@ZackerySpytz@corona10@vstinner@pitrou@csabella@serhiy-storchaka@marco-c@rhettinger@the-knights-who-say-ni@bedevere-bot@miss-islington