Uh oh!
There was an error while loading. Please reload this page.
[SPARK-14859][PYSPARK] Make Lambda Serializer Configurable - #12620
[SPARK-14859][PYSPARK] Make Lambda Serializer Configurable#12620njwhite wants to merge 1 commit into
Conversation
Store the serializer that we should use to serialize RDD transformation functions on the SparkContext, defaulting to a CloudPickleSerializer if not given. Allow a user to change this serializer when first constructing the SparkContext.
AmplabJenkins
commented
Apr 22, 2016
Can one of the admins verify this patch? |
holdenk
commented
Apr 25, 2016
Is this functionality we want to add? cc @davies ? |
holdenk
commented
Apr 25, 2016
If we do end up adding this we would probably want to add a test of using a custom serializer (but maybe don't rush to do this since I think if we want to expose this is maybe not yet clear). |
davies
commented
Apr 25, 2016
@njwhite We still use PickleSerializer to deserialize the functions, so it means the serializer MUST be compatible with Pickle, I'm not sure make it configurable will be really helpful (not a good API interface). If you really want to hack it in your case, I think you could have many ways to hack it in Python. |
njwhite
commented
Apr 25, 2016
@davies I'm using this to use the "dill" serializer, as it can pickle more things (and allows more fine-grained control) than the cloud-pickle serializer. What about making that the default for functions? |
davies
commented
Apr 25, 2016
Can we support dill directly and have a flag to choose from the two serializer? cloud-pickler could be the default one. |
holdenk
commented
Oct 7, 2016
I don't see much progress around this - would it maybe make sense to close and just focus on improving cloudpickle (or upgrading our cloudpickle)? |
HyukjinKwon
commented
Feb 9, 2017
@njwhite It seems inactive for few months. Would this be better to close this for now if you are currently not able to proceed this further? |
## What changes were proposed in this pull request? This PR proposes to close stale PRs. What I mean by "stale" here includes that there are some review comments by reviewers but the author looks inactive without any answer to them more than a month. I left some comments roughly a week ago to ping and the author looks still inactive in these PR below These below includes some PR suggested to be closed and a PR against another branch which seems obviously inappropriate. Given the comments in the last three PRs below, they are probably worth being taken over by anyone who is interested in it. Closesapache#7963Closesapache#8374Closesapache#11192Closesapache#11374Closesapache#11692Closesapache#12243Closesapache#12583Closesapache#12620Closesapache#12675Closesapache#12697Closesapache#12800Closesapache#13715Closesapache#14266Closesapache#15053Closesapache#15159Closesapache#15209Closesapache#15264Closesapache#15267Closesapache#15871Closesapache#15861Closesapache#16319Closesapache#16324Closesapache#16890Closesapache#12398Closesapache#12933Closesapache#14517 ## How was this patch tested? N/A Author: hyukjinkwon <gurwls223@gmail.com> Closesapache#16937 from HyukjinKwon/stale-prs-close.
What changes were proposed in this pull request?
Store the serializer that we should use to serialize RDD transformation
functions on the SparkContext, defaulting to a CloudPickleSerializer if not
given. Allow a user to change this serializer when first constructing the
SparkContext.
How was this patch tested?
Unit tests and manual integration tests.