Pickleable httpx.Request is not restored proper.

[Kludex]

Originally opened by @cloasdata on 2022-04-24 18:04:17 in encode/httpx

import pickle
import traceback
import httpx

class MyExtension:
 some_stuff = "stuff"


url = "https://github.com"
method = "GET"
extensions = {"a_proper_extension": MyExtension()}

request = httpx.Request(method, url, extensions=extensions)
print(request)

pickled_request = pickle.dumps(request)
print(pickled_request)

repickled_request = pickle.loads(pickled_request)
print(repickled_request)

try:
 assert repickled_request == request # is not the same but should be equal
except AssertionError as e:
 print(traceback.format_exc())

print("extensions:", repickled_request.extensions) # gone! why

It turns out that httpx.Request can be pickled but loading the serialized byte code will not return the same object at all.
I did not a lot of research on it so far and I cannot tell were exactly the problem occurs in httpx.

Reason for pickling is, that I'm running the client in several child processes and this leads to strange errors like
httpx.StreamClosed: Attempted to read or stream content, but the stream has been closed.

At least some warning should be given, that this class is not safe to pickle even if you can pickle it.

[Kludex]

Originally posted by @cloasdata on 2022-04-25T13:18:12Z

Today I found some time to look through the source and found

 def __getstate__(self) -> typing.Dict[str, typing.Any]:
 return {
 name: value
 for name, value in self.__dict__.items()
 if name not in ["extensions", "stream"]
 }

 def __setstate__(self, state: typing.Dict[str, typing.Any]) -> None:
 for name, value in state.items():
 setattr(self, name, value)
 self.extensions = {}
 self.stream = UnattachedStream()

That explains my above problem. But I don't know, why one should remove this data from the object. Some explanation or warning would be nice...

[Kludex]

Originally posted by @florimondmanca on 2022-04-25T15:43:57Z

@cloasdata Hi, just a heads up that you might want to review encode/httpx#1579 for background. It's still possible there are some developer experience niceties we could be adding...