Uh oh!
There was an error while loading. Please reload this page.
httpx.Response.iter_text has empty string for iterator's last item
#2995
Answered
byjamesbraza
jamesbraza
asked this question in
Potential Issue
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
When using Please see the below Python 3.11 code, its assertion checking for empty strings fails: """Demo of properly unit testing a starlette StreamingResponse.httpx==0.25.2pytest==7.4.3starlette==0.27.0uvicorn==0.24.0.post1"""importasyncioimportstatisticsimporttimefromcollections.abcimportIteratorfromthreadingimportThreadimporthttpximportpytestfromstarlette.responsesimportStreamingResponsefromuvicornimportConfig, Server# SEE: https://www.starlette.io/responses/#streamingresponseasyncdefslow_numbers(minimum, maximum):
yield"<html><body><ul>"fornumberinrange(minimum, maximum+1):
yield"<li>%d</li>"%numberawaitasyncio.sleep(0.5)
yield"</ul></body></html>"asyncdefapp(scope, receive, send):
assertscope["type"] =="http"response=StreamingResponse(slow_numbers(1, 5), media_type="text/html")
awaitresponse(scope, receive, send)
# SEE: https://github.com/encode/httpx/blob/0.25.2/tests/conftest.py#L230-L293# Workaround for https://github.com/encode/starlette/issues/1102classTestServer(Server):
@propertydefurl(self) ->httpx.URL:
protocol="https"ifself.config.is_sslelse"http"returnhttpx.URL(f"{protocol}://{self.config.host}:{self.config.port}/")
definstall_signal_handlers(self) ->None:
# Disable the default installation of handlers for signals such as SIGTERM,# because it can only be done in the main thread.passasyncdefserve(self, sockets=None):
self.restart_requested=asyncio.Event()
loop=asyncio.get_event_loop()
tasks= {
loop.create_task(super().serve(sockets=sockets)),
loop.create_task(self.watch_restarts()),
}
awaitasyncio.wait(tasks)
asyncdefrestart(self) ->None: # pragma: no cover# This coroutine may be called from a different thread than the one the# server is running on, and from an async environment that's not asyncio.# For this reason, we use an event to coordinate with the server# instead of calling shutdown()/startup() directly, and should not make# any asyncio-specific operations.self.started=Falseself.restart_requested.set()
whilenotself.started:
awaitasyncio.sleep(0.2)
asyncdefwatch_restarts(self) ->None: # pragma: no coverwhileTrue:
ifself.should_exit:
returntry:
awaitasyncio.wait_for(self.restart_requested.wait(), timeout=0.1)
exceptasyncio.TimeoutError:
continueself.restart_requested.clear()
awaitself.shutdown()
awaitself.startup()
defserve_in_thread(server: TestServer) ->Iterator[TestServer]:
thread=Thread(target=server.run)
thread.start()
try:
whilenotserver.started:
time.sleep(1e-3)
yieldserverfinally:
server.should_exit=Truethread.join()
@pytest.fixture(name="server", scope="session")deffixture_server() ->Iterator[TestServer]:
config=Config(app=app, lifespan="off", loop="asyncio")
server=TestServer(config=config)
yieldfromserve_in_thread(server)
# The actual testdeftest_streaming(server: TestServer) ->None:
client=httpx.Client(base_url=server.url)
withclient.stream("GET", "/") asresponse:
response: httpx.Responsetexts, times= [], []
tic=time.perf_counter()
fortextinresponse.iter_text():
texts.append(text)
times.append((toc:=time.perf_counter()) -tic)
tic=tocassertlen(times) >1, "Should be more than one chunk"asserttimes[0] <0.6, "Perhaps you streamed everything in first chunk"assertstatistics.mean(times) <0.6, "Should be streaming"assertall([bool(text) fortextintexts]), "Some text was empty" |
Answered by
jamesbraza
Dec 11, 2023
Replies: 1 comment 1 reply
Ah interesting. Yeah it'd be neater if it didn't ever return any empty components. The test cases for handling this are in |
1 reply
Thanks! Adding this test case to @pytest.mark.parametrize( ["data", "expected"], [((b"Hello,", b" world!"), ["Hello,", " world!"])],)deftest_streaming_iter_text(data: typing.Iterable[bytes], expected: typing.List[str]):
response=httpx.Response(200, content=iter(data))
assertlist(response.iter_text()) ==expectedThe issue seems to come from
I traced the cause to some missing logic in |
Answer selected bylovelydinosaur
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
Thanks! Adding this test case to
tests/test_decoders.pywill expose the bug:The issue seems to come from
TextChunker.decode:IdentityDecoderandTextDecoder) emits a empty valueByteChunker().decode(b"")returns[], butTextChunker().decode("")returns[""]I traced the cause to some missing logic in
TextChunker.decodethatByteChunker.decodehas, so I opened #2998