The connection is not returned to the httpx pool when using a stream

[sometastycake]

Confirm this is an issue with the Python library and not an underlying OpenAI API

• This is an issue with the Python library

Describe the bug

The connections is not returned to the pool when using a stream

In this case (see "To reproduce"), the error occurs in venv/lib/python3.11/site-packages/httpcore/_async/connection_pool.py in the wait_for_connection() method of the RequestStatus() class

And if after each end of reading the stream you call AsyncStream.response.aclose(), then everything will work correctly

To Reproduce

1. Initialize an AsyncOpenAI object with a maximum number of connections equal to 2
2. Send 2 requests (chat.completions) with stream=True and read the entire stream
3. Send a third request
4. We will receive an APITimeoutError error

Code snippets

import asyncio

import httpx
from openai import AsyncOpenAI


async def stream(client: AsyncOpenAI):
    response = await client.chat.completions.create(
        messages=[
            {'role': 'system', 'content': 'You are helpful assistant'},
            {'role': 'user', 'content': 'Hello'}
        ],
        model='gpt-4',
        stream=True,
        timeout=30,
    )
    result = ''
    async for message in response:
        result += message.choices[0].delta.content or ""


async def main():
    client = AsyncOpenAI(
        api_key='',
        http_client=httpx.AsyncClient(
            limits=httpx.Limits(
                max_connections=2,
            )
        )
    )
    for _ in range(3):
        await stream(client)


if __name__ == '__main__':
    asyncio.run(main())

OS

macOS, Ubuntu 22.04

Python version

Python v3.11.6

Library version

openai 1.2.0

[RobertCraigie]

Thanks for the bug report, I can reproduce the issue. Thanks for the minimal example! Looking into a fix.

[RobertCraigie]

While we're investigating this, you can workaround the issue by manually closing the response, e.g.

    stream = await client.chat.completions.create(
        messages=[{"role": "system", "content": "You are helpful assistant"}, {"role": "user", "content": "Hello"}],
        model="gpt-4",
        stream=True,
        timeout=10,
    )
    result = ""
    async for message in stream:
        if message.choices:
            result += message.choices[0].delta.content or ""
    await stream.response.aclose()

[RobertCraigie]

This will be fixed in the next release! https://github.com/openai/openai-python/pull/771/files

[RobertCraigie]

This has been released in v1.2.3!