- asyncpg version: 0.26.0
- PostgreSQL version: 12
- Do you use a PostgreSQL SaaS? If so, which? Can you reproduce
the issue with a local PostgreSQL install?: no, local installation - Python version: 3.9
- Platform: linux, centos8
- Do you use pgbouncer?: no
- Did you install asyncpg with pip?: we use poetry
- If you built asyncpg locally, which version of Cython did you use?: no
- Can the issue be reproduced under both asyncio and
uvloop?: didn't try, problem with reproducing, we use uvloop
We use connection pool. And connections are ending after while and our app is hang up (all new response has status code 500 and error "can not get connection from pool"). We observe this behavior when postgres process several queries simultaneous long time and we try to get new connection from the pool with timeout. This is not not stable and can happen 1 per week.
We debug and see following strangeness:
Our code
ctx: PoolAcquireContext=self.pool.acquire(timeout=timeoutBudget)
try: connection=awaitctx.__aenter__()
logger.debug("connection was received")
exceptasyncio.TimeoutError:
...asyncpg code (we added log)
asyncdef_acquire(self, timeout):
asyncdef_acquire_impl():
ch=awaitself._queue.get() # type: PoolConnectionHoldertry:
proxy=awaitch.acquire() # type: PoolConnectionProxyexcept (Exception, asyncio.CancelledError):
self._queue.put_nowait(ch)
raiseelse:
# Record the timeout, as we will apply it by default# in release().ch._timeout=timeoutlogger.debug("connection was gotten from queue")
returnproxyifself._closing:
raiseexceptions.InterfaceError('pool is closing')
self._check_init()
iftimeoutisNone:
returnawait_acquire_impl()
else:
returnawaitcompat.wait_for(
_acquire_impl(), timeout=timeout)We calculate "connection was received" messages count (let be A ) and "connection was gotten from queue" messages count (let be B ) after after app hanged up. B - A = connection pool size.
We have hypothesis that problem here . Future is completed but occurred timeout before compat.wait_for return result.
We try to use following code
ctx: PoolAcquireContext=self.pool.acquire()
connection=Nonetry:
asyncwithasync_timeout.timeout(timeoutBudget):
connection=awaitctx.__aenter__()
exceptaTimeoutErrorase:
ifconnection:
# connection valid but pool returns it too late. Exception was raised in timeout# __aexit__, return connection to loopawaitctx.__aexit__()
exceptasyncio.CancelledError:
# user canceled requestifconnection:
awaitctx.__aexit__()
[async-timeout](https://github.com/aio-libs/async-timeout) does not create additional task from ctx.__aenter__() coroutine. This code work for us (but maybe we're lucky).
the issue with a local PostgreSQL install?: no, local installation
uvloop?: didn't try, problem with reproducing, we use uvloop
We use connection pool. And connections are ending after while and our app is hang up (all new response has status code 500 and error "can not get connection from pool"). We observe this behavior when postgres process several queries simultaneous long time and we try to get new connection from the pool with timeout. This is not not stable and can happen 1 per week.
We debug and see following strangeness:
Our code
asyncpg code (we added log)
We calculate "connection was received" messages count (let be A ) and "connection was gotten from queue" messages count (let be B ) after after app hanged up.
B - A = connection pool size.We have hypothesis that problem here . Future is completed but occurred timeout before
compat.wait_forreturn result.We try to use following code
[async-timeout](https://github.com/aio-libs/async-timeout)does not create additional task fromctx.__aenter__()coroutine. This code work for us (but maybe we're lucky).