Skip to content

Connection pool aenter get connection from queue and not return connection to user. #955

Description

@matemax
  • asyncpg version: 0.26.0
  • PostgreSQL version: 12
  • Do you use a PostgreSQL SaaS? If so, which? Can you reproduce
    the issue with a local PostgreSQL install?
    : no, local installation
  • Python version: 3.9
  • Platform: linux, centos8
  • Do you use pgbouncer?: no
  • Did you install asyncpg with pip?: we use poetry
  • If you built asyncpg locally, which version of Cython did you use?: no
  • Can the issue be reproduced under both asyncio and
    uvloop?
    : didn't try, problem with reproducing, we use uvloop

We use connection pool. And connections are ending after while and our app is hang up (all new response has status code 500 and error "can not get connection from pool"). We observe this behavior when postgres process several queries simultaneous long time and we try to get new connection from the pool with timeout. This is not not stable and can happen 1 per week.

We debug and see following strangeness:

Our code

ctx: PoolAcquireContext=self.pool.acquire(timeout=timeoutBudget)
try: connection=awaitctx.__aenter__()
logger.debug("connection was received")
exceptasyncio.TimeoutError:
...

asyncpg code (we added log)

asyncdef_acquire(self, timeout):
asyncdef_acquire_impl():
ch=awaitself._queue.get() # type: PoolConnectionHoldertry:
proxy=awaitch.acquire() # type: PoolConnectionProxyexcept (Exception, asyncio.CancelledError):
self._queue.put_nowait(ch)
raiseelse:
# Record the timeout, as we will apply it by default# in release().ch._timeout=timeoutlogger.debug("connection was gotten from queue")
returnproxyifself._closing:
raiseexceptions.InterfaceError('pool is closing')
self._check_init()
iftimeoutisNone:
returnawait_acquire_impl()
else:
returnawaitcompat.wait_for(
_acquire_impl(), timeout=timeout)

We calculate "connection was received" messages count (let be A ) and "connection was gotten from queue" messages count (let be B ) after after app hanged up. B - A = connection pool size.

We have hypothesis that problem here . Future is completed but occurred timeout before compat.wait_for return result.

We try to use following code

ctx: PoolAcquireContext=self.pool.acquire()
connection=Nonetry:
asyncwithasync_timeout.timeout(timeoutBudget):
connection=awaitctx.__aenter__()
exceptaTimeoutErrorase:
ifconnection:
# connection valid but pool returns it too late. Exception was raised in timeout# __aexit__, return connection to loopawaitctx.__aexit__()
exceptasyncio.CancelledError:
# user canceled requestifconnection:
awaitctx.__aexit__()

[async-timeout](https://github.com/aio-libs/async-timeout) does not create additional task from ctx.__aenter__() coroutine. This code work for us (but maybe we're lucky).

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions