Skip to content

[Firestore] #3161

Description

@t0mstah

[READ] Step 1: Are you in the right place?

I believe this belongs in nodejs-firestore because the issue is specifically with Firestore Admin SDK document reads,
and firebase-admin is using @google-cloud/firestore underneath.

[REQUIRED] Step 2: Describe your environment

* Operating System version: macOS 26.3.1
* Firebase SDK version: `firebase-admin@13.10.0`
* `@google-cloud/firestore@7.11.6`
* `@grpc/grpc-js@1.14.4`
* `firebase-functions@7.2.5`
* `firebase-tools@15.18.0`
* Firebase Product: Firestore
* Node.js version: v22.22.2
* NPM version: 10.9.7

Runtime environment: Firebase Cloud Functions Gen 2 / Cloud Run, Node.js 22.

[REQUIRED] Step 3: Describe the problem

Steps to reproduce:

I am seeing intermittent very slow Firestore Admin SDK reads from Firebase Gen 2 public onRequest functions.

The function handles high-volume redirect/callback traffic. Inside the handler, it performs a simple Firestore document
read:

awaitadmin.firestore().collection('users').doc(userId).get()Mostreadsarenormal,usuallytensofmilliseconds.However,asmallpercentageintermittentlytakemultipleseconds,sometimes10s+,andIrecentlyobservedonesimpleusers/{userId}.get()taking~23s.Theslowreadsdonotappeartobeexplainedbyapplication-leveldocumentcontentionordocumentsize:
-TheslowreadsarenotconcentratedonasingleuserId.-Theslowuserdocumentsarenormal-sized,roughly4KB-14KBinthesampleschecked.-Afast-usersamplehadsimilarorlargerdocuments,includingp95around20KB.-Firestoreauditlogsdidnotshowsame-documentwritesduringtheslowreadwindows.-CPUandmemoryonCloudRunarenotsaturated.-CloudRunrequestconcurrencyislow.-ThesamecollectionandsimilarreadsfromonCallfunctionsappeartoreturnnormally.-Theissueseemsconcentratedinhigh-trafficpublicGen2onRequestfunctions.-Replacing/flushingtheCloudRuninstanceswithano-codegcloudrunservicesupdateclearstheissuetemporarily.ThismakesitlooklikesomewarminstancesmaygetintoadegradedFirestoreclient/channelstatewhereindividualreadsstallforseconds,buttheNodeprocessremainsotherwisehealthy.Exampletimingfromproductioninstrumentation:
{"operation": "users/{userId}.get()","userGetMs": 23220,"documentSizeBytes": 10278,"sameDocumentWritesDuringRead": 0}OtherslowexamplesinthesamewindowincludeduserGetMsaround3s-6s,whilep50/p95stayednormal.Expectedbehavior:
Asimpledoc().get()forasmallFirestoredocumentshouldnotintermittentlyhangfor5s-20s+onotherwisehealthywarmCloudRun/Gen2functioninstances.IftheFirestoreclient/channelbecomesunhealthy,Iwouldexpectittorecover,failclearly,orexposeanactionableerrorratherthancontinuingtoserverarebutveryslowreadsuntiltheinstanceisreplaced.Actual behavior:
Mostreadsarefast,butasmalllongtailofreadsstallsforseconds.Restarting/replacingCloudRuninstancestemporarilyresolvesit.
#### Relevant Code:
import*asadminfrom'firebase-admin'import{onRequest}from'firebase-functions/v2/https'import{performance}from'perf_hooks'admin.initializeApp()exportletrouterRedirect=onRequest(async(req,res)=>{letuserId: string=String(req.query.userId)letuserGetStartMs: number=performance.now()letuserDoc: admin.firestore.DocumentSnapshot=awaitadmin.firestore().collection('users').doc(userId).get()letuserGetMs: number=performance.now()-userGetStartMsconsole.log({message: 'firestoreUserGetTiming',
userId,
userGetMs,exists: userDoc.exists})res.status(200).send('ok')})

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions