aws-serverless + SST: LocalVariables integration / its usage of Node Inspector kills lambda functions and floods Sentry issues

[steffen-mueller]

Is there an existing issue for this?

• I have checked for existing issues https://github.com/getsentry/sentry-javascript/issues
• I have reviewed the documentation https://docs.sentry.io/
• I am using the latest SDK release https://github.com/getsentry/sentry-javascript/releases

How do you use Sentry?

Sentry Saas (sentry.io)

Which SDK are you using?

@sentry/aws-serverless

SDK Version

8.41.0

Framework Version

No response

Link to Sentry event

https://addcraft.sentry.io/issues/5987124956

Reproduction Example/SDK Setup

We've set up Sentry according to the NPM Package - ESM method in the docs. We are using SST v3.3.30.

Sentry is setup in a sentry.ts file:

/**
 * This file initializes Sentry. It has to be an own file so it is loaded before any other module.
 * The loading happens via NODE_OPTIONS in the infra/Helpers.ts file.
 * 
 * For more information, see https://docs.sentry.io/platforms/javascript/guides/aws-lambda/install/esm-npm/
 * This file is compiled into config/sentry-loader.js by running `npm run build-sentry-loader`.
 * This is done automatically on npm postinstall event.
 */

import * as Sentry from "@sentry/aws-serverless";

console.log("Initializing Sentry for DSN "+process.env.SENTRY_DSN);

Sentry.init({
 enabled: !!process.env.SENTRY_DSN,
 dsn: process.env.SENTRY_DSN,
 release: process.env.PLUGIN_VERSION || "0.1.0-SNAPSHOT",
 environment: process.env.PRESENTER_STAGE || "local",

 enableTracing: true,
 
 // Keep it on 1.0 on prod for now
 tracesSampleRate: 1.0,

 // Profiling Sample Rate 1 for now
 profilesSampleRate: 1.0,

 includeLocalVariables: true
});

As described in the code comment above, we compile this .ts file to .js on build time:

const esbuild = require('esbuild');

esbuild.build({
 entryPoints: ['src/sentry.ts'],
 bundle: true,
 platform: 'node',
 target: 'node18',
 outfile: 'config/sentry-loader.js',
 format: 'cjs'
});

The created sentry-loader.js is referenced in the lambda definition of the SST endpoint:

export const backendApi = new sst.aws.ApiGatewayV2("BackendApi", {
 domain: "api.myapp.com"
});

backendApi.route("$default", {
 handler: "src/index.handler",
 runtime: "nodejs20.x", // This was "nodejs18.x" before, see actual result for details
 timeout: "30 seconds",
 copyFiles: [
 { from: "./config" },
 { from: "./src/views" }
 ],
 environment: {
 DEPLOYED_REGION: aws.getRegionOutput().name,
 NODE_ENV: process.env.NODE_ENV || "development",
 PLUGIN_VERSION: process.env.PLUGIN_VERSION || "0.0.0",

 // Sentry configuration
 SENTRY_DSN: process.env.SENTRY_DSN || "",
 NODE_OPTIONS: "--import ./config/sentry-loader.js",
 SENTRY_TRACES_SAMPLE_RATE: "1.0"
 }
 link: [
 // Some boring dynamodb tables etc.
 ]
});

Steps to Reproduce

Setup a SST project with a sentry ESM loader as explained above and deploy to production.

Expected Result

Sentry logs errors and causes no problems.

Actual Result

The results differ between runtime versions (I assume - we've upgraded the runtime version along with a big bunch of NPM packages, including Sentry, going from 8.33.1 to 8.41.0, and I don't want to use the production environment to isolate the problem :)).

With runtime "nodejs20.x":

The lambda function does not start up at all, resulting in HTTP 503s after 30 seconds. The CloudWatch logs say:

Initializing Sentry for DSN https://[redacted]@[redacted].ingest.us.sentry.io/[redacted]
Debugger listening on ws://127.0.0.1:39651/3c7e36e4-ec77-4846-a745-7f5f5759e85f
For help, see: https://nodejs.org/en/docs/inspector
Debugger listening on ws://127.0.0.1:37035/481827cc-e29d-420a-9d45-8bb57a07511b
For help, see: https://nodejs.org/en/docs/inspector
Debugger listening on ws://127.0.0.1:36793/4cb1457e-a94c-406f-8006-80bddf296d28
For help, see: https://nodejs.org/en/docs/inspector
[... some dozens more ...]
Debugger listening on ws://127.0.0.1:38873/d5df3f49-ee27-41b7-b8dd-09ef21cb3c42
For help, see: https://nodejs.org/en/docs/inspector
Debugger listening on ws://127.0.0.1:38959/5191a315-bc8e-493e-b196-e2cdafded6d1
For help, see: https://nodejs.org/en/docs/inspector
Debugger listening on ws://127.0.0.1:45543/100f3533-56d2-476c-babc-b19a2a203d3a
For help, see: https://nodejs.org/en/docs/inspector
INIT_REPORT Init Duration: 30000.64 ms	Phase: invoke	Status: timeout

With runtime "nodejs18.x"

The application works as expected, but the Sentry issues are flooded with thousands (!) of issues affecting 0 users looking like that:

{
 "event_id": "d3234489202f43dca677175e8bf3fd5d",
 "project": [redacted],
 "release": "[redacted]",
 "dist": null,
 "platform": "node",
 "message": "",
 "datetime": "2024-11-27T16:38:44+00:00",
 "tags": [
 [
 "environment",
 "production"
 ],
 [
 "handled",
 "no"
 ],
 [
 "level",
 "error"
 ],
 [
 "mechanism",
 "onunhandledrejection"
 ],
 [
 "os.name",
 "Linux"
 ],
 [
 "runtime",
 "node v20.18.0"
 ],
 [
 "runtime.name",
 "node"
 ],
 [
 "release",
 "[redacted]"
 ],
 [
 "server_name",
 "[redacted]"
 ]
 ],
 "_dsc": {
 "environment": "[redacted]",
 "public_key": "[redacted]",
 "release": "[redacted]",
 "replay_id": null,
 "trace_id": "[redacted]",
 "transaction": null
 },
 "_metrics": {
 "bytes.ingested.event": 2792,
 "bytes.stored.event": 5297
 },
 "contexts": {
 "app": {
 "app_start_time": "2024-11-27T16:38:28.640Z",
 "app_memory": 582651904,
 "free_memory": 525627392,
 "type": "app"
 },
 "cloud_resource": {
 "cloud.platform": "AWS_Lambda_nodejs20.x",
 "cloud.provider": "aws",
 "cloud.region": "eu-central-1",
 "type": "cloud_resource"
 },
 "culture": {
 "locale": "en-US",
 "timezone": "UTC",
 "type": "culture"
 },
 "device": {
 "arch": "x64",
 "memory_size": 1249615872,
 "free_memory": 525627392,
 "boot_time": "2024-11-27T16:36:44.205Z",
 "processor_count": 2,
 "cpu_description": "Intel(R) Xeon(R) Processor @ 2.50GHz",
 "processor_frequency": 0,
 "type": "device"
 },
 "os": {
 "name": "Linux",
 "kernel_version": "5.10.227-239.884.amzn2.x86_64",
 "type": "os"
 },
 "runtime": {
 "runtime": "node v20.18.0",
 "name": "node",
 "version": "v20.18.0",
 "type": "runtime"
 },
 "trace": {
 "trace_id": "[redacted]",
 "span_id": "b194994bab6ec9c4",
 "status": "unknown",
 "type": "trace"
 }
 },
 "culprit": "#onMessage(node:inspector)",
 "environment": "production-de",
 "exception": {
 "values": [
 {
 "type": "Error",
 "value": "Inspector error -32000: Could not find object with given id",
 "stacktrace": {
 "frames": [
 {
 "function": "AsyncResource.runMicrotask",
 "module": "task_queues",
 "filename": "node:internal/process/task_queues",
 "abs_path": "node:internal/process/task_queues",
 "lineno": 137,
 "colno": 8,
 "in_app": false
 },
 {
 "function": "AsyncResource.runInAsyncScope",
 "module": "node:async_hooks",
 "filename": "node:async_hooks",
 "abs_path": "node:async_hooks",
 "lineno": 206,
 "colno": 9,
 "in_app": false
 },
 {
 "module": "task_queues",
 "filename": "node:internal/process/task_queues",
 "abs_path": "node:internal/process/task_queues",
 "lineno": 140,
 "colno": 7,
 "in_app": false
 },
 {
 "module": "node:inspector",
 "filename": "node:inspector",
 "abs_path": "node:inspector",
 "lineno": 75,
 "colno": 58,
 "in_app": false
 },
 {
 "function": "#onMessage",
 "module": "node:inspector",
 "filename": "node:inspector",
 "abs_path": "node:inspector",
 "lineno": 87,
 "colno": 15,
 "in_app": false
 }
 ]
 },
 "mechanism": {
 "type": "onunhandledrejection",
 "handled": false
 }
 }
 ]
 },
 "extra": {
 "unhandledPromiseRejection": true
 },
 "fingerprint": [
 "{{ default }}"
 ],
 "grouping_config": {
 "enhancements": "KLUv_SAYwQAAkwKRs25ld3N0eWxlOjIwMjMtMDEtMTGQ",
 "id": "newstyle:2023-01-11"
 },
 "hashes": [
 "[redacted]"
 ],
 "ingest_path": [
 {
 "version": "24.11.1",
 "public_key": "[redacted]"
 }
 ],
 "key_id": "3799414",
 "level": "error",
 "location": "node:inspector",
 "logger": "",
 "metadata": {
 "filename": "node:inspector",
 "function": "#onMessage",
 "in_app_frame_mix": "system-only",
 "type": "Error",
 "value": "Inspector error -32000: Could not find object with given id"
 },
 "nodestore_insert": 1732725526.58189,
 "received": 1732725524.25593,
 "sdk": {
 "name": "sentry.javascript.aws-serverless",
 "version": "8.33.1",
 "integrations": [
 "InboundFilters",
 "FunctionToString",
 "LinkedErrors",
 "RequestData",
 "Console",
 "Http",
 "NodeFetch",
 "OnUncaughtException",
 "OnUnhandledRejection",
 "ContextLines",
 "LocalVariablesAsync",
 "Context",
 "Modules",
 "Aws",
 "AwsLambda",
 "AWSLambda"
 ],
 "packages": [
 {
 "name": "npm:@sentry/aws-serverless",
 "version": "8.33.1"
 }
 ]
 },
 "timestamp": 1732725524.164,
 "title": "Error: Inspector error -32000: Could not find object with given id",
 "type": "error",
 "version": "7"
}

Workaround

Disabling includeLocalVariables ends the pain - the lambda loads as expected again.

[Lms24]

Hi, this looks indeed like a problem with our localVariables integration or probably with Node's inspector API. For the moment it's probably easiest to leave includeLocalVariables disabled.

I will tag @timfish - did you by any chance test local variables in lambda environments? I think this might very well be unexplored territory otherwise.

[timfish]

I don't know why this doesn't work with lambda but I suspect that it's because there are differences to plain Node.

Can you attach a debugger remotely to aws lambda? It would be very tricky/laborious to debug this if you can't!

[jonmast]

I spent some time debugging, and I believe this is because the NODE_OPTIONS environment variable is passed down into the worker, leading to an infinite spawn loop.

We use a nearly identical setup to @steffen-mueller, and I added an isMainThread check in our loader and it resolved the issue.

import { isMainThread } from "node:worker_threads";

if (isMainThread) {
 Sentry.init({
 includeLocalVariables: true,
 // ...
 });
} else {
 // skip setup in worker so we don't infinitely recurse
 console.warn("Sentry init skipped in worker thread");
}

[timfish]

I believe this is because the NODE_OPTIONS environment variable is passed down into the worker

Oh nice find! We specifically pass an empty execArgv: [] when we start workers so that --import doesn't get passed through and it looks like we need to exclude NODE_OPTIONS too.