[v8] Update for ESM monkeypatch + ESM file structure

[AbhiPrasad]

We’re going to be adding proper support for ESM in the Sentry repo.

Requirement:

• There is no require in ESM bundles at all
• ESM files have .mjs file extension
• CJS files have .cjs file extension

### Tasks
- [ ] https://github.com/getsentry/sentry-javascript/pull/10069
- [x] (pre-v8) Figure out how to use `dynamicRequire` with ESM
- [x] (pre-v8) Remove usage of `require.main.filename` for module name resolution #10061
- [x] (v8) Vendor `https-proxy-agent` which is cjs only #10088
- [x] (v8) Change build to output `mjs`/`cjs` for esm/cjs
- [x] (v8) Remove conditional import of `worker_threads` #10791
- [x] (v8) Update `inspector.d.ts` to work with esm and cjs
- [x] (v8) Remove require call in `require('inspector')` in `LocalVariables` integration
- [x] (v8) Update Rollup
- [ ] https://github.com/getsentry/sentry-javascript/pull/10928
- [ ] https://github.com/getsentry/sentry-javascript/issues/11066
- [ ] https://github.com/getsentry/sentry-javascript/issues/11067

Read below for some justification and information

Module Customization Hooks for Monkey Patching

For ESM bundles on server-side, we’ll require a minimum Node version of 18.6.0 or 20.6.0 or higher. Otherwise we’ll support CJS bundles for Node 14+ generally. This is because we want access to Node customization hooks, which allows us to add monkeypatching for esm modules programatically (doesn't require command-line loader).

We can take some inspiration here from https://github.com/DataDog/import-in-the-middle

import { register } from 'node:module';

register('./sentry-patch.mjs', import.meta.url);

Registering hooks to affect an import still needs to happen before the import is resolved, so there are ordering issues, but I think we can work around this by recommending that Sentry.init is called as soon as possible, and encouraging Sentry.addIntegration patterns for integrations that require objects from different libraries (like express with app).

The modules we need register via register API:

• Node node:http and node:https (diagnostics channels only works well Node 16+, so we need to maintain this for Node 14
• Remix @remix-run/server-runtime and react-router-dom - monkeypatch
• OTEL esm instrumentation
• GoogleCloudGrpc integration require('google-gax')
• GoogleCloudHttp integration with require('@google-cloud/common')
• AWSServices integration with require('aws-sdk/global')

We need to re-evaluate if we need monkeypatching for the database integrations now that we have OpenTelemetry.

Change File Structure

We’ll need to move our file structure to the following:

{
 // at build time we strip `build/X`
 "main": "build/cjs/index.cjs",
 "module": "build/esm/index.mjs",
 "types": "build/types/index.d.cts",
 // note: this is an opportunity for us to explore multiple subpaths, like
 // exporting from `@sentry/browser/utils`
 "exports": {
 "./package.json": "./package.json",
 ".": {
 // esm
 "import": {
 // path must be relative
 "types": "./build/types/index.d.mts",
 "default": "./build/esm/index.mjs"
 },
 // cjs
 "require": {
 // path must be relative
 "types": "./build/types/index.d.cts",
 "default": "./build/esm/index.mjs"
 }
 // we might need to expose an ESM only export for customization hooks
 // to use via node --loader=... API
 }
 },
 "typesVersions": {
 "<4.9": {
 "build/types/index.d.cts": [
 "build/types-ts3.8/index.d.cts"
 ],
 "build/types/index.d.mts": [
 "build/types-ts3.8/index.d.mts"
 ]
 }
 },
 "files": [
 "cjs",
 "esm",
 "types",
 "types-ts3.8"
 ]
}

Shameless promo: Watch my talk if you want more details about this.

We also should change all node standard library imports to import from node:X.

Make dynamicRequire/loadModule work with ESM import

This exists to trick webpack, but we use it all over the codebase. We need to add an esm compatible way.

One thing we can do is introduce an async dynamicRequire that simulates await import, and then build time replace the functionality of dynamicRequire to use import or require under the hood?

• packages/node/src/integrations/anr/index.ts - to conditionally import worker_threads. This is just because of Node 12 support, we’ll remove this import entirely
• packages/node/src/integrations/local-variables/local-variables-async.ts - to conditionally import node:inspector/promises.
• All of our database integrations - these are all going away
• packages/utils/src/time.ts - this needs to be refactored entirely. We still want an isomorphic solution because we need date-time helpers that work across core/browser/node and all other packages, but what should happen is that we look for the globalThis.performance instead of relying on perf_hooks. This means that if the global performance API is not available we will fall back to plain [Date.now](http://Date.now) and similar helpers (basically only Node 14 for us). We also need to add a smarter timeOrigin calculation that reset’s itself to help alleviate issues with drifting monotonic clock. See Provide a supported way to get anchored clock times open-telemetry/opentelemetry-js#3279 (comment)

[AbhiPrasad]

To validate our package.json setup, we'll need to add https://www.npmjs.com/package/@arethetypeswrong/cli as a validation check in CI.

[mydea]

I don't think we'll need to worry about node:http and node:https, as I guess for node this will be taken care of by the otel instrumentation?

[timfish]

There is also usage of require.main.filename for module name resolution

[AbhiPrasad]

I don't think we'll need to worry about node:http and node:https, as I guess for node this will be taken care of by the otel instrumentation?

We'll still need something to add breadcrumbs right? Or does that work?

[AbhiPrasad]

There is also usage of require.main.filename for module name resolution

Related GH issue: nodejs/node#49440

[mydea]

We'll still need something to add breadcrumbs right? Or does that work?

For this we hook into the otel instrumentation and add breadcrumbs from there - see the http integration in node-experimental!

[timfish]

There is also usage of require.main.filename for module name resolution

Turns out process.argv[1] will be and has always been the entry point in Node so we can use that over require.main.filename. The only place this doesn't hold true is Electron where this is automatically fetched from package.json > main and doesn't end up in argv

[timfish]

// note: this is an opportunity for us to explore multiple subpaths, like
// exporting from @sentry/browser/utils

package.json exports support was added in Node v12.7.0. Before that, the only way to support subpaths is to output the cjs files into a matching directory in the package root.

https://nodejs.org/api/packages.html#exports

The history suggests conditional exports was added even later but I guess it will still fallback to main.

[Lms24]

One thing we might need to consider is our usage of http-proxy-agent in the Node transport. Whenever I used ESM versions of our SDKs in Vite-based frameworks (Sveltekit, Astro) it seemed like Vite wasn't able to correctly work with this dependency because it's CJS only (so even after experimentally removing stuff like the http integrations to avoid dynamic requires, this seemed to be the problem I didn't find a workaround for).

Can we somehow get rid of this dependency?

Chances are that this problem is limited to Vite as #10042 shows. However, while we can change the respective vite config setting in Astro for our users, it's not as easy in other frameworks.

I'd vote we look at this once we made the other changes to check if it's actually still a problem by then. Maybe it was just a by-product of us using older package.json properties or the generic file extensions.