Add path mapping support to ESM and CJS loaders by geigerzaehler · Pull Request #1585 · TypeStrong/ts-node

geigerzaehler · 2021-12-29T14:37:10Z

The ESM loader now resolves import paths using TypeScripts path mapping feature.

EDIT:
Closes#1586

The ESM loader now resolves import paths using TypeScripts [path mapping][1] feature. [1]: https://www.typescriptlang.org/docs/handbook/module-resolution.html#path-mapping Signed-off-by: Thomas Scholtes <geigerzaehler@axiom.fm>

cspotcode · 2021-12-29T18:16:14Z

Thanks for this. Because of the American holidays, I have somewhat of a backlog of issues and PRs which need attention before I'll be able to properly review this. There may be some delay.

I see some refactoring in node-errors; can you explain the motivation for some of those changes? It's good to have an explanation for such changes when reviewing.

Your description says this enables path mapping for the ESM loader. We'd like to also extend the same functionality to the CommonJS loader. I see that your createPathMapper implementation looks clean and modular; do you anticipate any issues if we attempt to use it for CommonJS as well? We'd hook into Module._resolveFilename for this.

Is this feature-flagged? We should decide if it's enabled or disabled by default, and decide how users can opt-in or opt-out.

geigerzaehler · 2021-12-29T18:48:19Z

There may be some delay.

No problem.

I see some refactoring in node-errors; can you explain the motivation for some of those changes?

The new resolving code needs to detect ERR_MODULE_NOT_FOUND errors so it can try more mapped candidates. The builtin NodeError from Node’s internal error module has a code property that allows you to distinguish errors. The change in node-errors also sets this property on the error instances. The new logic for defining errors is similar to the code in Node’s error module.

do you anticipate any issues if we attempt to use it for CommonJS as well?

I don’t think there will be any issues. The path mapping code is does not include any logic that is specific to ESM resolution. The only inputs to it are the TypeScript compiler options.

Is this feature-flagged? We should decide if it's enabled or disabled by default, and decide how users can opt-in or opt-out.

It’s not feature-flagged at the moment because I couldn’t think of a scenario where you would want the path mapping to be disabled. If you have configured the paths compiler option and you are using it in your code, your code will only run if the path mapping is enabled. If path mapping is disabled the code will always fail at the module resolution stage which makes the option unusable. So I’d lean towards not feature-flagging this for simplicity. But if there’s a good reason to put this behind a feature flag it should be easy to accomplish

cspotcode · 2021-12-29T20:13:06Z

If you have configured the paths compiler option and you are using it in your code, your code will only run if the path mapping is enabled.

Good point. And I think someone could disable it like this:

{
"compilerOptions": {
"paths": { ... }
},
"ts-node": {
"compilerOptions": {
"paths": {} // empty object: disable path mappings in ts-node
}
}
}

codecov · 2021-12-29T20:24:43Z

Codecov Report

Attention: Patch coverage is 93.49593% with 8 lines in your changes missing coverage. Please review.

Project coverage is 80.78%. Comparing base (ff78b64) to head (6632481).
Report is 67 commits behind head on main.

Files with missing lines	Patch %	Lines
src/path-mapping.ts	92.85%	1 Missing and 2 partials ⚠️
src/esm.ts	92.30%	1 Missing and 1 partial ⚠️
src/index.ts	71.42%	1 Missing and 1 partial ⚠️
src/cjs-resolve-hooks.ts	96.00%	0 Missing and 1 partial ⚠️

Additional details and impacted files

Files with missing lines	Coverage Δ
dist-raw/node-internal-errors.js	`97.50% <100.00%> (+1.84%)`	⬆️
src/configuration.ts	`86.06% <ø> (ø)`
src/cjs-resolve-hooks.ts	`87.17% <96.00%> (+13.84%)`	⬆️
src/esm.ts	`84.72% <92.30%> (+1.38%)`	⬆️
src/index.ts	`79.56% <71.42%> (-0.13%)`	⬇️
src/path-mapping.ts	`92.85% <92.85%> (ø)`

... and 3 files with indirect coverage changes

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

cspotcode

I left comments from an initial code review. I plan to implement the requested changes myself, but if you are feeling enthusiastic and would like to do them before me, I will certainly appreciate the assistance. No pressure though.

Overall this looks great to me. I definitely want to add CommonJS support which shouldn't be too difficult. I'm already familiar with the hooking API.

cspotcode · 2022-01-24T05:26:07Z

+): PathMapper {
+ if (compilerOptions.paths) {
+ if (!compilerOptions.baseUrl) {
+ throw new Error(`Compiler option 'baseUrl' required when 'paths' is set`);


Is compilerOptions.baseUrl guaranteed to be an absolute path, or do we need to resolve it relative to the tsconfig's location?

Good point. I assumed that we always get the configuration from ts.readConfigFile() in src/configuration.ts and the function resolves the base URL relative to the config file. But there are probably other ways to set the base URL like through the command line, API, or environment.
To address this I think it make sense to resolve any base URL that is not a URL but a relative path to the current directory.

I see, we almost always get the configuration from ts.readConfigFile(), so we should be good. I think the only way to set it otherwise is via the API? Since we don't support all of tsc's command-line flags.
Maybe the correct approach is to resolve baseUrl once within create() so that we have a reliable, absolute baseUrl to be used elsewhere.

Maybe the correct approach is to resolve baseUrl once within create() so that we have a reliable, absolute baseUrl to be used elsewhere.
Alternatively, we could require baseUrl to be absolute and throw an error otherwise. This would allow us to avoid implicit, environment-dependent behavior for users of the API. This might prevent some confusion when path mapping doesn’t work as expected but only fails at the resolution stage instead of the construction stage. (For example a user may think the base URL they set on the API is considered relative to tsconfig.json and their scripts work whenever they are run from the project root. But then they run the script from a subdirectory and it fails.)

Sounds good, let's do that. In create() we throw an error if baseUrl is not absolute.

Actually, scratch that, seems we always pass options through TypeScript's API to be normalized, even when provided via our API:
$ node > require('ts-node').create({compilerOptions: {baseUrl: "foobar"}}).config.options.baseUrl '/home/ubuntu/dev/ts-node/ts-node/path-mapping/foobar'
So I guess we're all good.

Signed-off-by: Thomas Scholtes <geigerzaehler@axiom.fm>

geigerzaehler · 2022-01-24T11:15:08Z

I definitely want to add CommonJS support which shouldn't be too difficult. I'm already familiar with the hooking API.

I’d prefer to keep scope of this PR small and I’m not familiar enough with the CommonJS side of things so I’ll leave this to you.

One thing to consider is that we’d probably want the users to explicitly opt in to path mapping for CommonJS. They’ll be using tsconfig-paths at the moment and it is unclear if the ts-node implementation can be used side-by-side or compatible.

cspotcode · 2022-01-24T16:34:49Z

Fair point. Would it be a great hardship for those users to disable tsconfig-paths? I hope they would appreciate the simplicity of removing a component. I would like our default behaviour to be the best forward-thinking behaviour, but perhaps we add an optional flag which can be used to selectively disable it for commonjs? Or we somehow detect tsconfig-paths and throw a warning when both mappers are competing?

…

On Mon, Jan 24, 2022 at 6:15 AM Thomas Scholtes ***@***.***> wrote: I definitely want to add CommonJS support which shouldn't be too difficult. I'm already familiar with the hooking API. I’d prefer to keep scope of this PR small and I’m not familiar enough with the CommonJS side of things so I’ll leave this to you. One thing to consider is that we’d probably want the users to explicitly opt in to path mapping for CommonJS. They’ll be using tsconfig-paths at the moment and it is unclear if the ts-node implementation can be used side-by-side or compatible. — Reply to this email directly, view it on GitHub <#1585 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAC35OGJBEFQKHZNAXQ5F33UXUYERANCNFSM5K6CBQ2Q> . You are receiving this because you commented.Message ID: ***@***.***>

cspotcode · 2022-01-25T05:08:39Z

I made an attempt at adding the CommonJS resolver hook. Still some outstanding TODOs that I need to address, but this is looking good.

cspotcode · 2022-01-25T05:32:10Z

Looks like we have test failures on Windows due to module specifiers using forward slashes and windows paths using backslashes. I haven't checked the code yet, but we probably have some logic assuming a module specifier and a path will always look the same. Should be an easy fix.

https://github.com/TypeStrong/ts-node/runs/4932463748?check_suite_focus=true#step:15:167

geigerzaehler · 2022-01-25T11:43:17Z

Would it be a great hardship for those users to disable tsconfig-paths?

I don’t think it would. I just wanted to point out, that this is a breaking change. I do believe most users will appreciate this replacement.

Considering the release strategy it probably makes sense to have the commonjs mapping as opt-in and ask users to enable it to collect feedback. Once the feature is stable it can be enabled by default (or always) in a new major release.

Or we somehow detect tsconfig-paths and throw a warning when both mappers are competing?

This sounds complex and error-prone. I think I’d prefer a solution where it is explicitly configured

cspotcode · 2022-01-25T15:11:26Z

I like that plan: opt-in via a flag today, enable by default in the next major version, get feedback in the meantime.

How about experimentalPathMapping: 'both' | 'cjs' | 'esm' | 'none'? Where 'esm' is the default today, and 'both' is the default in the next major version, with the flag renamed to pathMapping?

kuksik · 2023-03-16T21:59:42Z

some news about this PR?

mprinc · 2023-03-16T22:48:32Z

Sad that the ts-node tool is too complex (including its dependencies) that it is not possible to handle it promptly enough but it rather takes year and half ... honestly the whole ESM model with ts-node + ts paths is broken because of that, and a lot of libs are now delivering ONLY ESM packages ... Thanks God, I am using that workaround (--experimental-specifier-resolution=node --loader ) and more less I am ok, but it is insane that it cannot be integrated properly and that I have to have that ugly scripts:

node --experimental-specifier-resolution=node --loader $COLABO/ts-esm-loader-with-tsconfig-paths.js index.ts rather than just ts-node index.ts ...

0x80 · 2023-03-17T23:33:43Z

@mprinc The workaround is not yet clear to me from skimming through this discussion. Could you share the content of your ts-esm-loader-with-tsconfig-paths.js please?

trim21 · 2023-03-17T23:37:17Z

suggesting tsx （and also esm/cjs loader in its description） for this feature

Naddiseo · 2023-03-18T17:53:57Z

@0x80

@mprinc The workaround is not yet clear to me from skimming through this discussion. Could you share the content of your ts-esm-loader-with-tsconfig-paths.js please?

This is what I'm using with node 16

/*loader.jsUsage (I'm also using dotenv, but you can omit the dotenv parts if needed):DOTENV_CONFIG_PATH=.env node -r dotenv/config --loader=./loader.js /bin/path/to/cli.ts*/import{pathToFileURL}from"node:url";import{getFormat,load,resolveasresolveTs,transformSource}from"ts-node/esm";import*astsConfigPathsfrom"tsconfig-paths";export{getFormat,transformSource,load};const{ absoluteBaseUrl, paths }=tsConfigPaths.loadConfig();constmatchPath=tsConfigPaths.createMatchPath(absoluteBaseUrl,paths);exportfunctionresolve(specifier,context,defaultResolver){constmappedSpecifier=matchPath(specifier);if(mappedSpecifier){specifier=`${mappedSpecifier}.js`;/* the resolve functionality can only work with file URLs, so we need to convert, this is especially the case on windows, where the path might not be a valid URL. */consturl=specifier.startsWith("file:") ? specifier : pathToFileURL(specifier.toString());returnresolveTs(url.toString(),context,defaultResolver);}else{// If we can't find a mapping, just pass it on to the default resolverreturnresolveTs(specifier,context,defaultResolver);}}

gudh · 2023-03-21T06:02:11Z

@trim21 thanks, tsx works like a charm, no configuration needed!

troywweber7 · 2023-03-21T14:19:53Z

@trim21 I might be missing something... Why tsx and not mts (similar to mjs)? Or maybe I'm confusing who this was directed at...

trim21 · 2023-03-21T14:52:07Z

@trim21 I might be missing something... Why tsx and not mts (similar to mjs)? Or maybe I'm confusing who this was directed at...

that's a link, click it

troywweber7 · 2023-03-21T15:18:13Z

@trim21 thanks. My client wasn't rendering the link. Boo. Will look in browser.

mprinc · 2023-03-22T18:37:36Z

Hi Thijs (@0x80 ) here is the code I am using ts-esm-loader-with-tsconfig-paths.js:

// https://github.com/TypeStrong/ts-node/discussions/1450#discussioncomment-1806115// "serve": "cross-env TS_NODE_PROJECT=\"tsconfig.build.json\" node --experimental-specifier-resolution=node --loader ./loader.js src/index.ts",// TS_NODE_PROJECT=\"tsconfig.build.json\" node --experimental-specifier-resolution=node --loader ./loader.js src/index.ts"import{resolveasresolveTs}from"ts-node/esm";import*astsConfigPathsfrom"tsconfig-paths";import{pathToFileURL}from"url";const{ absoluteBaseUrl, paths }=tsConfigPaths.loadConfig();constmatchPath=tsConfigPaths.createMatchPath(absoluteBaseUrl,paths);exportfunctionresolve(specifier,ctx,defaultResolve){constmatch=matchPath(specifier);returnmatch ? resolveTs(pathToFileURL(`${match}`).href,ctx,defaultResolve) : resolveTs(specifier,ctx,defaultResolve);}export{load,getFormat,transformSource}from'ts-node/esm'

together with the command again:

# $COLABO is a path to where the file is tored
node --experimental-specifier-resolution=node --loader $COLABO/ts-esm-loader-with-tsconfig-paths.js index.ts

mprinc · 2023-03-22T18:40:28Z

NOTE: I am investigating tsx and considering already using esBuild actively in my other projects like codemod tools, which I have a great experience with both efficiency and community, I might soon switch to it. It seems pretty straightforward to do it.

loynoir · 2023-04-24T03:57:11Z

suggesting tsx （and also esm/cjs loader in its description） for this feature

@trim21 Coming from privatenumber/tsx#113 😞

Any news about this PR?

parzhitsky · 2023-09-11T19:51:23Z

+exports.codes = {};
+
+function defineError(code, buildMessage) {
+ if (!buildMessage) {
+ buildMessage = (...args) => args.join(' ');
 }
-}

-function createErrorCtor(errorMessageCreator) {
- return class CustomError extends Error {
+ exports.codes[code] = class CustomError extends Error {
 constructor(...args) {
- super(errorMessageCreator(...args))
+ super(`${code}: ${buildMessage(...args)}`);
+ this.code = code;
 }
 }
 }
+
+defineError("ERR_INPUT_TYPE_NOT_ALLOWED");
+defineError("ERR_INVALID_ARG_VALUE");
+defineError("ERR_INVALID_MODULE_SPECIFIER");
+defineError("ERR_INVALID_PACKAGE_CONFIG");
+defineError("ERR_INVALID_PACKAGE_TARGET");
+defineError("ERR_MANIFEST_DEPENDENCY_MISSING");
+defineError("ERR_MODULE_NOT_FOUND", (path, base, type = 'package') => {
+ return `Cannot find ${type} '${path}' imported from ${base}`;
+});
+defineError("ERR_PACKAGE_IMPORT_NOT_DEFINED");
+defineError("ERR_PACKAGE_PATH_NOT_EXPORTED");
+defineError("ERR_UNSUPPORTED_DIR_IMPORT");
+defineError("ERR_UNSUPPORTED_ESM_URL_SCHEME");
+defineError("ERR_UNKNOWN_FILE_EXTENSION");


Why is this portion of the file exactly identical to dist-raw/node-errors.js?

Baterka · 2024-01-04T15:56:54Z

Any movement on this?

sthasam2 · 2024-01-05T06:24:23Z

Is this still in the works?

Dudeprogram · 2024-01-05T11:04:07Z

no dear

…

On Thu, Jan 4, 2024 at 8:57 PM Baterka ***@***.***> wrote: Any movement on this? — Reply to this email directly, view it on GitHub <#1585 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AWO46HSOCMJUSOCBYVE7RETYM3GOVAVCNFSM5K6CBQ22U5DIOJSWCZC7NNSXTN2JONZXKZKDN5WW2ZLOOQ5TCOBXG4ZTGNZSG44Q> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

billomore · 2024-02-11T21:19:54Z

tsx doesn't play well with monorepos, namely with decorators in monorepos. swcdoesn't play nice with ESM either, not sure if planned to fix. ts-node was the last citadel that worked out of the box until recently, but without any change or update it just stopped... sad :(

genesiscz · 2024-04-24T22:38:56Z

we really need this :(

cameralis · 2024-05-30T01:50:57Z

Tried to run ts. After 4 hours of debugging I ended up here. Where even am I???

ladal1 · 2024-07-20T15:54:07Z

We're two and a half year into this PR, loader hooks for esm are on release condidate stage, can we push this the final mile?

ethyaan · 2024-10-12T14:16:35Z

Really? 3 years in the making?? is this hard to push few lines of code and test it nowadays?

phips28 · 2025-04-08T19:21:13Z

🍿 4 years

geigerzaehler · 2025-04-11T09:46:27Z

Hi everybody. A lot has changed since I opened this PR. I’ve since switched to tsx and subpath imports and I have no motivation to continue this effort, so I’m closing the PR. The branch will remain available if anyone wants to pick it up.

To those frustrated by the lack of progress: open source runs on volunteer effort. If something matters to you, contribute. Drive-by complaints don’t help — they actively discourage maintainers and contributors from engaging.

That doesn’t mean there is nothing to criticize about how this PR was handled. But if you're just pointing out that a problem still exists, without offering help or insight, you're not moving things forward.

Add path mapping support to ESM loader
3be1fde
The ESM loader now resolves import paths using TypeScripts [path mapping][1] feature. [1]: https://www.typescriptlang.org/docs/handbook/module-resolution.html#path-mapping Signed-off-by: Thomas Scholtes <geigerzaehler@axiom.fm>

cspotcode mentioned this pull request Dec 29, 2021
[EPIC] Full TS-style resolution hooks #1514
Open
1 task

cspotcode mentioned this pull request Dec 29, 2021
Set correct code on errors returned by ESM resolver #1586
Open

geigerzaehler added 3 commits December 29, 2021 22:43

fixup! Add path mapping support to ESM loader
c542d98

fixup! fixup! Add path mapping support to ESM loader
0cbfa6c

fixup! fixup! fixup! Add path mapping support to ESM loader
e7082b1

cspotcode mentioned this pull request Jan 15, 2022
tsconfig.paths not working with ES modules #1596
Closed

cspotcode added this to the next milestone Jan 21, 2022

cspotcode requested changes Jan 24, 2022
View reviewed changes

cspotcodeand others added 4 commits January 24, 2022 00:31

Merge remote-tracking branch 'origin/main' into path-mapping
69a397b

address code review comments
2fcb982
Signed-off-by: Thomas Scholtes <geigerzaehler@axiom.fm>

map esm paths in all included files
48f5262
Signed-off-by: Thomas Scholtes <geigerzaehler@axiom.fm>

improve error message when mapped module is not found
a19454b
Signed-off-by: Thomas Scholtes <geigerzaehler@axiom.fm>

cspotcode mentioned this pull request Jan 24, 2022
ESM support: soliciting feedback #1007
Open
4 tasks

cspotcode reviewed Jan 24, 2022
View reviewed changes

Comment threadsrc/esm.ts Outdated
Comment threadsrc/esm.ts
Comment threadsrc/esm.ts Outdated
Comment threadsrc/esm.ts
Comment threadsrc/index.ts

Review changes; add CommonJS path mapping
32e26e8

fix failing tests
362935b

add path mapping to docs
e573fd7

cspotcode mentioned this pull request Jan 25, 2022
[META] v11 breaking changes #1505
Open
24 tasks

add flag to enable/disable path mapping in the two loaders
0012b22

deodad mentioned this pull request Dec 26, 2022
perf: use swc or esbuild toolchain to transpile code locally farcasterxyz/hub-monorepo#314
Closed

loynoir mentioned this pull request Apr 24, 2023
swc as tsx compile backend alternative privatenumber/tsx#226
Open

nextlevelbeard mentioned this pull request May 10, 2023
@wdio/cucumber-framework: Filter specs /w Cucumber Tag Expression before spawning workers webdriverio/webdriverio#10330
Merged
8 tasks

erwinheitzman mentioned this pull request Aug 24, 2023
[🐛 Bug]: Import giving error ERROR @wdio/runner: Error: Cannot find package with paths defined in tsconfig.json webdriverio/webdriverio#10857
Closed
3 tasks

parzhitsky reviewed Sep 11, 2023
View reviewed changes

Naddiseo mentioned this pull request Oct 3, 2023
ESM + path mapping #2068
Open

jaredperreault-okta mentioned this pull request Nov 14, 2023
fix: select authenticator auto selects when methodType differs okta/okta-auth-js#1470
Draft

geigerzaehler closed this Apr 11, 2025

Uh oh!

Uh oh!

Conversation

geigerzaehler commented Dec 29, 2021• edited by cspotcode Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cspotcode commented Dec 29, 2021

Uh oh!

geigerzaehler commented Dec 29, 2021

Uh oh!

cspotcode commented Dec 29, 2021

Uh oh!

codecovBot commented Dec 29, 2021• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

cspotcode left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

cspotcodeJan 24, 2022

Choose a reason for hiding this comment

Uh oh!

geigerzaehlerJan 24, 2022

Choose a reason for hiding this comment

Uh oh!

cspotcodeJan 25, 2022

Choose a reason for hiding this comment

Uh oh!

geigerzaehlerJan 25, 2022

Choose a reason for hiding this comment

Uh oh!

cspotcodeJan 25, 2022

Choose a reason for hiding this comment

Uh oh!

cspotcodeJan 25, 2022• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

geigerzaehler commented Jan 24, 2022

Uh oh!

cspotcode commented Jan 24, 2022 via email

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

cspotcode commented Jan 25, 2022

Uh oh!

cspotcode commented Jan 25, 2022• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

geigerzaehler commented Jan 25, 2022

Uh oh!

cspotcode commented Jan 25, 2022• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kuksik commented Mar 16, 2023

Uh oh!

mprinc commented Mar 16, 2023

Uh oh!

0x80 commented Mar 17, 2023

Uh oh!

trim21 commented Mar 17, 2023• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Naddiseo commented Mar 18, 2023

Uh oh!

gudh commented Mar 21, 2023

Uh oh!

troywweber7 commented Mar 21, 2023

Uh oh!

trim21 commented Mar 21, 2023

geigerzaehler commented Dec 29, 2021•
edited by cspotcode
Loading

codecovBot commented Dec 29, 2021•
edited
Loading

cspotcodeJan 25, 2022•
edited
Loading

cspotcode commented Jan 25, 2022•
edited
Loading

cspotcode commented Jan 25, 2022•
edited
Loading

trim21 commented Mar 17, 2023•
edited
Loading

mprinc commented Mar 22, 2023•
edited
Loading

loynoir commented Apr 24, 2023•
edited
Loading