transparent-or-not interop

[jdalton]

Seeing as #81 and #82 hinge on a call on transparent or non-transparent interop I think we should have an agenda item to discuss this (and possibly come to some decisions) at next week's meeting.

[weswigham]

More than just those - there's a bundle of other use cases in the use case doc that also hinge on it.

[ljharb]

To clarify, we need to make a decision on whether the defaults support transparent interop or not (as separate from “named exports from CJS”, which is a discussion worth having if we decide that transparent interop is a default)

[ljharb]

I’ll also request that we defer to the meeting after next - next week’s meeting conflicts with TC39, and a number of members will likely be unavailable.

[jdalton]

To clarify, we need to make a decision on whether the defaults support transparent interop or not

Correct.

I’ll also request that we defer to the meeting after next - next week’s meeting conflicts with TC39, and a number of members will likely be unavailable.

That's cool.

[GeoffreyBooth]

Rather than skip a meeting, where the next meeting would be June 6, could we just push next week’s meeting by one week? So the next meeting would be May 30.

[ljharb]

That sounds great to me; we’d have to get consensus from the group tho.

[jdalton]

@ljharb Geoffrey opened #92 to seek consensus to move next week's meeting to May 30th.

[GeoffreyBooth]

Could someone please explain exactly what the decision is that needs to be made? And the consequences of deciding one way or the other?

Or put another way: why wouldn’t we try to support transparent interoperability whenever we can and it doesn’t conflict with other priorities (spec compliance, performance) and via loaders for all other reasonable use cases?

And if the explanation could be written plainly enough to be understood by someone who has no familiarity with the inner workings of Node’s module system, that would be great. Asking for a friend.

[giltayar]

@GeoffreyBooth

Let's start with parse-goals: in JS (from ES2015 onward), a JS file is parsed and evaluated differently if it's a 'script' (this is all parsing pre-ESM) or a 'module' (i.e. an ES module, not a CJS). Note that NodeJS parses CJS modules as 'script', and this is unchangeable, as it would be a major breaking change.

Unfortunately, TC39 did not specify a way of saying what the parse goal of a file is. The browser world decided on two simple rule: 1. if the <script> has a type=module attribute, then it's a module, otherwise it's a script, and 2. if a file was import-ed by another module, then it is a module too.

This means that, in browsers, it is the "importer" that decides the parse goal of the file.

NodeJS decided differently: the parse goal of a file is decided by its extension: .js for script (i.e. a CJS module) and .mjs for ES modules. An interesting proposal to make it out of band in another way (i.e. a mode=esm in package.json is also floating around). All proposals of pre-parsing the file to determine what the parse goal is were rejected (e.g. "use module", or assuming it's a module if there's an export or import statement). Funnily enough, some babel contributors, it seem, agree that the file extension should be the way to determine parse-goals: https://twitter.com/loganfsmyth/status/996088384182411264.

What has this got to do with transparent interop? If we keep the transparent interop, then the runtime, when import-ing the file, needs to determine whether the parse goal is script or module, and thus needs the current mechanism, i.e. the mjs/cjs split. This necessarily precludes working the browser way, which is having the importer determine the parse goal.

You can't have it both ways. Either have transparent interop and force the mjs/js file extension split (which many people don't like), or don't have transparent interop, and have the ability for the importer to determine whether the file is a module or script.

That's one reason for people to dislike transparent interop. The other reason is this: why would we need transparent interop? If we had a meta.import.require that enables ESM to require CJS, we wouldn't really need transparent interop. Except that the babel world works this way. And if we want existing babel code to work transparently, we need transparent interop. This everyone agrees.

But, unfortunately, if we stick to the letter of the spec, we can only default import from a CJS (because determining named exports MUST occur before evaluating the code, and figuring out the named exports can only occur after code evaluation in CJS). And if we have only default imports from CJS, we're anyway not babel compatible.

So if we don't have babel compatibility, why would we need to have transparent interop?

Those are the two reasons I know of.

[GeoffreyBooth]

@giltayar Thanks for this, your explanation is clear and straightforward. So please correct me if I’m wrong, but it seems to me from your post that if we decide we want transparent interop, which it seems like is our default position unless the compromises that need to be made to achieve it are too unpalatable, then solutions must be found for two issues:

• How should Node know whether the file to be imported is ESM or CommonJS?

• How should Node handle named imports for CommonJS, such as import { shuffle } from 'underscore' as opposed to import _ from 'underscore'; const shuffle = _.shuffle?

With regard to how Node decides whether a file to be imported should be treated as a module or a script, I thought that options other than .mjs were back on the table. Besides the ones you mentioned (package.json, flag, etc.), I’m intrigued by the methods implemented in the npm proposal.

With regard to the named imports, it seems like a binary choice between either supporting named exports for CommonJS or maintaining spec compliance. But I read a lot of discussion among people trying to somehow achieve both; if someone can figure it out, is that a third option? I suppose we could always decide something like “if a spec-compliant way to do named imports for CommonJS can be found, sure, we do that; but if not . . .” and then decide on the backup option. The other backup option seems to be allowing it via a loader that can let users opt in to non-spec-compliant behavior (and by implication, building support for loaders that can allow such things).

And I guess the last decision to be made is what to do if we find a solution to one problem but not the other: ship one solution or neither.

Does this sum up all our decisions and options, or am I missing any?

[benjamingr]

or don't have transparent interop,

One thing that can be done is transparent interop with a loader rather than by default like Myles suggested.