"Transparent" Interop is a confusing term

[bmeck]

We should try and find a better set of terms as this usage of the word "transparent" is proving to be different across multiple discussions / viewpoints. I'd suggest we split up different points and avoid using the word "transparent" when talking about things. How do other people feel about deprecating using the term "transparent" when talking about interoperability?

[MylesBorins]

+1 to deprecation

Do you have a suggestion for alternative?

Mixed Mode Interoperability?
Overloaded Import?

[hybrist]

How about just "interop"? In the context of the module discussion that seems sufficient. Potentially with "backward interop" and "forward interop" to describe the two directions?

[guybedford]

I don’t really understand where the confusion comes here - it seems an incredibly well defined term to me: Given a parent module written in a module format, the ability to load a child module of a different module format from the parent using the natural native module syntax of the parent module. The term interior covers the fact we are bridging module systems. The term transparent covers that we are using the existing module format machinery. What are the confusions we’re seeing here with this definition? Can we clear them up somehow?

…

On Mon, 25 Jun 2018 at 19:35, Myles Borins ***@***.***> wrote: +1 to deprecation Do you have a suggestion for alternative? Mixed Mode Interoperability? Overloaded Import? — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#138 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAkiytk1yNXD3DHbYR6ptlIqQC_GCwZEks5uAR9ZgaJpZM4U2ira> .

[GeoffreyBooth]

I interpret “transparent” to mean, “as a user, I don’t need to know the module type.”

I agree it can be confusing, but I think maybe it really is this simple? The core of it is that I can import a package without needing to know whether that package is ESM or CommonJS. The rest of transparent interop flows from that: I need to be able to export from a package such that this “blind” import works, etc. Ditto for other ways of importing and exporting modules, such as importing or running single files or strings.

One especially confusing area is the “drop the file extension” part of the .mjs proposal, where it lets me do e.g. import './index' and in the code it’s transparent whether I’m importing index.js or index.mjs, but it’s obviously not transparent in the filenames on disk.

This assumes that interoperability is limited to importing and exporting, and perhaps there are other forms of interop that I’m not considering, in which case maybe those other parts should get another term? And I’m sure there are other things that people have lumped under this rubric that perhaps should have other terms, but I think “transparent interop” is definable.

[bmeck]

I don't think we can have a single term.

The problem I think is that it covers too many things, we need to classify different aspects of interoperability. Per my thoughts on how to classify things:

• require interoperability : the ability to use require() to load ESM
• import interoperability : the ability to use import to load non-ESM (JSON/C++/CJS/etc.)

Part of the problem also is the usage of interoperability when paired with transparent also brings up things that are not about interoperability but specifics about how isomorphism could work inside of an interoperability story:

• module isomorphism : the ability for a module to present the same API if loaded via import or require()

  This isomorphism has several topics of discussion since there are limits and constraints to discuss for each. Only having a limited story for cross module system Isomorphism seems fine:

  • named export isomorphism : the ability for a non-ESM module to present the same API as an ESM containing arbitrarily named exports
  • platform isomorphism : the ability for a module to be loaded and run properly into all platforms without alteration of source code
  • timing isomorphism : the ability for a module to execute in the same timing regardless of being ESM or not

• specifier isomorphism : the ability for a specifier to resolve in the same manner in all platforms

We often treat the idea of isomorphism as all or nothing when using the word "transparent" which makes it hard to discuss transition paths and compatibility for things that only wish to have a path, even if not universal:

• forwards compatibility : the transition path/overlap created that allows isomorphism under all the types above. This does not necessarily apply to all forms of modules CJS/ESM/C++/etc.

In addition we are also using the term to discuss specific, often configurable, implementations of loading steps:

• resolution mechanisms : mechanisms used to locate a resource
• translation mechanisms : mechanisms used to parse, produce a module record for, and evaluate a resource. These must use facades for non-ESM.
  • this topic generally seems to only have real debate around how to decide on which translator to use
• loaders : configurable mechanisms used to alter the module records being loaded via hooks

[ljharb]

@GeoffreyBooth “as a user, I don’t need to know the module type" is why .mjs is transparent (because only the author needs to know that, not the consumer).

I think we should differentiate between, to name a few:

• consumers need to know the module type, or not
• authors need to know the module type, or not (i can't conceive of how this can be avoided, ofc)
• require('esm')
• import 'cjs'
  • named imports from CJS

All of these are facets of "transparent", but not its entirety.

[robpalme]

Some people think transparent interop is obvious/well-defined and others interpret it as some combination of independent attributes. I believe the simplicity of the term is unhelpful and is preventing us from communicating clearly in discussions. Maybe if we define the terms precisely we can better debate the problem space.

I've made an attempt at defining these terms loosely based on @bmeck and @ljharb 's breakdown using @SMotaal 's latest glossary document as a place to hold the definitions. I would invite people to comment/edit the document so we can iterate on the terms one place, as opposed to building ever-longer issues.

https://docs.google.com/document/d/150lG_qm7GdylZ08akrk8gdIUgZC8HtnTGAfgD9R5QEI/edit#heading=h.mn114lmmcllh

To be clear: my opinion is that the term transparent interop has now become so over-used for different things that we should (a) deprecate/suspend using the term until we can agree that it has one written definition, and (b) encourage the usage of more precise terms where possible.

I'm not attached to the new terms in the doc or their definitions so feel free to refactor vigorously until they make sense.

[bmeck]

@robpalme that is a good direction to take this I think. I personally would like it to be split up more, but think adding terms separately might be enough.

[zenparsing]

@robpalme Thanks. I think it's interesting that we are converging around the definitions of "require interop" and "import interop".

In your definitions, you specifically call out some challenges with each interop direction:

• For "require interop", the challenge of returning a Promise from require, because of async ESM modules.
  • Comment 1: I would say that returning a Promise from require is an interop failure. Async is infectious. If I have to wait on a dependency, then I become async and can no longer satisfy the same module API.
  • Comment 2: I think it's important to note that the only essential asynchronicity is the one introduced by top-level await. In that sense, I would say that dependency graphs containing top-level await are not "require interoperable", even through transpilers or transpiling loaders.
• For "import interop" the challenge of returning named imports.
  • Comment: Given that we don't want to mess with module evaluation order, I don't see a way to support named imports "out of the box". Perhaps it is enough for CJS authors to "opt-in" to named exports by publishing a "stub" ESM entry point which does require("./") and re-exports the correct names?

[robpalme]

@zenparsing thanks for the comments.

• For require interop, I agree and am inclined to think that the existence of dynamic import() negates the desire for a version of require() that returns promise-wrapped modules. So I have redefined require interop to mean returning the namespace directly as a value. I'll add the promise-wrapping variant back if anyone says they want it.
• For import interop, I think we still need both explicit qualifiers (import interop without named exports & import interop with named exports) because some desire exists even if we believe ultimately it is not feasible.

[bmeck]

@robpalme I'd add the Promise wrapping variant because I don't think that the ability for APIs interoperate deals with isomorphism of the APIs.

[robpalme]

@bmeck Sure. We can go for completeness if you think it's important. But is there anyone that would ever want a variant of require() that returns promise-wrapped ESM namespaces? I'd prefer succinct terms for things that we all agree only have one desired meaning in practice.

[bmeck]

@robpalme people shipping libraries to versions of Node that don't support ESM and want to create an API that works the same as in a version of Node that supports ESM. They could be compiling ESM down to Promises using something like babel, or they could be directly writing CJS that is isomorphic to people on versions of Node supporting ESM. Basically, anyone who wishes to continue support consumers using require() (most likely due to shipping to older versions of Node).

[zenparsing]

@bmeck I agree that "require interop" does not necessitate isomophism of the module's API. On the other hand, I'm not sure we should call Promise-returning require "interop".

Let's assume for the sake of argument that we have a version of "require interop" that returns promises for module namespace objects. If I have a module C which takes such a dependency, then C's exports will also need to be made asynchronous. And then any module that requires C will also need to be made asynchronous. Even if I keep the shape of my module the same, consumers of that module are forced into an async mode.

In such a case, package authors and consumers are forced to agree on the module "mode", which is something I think we are trying to avoid with interop.