This content originally appeared on Level Up Coding - Medium and was authored by Max Boguslavskiy
Mistakes. They are everywhere
The reason for writing the article was errors like:
Error [ERR_REQUIRE_ESM]: require() of ES Module …/index.js from …/file.ts not supported.
Instead change the require of index.js in …/file.ts to a dynamic import() which is available in all CommonJS modules.
This happens because you are trying to connect an ES module from the CommonJS module. Alas, in recent years, the number of such errors when updating packages and switching to new releases has been growing inexorably. Why do such errors occur? Let’s understand the nature of this phenomenon. To do this, you need to plunge into the history of JS.
JS
JS was born in 1995. Was written in a few months. It was initially loaded on the page via a tag script. All JS loaded on the page fell into the global scope. Because of this, when loading several external scripts, there were collisions, variable overloads, and other side effects. There were no imports and exports as such. Although keywords have been reserved in the language.
Node.js and CommonJS
By 2009, the technology had become so popular that there was a request why not use JS on the backend. Thus, Node.js appeared. Despite the similar syntax, it was arranged a little differently. Since, unlike the browser, run-time Node lacks html and the script tag, some other mechanism for loading JS was required to load the module. Such a mechanism was require. It’s worth noting that require is not a new language property. This mechanism was built into run-time.
Require.js parsed a module, built a syntax tree in it, found connections to other modules in it, chose the first one, connected it to the syntax tree, then parsed the next module. This process continued until the entire tree was built. Until this moment, the module was not considered loaded and it was not possible to analyze it. After parsing and building the tree, require exposed the module.exports object and provided an API to work with it, hiding the implementation details.
It is also worth noting that, unlike the browser, node.js could not search by URL, but searched by the path to the file, if the path was not provided, then the module was searched in the local node_module folder, if it was empty, then it was searched in the folder with the same name, but a level higher, and so on, until node.js reaches the root folder. This convention has been called CommonJS. To load external libraries, npm was created, which looked for a package in an external shared repository, or in those specified by the user, and pulled the necessary external libraries into the corresponding node_modules folders.
Require.js and AMD, etc
Some time after the release of Node.js and the appearance of the require mechanism, dudes from the world of browsers appreciated this approach and took steps to drag this approach to themselves. Porting 1in1 CommonJS to the browser is not possible. The first reason: in the work of JS within Node.js and the browser, is that the browser does not have access to the file system and operates on the url, while node.js operates on the file system. Reason two: run-time can add any variable to the script. Module.exports is not available in the browser. Therefore, in 2010, the RequireJS library was created. Which offered an adapted version of CommonJS for a browser called AMD. It is important to note that AMD and CommonJS did not change the language itself.
ES6 and ESM:
In 2015, ES6 appeared, which included many of the changes that the community asked for. Among other things, it described import and export. In 2017, these standards were implemented experimentally in version 8 of Node.js . They were ESM titles. The main difference between ESM and CommonJS was dynamic import, which allowed loading individual parts of the module as they were used. At the same time, the possibility of static import was preserved using the await import construct, which, in terms of implementation, resembles the implementation of require in CommonJS
Chronology of events:
Chronologically described events can be represented as follows.
1995:
- Browser (Script)
2009:
- Node.js
- CommonJS (Node.JS): Require | module.exports
2010:
- AMD (Require.JS | Browser): Require | module.exports | Define
2015:
- ES6 Standard: import | Export
2017:
- ESM (Node.js 8.0.0 2017 Experimental | Node.js 12 2018): import | Export
Typescript
Let’s add one more level of complexity. Historically, Typescript has been transpiled to CommonJS and has been a strict syntactic superset of JS. That is, it supported all the properties of the JS language, but at the same time added its own syntactic constructions to the language. It is important to note that natively browser and node do not support Typescript. To work with TS, you must first translate TS into a language understandable by node and browser. The second important point is that ESM support became available from April 8, 2022 for nodeResolution NodeNext.
Up to this point, TS could not:
- Create correct imports with extensions;
- Parse exports and imports sections in package.json; Documentation link: https://www.typescriptlang.org/docs/handbook/esm-node.html#packagejson-exports-imports-and-self-referencing
It is also important to note that jest still does not fully support ESM.
Why the transition between CommonJS and ESM can be painful
- It is very easy to import CommonJS packages into ESM. At the same time, reverse porting can be associated with difficulties.
- Node.js was originally oriented towards CommonJS
- There are a huge number of packages written in CommonJS. For example, all tooling (ts-node, typescript, etc), etc.
- CommonJS made it possible to implement modularity in packages in different ways. As a consequence, different tools have different implementations of modularity.
- CommonJS allowed external processes to modify node, for example, ts-node could modify node.js at runtime. Now this is impossible.
- ESM imposes a certain kind of modularity, so a smooth transition is not possible.
Node.js recommendations for fresh packages:
It is important to note that for new packages, ESM is the official standard format for packages to reuse code. Modules must be defined using import and export statements. Link to official documentation: https://nodejs.org/api/esm.html Thus, we come to the conclusion that the transition is inevitable and unpleasant. What changes in Node.js make this transition more favorable at the moment? Experimental support for running ESM on the fly appeared in another 14 versions of Node.js, but only in version 18 did it become stable. In addition to this, the latest version of ts-node introduced on-the-fly transpilation.
What have we done?
Created a repository https://src.aligntech.com/projects/CCW/repos/ipl-shared/browse We created an ESM package in it on the latest JS stack: NodeJS 18. Checked launch. All OK. We tried to drag it into the infrastructure: Jest, Eslint. Added unit tests. Everything is working. We converted the package to Typescript and tried to run it through TS-node. Everything is working. There was a subtle point related to the fact that in the 18th version in the early time it is impossible to influence the work of its internal structure, so we had to write the following wrapper to run Typescript:
But all this machinery worked only within the framework of a mono-repository. Why? Because for external packages, it was necessary to prepare a project either in the form of CommonJS or in the form of ESM.
Of course, we could solve all this through static imports const X = await import (‘Y’); but it’s not productive.
To do this, we set up the transpiler to generate esm and commonJS modules.
The following structure came out:
dist/
- esm/
- commonJS/
src/
- Package.json (where the module type was specified)
In our situation, we can import a package from three points:
- from a neighbouring workspace via the ./src/index.ts folder
2. when we import the transpiled code from another repository:
a. From ESM
b. From CommonJS
The scheme is working, but alas, it is not enough for full-fledged work. I will illustrate this thesis with an example:
Of course, you can use createRequire and await import, but I would like to connect in the traditional way: Import { something } from “@aligntech-ipl/translations-core”; const { something } = require(“@aligntech-ipl/translations-core”);
Why is it better not to do this?
- Getting into the implementation details
- Human factor as we add potential bugs in case the package structure changes
Conditional exports
To simplify imports in code, we use two constructs in package.json:
- PublishConfig — this field contains an object with properties that will be replaced with properties with the same names after the package is built for publication
It turns out like this:
I’ll add some details to the screenshot:
- “.” is subpaths export. Thanks to him, we can implement different export conditions along the way
- “types” — allow you to specify different types depending on the conditions
- “default” is the generic entry point
What else did we have to do?
- Even in the case of using conditional exports, we will have to change the js extension to mjs (cjs), either in ESM or in the CommonJS package.
- By default js files will be interpreted by node.js depending on the value of the type field in package.json.
- Additionally, we will need to replace the non-dual packages with their respective counterparts for the CommonJS and ESM versions of the package. For example, in ESM use lodash-es, and in commonJS lodash.
Summary:
- We managed to set up ts-node launch
- Transpilation on the fly turned out:
- Acceleration of test launch compared to a similar launch by 6–10 times.
- It turned out to set up dual-package (commonJS | esm)
Links:
- https://nodejs.org/docs/latest-v18.x/api/packages.html
- https://medium.com/sungthecoder/javascript-module-module-loader-module-bundler-es6-module-confused-yet-6343510e7bde
- https://nodejs.org/api/packages.html#dual-commonjses-module-packages
- https://nodejs.org/api/packages.html#conditional-exports
- https://www.typescriptlang.org/docs/handbook/release-notes/typescript-4-7.html#packagejson-exports-imports-and-self-referencing
- https://2ality.com/2019/10/hybrid-npm-packages.html
- https://www.typescriptlang.org/docs/handbook/esm-node.html#packagejson-exports-imports-and-self-referencing
Level Up Coding
Thanks for being a part of our community! Before you go:
- 👏 Clap for the story and follow the author 👉
- 📰 View more content in the Level Up Coding publication
- 🔔 Follow us: Twitter | LinkedIn | Newsletter
🚀👉 Placing developers like you at top startups and tech companies
Transpiling Typescript into double packages (CommonJS + ESM) was originally published in Level Up Coding on Medium, where people are continuing the conversation by highlighting and responding to this story.
This content originally appeared on Level Up Coding - Medium and was authored by Max Boguslavskiy
Max Boguslavskiy | Sciencx (2022-09-22T15:36:29+00:00) Transpiling Typescript into double packages (CommonJS + ESM). Retrieved from https://www.scien.cx/2022/09/22/transpiling-typescript-into-double-packages-commonjs-esm/
Please log in to upload a file.
There are no updates yet.
Click the Upload button above to add an update.