lib,src: split up big builtin scripts

[bnoordhuis]

Reduces the body of parsed source code from 210 kb to 80 kb for a null
program (node -e 0) and from 255 kb to 190 kb for a one-liner that
prints some output to the console.

It doesn't seem to have much impact on the size of the generated code
(on my system I see an 8% reduction with --nolazy and 3% without) but
hopefully it speeds up startup on slow systems because there is now
less source code to parse.

Do not merge yet; this is an experiment and for soliciting feedback.

See #2138.

CI: https://ci.nodejs.org/job/node-test-pull-request/255/

[cjihrig]

Looks like this PR is affected by the issue with 617ee32.

[ChALkeR]

@cjihrig All the PRs that were published (or rebased) in the last few days need to be rebased now.

[bnoordhuis]

Ah, it wasn't like that when I opened the PR this morning. Anyway, rebased and new CI started: https://ci.nodejs.org/job/node-test-pull-request/256/

[thefourtheye]

EventEmitter at the end could be removed along with this change I guess.

[bnoordhuis]

make test-npm is failing rather spectacularly. Looks like it hasn't updated to a version of graceful-fs that doesn't monkey-patch fs? That change landed in graceful-fs in June IIRC. /cc @isaacs@zkat

[thefourtheye]

@bnoordhuis npm 2.14.2 has graceful-fs ~4.1.2 as its dependency. That has the fix which you proposed.

[Fishrock123]

@bnoordhuis could you send me some make test-npm output?

[bnoordhuis]

@Fishrock123 E.g. here:

Error: Cannot find module 'internal/errno_exception'
at Function.Module._resolveFilename (module.js:339:15)
at Function.Module._load (module.js:289:25)
at Module.require (module.js:368:17)
at require (module.js:387:17)
at evalmachine.<anonymous>:36:24
at Object.<anonymous> (/home/iojs/build/workspace/node-test-commit-linux/nodes/fedora22/deps/npm/node_modules/node-gyp/node_modules/graceful-fs/fs.js:11:1)
at Module._compile (module.js:437:26)
at Object.Module._extensions..js (module.js:455:10)
at Module.load (module.js:358:32)
at Function.Module._load (module.js:313:12)

deps/npm/node_modules/node-gyp/node_modules/graceful-fs/fs.js is still monkey-patching things.

[bnoordhuis]

I guess the problem is that npm and node-gyp depend on different versions of graceful-fs....

[bnoordhuis]

@thefourtheye Thanks for filing that node-gyp pull request but I don't think that's going to be enough, there's a whole bunch of npm dependencies that depend on outdated versions of graceful-fs:

$ find deps/npm/ -name package.json | xargs grep -E '"graceful-fs"\s*:' | grep -v '4\.1\.2'
deps/npm/node_modules/fs-vacuum/package.json: "graceful-fs": "^3.0.2",
deps/npm/node_modules/node-gyp/node_modules/tar/package.json: "graceful-fs": "^3.0.2",
deps/npm/node_modules/node-gyp/package.json: "graceful-fs": "3",
deps/npm/node_modules/sha/package.json: "graceful-fs": "2 || 3",
deps/npm/node_modules/sha/package.json: "graceful-fs": "2 || 3",
deps/npm/node_modules/fs-write-stream-atomic/package.json: "graceful-fs": "^3.0.2"
deps/npm/node_modules/read-installed/node_modules/readdir-scoped-modules/package.json: "graceful-fs": "^3.0.4",
deps/npm/node_modules/read-installed/package.json: "graceful-fs": "2 || 3"
deps/npm/node_modules/read-installed/package.json: "graceful-fs": "2 || 3"
deps/npm/node_modules/read-installed/test/fixtures/package.json: "graceful-fs": "~2"
deps/npm/node_modules/fstream/package.json: "graceful-fs": "3",
deps/npm/node_modules/tar/package.json: "graceful-fs": "^3.0.2",
deps/npm/node_modules/write-file-atomic/package.json: "graceful-fs": "^3.0.2",
deps/npm/node_modules/read-package-json/package.json: "graceful-fs": "2 || 3"
deps/npm/node_modules/read-package-json/package.json: "graceful-fs": "2 || 3"
deps/npm/node_modules/cmd-shim/package.json: "graceful-fs": ">3.0.1 <4.0.0-0",
deps/npm/node_modules/npm-registry-client/package.json: "graceful-fs": "^3.0.0",

[thefourtheye]

@bnoordhuis I'll submit a patch to those packages now. But then even if they upgrade, we can land this only after npm picks up the upgraded dependencies, right?

[bnoordhuis]

I'll submit a patch to those packages now. But then even if they upgrade, we can land this only after npm picks up the upgraded dependencies, right?

Yes, that's right.

[othiym23]

Just so everyone is clear, even were we to land all those PRs today (which we're not going to do, because this is the middle of a long holiday weekend in the US, and also those changes need to go through our release process, and probably, y,know, be tested), it's at least two Thursdays until the patched version of npm would be ready for downstream use. This strongly argues against including this PR in Node 4.0.0, unless everyone is cool with waiting that long.

Big thanks to @thefourtheye for taking the time to put those PRs together.

[thefourtheye]

This strongly argues against including this PR in Node 4.0.0, unless everyone is cool with waiting that long.

As mentioned in the PR,

Do not merge yet; this is an experiment and for soliciting feedback.

we can assume that it may not be targeted for 4.0.0 (@bnoordhuis please correct if I am wrong).

Big thanks to @thefourtheye for taking the time to put those PRs together.

You are welcome 😊

[bnoordhuis]

we can assume that it may not be targeted for 4.0.0 (@bnoordhuis please correct if I am wrong)

Nope, that's right; I wasn't planning on landing this in v4.x.

[chrisdickinson]

This seems like it should be unnecessary, since _stream_duplex's export is hoisted.

[Fishrock123]

@Trott You know you can make a new label if you need to right? :)

[othiym23]

In npm@2.14.4, npm has landed all of the PRs that @thefourtheye made with graceful-fs updates for packages over which we have control. The only remaining holdouts are sha and cmd-shim, which @ForbesLindesay maintains, and in the case of sha, I'm trying to work out an issue related to npm's need to continue to support Node 0.8. In a release or two of npm, it should be clean enough to support this patch landing.

[Fishrock123]

@bnoordhuis npm now ships with all those patches by default, so the above concerns seem to be met now.

[Fishrock123]

@bnoordhuis is this still a thing you'd like to do?

[bnoordhuis]

Closing, stale. Now that it's possible to add our own code to the snapshot, that's probably a better solution anyway.