src: don't overwrite environment from .env file

[philnash]

This commit adds a check to see if an environment variable that is found in the .env file is already set in the environment. If it is, then the value from the .env file is not used.

The docs from #48890 say:

If the same variable is defined in the environment and in the file, the value from the environment takes precedence.

In my testing, I found that not to be the case, so I went to look for a solution.

Caveats. I am not a C++ programmer and have never tried to read, understand or contribute back to a C++ code base before. This is a naive solution. Please feel free to criticize and correct my C++.

I know the v20.6.0 release is very close, but as dotenv support is a notable feature, I think this should join the release so that we don't push a feature that contradicts its own documentation.

[ghost]

This seems to contradict what the average expectation would be when using the --env-file option. If I ran something with the option and then logged different values from what was in my .env file, I would be extremely confused or think something is broken. A very common case of using an environment file would be to overwrite/ignore any existing variables with the same name.

On a side note, can you link to where you found that quote? I couldn't find that anywhere in the linked thread. The closest I saw was an incomplete action item that said needed more discussion.

[cjihrig]

On a side note, can you link to where you found that quote?

https://github.com/nodejs/node/pull/48890/files#diff-7349c17bce4d97526eefcb6f8291d617e7edb92ed9927fd3b9264e8014fc4e7cR998-R999

I agree with @tacozilla here. I think we should change the docs, not the code. More locally defined config should take precedence over less locally defined config.

[philnash]

All the existing implementations of dotenv behave this way. You can see that the Node.js dotenv library has an override option that is false by default. The Ruby library has an overload method, but won't do by it by default. The Python dotenv behaves this way too.

I think this feature should follow the prior implementations.

Though we should also consider a way to override the environment for a future release too.

[anonrig]

I agree with @philnash. I didn't had any time to check the code but I think this is the correct path.

[cjihrig]

TIL. I agree we should follow what all of the other implementations do.

Side note: It seems like a design decision with obvious downsides. I'm curious if there are advantages I'm just missing to overriding by default other than the consistency aspect.

[ghost]

The significant difference to me is that libraries like dotenv are used/required/imported during runtime in application code. This feature is a CLI option you pass to the node executable. While I understand their end goals are similar (e.g. expose environment variables to your application), I'd argue the implementation matters.

If node was offering a module that you used similarly to dotenv, I'd be fine matching the existing behavior, but it feels wrong in the current implementation. This conflicts (albeit maybe not in the conventional way of thinking) with the precedent that CLI options always override environment variables:

Options from the command line take precedence over options passed through the NODE_OPTIONS environment variable.

On one hand, you could make the argument this isn't actually violating that because the option is always used even if the end result is a no-op, but on the other hand, I'd argue it does violate things in spirit. If I pass a CLI option, I expect it to have highest priority / override anything previously set, but it's entirely possible I'm just weird :P

There should definitely be some way to override existing environment variables, so I think everyone at least agrees on that in some form.

[philnash]

Honestly, I'm sort of torn here. I've never really liked the default way that any dotenv implementation works, I expect my local settings to override the environment. I've been caught out by that many times before and I don't know why that was the original decision that was taken.

However, as I've said above, it is the default for many, if not all, implementations of dotenv. If Node is to provide an implementation of dotenv, even a simple one compared to the options that are out there, then I think it should match what already exists. I think that if someone who is using dotenv-node with default settings decides to drop it in place of using --env-file then it should work the same for them.

And that's why I wrote this pull request. It goes against how I think the feature should work, but it follows how everyone who does know the feature already knows how it works.

(And yes, we should add override as an option somehow, but that can be done in a further PR.)

[nodejs-github-bot]

CI: https://ci.nodejs.org/job/node-test-pull-request/53693/

[rluvaton]

Important note:

I think we need to make an exception for NODE_OPTIONS:

# .env
NODE_OPTIONS='--max-old-space-size=3096'

$ NODE_OPTIONS='--env-file=./.env' node something.js

I think we should override the NODE_OPTIONS

[anonrig]

I think we should override the NODE_OPTIONS

NODE_OPTIONS has already an exception (defined at 769823e#diff-6c1386cbc6e4a11941649af31a0257d5c3e40b5f41223cdddf87ab2e984b705aR851), but you're right that we need to also make an exception in here, since we want to get the correct value for process.env.NODE_OPTIONS

[GeoffreyBooth]

I think we need to make an exception for NODE_OPTIONS:

I disagree, I think NODE_OPTIONS needs to behave like any other environment variable. It would be very confusing otherwise.

Yes we need to prevent an infinite loop, but that should be the only exceptional handling that we do.

NODE_OPTIONS has already an exception (defined at 769823e#diff-6c1386cbc6e4a11941649af31a0257d5c3e40b5f41223cdddf87ab2e984b705aR851)

My reading of this is that NODE_OPTIONS was behaving “correctly” (as in, the environment takes precedence over the file) because we carved out an exception to do exactly that for this one variable; but all other variables were behaving incorrectly. So it seems like we just need to make this exception the rule for all environment variables.

[GeoffreyBooth]

Honestly, I’m sort of torn here. I’ve never really liked the default way that any dotenv implementation works, I expect my local settings to override the environment. I’ve been caught out by that many times before and I don’t know why that was the original decision that was taken.

I understand the sentiment. I think you can achieve your desired behavior, at least in Bash and similar shells, via:

set -a;source .env;set +a; node app.js

Which is a bit clunky but gets the job done. Maybe this is why the default is the way it is in those other libraries.

[ghost]

I understand the sentiment. I think you can achieve your desired behavior, at least in Bash and similar shells, via:

set -a;source .env;set +a; node app.js

Which is a bit clunky but gets the job done. Maybe this is why the default is the way it is in those other libraries.

Isn't that argument somewhat self-defeating? Using the same argument, you could ask why node is implementing this as an option in the first place?

With that being said, I think framing this as should node match dotenv is missing the point. A better question is why dotenv implementations behave the way they do and then ask if that same logic holds true for this case.

From the Ruby implementation's docs:

By default, it won't overwrite existing environment variables as dotenv assumes the deployment environment has more knowledge about configuration than the application does.

I would argue that reasoning doesn't hold true for node's implementation. Passing a CLI option to the node executable would live pretty firmly in the context of the deployment environment, as such, it should take precedence.

At the end of the day, if node's priority is simply to blindly match dotenv as closely as possible, then this entire discussion is moot. I don't agree with blindly matching for the sake of matching, but I do understand that line of reasoning, but in that case I think we should be very clear and acknowledge it's not necessarily the actual best way to implement it.

On a slight tangent, I really don't understand when this default behavior would be useful in the average case. If a user is explicitly passing in an environment file, the chances of then wanting already set environment variables to take precedence would be very low if not directly contradict using the CLI option to begin with.

[GeoffreyBooth]

I would argue that reasoning doesn’t hold true for node’s implementation. Passing a CLI option to the node executable would live pretty firmly in the context of the deployment environment, as such, it should take precedence.

In the initial discussions for the feature we had discussed making loading .env happen automatically, as in node --env-file=.env would be the default behavior and wouldn’t need a flag. This would be a semver-major change, so we added the flag instead as a way to opt-in; the value for the flag, to load a file with a name other than .env, was a later addition. We might still want to auto-load .env files in a future major version of Node, in which case this “because it’s via CLI option” argument no longer holds.

That doesn’t mean we couldn’t have one precedence behavior for autoloaded files versus CLI-referenced files, but I’m assuming that we wouldn’t want such a divergence in behavior.

Isn’t that argument somewhat self-defeating? Using the same argument, you could ask why node is implementing this as an option in the first place?

Because there’s a difference between --env-file and set -a; source .env; set +a; node: the former gives variables already present in the environment precedence, the latter gives variables defined in the file precedence.

By default, it won’t overwrite existing environment variables as dotenv assumes the deployment environment has more knowledge about configuration than the application does.

What the Ruby docs describe is exactly how I’d use this feature myself. I can imagine having a .env file committed into the repo with contents like:

ROOT_URL=http://localhost:3000
DATABASE_USER=postgres
DATABASE_PASS=postgres

And these obviously wouldn’t be the values in the prod environment. I could have an npm start command of node --env-file=.env app.js and it can run everywhere without changes, because in production the ROOT_URL etc variables defined in that environment would override what’s in the file.

[GeoffreyBooth]

One more point of comparison: Bun behaves like dotenv and Ruby and so on:

echo'FOO=bar'> .env
echo'console.log(process.env.FOO)'> test.js
bun test.js # bar
FOO=baz bun test.js # baz

[nodejs-github-bot]

CI: https://ci.nodejs.org/job/node-test-pull-request/53800/

[philnash]

I think I'm getting the hang of analysing the Jenkins builds.

This fail seems to be the flaky IPv6 test again.

13:08:24 C:\workspace\node-compile-windows\node\test\cctest\test_inspector_socket.cc(323): error: Value of: (!expectation.read_expected)
13:08:24 Actual: true
13:08:24 Expected: false
13:08:24 13:08:24 [ FAILED ] InspectorSocketTest.HostIPv6NonRoutable (5000 ms)

[nodejs-github-bot]

CI: https://ci.nodejs.org/job/node-test-pull-request/53804/

[nodejs-github-bot]

CI: https://ci.nodejs.org/job/node-test-pull-request/53805/

[philnash]

Live view of me hoping this time the CI will pass.

I think one of the suites just straight failed to run this time.

[nodejs-github-bot]

CI: https://ci.nodejs.org/job/node-test-pull-request/53806/

[philnash]

[nodejs-github-bot]

Landed in 015e27b