fs.writeFileSync intermittently does not create file on disk when called in parallel

[cratter49]

• Node.js Version: v8.8.0
• OS: Linux 729a00fd3f64 4.4.74-boot2docker Update README for help #1 SMP Mon Jun 26 18:01:14 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
• Scope (install, code, runtime, meta, other?): runtime
• Module (and version) (if relevant): fs

I'm currently running a custom Jupyter notebook server within a Docker container. In this server there is code that executes a NodeJS script one or more times which launches an instance of headless chrome, extracts some information from the DOM of a page that instance of headless chrome points to, and then writes that information to a file in a temporary directory using fs.writeFileSync from the fs module. When this script is executed multiple times in parallel sometimes the fs.writeFileSync does not seem to actually write the file to disk even though there are no errors thrown from the script. For example, when the script is executed six times in parallel on my Docker system one NodeJS process out of the six I launched will fail to write the file to disk. Additionally, I know that the file is written to a unique path with each execution of the script and using the asynchronous version of writeFile does not work.

Also, it's important to note that the file does not exist before this call is made.

Any ideas why the file write my not occur?

[ORESoftware]

can you share you code?

fs.XSync won't actually run in parallel, one call should block another. Any sync call will block.

[cratter49]

@ORESoftware

There isn't a whole lot of code I can share but I start the NodeJS script using the nvm (node version manager) package like so in Python:

 generateReportCommand = ("%s use 8.8.1 %s '%s'" % (nBinary, pipes.quote(reportScriptPath), reporting_url)) 
subprocess.check_output(generateReportCommand, stderr=subprocess.STDOUT, shell=True).decode('utf-8')

where nBinary is n, reportScriptPath is the path of the NodeJS script, and reporting_url is the URL I point my headless chrome browser to.

Then in the NodeJS script at reportScriptPath I write the file to the file:

fileSystem.writeFileSync("jsonObjForLatex.json", rectanglesAndInfoTuple.jsonObjForLatex);

where fileSystem contains the fs module and rectanglesAndInfoTuple.jsonObjForLatex contains a string with JSON formatting.

All of this code is running multiple times in parallel on the same Docker container

If what you're saying about fs.writeFileSync methods blocking each other is occurring then wouldn't the writes still eventually happen once each call has gotten its turn?

I'm currently trying to flush the buffer cache using fsyncSync which successfully creates the files but the contents of the files seem to be completely erased so the files are blank.

[ORESoftware]

how many Node.js processes are running? Any fs.xSync call will block in a given Node.js process, but if there is more than one process, then the fs calls will run independently / in parallel.

[ORESoftware]

by the way it's probably not nvm that launches the node executable...nvm is only there to switch node.js versions in the shell.

[ORESoftware]

can you dump the actual string that generateReportCommand represents? Paste the actual command to this conversation. That variable seems to be the actual node command that you are using.

[cratter49]

@ORESoftware

There's always more than one Node.js process running. In my testing it usually ranges from 3-7 processes. Each Node.js process will call fs.writeFileSync 1 to X times.

The command is n use 8.8.1 /home/ubuntu/report.js https://localhost:8443/notebooks/prebuilt%20notebooks/0.%20Getting%20Started.ipynb

[ORESoftware]

Yeah you are using 'n' not nvm to manage node versions - the report.js file probably has a shebang/hashbang that makes it use the node executable. Your problem is likely cause because more than one node.js processes is writing to the same file at the same time. Without more info, one thing I can suggest is using uuids for unique filenames.

…

On Nov 29, 2017 5:13 PM, "Wise" ***@***.***> wrote: @ORESoftware <https://github.com/oresoftware> There's always more than one Node.js process running. In my testing it usually ranges from 3-7 processes. Each Node.js process will call fs.writeFileSync 1 to X times. The command is n use 8.8.1 /home/ubuntu/report.js https://dswb.soasta.com:8443/notebooks/prebuilt%20notebooks/0.%20Getting% 20Started.ipynb — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1002 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AKn56HYJRPJWPy0HK91TTxkrWfJ6rRc4ks5s7gEygaJpZM4QviCR> .

[cratter49]

@ORESoftware

Actually each process initially starts in the same directory but os.chdir changes the path to a specific temporary directory that is unique to each process. This means that the jsonObjForLatex.json file being written to is unique for each process.

[ORESoftware]

Can you verify that it's actually changing directories properly? what exactly is the issue where the files are "overwriting" each other, if they are actually writing to different directories?

[cratter49]

@ORESoftware

I printed out the current working directory using a utility function and every process was operating within its own unique path. The overwriting issue revolves around the use of fsync which flushes the buffer cache to the file descriptor passed in as a parameter. Using fsync makes sure that all of the files are created but does not succeed in writing data to each file.

[gireeshpunathil]

@cratter49 - is this still outstanding?

[gireeshpunathil]

closing due to inactivity

[douglasg14b]

I'm also noticing this behavior in unit tests. However, it seems to be a timing issue.

The file is created, but writeFileSync returns control BEFORE the actual write completes. But only in parallel scenarios. If I "sleep" await new Promise((resolve) => setTimeout(resolve, 100)); after writeFileSync then all tests pass, and access to the written file succeeds. But as I lower that sleep time the rate of failures gradually increases until it fails every test run.

This is weird behavior.