file-processor

Node.js utility for mass-processing files in parallel.

Usage

Files are handled in parallel running workers. In order to use the library you need 2 modules.

Worker module

Must export single function, accepting fileName and callback. This function must process the file and call the callback when it is done. Function can be asynchronous.

module.exports=function(fileName,callback){constresult=doExpensiveProcessing(fileName);callback(null,result);};

Main module

Must use FileProcessor class and provide a it one or more glob patterns and path to worker module. Each file, matching the pattern will be processed by worker module.

constFileProcessor=require('@researchgate/file-processor');constprocessor=newFileProcessor(['path/to/some/files/*.txt','some/other/path/*.js'],require.resolve('./worker'));processor.on('processed',(fileName,result)=>{console.log(`result for ${fileName}: ${result}`);});

FileProcessor instace emits following events:

• queued - file is queued for processing.

  Arguments:

  • fileName

• processed - file is successfully processed by worker.

  Arguments:

  • fileName
  • result - the result, returned by worker module

• error - worker failed to process the file

  Arguments:

  • error

• allQueued - all files, matching the pattern are queued for processing.

  Arguments:

  • stats - object with the following field

    • queuedCount - total number of queued files
    • processedCount - total number of files which are already processed

• end - all files are processed.