node-lambda run output truncated when output data is large

[MkFoster]

I am using the node-lambda run command to execute a function that outputs a large amount of data (70KB+). The output is incomplete/truncated. Requests that output smaller amounts of data work fine.

It appears the process is exited before the output is completed. To work around this I removed the process.exit(0) call in the _runHandler function in /lib/main.js. It is probably there for a reason so this might not be a good fix but it worked for me.

`Lambda.prototype._runHandler = function (handler, event, runtime, context) {

var callback = function (err, result) {
if (err) {
console.log('Error: ' + err);
process.exit(-1);
} else {
console.log('Success:');
if (result) {
console.log(JSON.stringify(result));
}
//process.exit(0); (kills the above console.log output before it is done when there is a lot of data returned)
}
};`

[DeviaVir]

@MkFoster it sounds like you're returning the callback before you're done processing (this is the function that exits) and not like a bug in node-lambda, we might be able to help you debug your problem if you share relevant pieces of code?

[DeviaVir]

@MkFoster is this still an active problem?

[MkFoster]

I found a workaround for myself but I am wondering if this is intended behavior. Instead posting the whole project I am working on I made a simple lambda project that just outputs a lot of text to duplicate the behavior.

From the root of the project I run this (on CentOS 7 using node v4.3.2) :
node_modules/node-lambda/bin/node-lambda run --eventFile event.json --handler test.handler

The behavior I am expecting is to see the whole string output to my shell including "does this show up in the output?". I only see part of the string. If I redirect the output to a file "I.e. > mytext.txt" I get all of it. If I comment out the process.exit(0) line in _runHandler.js, I get all of it. This has been my workaround.

If you don't want to download the zip here is what the code looks like in a nutshell minus all the text/giant string:

// For development/testing purposes
exports.handler = function( event, context, callback ) {
    console.log( "Running index.handler" );
    console.log( "==================================");
    console.log( "event", event );
    console.log( "==================================");
    var textblock = `...70K of text...`;
    callback( null, textblock );
};

marktest.zip

[GitTom]

I found a workaround for myself but I am wondering if this is intended behavior.

This is something that @DeviaVir and I disagreed on a while ago when I raised it as an issue:
#110

I haven't changed my opinion - that AWS Lambda allows processing to continue after callback - and I continue to use AWS Lambda in the way that I described. To continue using node-lambda I made some changes and stopped updating. Here is my code, in case anyone else wants compatibility with this behavior. I switched to node6 recently so I updated it for that. It goes in "Lambda.prototype._runHandler" in main.js.

  context.callbackWaitsForEmptyEventLoop = true;

  var startTime = new Date();
  var timeout = Math.min(program.timeout, 300) * 1000; // convert the timeout into milliseconds

  var callback = function (err, result) {
    if (err) {
      // GitTom -  added this 'if' statement
      if (!context.callbackWaitsForEmptyEventLoop) {
        process.exit(-1);
      } // else allow process to continue
    } else {
      //console.log('Success:');
      if (result) {
      }
      // GitTom added this 'if' statement
      if (!context.callbackWaitsForEmptyEventLoop) {
        process.exit(0);
      } // else allow process to continue
    }
  };

  context.getRemainingTimeInMillis = function () {
    var currentTime = new Date();
    return timeout - (currentTime - startTime);
  };

  switch(program.runtime) {
    case "nodejs4.3":
      handler(event, context, callback);
    break;
    case "nodejs6.10":
      handler(event, context, callback);
    break;
    default:
      console.error("Runtime [" + program.runtime + "] is not supported.");
  }

[MkFoster]

For what it's worth, in my case I am interested in emulating the behavior of AWS as closely as possible. We use this so we can develop locally quickly and then we do a Jenkins build out to a real Lambda function in AWS to verify things behave as expected. This module has saved us a lot of time. Thank you for putting it out there.

In this case though it was working as expected in the real AWS environment but breaking in my sandbox which uses node-lambda. When I first hit this it took a bit to debug because it otherwise passed our tests and didn't occur to me that the size of output would matter. I humbly suggest if it is going to stay the way it is, it might be worth a note in the README.md that longer output may truncated because the process may terminate before the output is completed.

[abetomo]

With nodejs (v4.3.2) it was confirmed that it was cut in the middle.
It worked as expected with nodejs (v6.10.1).
(When I search, I found something similar to the phenomenon.)

Even without executing exit(0), it ends with exitCode=0.
Should I erase exit(0)?

[abetomo]

@MkFoster Is the problem still occurring now?

[MkFoster]

@abetomo We grabbed 0.11.3 today via npm and it fixed our problem outputting large amounts of text (70KB+). We can now use node-lambda without modification. Thanks!