python stdin buffering issue

[punit-dato]

Hey - I'm trying to the send/message functionality on python-shell to exchange data between a long-running python script and node, but I'm seeing that the messages don't get handled till I end the input stream. For example, the following sample code does not invoke the 'message' callback (or 'close' or 'error):

var pyshell = new PythonShell('python/helloworld.py', options);

pyshell.send({'hello': 'world'});

pyshell.on('message', function(message) {
 console.log("message - ", message);
}).on('close', function (result) {
 console.log("close - ", result);
}).on('error', function (err) {
 console.log("error")
});

The python is the same as your echo script.

import sys, json

# simple JSON echo script
for line in sys.stdin:
 print json.dumps(json.loads(line))

Now, if I close the connection after the above code, then I get all the messages:

pyshell.end(function (err) {
 if (err) throw err;
 console.log('finished');
});

This won't work for me as my python script maintains state and I don't want to re-run the script every time. I'd rather have a long running child process I could pass messages back and forth from. This seems like a bug. Any suggestions?

[extrabacon]

Stdout is being buffered by default, which is good for most apps. If your Node app does not receive messages, it means not enough data has been written.

You can explicitly flush the buffer to force the data to reach your Node app immediately.

See: http://stackoverflow.com/questions/10019456/usage-of-sys-stdout-flush-method

[punit-dato]

Thanks for the quick response. Unfortunately even after calling flush after every print in my python code this still isn't working.

import sys, json

# simple JSON echo script
for line in sys.stdin:
 print json.dumps(json.loads(line))
 sys.stdout.flush()

I also tried the -u option, and the Unbuffered wrapper (which just does a flush as well) as per suggestions here: http://stackoverflow.com/questions/107705/disable-output-buffering

None of these seem to work. Any other thoughts?

[extrabacon]

Both stdin and stdout are buffered. This code writes to stdout what comes from stdin, so now you may want to flush on both sides.

[punit-dato]

I can't find anything on flushing stdin. thoughts?

[extrabacon]

True, writable streams in node cannot be flushed that simply. Maybe this would help: http://stackoverflow.com/questions/18849112/stream-child-process-output-in-flowing-mode

Let me know if you can make it work.

[punit-dato]

I spent way too long on this. Apparently "for line in stdin:" blocks until it sees an EOF in Python 2 (I haven't verified on Python 3).

The following code seems to work for me and will also terminate itself when it sees the EOF character. Also included is a flush after every print.

line=' '
while line:
 line = sys.stdin.readline()
 print line
 sys.stdout.flush()

[extrabacon]

I see, thank you for sharing your experience.

[JonathanLehner]

this should be way more prominent in the documentation. I spent 4 hours wondering why the send is only received after end() and if this is by design

[JonathanLehner]

however line='' didn't work for me.
https://stackoverflow.com/questions/31696444/iterating-over-standard-in-blocks-until-eof-is-read should be
-u Force stdin, stdout and stderr to be totally unbuffered. On systems where it matters, also put stdin, stdout and stderr in binary mode. Note that there is internal buffering in xread‐ lines(), readlines() and file-object iterators ("for line in sys.stdin") which is not influenced by this option. To work around this, you will want to use "sys.stdin.readline()" inside a "while 1:" loop.

[Almenon]

@JonathanLehner what do you mean by

however line='' didn't work for me

-u is already mentioned in the documentation I believe. You can submit a pull request if you want to change it.