Streaming request body parsing

[WyriHaximus]

This PR is the follow up for #13. It started out to make multipart streaming but ended up making all bodies streaming.

The parsers emit a post event with the key and value of a post variable and file on uploaded files found in the request. On the request object getFiles is gone due to the streaming nature of the parsers. getPost is still there but it won't have everything until the entire request has been parsed.

Todo:

• Normal body streaming
• Multipart body streaming
• Form URL Encoded body streaming
• File object for files in the request
• Make sure all parsers behave the same
• Make the form parsers optional
• Add buffered sink like helpers for request post fields and files
• Update readme with an example

[nazar-pc]

I saw gPsr, are request/response objects PSR-7 compatible now?

[WyriHaximus]

Not yet, those are also in the works though. But this is a step in that direction. The gPsr you're referring to is only used to parse request headers.

[clue]

The parsers emit a post event with the key and value of a post variable and file on uploaded files found in the request.

The HTTP message body can potentially have any size (even single fields can be huge), do we really want to store this automatically?

As an alternative, nodejs only exposes the body stream and leaves it up to the consumer to pass this stream to the correct parser (for example http://stackoverflow.com/questions/4295782/how-do-you-extract-post-data-in-node-js).

[WyriHaximus]

The HTTP message body can potentially have any size (even single fields can be huge), do we really want to store this automatically?

No I want to go nodes way where a string or stream is emitted? Those can just be gathered using the buffered sink. Not sure yet about the stream part. It would result in less overhead for small fields but more code on the implementing side.

[clue]

My personal vote here would be small, independent, composable parts instead of convenience built in.

Composable parts enable convenience on a higher level, such as this draft API:

$http->on('request', function (Request$request, Response$response) use ($formParser) {
$formParser->parseDeferredStream($request)->then(function ($fields) use ($response) {
$response->end('hello ' . $field['name']);
}, function ($error) use ($response) {
$response->writeHead(400);
});
});

Once we look into PSR-7 support, we could probably build a convenient middleware around this concept in order to make this available to each request handler.

[WyriHaximus]

That looks good, I'm assuming that is just listening to form events emitted from $request. Or are you suggesting that $request just emits data and the form parser listens into that handling the parsing. Do like that even more 👍 . Should be fairly easy to do as well, I'll make sure both parsers behave the before refactoring into that

[clue]

suggesting that $request just emits data and the form parser listens into that handling the parsing. Do like that even more 👍

This exactly 👍

Should be fairly easy to do as well […]

Yeah, I too suppose this should be easier than auto-wiring all parsers 👍

[WyriHaximus]

Thanks for clarifying that @clue, working on that refactor right now 👍

[WyriHaximus]

@clue what I'm working out now would have this API approximately:

$http->on('request', function (Request$request, Response$response) {
FormParserFactory::create($request)->deferredStream()->then(function ($fields) use ($response) {
$response->end('hello ' . $field['name']);
}, function ($error) use ($response) {
$response->writeHead(400);
});
});

Currently decoupling all that auto-wiring code

[WyriHaximus]

The last few commits remove the right wiring between the form parsers and request parser. They also add a form parser factory. Next step is adding methods like deferredStream to them.

[WyriHaximus]

Yeah Github won't even let me merge it from the site 😝 . This PR is my top ReactPHP priority at the moment, would prefer to get reactphp/http-client#58 in A.S.A.P. so I can fully focus on this right here.

I'll discus with @clue how exactly we're going to cut it up, but since most of the discussion has already taken place here we can move relatively quick

[mu578]

Hello gents,

if I may, some remarks about the adopted design :

Input:
-- Any request may have query parameters HEAD, PUT, POST, DELETE and GET... (1)
-- It will make sense to me to have an onEvent taking the HTTP verb as filter not "request"
-- You are inlining the buffer of uploaded files (Apply to BODY and PUT) (2)
-- What happens if the Content-Length header is missing? or set to an offset out of range? bad things.

• (1) Thus, you are forcing the user to juggle between one to another object representation to get everything.
• (2) which bars you to handle large file ; also a security threat. In my opinion, it would be better to redirect the raw-input to a FIFO file session then carefully extract and parse per chunk from there ; everything in this safe space, even if the FS is involved ; the full in memory solution is an utopia.

Suggestions :
you should only pre-parse in memory the part-offsets from the FIFO file space ; then give the result totally parsed on demand ; files should be carefully extracted per chunk-unit-buffer (let's say 1024 or 2048) to their destination if the user request it, not inlined in memory.

Output:
Same concepts apply to the response object (streamable and not buffered) i.e handling file-downloads, ranged-streams or large body-response without starving the machine.

Regards.

[WyriHaximus]

Been spending some time splitting this PR up into smaller ones, that resulted in the following pull requests:
#62: File object
#69: Streaming body parsers foundation
#70: Content length buffered sink
#71: Parser: urlencoded (depends on #69 and #70)
#72: Parser: multipart (depends on #69 and #62)
#73: Streaming parser bufferedsink (depends on #69 and #62)

-----------

#62: Uploaded File object that doesn't make sense on it's own but it provides something needed by #72 and #73.
#69: Provides the base factory and NoBody and RawBody parsers to get started.
#70: Mainly a buffered sink like the one from react/stream but cuts off at a set length or when the stream ends.
#71: Urlencoded parser that requires the base (#69) and the sink (#70).
#72: Multipart parser that requires the file object (#62) and parser factory (#69), will be updated once the factory is in and added to the factory.
#73: Buffered sink that takes a streaming parser and will resolve the promise when the parser is done parsing the incoming request.

-----------

Order of merging (all PR's will be squashed on merge keeping the history clear): #69, #62, #70, #73, #71, #72

-----------

Will go over @moe123's comment carefully and see where adjustment is necessary. One of the things I've already done due to @moe123's is make all the parsers cancelable.

[andig]

Checking back with how close we are to merging 😮. From what I understand PRs should be fine up to #62 and #69, open comment on #70 and open tasks on #71, #72, #73.

Anything userland could do to get closer to merging apart from friendly nagging?

[WyriHaximus]

@andig yes #62 and #69 are done as far as I'm concerned, unless @jsor or @clue thinks otherwise. And I like to get them in soon, I'll ping them on IRC tonight and see how they look at it.

Once that is in, start working on completing the other PR's. One of the issues I came across is that the urlencoded parser (#71) is going to be interesting as I can't use build in PHP functions to do the parsing without buffering.

[andig]

One of the issues I came across is that the urlencoded parser (#71) is going to be interesting as I can't use build in PHP functions to do the parsing without buffering.

Can't we assume- for the time being- that buffering for this case is ok, i.e. you either have a POST blob which doesn't need decoding or you have urlencoded data that will most likely not exceed a certain size?

[WyriHaximus]

One of the issues I came across is that the urlencoded parser (#71) is going to be interesting as I can't use build in PHP functions to do the parsing without buffering.

Can't we assume- for the time being- that buffering for this case is ok, i.e. you either have a POST blob which doesn't need decoding or you have urlencoded data that will most likely not exceed a certain size?

We could do that, I've set up several milestones that allows us to release this in parts. For example first getting the foundation out in 0.5.0, then multipart in 0.5.1, and then urlencoded parser in 0.5.2.

[bweston92]

Any update?

[WyriHaximus]

@bweston92 See this issue for our roadmap: #120

[kelunik]

This might result in a large string operation. Better use the previous strpos and check that against the $this->maxSize before. You might also want to use substr as you already have the position then.