Add a SAX parser

[nlohmann]

The library currently only supports DOM-like parsing. This does not scale when the input files are enormous (#927). I would like to discuss how a SAX-like parser could look like.

My proposal (heavily motivated by RapidJSON) is as follows:

struct SAX
{
    // a null value was read
    bool null();

    // a boolean value was read
    bool boolean(bool);

    // an integer number was read
    bool number_integer(number_integer_t);

    // an unsigned integer number was read
    bool number_unsigned(number_unsigned_t);

    // a floating-point number was read
    // the string parameter contains the raw number value
    bool number_float(number_float_t, const std::string&);

    // a string value was read
    bool string(const std::string&);

    // the beginning of an object was read
    // binary formats may report the number of elements
    bool start_object(std::size_t elements);

    // an object key was read
    bool key(const std::string&);

    // the end of an object was read
    bool end_object();

    // the beginning of an array was read
    // binary formats may report the number of elements
    bool start_array(std::size_t elements);

    // the end of an array was read
    bool end_array();

    // a binary value was read
    // examples are CBOR type 2 strings, MessagePack bin, and maybe UBJSON array<uint8t>
    bool binary(const std::vector<uint8_t>& vec);

    // a parse error occurred
    // the byte position and the last token are reported
    bool parse_error(int position, const std::string& last_token);
};

Some remarks:

• All functions return a bool: true if the parser should continue or false if the parser should stop processing the input.
• The proposal covers parsing of JSON, but also of CBOR, MessagePack, and UBJSON. Therefore, it contains extensions like array or object sizes as well as a binary type which do not occur when parsing JSON.
• The idea is that the user would implement the above struct (we need to discuss whether we make all functions virtual, define a default implementation, etc.) and pass it to new parse functions, e.g. void parse_json(SAX &sax); or void parse_ubjson(SAX &sax);.

What do you think?

[gregmarr]

The idea is that the user would implement the above struct (we need to discuss whether we make all functions virtual, define a default implementation, etc.) and pass it to new parse functions, e.g. void parse_json(SAX &sax); or void parse_ubjson(SAX &sax);.

For "define a default implementation", were you thinking that the library would include the struct declaration in the headers, and the implementation in a separate .cpp file that could be used or replaced as needed?

[nlohmann]

For "define a default implementation", were you thinking that the library would include the struct declaration in the headers, and the implementation in a separate .cpp file that could be used or replaced as needed?

I think I would rather say the library should define an interface that client code needs to implement.

[theodelrieu]

Maybe the "default implementation" could be a template argument of some sort, like adl_serializer currently is?

I have no experience with SAX at all but it could be a good occasion to continue the discussion which began in #605 and #774 with @vinniefalco.

[nlohmann]

I think "default implementation" makes little sense here, because it is an extension point for user code after all. Sorry for the confusion.

[theodelrieu]

So you could not use the SAX version without reimplementing the SAX struct?

[gregmarr]

I think pure virtuals would provide the most flexibility. That way a code base can have multiple objects that implement the SAX interface.

[nlohmann]

@theodelrieu Yes, the user code needs to implement the SAX interface and pass the object to the SAX parser.

[nlohmann]

I think an eof() event would be helpful to detect situations like false 10 which currently would just trigger boolean(false) and end. With an eof() event, one can detect that there is still some non-whitespace input.

[NickeZ]

I'm not sure where to write this, but I saw #601 #862 and thought that I would point out that there is a draft for typed arrays in cbor which might be interesting to implement. https://github.com/cbor-wg/array-tags

[dvhwgumby]

I am not a fan of this SAX struct approach (but I am not submitting a pull request so this is only my opinion). As a side point, in terms of C++ design I agree with gregmarr that pure virtuals would be a better approach.

I was using rapidjson and switched to this package because it's easier and more intuitive. (I have had to keep rapidJSON for our sax-based serializer/deserializer). I would hope any SAX implementation would keep the spirit of nlohmann/json.

As the 'S' in SAX is for ,,stream'', why not use the stream model which is more C++-isch anyway and fits what this package already does? That is, if the system encounters the key "foo" it could call std::istream& handle_key(std::istream& is, std::string const& key) (similar to the stream extraction operator -- but notice the second argument is ```const``) which could in most cases recursively call the appropriate stream functions, and set stream errors on failure. Also means I could read an array of object handles (which already have from_json handlers) in an intuitive way consistent with the non-SAX version (and consistent with reading objects from any other stream).

The downside of this, and all such parsers, is that handle_key has to know about every possible key, but the user can easily handle this by managing a list of expected keys and their handlers on the call stack.

For writing then you could perhaps do something like:

std::vector<things> the_things {thing_one, thing_two};
json_ostream["the_things"] << the_things;

OK, perhaps that's too much!

PS: I don't know what to call this package since its name is generic so that's why I say ,,this package'' or ,,nlohmann/json'' vs ,,rapidJSON''

[nlohmann]

Overview

So about the SAX parsing, or parsing in general:

The library currently supports any combination of the following items:

1. input: string, input stream, iterator range, container
2. strict parsing (parsing must end with EOF): yes/no
3. exceptions on parse error: yes/no
4. using a callback function to filter values: yes/no

In addition, we also have an acceptor which only returns a boolean to indicate whether the input was a valid JSON text (works with any of 1. and 2.).

SAX Parsing

The current take on a SAX parser can also be combined with 1., 2., and 3. I think, this not too far from the spirit of the library :-)

For testing purposes, I implemented a SAX parser which creates a basic_json value - so basically a "SAX-to-DOM" parser. Surprisingly (for me), its performance was at least as good as the original parser.

Planning

I am planning to replace the original parser (well, the part where no callback (4.) is used) with this SAX-DOM parser, and also use the SAX approach for the binary formats CBOR, MessagePack, and UBJSON. As a result, we would have SAX and DOM parsers for all formats without code duplication on the parser classes. Furthermore, I think we can also realize the acceptor with a SAX parser.

What do you think?

• @dvhwgumby Would this be a feasible approach for you? If not, could you please give more details for your proposed syntax?
• @dvhwgumby What do you propose instead of pure virtuals?
• @gregmarr What do you think of RapidJSON's approach to use templates like struct MyHandler : public BaseReaderHandler<UTF8<>, MyHandler>?