spec: Variant lower/upper bounds

[aihuaxu]

This is to revise the bounds specification for Variant. In summary:

The writer determines which fields to collect bounds for in a Variant column. Field bounds are stored as serialized Variant objects, where each key is a normalized JSON path identifying a field, and each value is the corresponding lower or upper bound.

E.g.

For a Variant column with the schema as follows:

{
"event_type": "login",
"user.name": "Alex", "tags": ["action", "drama"]
}

The collected bound object looks like:

{
"$['event_type']": "login",
"$['user.name']": "Alex",
"$['tags']": "action"
}

[aihuaxu]

cc @rdblue, @danielcweeks, @RussellSpitzer

[xxubai]

LGTM

[danielcweeks]

@aihuaxu and @rdblue is there a reason we need to explicitly restrict the lower/upper bounds to shredded fields? I would think that the stats pruning would be useful for any field that a writer would want to include in the bound (regardless of whether it was shredded or not).

[aihuaxu]

@aihuaxu and @rdblue is there a reason we need to explicitly restrict the lower/upper bounds to shredded fields? I would think that the stats pruning would be useful for any field that a writer would want to include in the bound (regardless of whether it was shredded or not).

What we were thinking is that the bounds are collected from shredded column stats during shredding process. But it does seem reasonable to me to bounds and shredding can be separated: if a writer has the knowledge of the bounds and chooses not to shred, the bounds can still be used in pruning.

[RussellSpitzer]

Looks good to me, I think we need to clarify a bit here

[rdblue]

When I read the thread you raised on the dev list, I liked that you used "tags" as the example. Maybe we should change some of these to match the examples in the variant shredding spec?

[aihuaxu]

Let me do that to match as much as possible.

[amogh-jahagirdar]

Thanks @RussellSpitzer@rdblue@Fokko@flyrain@huaxingao @XBaith for reviewing and everyone for voting. Since the vote passed, I'll go ahead and merge