ARROW-534 - WIP

[tebeka]

Just for code review, not final code.

[xhochy]

Timestamps before 1970 are negative, so: yes.

[tebeka]

OK :)

[wesm]

Can you also add data generator fixtures to integration_test.py?

[tebeka]

OK

[wesm]

This will have some minor rebase conflicts with #347

[wesm]

Small rebase required

[wesm]

You need to add date and time test cases for https://github.com/apache/arrow/blob/master/cpp/src/arrow/ipc/ipc-adapter-test.cc#L178 and https://github.com/apache/arrow/blob/master/cpp/src/arrow/ipc/ipc-file-test.cc#L183.

see e.g. https://github.com/apache/arrow/blob/master/cpp/src/arrow/ipc/test-common.h#L283

Integration tests are blocked until ipc-file-test and ipc-adapter-test are passing , then we can add the data generators to integration_test.py. We'll also need to wait for ARROW-582 to get done

[wesm]

How does rapidjson handle integers over 2^53? If it doesn't have any problems then this is OK

[tebeka]

See so, the following code works:

#include<iostream>
#include<sstream>
#include"rapidjson/document.h"
#include"rapidjson/istreamwrapper.h"usingnamespacerapidjson;intmain() {
std::stringstream in;
in << "{";
// 4611686018427387917
in << "\"x\": " << ((uint64_t(1)<<62) + 13) << ",";
in << "\"y\": 2";
in << "}";
IStreamWrapper isw(in);
Document doc;
doc.ParseStream(isw);
int64_t x = doc["x"].GetInt64();
std::cout << "x = " << x << std::endl;
int64_t y = doc["y"].GetInt64();
std::cout << "y = " << y << std::endl;
}

[wesm]

Good to know. Suppose we'll have the proof in the working integration tests anyway

[wesm]

Timestamps are always signed

[tebeka]

OK, fixing

[wesm]

Can you combine the logic for TimeType and TimestampType into a single function?

[tebeka]

Sure. WIll do

[wesm]

Can you use the same parsing code as for integers?

[tebeka]

Good idea, will try to add is_base_of to the ReadArray method for PrimitiveCType and BooleanType.

[wesm]

This can be deleted

[wesm]

TimeUnit is a strongly typed enum, so these MIN/MAX fields aren't necessary. In Flatbuffers, the MIN/MAX values are these to help account for NULL.

[wesm]

If unit is an invalid value, I don't believe it will have made it this far

[tebeka]

Where will we fail? In the flatbuffer side of things?

[wesm]

Right, i.e. it should not be possible to return an invalid TimeUnit from a flatbuffer

[wesm]

I note that a similar test fixture is missing in https://github.com/apache/arrow/blob/master/cpp/src/arrow/ipc/ipc-json-test.cc

created https://issues.apache.org/jira/browse/ARROW-591

[wesm]

@tebeka I opened https://issues.apache.org/jira/browse/ARROW-620 -- since you started on the JSON support in this patch, if you have time can you take this up in a new patch?

[tebeka]

@wesm OK, will look into ARROW-620

[tebeka]

Just for code review, not final code.

[xhochy]

Timestamps before 1970 are negative, so: yes.

[tebeka]

OK :)

[wesm]

Can you also add data generator fixtures to integration_test.py?

[tebeka]

OK

[wesm]

This will have some minor rebase conflicts with #347

[wesm]

Small rebase required

[wesm]

You need to add date and time test cases for https://github.com/apache/arrow/blob/master/cpp/src/arrow/ipc/ipc-adapter-test.cc#L178 and https://github.com/apache/arrow/blob/master/cpp/src/arrow/ipc/ipc-file-test.cc#L183.

see e.g. https://github.com/apache/arrow/blob/master/cpp/src/arrow/ipc/test-common.h#L283

Integration tests are blocked until ipc-file-test and ipc-adapter-test are passing , then we can add the data generators to integration_test.py. We'll also need to wait for ARROW-582 to get done

[wesm]

How does rapidjson handle integers over 2^53? If it doesn't have any problems then this is OK

[tebeka]

See so, the following code works:

#include<iostream>
#include<sstream>
#include"rapidjson/document.h"
#include"rapidjson/istreamwrapper.h"usingnamespacerapidjson;intmain() {
std::stringstream in;
in << "{";
// 4611686018427387917
in << "\"x\": " << ((uint64_t(1)<<62) + 13) << ",";
in << "\"y\": 2";
in << "}";
IStreamWrapper isw(in);
Document doc;
doc.ParseStream(isw);
int64_t x = doc["x"].GetInt64();
std::cout << "x = " << x << std::endl;
int64_t y = doc["y"].GetInt64();
std::cout << "y = " << y << std::endl;
}

[wesm]

Good to know. Suppose we'll have the proof in the working integration tests anyway

[wesm]

Timestamps are always signed

[tebeka]

OK, fixing

[wesm]

Can you combine the logic for TimeType and TimestampType into a single function?

[tebeka]

Sure. WIll do

[wesm]

Can you use the same parsing code as for integers?

[tebeka]

Good idea, will try to add is_base_of to the ReadArray method for PrimitiveCType and BooleanType.

[wesm]

This can be deleted

[wesm]

TimeUnit is a strongly typed enum, so these MIN/MAX fields aren't necessary. In Flatbuffers, the MIN/MAX values are these to help account for NULL.

[wesm]

If unit is an invalid value, I don't believe it will have made it this far

[tebeka]

Where will we fail? In the flatbuffer side of things?

[wesm]

Right, i.e. it should not be possible to return an invalid TimeUnit from a flatbuffer

[wesm]

I note that a similar test fixture is missing in https://github.com/apache/arrow/blob/master/cpp/src/arrow/ipc/ipc-json-test.cc

created https://issues.apache.org/jira/browse/ARROW-591

[wesm]

@tebeka I opened https://issues.apache.org/jira/browse/ARROW-620 -- since you started on the JSON support in this patch, if you have time can you take this up in a new patch?

[tebeka]

@wesm OK, will look into ARROW-620

[tebeka]

Just for code review, not final code.

[xhochy]

Timestamps before 1970 are negative, so: yes.

[tebeka]

OK :)

[wesm]

Can you also add data generator fixtures to integration_test.py?

[tebeka]

OK

[wesm]

This will have some minor rebase conflicts with #347

[wesm]

Small rebase required

[wesm]

You need to add date and time test cases for https://github.com/apache/arrow/blob/master/cpp/src/arrow/ipc/ipc-adapter-test.cc#L178 and https://github.com/apache/arrow/blob/master/cpp/src/arrow/ipc/ipc-file-test.cc#L183.

see e.g. https://github.com/apache/arrow/blob/master/cpp/src/arrow/ipc/test-common.h#L283

Integration tests are blocked until ipc-file-test and ipc-adapter-test are passing , then we can add the data generators to integration_test.py. We'll also need to wait for ARROW-582 to get done

[wesm]

How does rapidjson handle integers over 2^53? If it doesn't have any problems then this is OK

[tebeka]

See so, the following code works:

#include<iostream>
#include<sstream>
#include"rapidjson/document.h"
#include"rapidjson/istreamwrapper.h"usingnamespacerapidjson;intmain() {
std::stringstream in;
in << "{";
// 4611686018427387917
in << "\"x\": " << ((uint64_t(1)<<62) + 13) << ",";
in << "\"y\": 2";
in << "}";
IStreamWrapper isw(in);
Document doc;
doc.ParseStream(isw);
int64_t x = doc["x"].GetInt64();
std::cout << "x = " << x << std::endl;
int64_t y = doc["y"].GetInt64();
std::cout << "y = " << y << std::endl;
}

[wesm]

Good to know. Suppose we'll have the proof in the working integration tests anyway

[wesm]

Timestamps are always signed

[tebeka]

OK, fixing

[wesm]

Can you combine the logic for TimeType and TimestampType into a single function?

[tebeka]

Sure. WIll do

[wesm]

Can you use the same parsing code as for integers?

[tebeka]

Good idea, will try to add is_base_of to the ReadArray method for PrimitiveCType and BooleanType.

[wesm]

This can be deleted

[wesm]

TimeUnit is a strongly typed enum, so these MIN/MAX fields aren't necessary. In Flatbuffers, the MIN/MAX values are these to help account for NULL.

[wesm]

If unit is an invalid value, I don't believe it will have made it this far

[tebeka]

Where will we fail? In the flatbuffer side of things?

[wesm]

Right, i.e. it should not be possible to return an invalid TimeUnit from a flatbuffer

[wesm]

I note that a similar test fixture is missing in https://github.com/apache/arrow/blob/master/cpp/src/arrow/ipc/ipc-json-test.cc

created https://issues.apache.org/jira/browse/ARROW-591

[wesm]

@tebeka I opened https://issues.apache.org/jira/browse/ARROW-620 -- since you started on the JSON support in this patch, if you have time can you take this up in a new patch?

[tebeka]

@wesm OK, will look into ARROW-620

[tebeka]

Just for code review, not final code.

[xhochy]

Timestamps before 1970 are negative, so: yes.

[tebeka]

OK :)

[wesm]

Can you also add data generator fixtures to integration_test.py?

[tebeka]

OK

[wesm]

This will have some minor rebase conflicts with #347

[wesm]

Small rebase required

[wesm]

You need to add date and time test cases for https://github.com/apache/arrow/blob/master/cpp/src/arrow/ipc/ipc-adapter-test.cc#L178 and https://github.com/apache/arrow/blob/master/cpp/src/arrow/ipc/ipc-file-test.cc#L183.

see e.g. https://github.com/apache/arrow/blob/master/cpp/src/arrow/ipc/test-common.h#L283

Integration tests are blocked until ipc-file-test and ipc-adapter-test are passing , then we can add the data generators to integration_test.py. We'll also need to wait for ARROW-582 to get done

[wesm]

How does rapidjson handle integers over 2^53? If it doesn't have any problems then this is OK

[tebeka]

See so, the following code works:

#include<iostream>
#include<sstream>
#include"rapidjson/document.h"
#include"rapidjson/istreamwrapper.h"usingnamespacerapidjson;intmain() {
std::stringstream in;
in << "{";
// 4611686018427387917
in << "\"x\": " << ((uint64_t(1)<<62) + 13) << ",";
in << "\"y\": 2";
in << "}";
IStreamWrapper isw(in);
Document doc;
doc.ParseStream(isw);
int64_t x = doc["x"].GetInt64();
std::cout << "x = " << x << std::endl;
int64_t y = doc["y"].GetInt64();
std::cout << "y = " << y << std::endl;
}

[wesm]

Good to know. Suppose we'll have the proof in the working integration tests anyway

[wesm]

Timestamps are always signed

[tebeka]

OK, fixing

[wesm]

Can you combine the logic for TimeType and TimestampType into a single function?

[tebeka]

Sure. WIll do

[wesm]

Can you use the same parsing code as for integers?

[tebeka]

Good idea, will try to add is_base_of to the ReadArray method for PrimitiveCType and BooleanType.

[wesm]

This can be deleted

[wesm]

TimeUnit is a strongly typed enum, so these MIN/MAX fields aren't necessary. In Flatbuffers, the MIN/MAX values are these to help account for NULL.

[wesm]

If unit is an invalid value, I don't believe it will have made it this far

[tebeka]

Where will we fail? In the flatbuffer side of things?

[wesm]

Right, i.e. it should not be possible to return an invalid TimeUnit from a flatbuffer

[wesm]

I note that a similar test fixture is missing in https://github.com/apache/arrow/blob/master/cpp/src/arrow/ipc/ipc-json-test.cc

created https://issues.apache.org/jira/browse/ARROW-591

[wesm]

@tebeka I opened https://issues.apache.org/jira/browse/ARROW-620 -- since you started on the JSON support in this patch, if you have time can you take this up in a new patch?

[tebeka]

@wesm OK, will look into ARROW-620

[tebeka]

Just for code review, not final code.

[xhochy]

Timestamps before 1970 are negative, so: yes.

[tebeka]

OK :)

[wesm]

Can you also add data generator fixtures to integration_test.py?

[tebeka]

OK

[wesm]

This will have some minor rebase conflicts with #347

[wesm]

Small rebase required

[wesm]

You need to add date and time test cases for https://github.com/apache/arrow/blob/master/cpp/src/arrow/ipc/ipc-adapter-test.cc#L178 and https://github.com/apache/arrow/blob/master/cpp/src/arrow/ipc/ipc-file-test.cc#L183.

see e.g. https://github.com/apache/arrow/blob/master/cpp/src/arrow/ipc/test-common.h#L283

Integration tests are blocked until ipc-file-test and ipc-adapter-test are passing , then we can add the data generators to integration_test.py. We'll also need to wait for ARROW-582 to get done

[wesm]

How does rapidjson handle integers over 2^53? If it doesn't have any problems then this is OK

[tebeka]

See so, the following code works:

#include<iostream>
#include<sstream>
#include"rapidjson/document.h"
#include"rapidjson/istreamwrapper.h"usingnamespacerapidjson;intmain() {
std::stringstream in;
in << "{";
// 4611686018427387917
in << "\"x\": " << ((uint64_t(1)<<62) + 13) << ",";
in << "\"y\": 2";
in << "}";
IStreamWrapper isw(in);
Document doc;
doc.ParseStream(isw);
int64_t x = doc["x"].GetInt64();
std::cout << "x = " << x << std::endl;
int64_t y = doc["y"].GetInt64();
std::cout << "y = " << y << std::endl;
}

[wesm]

Good to know. Suppose we'll have the proof in the working integration tests anyway

[wesm]

Timestamps are always signed

[tebeka]

OK, fixing

[wesm]

Can you combine the logic for TimeType and TimestampType into a single function?

[tebeka]

Sure. WIll do

[wesm]

Can you use the same parsing code as for integers?

[tebeka]

Good idea, will try to add is_base_of to the ReadArray method for PrimitiveCType and BooleanType.

[wesm]

This can be deleted

[wesm]

TimeUnit is a strongly typed enum, so these MIN/MAX fields aren't necessary. In Flatbuffers, the MIN/MAX values are these to help account for NULL.

[wesm]

If unit is an invalid value, I don't believe it will have made it this far

[tebeka]

Where will we fail? In the flatbuffer side of things?

[wesm]

Right, i.e. it should not be possible to return an invalid TimeUnit from a flatbuffer

[wesm]

I note that a similar test fixture is missing in https://github.com/apache/arrow/blob/master/cpp/src/arrow/ipc/ipc-json-test.cc

created https://issues.apache.org/jira/browse/ARROW-591

[wesm]

@tebeka I opened https://issues.apache.org/jira/browse/ARROW-620 -- since you started on the JSON support in this patch, if you have time can you take this up in a new patch?

[tebeka]

@wesm OK, will look into ARROW-620

[tebeka]

Just for code review, not final code.

[xhochy]

Timestamps before 1970 are negative, so: yes.

[tebeka]

OK :)

[wesm]

Can you also add data generator fixtures to integration_test.py?

[tebeka]

OK

[wesm]

This will have some minor rebase conflicts with #347

[wesm]

Small rebase required

[wesm]

You need to add date and time test cases for https://github.com/apache/arrow/blob/master/cpp/src/arrow/ipc/ipc-adapter-test.cc#L178 and https://github.com/apache/arrow/blob/master/cpp/src/arrow/ipc/ipc-file-test.cc#L183.

see e.g. https://github.com/apache/arrow/blob/master/cpp/src/arrow/ipc/test-common.h#L283

Integration tests are blocked until ipc-file-test and ipc-adapter-test are passing , then we can add the data generators to integration_test.py. We'll also need to wait for ARROW-582 to get done

[wesm]

How does rapidjson handle integers over 2^53? If it doesn't have any problems then this is OK

[tebeka]

See so, the following code works:

#include<iostream>
#include<sstream>
#include"rapidjson/document.h"
#include"rapidjson/istreamwrapper.h"usingnamespacerapidjson;intmain() {
std::stringstream in;
in << "{";
// 4611686018427387917
in << "\"x\": " << ((uint64_t(1)<<62) + 13) << ",";
in << "\"y\": 2";
in << "}";
IStreamWrapper isw(in);
Document doc;
doc.ParseStream(isw);
int64_t x = doc["x"].GetInt64();
std::cout << "x = " << x << std::endl;
int64_t y = doc["y"].GetInt64();
std::cout << "y = " << y << std::endl;
}

[wesm]

Good to know. Suppose we'll have the proof in the working integration tests anyway

[wesm]

Timestamps are always signed

[tebeka]

OK, fixing

[wesm]

Can you combine the logic for TimeType and TimestampType into a single function?

[tebeka]

Sure. WIll do

[wesm]

Can you use the same parsing code as for integers?

[tebeka]

Good idea, will try to add is_base_of to the ReadArray method for PrimitiveCType and BooleanType.

[wesm]

This can be deleted

[wesm]

TimeUnit is a strongly typed enum, so these MIN/MAX fields aren't necessary. In Flatbuffers, the MIN/MAX values are these to help account for NULL.

[wesm]

If unit is an invalid value, I don't believe it will have made it this far

[tebeka]

Where will we fail? In the flatbuffer side of things?

[wesm]

Right, i.e. it should not be possible to return an invalid TimeUnit from a flatbuffer

[wesm]

I note that a similar test fixture is missing in https://github.com/apache/arrow/blob/master/cpp/src/arrow/ipc/ipc-json-test.cc

created https://issues.apache.org/jira/browse/ARROW-591

[wesm]

@tebeka I opened https://issues.apache.org/jira/browse/ARROW-620 -- since you started on the JSON support in this patch, if you have time can you take this up in a new patch?

[tebeka]

@wesm OK, will look into ARROW-620

[tebeka]

Just for code review, not final code.

[xhochy]

Timestamps before 1970 are negative, so: yes.

[tebeka]

OK :)

[wesm]

Can you also add data generator fixtures to integration_test.py?

[tebeka]

OK

[wesm]

This will have some minor rebase conflicts with #347

[wesm]

Small rebase required

[wesm]

You need to add date and time test cases for https://github.com/apache/arrow/blob/master/cpp/src/arrow/ipc/ipc-adapter-test.cc#L178 and https://github.com/apache/arrow/blob/master/cpp/src/arrow/ipc/ipc-file-test.cc#L183.

see e.g. https://github.com/apache/arrow/blob/master/cpp/src/arrow/ipc/test-common.h#L283

Integration tests are blocked until ipc-file-test and ipc-adapter-test are passing , then we can add the data generators to integration_test.py. We'll also need to wait for ARROW-582 to get done

[wesm]

How does rapidjson handle integers over 2^53? If it doesn't have any problems then this is OK

[tebeka]

See so, the following code works:

#include<iostream>
#include<sstream>
#include"rapidjson/document.h"
#include"rapidjson/istreamwrapper.h"usingnamespacerapidjson;intmain() {
std::stringstream in;
in << "{";
// 4611686018427387917
in << "\"x\": " << ((uint64_t(1)<<62) + 13) << ",";
in << "\"y\": 2";
in << "}";
IStreamWrapper isw(in);
Document doc;
doc.ParseStream(isw);
int64_t x = doc["x"].GetInt64();
std::cout << "x = " << x << std::endl;
int64_t y = doc["y"].GetInt64();
std::cout << "y = " << y << std::endl;
}

[wesm]

Good to know. Suppose we'll have the proof in the working integration tests anyway

[wesm]

Timestamps are always signed

[tebeka]

OK, fixing

[wesm]

Can you combine the logic for TimeType and TimestampType into a single function?

[tebeka]

Sure. WIll do

[wesm]

Can you use the same parsing code as for integers?

[tebeka]

Good idea, will try to add is_base_of to the ReadArray method for PrimitiveCType and BooleanType.

[wesm]

This can be deleted

[wesm]

TimeUnit is a strongly typed enum, so these MIN/MAX fields aren't necessary. In Flatbuffers, the MIN/MAX values are these to help account for NULL.

[wesm]

If unit is an invalid value, I don't believe it will have made it this far

[tebeka]

Where will we fail? In the flatbuffer side of things?

[wesm]

Right, i.e. it should not be possible to return an invalid TimeUnit from a flatbuffer

[wesm]

I note that a similar test fixture is missing in https://github.com/apache/arrow/blob/master/cpp/src/arrow/ipc/ipc-json-test.cc

created https://issues.apache.org/jira/browse/ARROW-591

[wesm]

@tebeka I opened https://issues.apache.org/jira/browse/ARROW-620 -- since you started on the JSON support in this patch, if you have time can you take this up in a new patch?

[tebeka]

@wesm OK, will look into ARROW-620

[tebeka]

Just for code review, not final code.

[xhochy]

Timestamps before 1970 are negative, so: yes.

[tebeka]

OK :)

[wesm]

Can you also add data generator fixtures to integration_test.py?

[tebeka]

OK

[wesm]

This will have some minor rebase conflicts with #347

[wesm]

Small rebase required

[wesm]

You need to add date and time test cases for https://github.com/apache/arrow/blob/master/cpp/src/arrow/ipc/ipc-adapter-test.cc#L178 and https://github.com/apache/arrow/blob/master/cpp/src/arrow/ipc/ipc-file-test.cc#L183.

see e.g. https://github.com/apache/arrow/blob/master/cpp/src/arrow/ipc/test-common.h#L283

Integration tests are blocked until ipc-file-test and ipc-adapter-test are passing , then we can add the data generators to integration_test.py. We'll also need to wait for ARROW-582 to get done

[wesm]

How does rapidjson handle integers over 2^53? If it doesn't have any problems then this is OK

[tebeka]

See so, the following code works:

#include<iostream>
#include<sstream>
#include"rapidjson/document.h"
#include"rapidjson/istreamwrapper.h"usingnamespacerapidjson;intmain() {
std::stringstream in;
in << "{";
// 4611686018427387917
in << "\"x\": " << ((uint64_t(1)<<62) + 13) << ",";
in << "\"y\": 2";
in << "}";
IStreamWrapper isw(in);
Document doc;
doc.ParseStream(isw);
int64_t x = doc["x"].GetInt64();
std::cout << "x = " << x << std::endl;
int64_t y = doc["y"].GetInt64();
std::cout << "y = " << y << std::endl;
}

[wesm]

Good to know. Suppose we'll have the proof in the working integration tests anyway

[wesm]

Timestamps are always signed

[tebeka]

OK, fixing

[wesm]

Can you combine the logic for TimeType and TimestampType into a single function?

[tebeka]

Sure. WIll do

[wesm]

Can you use the same parsing code as for integers?

[tebeka]

Good idea, will try to add is_base_of to the ReadArray method for PrimitiveCType and BooleanType.

[wesm]

This can be deleted

[wesm]

TimeUnit is a strongly typed enum, so these MIN/MAX fields aren't necessary. In Flatbuffers, the MIN/MAX values are these to help account for NULL.

[wesm]

If unit is an invalid value, I don't believe it will have made it this far

[tebeka]

Where will we fail? In the flatbuffer side of things?

[wesm]

Right, i.e. it should not be possible to return an invalid TimeUnit from a flatbuffer

[wesm]

I note that a similar test fixture is missing in https://github.com/apache/arrow/blob/master/cpp/src/arrow/ipc/ipc-json-test.cc

created https://issues.apache.org/jira/browse/ARROW-591

[wesm]

@tebeka I opened https://issues.apache.org/jira/browse/ARROW-620 -- since you started on the JSON support in this patch, if you have time can you take this up in a new patch?

[tebeka]

@wesm OK, will look into ARROW-620