ARROW-510 ARROW-582 ARROW-663 ARROW-729: [Java] Added units for Time and Date types, and integration tests - #475

Closed
leifwalsh wants to merge 4 commits into
apache:masterfrom
leifwalsh:feature/java-date-time-types
Closed

ARROW-510 ARROW-582 ARROW-663 ARROW-729: [Java] Added units for Time and Date types, and integration tests#475
leifwalsh wants to merge 4 commits into
apache:masterfrom
leifwalsh:feature/java-date-time-types

Conversation

@leifwalsh

@leifwalshleifwalsh commented Apr 1, 2017•
edited
Loading

Copy link
Copy Markdown
Contributor

closes#366

@leifwalshleifwalsh changed the title ARROW-670: [Java] Added day and milli implementations for date, change Time to TimeMilliARROW-729: [Java] Added day and milli implementations for date, change Time to TimeMilliApr 1, 2017
@leifwalsh
leifwalshforce-pushed the feature/java-date-time-types branch from 8a85ab9 to d10713eCompareApril 1, 2017 00:13
@icexelloss

icexelloss commented Apr 1, 2017

Copy link
Copy Markdown
Contributor

This is how I would do it.

@leifwalsh

leifwalsh commented Apr 1, 2017

Copy link
Copy Markdown
ContributorAuthor

@julienledem@wesm where are the roundtrip integration tests so I can add these to them and how do I run them?

@icexelloss

icexelloss commented Apr 1, 2017•
edited
Loading

Copy link
Copy Markdown
Contributor

@leifwalsh , you first want to look at the java integration test:
https://github.com/apache/arrow/blob/master/java/tools/src/main/java/org/apache/arrow/tools/Integration.java

Then there is https://github.com/apache/arrow/blob/master/integration/integration_test.py#L659 which invokes the Java integration thing.

@leifwalsh

leifwalsh commented Apr 1, 2017

Copy link
Copy Markdown
ContributorAuthor

It seems to work but I think the test data doesn't have any timestamps, dates, or times in it. Were the json files constructed manually or programmatically?

@wesm

wesm commented Apr 1, 2017

Copy link
Copy Markdown
Member

The integration tests for date/times are in the patch for ARROW-510. There's a line commented out that generates the test cases

@leifwalsh

leifwalsh commented Apr 1, 2017

Copy link
Copy Markdown
ContributorAuthor

@wesm awesome. I created a branch off this and merged yours into that, uncommented the test, and got a failure:

Command failed: java -cp /home/leif/git/arrow/java/tools/target/arrow-tools-0.2.1-SNAPSHOT-jar-with-dependencies.jar org.apache.arrow.tools.Integration -a /tmp/tmp4dpiwmfk/c377ba55676a45d28e09e9a5ea70b617 -j /tmp/tmpadyd70vm/628548c02e5440eba693410d754f7c31.json -c VALIDATE
With output:
--------------
12:32:16.427 [main] DEBUG i.n.u.i.l.InternalLoggerFactory - Using SLF4J as the default logging framework
12:32:16.432 [main] DEBUG io.netty.buffer.AbstractByteBuf - -Dio.netty.buffer.bytebuf.checkAccessible: true
12:32:16.434 [main] DEBUG io.netty.util.ResourceLeakDetector - -Dio.netty.leakDetection.level: simple
12:32:16.434 [main] DEBUG io.netty.util.ResourceLeakDetector - -Dio.netty.leakDetection.maxRecords: 4
12:32:16.447 [main] DEBUG i.n.util.internal.PlatformDependent0 - java.nio.Buffer.address: available
12:32:16.447 [main] DEBUG i.n.util.internal.PlatformDependent0 - sun.misc.Unsafe.theUnsafe: available
12:32:16.448 [main] DEBUG i.n.util.internal.PlatformDependent0 - sun.misc.Unsafe.copyMemory: available
12:32:16.448 [main] DEBUG i.n.util.internal.PlatformDependent0 - direct buffer constructor: available
12:32:16.449 [main] DEBUG i.n.util.internal.PlatformDependent0 - java.nio.Bits.unaligned: available, true
12:32:16.449 [main] DEBUG i.n.util.internal.PlatformDependent0 - java.nio.DirectByteBuffer.<init>(long, int): available
12:32:16.450 [main] DEBUG io.netty.util.internal.Cleaner0 - java.nio.ByteBuffer.cleaner(): available
12:32:16.451 [main] DEBUG i.n.util.internal.PlatformDependent - Java version: 8
12:32:16.451 [main] DEBUG i.n.util.internal.PlatformDependent - -Dio.netty.noUnsafe: false
12:32:16.451 [main] DEBUG i.n.util.internal.PlatformDependent - sun.misc.Unsafe: available
12:32:16.451 [main] DEBUG i.n.util.internal.PlatformDependent - -Dio.netty.noJavassist: false
12:32:16.452 [main] DEBUG i.n.util.internal.PlatformDependent - Javassist: unavailable
12:32:16.452 [main] DEBUG i.n.util.internal.PlatformDependent - You don't have Javassist in your class path or you don't have enough permission to load dynamically generated classes. Please check the configuration for better performance.
12:32:16.452 [main] DEBUG i.n.util.internal.PlatformDependent - -Dio.netty.tmpdir: /tmp (java.io.tmpdir)
12:32:16.453 [main] DEBUG i.n.util.internal.PlatformDependent - -Dio.netty.bitMode: 64 (sun.arch.data.model)
12:32:16.453 [main] DEBUG i.n.util.internal.PlatformDependent - -Dio.netty.noPreferDirect: false
12:32:16.453 [main] DEBUG i.n.util.internal.PlatformDependent - io.netty.maxDirectMemory: 1836580864 bytes
12:32:16.454 [main] DEBUG i.n.util.ResourceLeakDetectorFactory - Loaded default ResourceLeakDetector: io.netty.util.ResourceLeakDetector@1c6b6478
12:32:16.464 [main] DEBUG i.n.buffer.PooledByteBufAllocator - -Dio.netty.allocator.numHeapArenas: 8
12:32:16.464 [main] DEBUG i.n.buffer.PooledByteBufAllocator - -Dio.netty.allocator.numDirectArenas: 8
12:32:16.464 [main] DEBUG i.n.buffer.PooledByteBufAllocator - -Dio.netty.allocator.pageSize: 8192
12:32:16.465 [main] DEBUG i.n.buffer.PooledByteBufAllocator - -Dio.netty.allocator.maxOrder: 11
12:32:16.465 [main] DEBUG i.n.buffer.PooledByteBufAllocator - -Dio.netty.allocator.chunkSize: 16777216
12:32:16.465 [main] DEBUG i.n.buffer.PooledByteBufAllocator - -Dio.netty.allocator.tinyCacheSize: 512
12:32:16.465 [main] DEBUG i.n.buffer.PooledByteBufAllocator - -Dio.netty.allocator.smallCacheSize: 256
12:32:16.465 [main] DEBUG i.n.buffer.PooledByteBufAllocator - -Dio.netty.allocator.normalCacheSize: 64
12:32:16.465 [main] DEBUG i.n.buffer.PooledByteBufAllocator - -Dio.netty.allocator.maxCachedBufferCapacity: 32768
12:32:16.465 [main] DEBUG i.n.buffer.PooledByteBufAllocator - -Dio.netty.allocator.cacheTrimInterval: 8192
12:32:16.724 [main] DEBUG o.a.arrow.vector.file.ReadChannel - Reading buffer with size: 10
12:32:16.726 [main] DEBUG o.a.a.vector.file.ArrowFileReader - Footer starts at 5968, length: 1040
12:32:16.726 [main] DEBUG o.a.arrow.vector.file.ReadChannel - Reading buffer with size: 1040
12:32:16.808 [main] DEBUG org.apache.arrow.tools.Integration - Arrow Input file size: 7018
12:32:16.809 [main] DEBUG org.apache.arrow.tools.Integration - ARROW schema: Schema<f0: Date(DAY), f1: Date(MILLISECOND), f2: Time(SECOND, 0), f3: Time(MILLISECOND, 0), f4: Time(MICROSECOND, 0), f5: Time(NANOSECOND, 0), f6: Timestamp(SECOND, null), f7: Timestamp(MILLISECOND, null), f8: Timestamp(MICROSECOND, null), f9: Timestamp(NANOSECOND, null), f10: Timestamp(MILLISECOND, America/New_York)>
12:32:16.809 [main] DEBUG org.apache.arrow.tools.Integration - JSON Input file size: 15314
12:32:16.809 [main] DEBUG org.apache.arrow.tools.Integration - JSON schema: Schema<f0: Date(DAY), f1: Date(MILLISECOND), f2: Time(SECOND, 32), f3: Time(MILLISECOND, 32), f4: Time(MICROSECOND, 64), f5: Time(NANOSECOND, 64), f6: Timestamp(SECOND, null), f7: Timestamp(MILLISECOND, null), f8: Timestamp(MICROSECOND, null), f9: Timestamp(NANOSECOND, null), f10: Timestamp(MILLISECOND, America/New_York)>
Incompatible files
Different schemas:
Schema<f0: Date(DAY), f1: Date(MILLISECOND), f2: Time(SECOND, 0), f3: Time(MILLISECOND, 0), f4: Time(MICROSECOND, 0), f5: Time(NANOSECOND, 0), f6: Timestamp(SECOND, null), f7: Timestamp(MILLISECOND, null), f8: Timestamp(MICROSECOND, null), f9: Timestamp(NANOSECOND, null), f10: Timestamp(MILLISECOND, America/New_York)>
Schema<f0: Date(DAY), f1: Date(MILLISECOND), f2: Time(SECOND, 32), f3: Time(MILLISECOND, 32), f4: Time(MICROSECOND, 64), f5: Time(NANOSECOND, 64), f6: Timestamp(SECOND, null), f7: Timestamp(MILLISECOND, null), f8: Timestamp(MICROSECOND, null), f9: Timestamp(NANOSECOND, null), f10: Timestamp(MILLISECOND, America/New_York)>
12:32:16.815 [main] ERROR org.apache.arrow.tools.Integration - Incompatible files
java.lang.IllegalArgumentException: Different schemas:
Schema<f0: Date(DAY), f1: Date(MILLISECOND), f2: Time(SECOND, 0), f3: Time(MILLISECOND, 0), f4: Time(MICROSECOND, 0), f5: Time(NANOSECOND, 0), f6: Timestamp(SECOND, null), f7: Timestamp(MILLISECOND, null), f8: Timestamp(MICROSECOND, null), f9: Timestamp(NANOSECOND, null), f10: Timestamp(MILLISECOND, America/New_York)>
Schema<f0: Date(DAY), f1: Date(MILLISECOND), f2: Time(SECOND, 32), f3: Time(MILLISECOND, 32), f4: Time(MICROSECOND, 64), f5: Time(NANOSECOND, 64), f6: Timestamp(SECOND, null), f7: Timestamp(MILLISECOND, null), f8: Timestamp(MICROSECOND, null), f9: Timestamp(NANOSECOND, null), f10: Timestamp(MILLISECOND, America/New_York)>
at org.apache.arrow.vector.util.Validator.compareSchemas(Validator.java:43) ~[arrow-tools-0.2.1-SNAPSHOT-jar-with-dependencies.jar:na]
at org.apache.arrow.tools.Integration$Command$3.execute(Integration.java:177) ~[arrow-tools-0.2.1-SNAPSHOT-jar-with-dependencies.jar:na]
at org.apache.arrow.tools.Integration.run(Integration.java:101) ~[arrow-tools-0.2.1-SNAPSHOT-jar-with-dependencies.jar:na]
at org.apache.arrow.tools.Integration.main(Integration.java:62) ~[arrow-tools-0.2.1-SNAPSHOT-jar-with-dependencies.jar:na]

I'm not sure if I'm reading this right but it looks like the schema in the json file is specifying bit widths for Times but the arrow file that C++ generated sets them to 0?

@wesm

wesm commented Apr 1, 2017•
edited
Loading

Copy link
Copy Markdown
Member

Yeah, that looks like the case: https://github.com/apache/arrow/blob/master/cpp/src/arrow/ipc/metadata.cc#L389

Let me fix that in #458 right now and add a check to validate that the bit width is what we expect for each unit

@leifwalshleifwalsh changed the title ARROW-729: [Java] Added day and milli implementations for date, change Time to TimeMilliARROW-729: [Java] Added units for Time and Date typesApr 1, 2017
@wesm

wesm commented Apr 1, 2017

Copy link
Copy Markdown
Member

done 06c4eed

@leifwalsh

leifwalsh commented Apr 1, 2017

Copy link
Copy Markdown
ContributorAuthor

Okay, I implemented these types in JsonFileReader and JsonFileWriter and now I get this failure:

-- Java producing, C++ consuming
Testing file /home/leif/git/arrow/integration/data/struct_example.json
-- Creating binary inputs
-- Validating file
-- Validating stream
Testing file /home/leif/git/arrow/integration/data/simple.json
-- Creating binary inputs
-- Validating file
-- Validating stream
Testing file /tmp/tmpugxk26ua/053a211f7b8f448cb7093a44b16258c5.json
-- Creating binary inputs
-- Validating file
-- Validating stream
Testing file /tmp/tmpugxk26ua/9b7a257132e74a5f81364c88d94c1b12.json
-- Creating binary inputs
-- Validating file
-- Validating stream
Testing file /tmp/tmpugxk26ua/34173bff0e02405eb44015751d80d2cd.json
-- Creating binary inputs
-- Validating file
-- Validating stream
Testing file /tmp/tmpugxk26ua/1e0a01f57e7740a7bc715f95d6b9ea7c.json
-- Creating binary inputs
-- Validating file
Command failed: /home/leif/git/arrow/cpp/test-build/debug/json-integration-test --integration --arrow=/tmp/tmp4wae8kve/641c555737874c5e80c5f2ea5aa91179 --json=/tmp/tmpugxk26ua/1e0a01f57e7740a7bc715f95d6b9ea7c.json --mode=VALIDATE
With output:
--------------
Error message: Invalid: Record batch 0 did not match
JSON:
f0: [null, -1534653433, null, -471089378, null, null, 1726023356]
f1: [1218641917, -2011903818, -470743102, null, -1756786353, null, null]
f2: [810115012, null, null, -883898073, -1982197003, null, -1729996989]
f3: [1591774331, -314120230, null, 1196117999, -1539384497, null, 872289996]
f4: [658927759, null, null, -961381010, -1231869285, -1369355730, null]
f5: [null, 552443512, null, 2013848155, 709970719, null, -1972820288]
f6: [null, null, -102932588, 1536992262, null, null, null]
f7: [-1730827518, 1939266448, -678382409, null, null, null, null]
f8: [null, -264948505, -1250649123, 2006597095, null, 1050350308, -394676957]
f9: [33833774, null, 114513685, null, 1900471701, -669924741, 1015463225]
f10: [399170240, null, null, null, null, null, -548617532]
Arrow:
f0: [null, -1534653433, null, -471089378, null, null, 1726023356]
f1: [1218641917, -2011903818, -470743102, null, -1756786353, null, null]
f2: [810115012, null, null, -883898073, -1982197003, null, -1729996989]
f3: [1591774331, -314120230, null, 1196117999, -1539384497, null, 872289996]
f4: [658927759, null, null, -961381010, -1231869285, -1369355730, null]
f5: [null, 552443512, null, 2013848155, 709970719, null, -1972820288]
f6: [null, null, -102932588, 1536992262, null, null, null]
f7: [-1730827518, 1939266448, -678382409, null, null, null, null]
f8: [null, -264948505, -1250649123, 2006597095, null, 1050350308, -394676957]
f9: [33833774, null, 114513685, null, 1900471701, -669924741, 1015463225]
f10: [399170240, null, null, null, null, null, -548617532]

Funny thing is, those are identical according to diff.

@wesm

wesm commented Apr 1, 2017

Copy link
Copy Markdown
Member

Interesting. Can you pull my commits from #458 into this patch and I'll take a look to see what's up?

@leifwalsh

leifwalsh commented Apr 1, 2017

Copy link
Copy Markdown
ContributorAuthor

Why do you not check if right.IsNull() here? https://github.com/apache/arrow/blob/master/cpp/src/arrow/compare.cc#L472

Is the expectation that left.IsNull() iff right.IsNull()?

@leifwalsh

leifwalsh commented Apr 1, 2017

Copy link
Copy Markdown
ContributorAuthor

Sure.

@leifwalsh

leifwalsh commented Apr 1, 2017

Copy link
Copy Markdown
ContributorAuthor

Pushed

@wesm

wesm commented Apr 1, 2017

Copy link
Copy Markdown
Member

looking

@wesm

wesm commented Apr 1, 2017

Copy link
Copy Markdown
Member

Found the issue, the bit width on Date32Type in arrow/type.h was incorrect. fixed at wesm@2e5570c. integration tests pass now... awesome!

@wesmwesm mentioned this pull request Apr 1, 2017
Closed
@leifwalsh

leifwalsh commented Apr 1, 2017 via email •
edited
Loading

Copy link
Copy Markdown
ContributorAuthor

@wesm

wesm commented Apr 1, 2017

Copy link
Copy Markdown
Member

I closed my PR -- do you mind doing all the code reviews in this patch? Feel free to review the integration test / C++ stuff (the changes were pretty minimal)

@wesm

wesm commented Apr 1, 2017

Copy link
Copy Markdown
Member

Also, if you can add "closes #366" into your PR description then that will close that PR when this is merged

@leifwalsh

leifwalsh commented Apr 1, 2017 via email •
edited
Loading

Copy link
Copy Markdown
ContributorAuthor

@leifwalshleifwalsh changed the title ARROW-729: [Java] Added units for Time and Date typesARROW-510 ARROW-582 ARROW-729: [Java] Added units for Time and Date types, and integration testsApr 1, 2017
@leifwalshleifwalsh changed the title ARROW-510 ARROW-582 ARROW-729: [Java] Added units for Time and Date types, and integration testsARROW-510 ARROW-582 ARROW-663 ARROW-729: [Java] Added units for Time and Date types, and integration testsApr 1, 2017
@icexelloss

icexelloss commented Apr 1, 2017

Copy link
Copy Markdown
Contributor

Wow, awesome! Looks like you guys are pretty much all set. Let me know if I can help with anything.

@leifwalsh

leifwalsh commented Apr 1, 2017

Copy link
Copy Markdown
ContributorAuthor

@icexelloss would appreciate review, especially of what I'm doing with micros and nanos to create joda objects here #475 (comment)

@leifwalsh
leifwalshforce-pushed the feature/java-date-time-types branch from 2e5570c to 0bfe92eCompareApril 1, 2017 17:58
@leifwalsh

leifwalsh commented Apr 1, 2017

Copy link
Copy Markdown
ContributorAuthor

Reorganized commits a little

@wesm

wesm commented Apr 1, 2017

Copy link
Copy Markdown
Member

@leifwalsh looks like you pulled in a few unintentional commits, can you rebase on apache/master?

@leifwalsh
leifwalshforce-pushed the feature/java-date-time-types branch from 27c6853 to 49392f2CompareApril 2, 2017 21:59
@wesmwesm mentioned this pull request Apr 3, 2017
Closed
julienledem
julienledem reviewed Apr 3, 2017
View reviewed changes

@julienledemjulienledem left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall, this looks good to me.
@jacques-n ?

@wesm

wesm commented Apr 3, 2017

Copy link
Copy Markdown
Member

@leifwalsh there is a conflict (hopefully not too horrible to resolve) after merging #409 -- the idea is that this will make it easier to add more types that have additional metadata

@jacques-n

jacques-n commented Apr 3, 2017

Copy link
Copy Markdown
Contributor

@julienledem, I think this patch further propagates the issues we've with using DateTime from Joda instead of LocalDateTime. I think those should probably be addressed separately but wanted to confirm your thoughts?

@julienledem

julienledem commented Apr 3, 2017

Copy link
Copy Markdown
Member

@jacques-n: right. This should probably be fixed in a separate change. I'd suggest the new vectors should just use Integer and Long in getObject().

@wesm

wesm commented Apr 3, 2017

Copy link
Copy Markdown
Member

I just opened https://issues.apache.org/jira/browse/ARROW-768 and made it a blocker for 0.3

@leifwalsh

leifwalsh commented Apr 4, 2017

Copy link
Copy Markdown
ContributorAuthor

@julienledem do you want me to change getObject() to return Integer and Long or leave the way it is for now and address later?

@leifwalsh

leifwalsh commented Apr 4, 2017

Copy link
Copy Markdown
ContributorAuthor

Resolved merge conflicts by rebasing on master, passes java unit and c++/java integration tests.

leifwalshand others added 3 commits April 3, 2017 23:23
@leifwalsh
ARROW-729: [Java] Added units for date/time types
2dca474
@wesm@leifwalsh
ARROW-733: [C++/Python] Rename FixedWidthBinary to FixedSizeBinary fo…
6e86422
…r consistency with FixedSizeList
As discussed on JIRA
Author: Wes McKinney <wes.mckinney@twosigma.com>
Closesapache#473 from wesm/ARROW-733 and squashes the following commits:
0e30af3 [Wes McKinney] Rename FixedWidthBinary to FixedSizeBinary for consistency with FixedSizeList type
@wesm@leifwalsh
Integration tests for all date and time combinations
47f83a8
Change-Id: I79f5d942f64c275c87568703f9edf6a7e89467ac
@leifwalsh
leifwalshforce-pushed the feature/java-date-time-types branch from 49392f2 to 47f83a8CompareApril 4, 2017 03:26
@julienledem

julienledem commented Apr 4, 2017

Copy link
Copy Markdown
Member

@leifwalsh: Yes, please make getObject() return Integer and Long (as appropriate) for new vectors.

@leifwalsh

leifwalsh commented Apr 4, 2017

Copy link
Copy Markdown
ContributorAuthor

@julienledem should I also do that to the existing TimeStamp* types?

@leifwalsh

leifwalsh commented Apr 4, 2017

Copy link
Copy Markdown
ContributorAuthor

Also, ComplexReaders.java defines read${minor.boxedType} for all the timestamp types and my new types. Should I remove that as well?

@julienledem

julienledem commented Apr 5, 2017

Copy link
Copy Markdown
Member

@leifwalsh I think for now it's better to do it only for new vectors. Changing the behavior of existing vectors requires more work on the code using them. Which is why I'd rather have those as separate changes.

@leifwalsh

leifwalsh commented Apr 5, 2017

Copy link
Copy Markdown
ContributorAuthor

Okay then I think this is good to go.

@wesm

wesm commented Apr 5, 2017

Copy link
Copy Markdown
Member

Cool. +1 -- thanks @leifwalsh for your contribution!

@asfgitasfgit closed this in f4fcb42Apr 5, 2017
julienledem
julienledem reviewed Apr 6, 2017
View reviewed changes

@julienledemjulienledem left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we'd better not change the behavior of dateVector.getAccessor().getObject() just yet.
Otherwise, this looks good to me.

Comment threadjava/vector/src/main/codegen/data/ValueVectorTypes.tdd
{ class: "TimeStampMilli", javaType: "long", boxedType: "Long", friendlyType: "DateTime" }
{ class: "TimeStampMicro", javaType: "long", boxedType: "Long", friendlyType: "DateTime" }
{ class: "TimeStampNano", javaType: "long", boxedType: "Long", friendlyType: "DateTime" }
{ class: "DateMilli" },

@julienledemjulienledemApr 6, 2017

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is changing an existing vector, right?

Comment threadjava/vector/src/main/codegen/templates/FixedValueVectors.java

</#if>

<#if minor.class == "Date">

@julienledemjulienledemApr 6, 2017

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here we're changing the behavior of Date.

@leifwalsh

leifwalsh commented Apr 6, 2017 via email

Copy link
Copy Markdown
ContributorAuthor

@wesm

wesm commented Apr 6, 2017

Copy link
Copy Markdown
Member

@julienledem sorry for merging early, should have waited for your final review. Let's revert what needs to be reverted for 0.3?

@wesm

wesm commented Apr 6, 2017

Copy link
Copy Markdown
Member

I opened https://issues.apache.org/jira/browse/ARROW-777, will leave it to you all to resolve ahead of 0.3

@leifwalsh
leifwalsh deleted the feature/java-date-time-types branch April 7, 2017 01:44
This was referenced Apr 5, 2017
Closed
Closed
pribor pushed a commit to GlobalWebIndex/arrow that referenced this pull request Oct 24, 2025
@leifwalsh@kou
ARROW-510 ARROW-582 ARROW-663 ARROW-729: [Java] Added units for Time …
58dafc2
…and Date types, and integration tests
closesapache#366
Author: Leif Walsh <leif.walsh@gmail.com>
Author: Wes McKinney <wes.mckinney@twosigma.com>
Closesapache#475 from leifwalsh/feature/java-date-time-types and squashes the following commits:
2e2a4cf [Leif Walsh] ARROW-729: [Java] removed Joda DateTime getters from Date* and Time* types
47f83a8 [Wes McKinney] Integration tests for all date and time combinations
6e86422 [Wes McKinney] ARROW-733: [C++/Python] Rename FixedWidthBinary to FixedSizeBinary for consistency with FixedSizeList
2dca474 [Leif Walsh] ARROW-729: [Java] Added units for date/time types
pribor pushed a commit to GlobalWebIndex/arrow that referenced this pull request Oct 24, 2025
@lidavidm
apacheGH-447: Port fix from apache/arrow that was missed (apache#475)
5b6ee22
apache@ded148c
was merged after the migration.
Fixesapache#447.
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@wesmwesmwesm left review comments
@julienledemjulienledemjulienledem left review comments
@cpcloudcpcloudcpcloud left review comments
@icexellossicexellossicexelloss left review comments

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@leifwalsh@icexelloss@wesm@jacques-n@julienledem@cpcloud
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Add copy buttons to all 
 blocks\n(function() {\n function addCopyButtons() {\n document.querySelectorAll('pre code').forEach(function(codeBlock) {\n if (codeBlock.parentElement.hasAttribute('data-copy-added')) return;\n codeBlock.parentElement.setAttribute('data-copy-added', 'true');\n \n var btn = document.createElement('button');\n btn.textContent = 'Copy';\n btn.style.cssText = 'position:absolute;top:4px;right:4px;padding:2px 8px;font-size:11px;background:#4ecdc4;border:none;border-radius:4px;color:#1a1a2e;cursor:pointer;opacity:0.7;transition:opacity 0.2s;';\n btn.onmouseover = function() { this.style.opacity = '1'; };\n btn.onmouseout = function() { this.style.opacity = '0.7'; };\n btn.onclick = function() {\n navigator.clipboard.writeText(codeBlock.textContent).then(function() {\n btn.textContent = 'Copied!';\n setTimeout(function() { btn.textContent = 'Copy'; }, 1500);\n });\n };\n codeBlock.parentElement.style.position = 'relative';\n codeBlock.parentElement.appendChild(btn);\n });\n }\n \n addCopyButtons();\n \n // Re-run on dynamic content\n var observer = new MutationObserver(addCopyButtons);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Add Copy Buttons to Code Blocks");
}
} catch(__e) { console.warn('[Userscript:Add Copy Buttons to Code Blocks]', __e); }
})();
(function(){
try {
var __m = "github.com";
var __re = new RegExp('^' + "github\\.com" + '

Skip to content
ARROW-510 ARROW-582 ARROW-663 ARROW-729: [Java] Added units for Time and Date types, and integration tests - #475
Closed
leifwalsh wants to merge 4 commits into
apache:masterfrom 
leifwalsh:feature/java-date-time-types
Closed
ARROW-510 ARROW-582 ARROW-663 ARROW-729: [Java] Added units for Time and Date types, and integration tests#475
leifwalsh wants to merge 4 commits into
apache:masterfrom 
leifwalsh:feature/java-date-time-types
Conversation
@leifwalsh
@leifwalshleifwalsh
commented
Apr 1, 2017•

edited

Loading

Copy link


Copy Markdown

Contributor
closes#366
@leifwalshleifwalsh
changed the title
ARROW-670: [Java] Added day and milli implementations for date, change Time to TimeMilliARROW-729: [Java] Added day and milli implementations for date, change Time to TimeMilliApr 1, 2017
@leifwalsh
leifwalshforce-pushed
the
feature/java-date-time-types
branch
from
8a85ab9 to
d10713eCompareApril 1, 2017 00:13
@icexelloss
icexelloss
commented
Apr 1, 2017

Copy link


Copy Markdown

Contributor
This is how I would do it.
@leifwalsh
leifwalsh
commented
Apr 1, 2017

Copy link


Copy Markdown

ContributorAuthor
@julienledem@wesm where are the roundtrip integration tests so I can add these to them and how do I run them?
@icexelloss
icexelloss
commented
Apr 1, 2017•

edited

Loading

Copy link


Copy Markdown

Contributor
@leifwalsh , you first want to look at the java integration test:
https://github.com/apache/arrow/blob/master/java/tools/src/main/java/org/apache/arrow/tools/Integration.java
Then there is https://github.com/apache/arrow/blob/master/integration/integration_test.py#L659 which invokes the Java integration thing.
@leifwalsh
leifwalsh
commented
Apr 1, 2017

Copy link


Copy Markdown

ContributorAuthor
It seems to work but I think the test data doesn't have any timestamps, dates, or times in it. Were the json files constructed manually or programmatically?
@wesm
wesm
commented
Apr 1, 2017

Copy link


Copy Markdown

Member
The integration tests for date/times are in the patch for ARROW-510. There's a line commented out that generates the test cases
@leifwalsh
leifwalsh
commented
Apr 1, 2017

Copy link


Copy Markdown

ContributorAuthor
@wesm awesome. I created a branch off this and merged yours into that, uncommented the test, and got a failure:
Command failed: java -cp /home/leif/git/arrow/java/tools/target/arrow-tools-0.2.1-SNAPSHOT-jar-with-dependencies.jar org.apache.arrow.tools.Integration -a /tmp/tmp4dpiwmfk/c377ba55676a45d28e09e9a5ea70b617 -j /tmp/tmpadyd70vm/628548c02e5440eba693410d754f7c31.json -c VALIDATE
With output:
--------------
12:32:16.427 [main] DEBUG i.n.u.i.l.InternalLoggerFactory - Using SLF4J as the default logging framework
12:32:16.432 [main] DEBUG io.netty.buffer.AbstractByteBuf - -Dio.netty.buffer.bytebuf.checkAccessible: true
12:32:16.434 [main] DEBUG io.netty.util.ResourceLeakDetector - -Dio.netty.leakDetection.level: simple
12:32:16.434 [main] DEBUG io.netty.util.ResourceLeakDetector - -Dio.netty.leakDetection.maxRecords: 4
12:32:16.447 [main] DEBUG i.n.util.internal.PlatformDependent0 - java.nio.Buffer.address: available
12:32:16.447 [main] DEBUG i.n.util.internal.PlatformDependent0 - sun.misc.Unsafe.theUnsafe: available
12:32:16.448 [main] DEBUG i.n.util.internal.PlatformDependent0 - sun.misc.Unsafe.copyMemory: available
12:32:16.448 [main] DEBUG i.n.util.internal.PlatformDependent0 - direct buffer constructor: available
12:32:16.449 [main] DEBUG i.n.util.internal.PlatformDependent0 - java.nio.Bits.unaligned: available, true
12:32:16.449 [main] DEBUG i.n.util.internal.PlatformDependent0 - java.nio.DirectByteBuffer.<init>(long, int): available
12:32:16.450 [main] DEBUG io.netty.util.internal.Cleaner0 - java.nio.ByteBuffer.cleaner(): available
12:32:16.451 [main] DEBUG i.n.util.internal.PlatformDependent - Java version: 8
12:32:16.451 [main] DEBUG i.n.util.internal.PlatformDependent - -Dio.netty.noUnsafe: false
12:32:16.451 [main] DEBUG i.n.util.internal.PlatformDependent - sun.misc.Unsafe: available
12:32:16.451 [main] DEBUG i.n.util.internal.PlatformDependent - -Dio.netty.noJavassist: false
12:32:16.452 [main] DEBUG i.n.util.internal.PlatformDependent - Javassist: unavailable
12:32:16.452 [main] DEBUG i.n.util.internal.PlatformDependent - You don't have Javassist in your class path or you don't have enough permission to load dynamically generated classes. Please check the configuration for better performance.
12:32:16.452 [main] DEBUG i.n.util.internal.PlatformDependent - -Dio.netty.tmpdir: /tmp (java.io.tmpdir)
12:32:16.453 [main] DEBUG i.n.util.internal.PlatformDependent - -Dio.netty.bitMode: 64 (sun.arch.data.model)
12:32:16.453 [main] DEBUG i.n.util.internal.PlatformDependent - -Dio.netty.noPreferDirect: false
12:32:16.453 [main] DEBUG i.n.util.internal.PlatformDependent - io.netty.maxDirectMemory: 1836580864 bytes
12:32:16.454 [main] DEBUG i.n.util.ResourceLeakDetectorFactory - Loaded default ResourceLeakDetector: io.netty.util.ResourceLeakDetector@1c6b6478
12:32:16.464 [main] DEBUG i.n.buffer.PooledByteBufAllocator - -Dio.netty.allocator.numHeapArenas: 8
12:32:16.464 [main] DEBUG i.n.buffer.PooledByteBufAllocator - -Dio.netty.allocator.numDirectArenas: 8
12:32:16.464 [main] DEBUG i.n.buffer.PooledByteBufAllocator - -Dio.netty.allocator.pageSize: 8192
12:32:16.465 [main] DEBUG i.n.buffer.PooledByteBufAllocator - -Dio.netty.allocator.maxOrder: 11
12:32:16.465 [main] DEBUG i.n.buffer.PooledByteBufAllocator - -Dio.netty.allocator.chunkSize: 16777216
12:32:16.465 [main] DEBUG i.n.buffer.PooledByteBufAllocator - -Dio.netty.allocator.tinyCacheSize: 512
12:32:16.465 [main] DEBUG i.n.buffer.PooledByteBufAllocator - -Dio.netty.allocator.smallCacheSize: 256
12:32:16.465 [main] DEBUG i.n.buffer.PooledByteBufAllocator - -Dio.netty.allocator.normalCacheSize: 64
12:32:16.465 [main] DEBUG i.n.buffer.PooledByteBufAllocator - -Dio.netty.allocator.maxCachedBufferCapacity: 32768
12:32:16.465 [main] DEBUG i.n.buffer.PooledByteBufAllocator - -Dio.netty.allocator.cacheTrimInterval: 8192
12:32:16.724 [main] DEBUG o.a.arrow.vector.file.ReadChannel - Reading buffer with size: 10
12:32:16.726 [main] DEBUG o.a.a.vector.file.ArrowFileReader - Footer starts at 5968, length: 1040
12:32:16.726 [main] DEBUG o.a.arrow.vector.file.ReadChannel - Reading buffer with size: 1040
12:32:16.808 [main] DEBUG org.apache.arrow.tools.Integration - Arrow Input file size: 7018
12:32:16.809 [main] DEBUG org.apache.arrow.tools.Integration - ARROW schema: Schema<f0: Date(DAY), f1: Date(MILLISECOND), f2: Time(SECOND, 0), f3: Time(MILLISECOND, 0), f4: Time(MICROSECOND, 0), f5: Time(NANOSECOND, 0), f6: Timestamp(SECOND, null), f7: Timestamp(MILLISECOND, null), f8: Timestamp(MICROSECOND, null), f9: Timestamp(NANOSECOND, null), f10: Timestamp(MILLISECOND, America/New_York)>
12:32:16.809 [main] DEBUG org.apache.arrow.tools.Integration - JSON Input file size: 15314
12:32:16.809 [main] DEBUG org.apache.arrow.tools.Integration - JSON schema: Schema<f0: Date(DAY), f1: Date(MILLISECOND), f2: Time(SECOND, 32), f3: Time(MILLISECOND, 32), f4: Time(MICROSECOND, 64), f5: Time(NANOSECOND, 64), f6: Timestamp(SECOND, null), f7: Timestamp(MILLISECOND, null), f8: Timestamp(MICROSECOND, null), f9: Timestamp(NANOSECOND, null), f10: Timestamp(MILLISECOND, America/New_York)>
Incompatible files
Different schemas:
Schema<f0: Date(DAY), f1: Date(MILLISECOND), f2: Time(SECOND, 0), f3: Time(MILLISECOND, 0), f4: Time(MICROSECOND, 0), f5: Time(NANOSECOND, 0), f6: Timestamp(SECOND, null), f7: Timestamp(MILLISECOND, null), f8: Timestamp(MICROSECOND, null), f9: Timestamp(NANOSECOND, null), f10: Timestamp(MILLISECOND, America/New_York)>
Schema<f0: Date(DAY), f1: Date(MILLISECOND), f2: Time(SECOND, 32), f3: Time(MILLISECOND, 32), f4: Time(MICROSECOND, 64), f5: Time(NANOSECOND, 64), f6: Timestamp(SECOND, null), f7: Timestamp(MILLISECOND, null), f8: Timestamp(MICROSECOND, null), f9: Timestamp(NANOSECOND, null), f10: Timestamp(MILLISECOND, America/New_York)>
12:32:16.815 [main] ERROR org.apache.arrow.tools.Integration - Incompatible files
java.lang.IllegalArgumentException: Different schemas:
Schema<f0: Date(DAY), f1: Date(MILLISECOND), f2: Time(SECOND, 0), f3: Time(MILLISECOND, 0), f4: Time(MICROSECOND, 0), f5: Time(NANOSECOND, 0), f6: Timestamp(SECOND, null), f7: Timestamp(MILLISECOND, null), f8: Timestamp(MICROSECOND, null), f9: Timestamp(NANOSECOND, null), f10: Timestamp(MILLISECOND, America/New_York)>
Schema<f0: Date(DAY), f1: Date(MILLISECOND), f2: Time(SECOND, 32), f3: Time(MILLISECOND, 32), f4: Time(MICROSECOND, 64), f5: Time(NANOSECOND, 64), f6: Timestamp(SECOND, null), f7: Timestamp(MILLISECOND, null), f8: Timestamp(MICROSECOND, null), f9: Timestamp(NANOSECOND, null), f10: Timestamp(MILLISECOND, America/New_York)>
at org.apache.arrow.vector.util.Validator.compareSchemas(Validator.java:43) ~[arrow-tools-0.2.1-SNAPSHOT-jar-with-dependencies.jar:na]
at org.apache.arrow.tools.Integration$Command$3.execute(Integration.java:177) ~[arrow-tools-0.2.1-SNAPSHOT-jar-with-dependencies.jar:na]
at org.apache.arrow.tools.Integration.run(Integration.java:101) ~[arrow-tools-0.2.1-SNAPSHOT-jar-with-dependencies.jar:na]
at org.apache.arrow.tools.Integration.main(Integration.java:62) ~[arrow-tools-0.2.1-SNAPSHOT-jar-with-dependencies.jar:na]

I'm not sure if I'm reading this right but it looks like the schema in the json file is specifying bit widths for Times but the arrow file that C++ generated sets them to 0?
@wesm
wesm
commented
Apr 1, 2017•

edited

Loading

Copy link


Copy Markdown

Member
Yeah, that looks like the case: https://github.com/apache/arrow/blob/master/cpp/src/arrow/ipc/metadata.cc#L389
Let me fix that in #458 right now and add a check to validate that the bit width is what we expect for each unit
@leifwalshleifwalsh
changed the title
ARROW-729: [Java] Added day and milli implementations for date, change Time to TimeMilliARROW-729: [Java] Added units for Time and Date typesApr 1, 2017
@wesm
wesm
commented
Apr 1, 2017

Copy link


Copy Markdown

Member
done 06c4eed
@leifwalsh
leifwalsh
commented
Apr 1, 2017

Copy link


Copy Markdown

ContributorAuthor
Okay, I implemented these types in JsonFileReader and JsonFileWriter and now I get this failure:
-- Java producing, C++ consuming
Testing file /home/leif/git/arrow/integration/data/struct_example.json
-- Creating binary inputs
-- Validating file
-- Validating stream
Testing file /home/leif/git/arrow/integration/data/simple.json
-- Creating binary inputs
-- Validating file
-- Validating stream
Testing file /tmp/tmpugxk26ua/053a211f7b8f448cb7093a44b16258c5.json
-- Creating binary inputs
-- Validating file
-- Validating stream
Testing file /tmp/tmpugxk26ua/9b7a257132e74a5f81364c88d94c1b12.json
-- Creating binary inputs
-- Validating file
-- Validating stream
Testing file /tmp/tmpugxk26ua/34173bff0e02405eb44015751d80d2cd.json
-- Creating binary inputs
-- Validating file
-- Validating stream
Testing file /tmp/tmpugxk26ua/1e0a01f57e7740a7bc715f95d6b9ea7c.json
-- Creating binary inputs
-- Validating file
Command failed: /home/leif/git/arrow/cpp/test-build/debug/json-integration-test --integration --arrow=/tmp/tmp4wae8kve/641c555737874c5e80c5f2ea5aa91179 --json=/tmp/tmpugxk26ua/1e0a01f57e7740a7bc715f95d6b9ea7c.json --mode=VALIDATE
With output:
--------------
Error message: Invalid: Record batch 0 did not match
JSON:
f0: [null, -1534653433, null, -471089378, null, null, 1726023356]
f1: [1218641917, -2011903818, -470743102, null, -1756786353, null, null]
f2: [810115012, null, null, -883898073, -1982197003, null, -1729996989]
f3: [1591774331, -314120230, null, 1196117999, -1539384497, null, 872289996]
f4: [658927759, null, null, -961381010, -1231869285, -1369355730, null]
f5: [null, 552443512, null, 2013848155, 709970719, null, -1972820288]
f6: [null, null, -102932588, 1536992262, null, null, null]
f7: [-1730827518, 1939266448, -678382409, null, null, null, null]
f8: [null, -264948505, -1250649123, 2006597095, null, 1050350308, -394676957]
f9: [33833774, null, 114513685, null, 1900471701, -669924741, 1015463225]
f10: [399170240, null, null, null, null, null, -548617532]
Arrow:
f0: [null, -1534653433, null, -471089378, null, null, 1726023356]
f1: [1218641917, -2011903818, -470743102, null, -1756786353, null, null]
f2: [810115012, null, null, -883898073, -1982197003, null, -1729996989]
f3: [1591774331, -314120230, null, 1196117999, -1539384497, null, 872289996]
f4: [658927759, null, null, -961381010, -1231869285, -1369355730, null]
f5: [null, 552443512, null, 2013848155, 709970719, null, -1972820288]
f6: [null, null, -102932588, 1536992262, null, null, null]
f7: [-1730827518, 1939266448, -678382409, null, null, null, null]
f8: [null, -264948505, -1250649123, 2006597095, null, 1050350308, -394676957]
f9: [33833774, null, 114513685, null, 1900471701, -669924741, 1015463225]
f10: [399170240, null, null, null, null, null, -548617532]

Funny thing is, those are identical according to diff.
@wesm
wesm
commented
Apr 1, 2017

Copy link


Copy Markdown

Member
Interesting. Can you pull my commits from #458 into this patch and I'll take a look to see what's up?
@leifwalsh
leifwalsh
commented
Apr 1, 2017

Copy link


Copy Markdown

ContributorAuthor
Why do you not check if right.IsNull() here? https://github.com/apache/arrow/blob/master/cpp/src/arrow/compare.cc#L472
Is the expectation that left.IsNull() iff right.IsNull()?
@leifwalsh
leifwalsh
commented
Apr 1, 2017

Copy link


Copy Markdown

ContributorAuthor
Sure.
@leifwalsh
leifwalsh
commented
Apr 1, 2017

Copy link


Copy Markdown

ContributorAuthor
Pushed
@wesm
wesm
commented
Apr 1, 2017

Copy link


Copy Markdown

Member
looking
@wesm
wesm
commented
Apr 1, 2017

Copy link


Copy Markdown

Member
Found the issue, the bit width on Date32Type in arrow/type.h was incorrect. fixed at wesm@2e5570c. integration tests pass now... awesome!
@wesmwesm
mentioned this pull request
Apr 1, 2017
 Closed

@leifwalsh
leifwalsh
commented
Apr 1, 2017
via email •

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
@wesm
wesm
commented
Apr 1, 2017

Copy link


Copy Markdown

Member
I closed my PR -- do you mind doing all the code reviews in this patch? Feel free to review the integration test / C++ stuff (the changes were pretty minimal)
@wesm
wesm
commented
Apr 1, 2017

Copy link


Copy Markdown

Member
Also, if you can add "closes #366" into your PR description then that will close that PR when this is merged
@leifwalsh
leifwalsh
commented
Apr 1, 2017
via email •

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
@leifwalshleifwalsh
changed the title
ARROW-729: [Java] Added units for Time and Date typesARROW-510 ARROW-582 ARROW-729: [Java] Added units for Time and Date types, and integration testsApr 1, 2017
@leifwalshleifwalsh
changed the title
ARROW-510 ARROW-582 ARROW-729: [Java] Added units for Time and Date types, and integration testsARROW-510 ARROW-582 ARROW-663 ARROW-729: [Java] Added units for Time and Date types, and integration testsApr 1, 2017
@icexelloss
icexelloss
commented
Apr 1, 2017

Copy link


Copy Markdown

Contributor
Wow, awesome! Looks like you guys are pretty much all set. Let me know if I can help with anything.
@leifwalsh
leifwalsh
commented
Apr 1, 2017

Copy link


Copy Markdown

ContributorAuthor
@icexelloss would appreciate review, especially of what I'm doing with micros and nanos to create joda objects here #475 (comment)
@leifwalsh
leifwalshforce-pushed
the
feature/java-date-time-types
branch
from
2e5570c to
0bfe92eCompareApril 1, 2017 17:58
@leifwalsh
leifwalsh
commented
Apr 1, 2017

Copy link


Copy Markdown

ContributorAuthor
Reorganized commits a little
@wesm
wesm
commented
Apr 1, 2017

Copy link


Copy Markdown

Member
@leifwalsh looks like you pulled in a few unintentional commits, can you rebase on apache/master?
@leifwalsh
leifwalshforce-pushed
the
feature/java-date-time-types
branch
from
27c6853 to
49392f2CompareApril 2, 2017 21:59
@wesmwesm
mentioned this pull request
Apr 3, 2017
 Closed

julienledem
julienledem
reviewed
Apr 3, 2017
View reviewed changes
@julienledemjulienledem
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall, this looks good to me.
@jacques-n ?
@wesm
wesm
commented
Apr 3, 2017

Copy link


Copy Markdown

Member
@leifwalsh there is a conflict (hopefully not too horrible to resolve) after merging #409 -- the idea is that this will make it easier to add more types that have additional metadata
@jacques-n
jacques-n
commented
Apr 3, 2017

Copy link


Copy Markdown

Contributor
@julienledem, I think this patch further propagates the issues we've with using DateTime from Joda instead of LocalDateTime. I think those should probably be addressed separately but wanted to confirm your thoughts?
@julienledem
julienledem
commented
Apr 3, 2017

Copy link


Copy Markdown

Member
@jacques-n: right. This should probably be fixed in a separate change. I'd suggest the new vectors should just use Integer and Long in getObject().
@wesm
wesm
commented
Apr 3, 2017

Copy link


Copy Markdown

Member
I just opened https://issues.apache.org/jira/browse/ARROW-768 and made it a blocker for 0.3
@leifwalsh
leifwalsh
commented
Apr 4, 2017

Copy link


Copy Markdown

ContributorAuthor
@julienledem do you want me to change getObject() to return Integer and Long or leave the way it is for now and address later?
@leifwalsh
leifwalsh
commented
Apr 4, 2017

Copy link


Copy Markdown

ContributorAuthor
Resolved merge conflicts by rebasing on master, passes java unit and c++/java integration tests.
leifwalshand others
added 3 commits
April 3, 2017 23:23
@leifwalsh
ARROW-729: [Java] Added units for date/time types
2dca474
@wesm@leifwalsh
ARROW-733: [C++/Python] Rename FixedWidthBinary to FixedSizeBinary fo…
6e86422
…r consistency with FixedSizeList
As discussed on JIRA
Author: Wes McKinney <wes.mckinney@twosigma.com>
Closesapache#473 from wesm/ARROW-733 and squashes the following commits:
0e30af3 [Wes McKinney] Rename FixedWidthBinary to FixedSizeBinary for consistency with FixedSizeList type
@wesm@leifwalsh
Integration tests for all date and time combinations
47f83a8
Change-Id: I79f5d942f64c275c87568703f9edf6a7e89467ac
@leifwalsh
leifwalshforce-pushed
the
feature/java-date-time-types
branch
from
49392f2 to
47f83a8CompareApril 4, 2017 03:26
@julienledem
julienledem
commented
Apr 4, 2017

Copy link


Copy Markdown

Member
@leifwalsh: Yes, please make getObject() return Integer and Long (as appropriate) for new vectors.
@leifwalsh
leifwalsh
commented
Apr 4, 2017

Copy link


Copy Markdown

ContributorAuthor
@julienledem should I also do that to the existing TimeStamp* types?
@leifwalsh
leifwalsh
commented
Apr 4, 2017

Copy link


Copy Markdown

ContributorAuthor
Also, ComplexReaders.java defines read${minor.boxedType} for all the timestamp types and my new types. Should I remove that as well?
@julienledem
julienledem
commented
Apr 5, 2017

Copy link


Copy Markdown

Member
@leifwalsh I think for now it's better to do it only for new vectors. Changing the behavior of existing vectors requires more work on the code using them. Which is why I'd rather have those as separate changes.
@leifwalsh
leifwalsh
commented
Apr 5, 2017

Copy link


Copy Markdown

ContributorAuthor
Okay then I think this is good to go.
@wesm
wesm
commented
Apr 5, 2017

Copy link


Copy Markdown

Member
Cool. +1 -- thanks @leifwalsh for your contribution!
@asfgitasfgit
closed this
in
f4fcb42Apr 5, 2017
julienledem
julienledem
reviewed
Apr 6, 2017
View reviewed changes
@julienledemjulienledem
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we'd better not change the behavior of dateVector.getAccessor().getObject() just yet.

Otherwise, this looks good to me.
Comment threadjava/vector/src/main/codegen/data/ValueVectorTypes.tdd
 { class: "TimeStampMilli", javaType: "long", boxedType: "Long", friendlyType: "DateTime" }
 { class: "TimeStampMicro", javaType: "long", boxedType: "Long", friendlyType: "DateTime" }
 { class: "TimeStampNano", javaType: "long", boxedType: "Long", friendlyType: "DateTime" }
 { class: "DateMilli" },
@julienledemjulienledemApr 6, 2017

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is changing an existing vector, right?
Comment threadjava/vector/src/main/codegen/templates/FixedValueVectors.java

 </#if>

 <#if minor.class == "Date">
@julienledemjulienledemApr 6, 2017

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here we're changing the behavior of Date.
@leifwalsh
leifwalsh
commented
Apr 6, 2017
via email 

Copy link


Copy Markdown

ContributorAuthor
@wesm
wesm
commented
Apr 6, 2017

Copy link


Copy Markdown

Member
@julienledem sorry for merging early, should have waited for your final review. Let's revert what needs to be reverted for 0.3?
@wesm
wesm
commented
Apr 6, 2017

Copy link


Copy Markdown

Member
I opened https://issues.apache.org/jira/browse/ARROW-777, will leave it to you all to resolve ahead of 0.3
@leifwalsh
leifwalsh
deleted the

feature/java-date-time-types

branch
April 7, 2017 01:44

This was referenced Apr 5, 2017

Closed


Closed

pribor pushed a commit
to GlobalWebIndex/arrow
that referenced
this pull request
Oct 24, 2025
@leifwalsh@kou
ARROW-510 ARROW-582 ARROW-663 ARROW-729: [Java] Added units for Time …
58dafc2
…and Date types, and integration tests
closesapache#366
Author: Leif Walsh <leif.walsh@gmail.com>
Author: Wes McKinney <wes.mckinney@twosigma.com>
Closesapache#475 from leifwalsh/feature/java-date-time-types and squashes the following commits:
2e2a4cf [Leif Walsh] ARROW-729: [Java] removed Joda DateTime getters from Date* and Time* types
47f83a8 [Wes McKinney] Integration tests for all date and time combinations
6e86422 [Wes McKinney] ARROW-733: [C++/Python] Rename FixedWidthBinary to FixedSizeBinary for consistency with FixedSizeList
2dca474 [Leif Walsh] ARROW-729: [Java] Added units for date/time types
pribor pushed a commit
to GlobalWebIndex/arrow
that referenced
this pull request
Oct 24, 2025
@lidavidm
apacheGH-447: Port fix from apache/arrow that was missed (apache#475)
5b6ee22
apache@ded148c
was merged after the migration.
Fixesapache#447.
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@wesmwesmwesm left review comments
@julienledemjulienledemjulienledem left review comments
@cpcloudcpcloudcpcloud left review comments
@icexellossicexellossicexelloss left review comments

Assignees


No one assigned


Labels


None yet


Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

6 participants

@leifwalsh@icexelloss@wesm@jacques-n@julienledem@cpcloud

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Force GitHub README to respect dark mode\n(function() {\n var style = document.createElement('style');\n style.textContent = '\n .markdown-body {\n color-scheme: dark light;\n }\n .markdown-body pre { background: #161b22 !important; }\n .markdown-body code { background: rgba(110, 118, 129, 0.4) !important; }\n .markdown-body table th, .markdown-body table td { border-color: #30363d !important; }\n .markdown-body img { background: #0d1117; }\n .markdown-body blockquote { border-left-color: #8b949e; }\n .markdown-body hr { border-color: #30363d; }\n ';\n document.head.appendChild(style);\n})();", "GitHub Dark Mode README Fix");
}
} catch(__e) { console.warn('[Userscript:GitHub Dark Mode README Fix]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
ARROW-510 ARROW-582 ARROW-663 ARROW-729: [Java] Added units for Time and Date types, and integration tests - #475
Closed
leifwalsh wants to merge 4 commits into
apache:masterfrom 
leifwalsh:feature/java-date-time-types
Closed
ARROW-510 ARROW-582 ARROW-663 ARROW-729: [Java] Added units for Time and Date types, and integration tests#475
leifwalsh wants to merge 4 commits into
apache:masterfrom 
leifwalsh:feature/java-date-time-types
Conversation
@leifwalsh
@leifwalshleifwalsh
commented
Apr 1, 2017•

edited

Loading

Copy link


Copy Markdown

Contributor
closes#366
@leifwalshleifwalsh
changed the title
ARROW-670: [Java] Added day and milli implementations for date, change Time to TimeMilliARROW-729: [Java] Added day and milli implementations for date, change Time to TimeMilliApr 1, 2017
@leifwalsh
leifwalshforce-pushed
the
feature/java-date-time-types
branch
from
8a85ab9 to
d10713eCompareApril 1, 2017 00:13
@icexelloss
icexelloss
commented
Apr 1, 2017

Copy link


Copy Markdown

Contributor
This is how I would do it.
@leifwalsh
leifwalsh
commented
Apr 1, 2017

Copy link


Copy Markdown

ContributorAuthor
@julienledem@wesm where are the roundtrip integration tests so I can add these to them and how do I run them?
@icexelloss
icexelloss
commented
Apr 1, 2017•

edited

Loading

Copy link


Copy Markdown

Contributor
@leifwalsh , you first want to look at the java integration test:
https://github.com/apache/arrow/blob/master/java/tools/src/main/java/org/apache/arrow/tools/Integration.java
Then there is https://github.com/apache/arrow/blob/master/integration/integration_test.py#L659 which invokes the Java integration thing.
@leifwalsh
leifwalsh
commented
Apr 1, 2017

Copy link


Copy Markdown

ContributorAuthor
It seems to work but I think the test data doesn't have any timestamps, dates, or times in it. Were the json files constructed manually or programmatically?
@wesm
wesm
commented
Apr 1, 2017

Copy link


Copy Markdown

Member
The integration tests for date/times are in the patch for ARROW-510. There's a line commented out that generates the test cases
@leifwalsh
leifwalsh
commented
Apr 1, 2017

Copy link


Copy Markdown

ContributorAuthor
@wesm awesome. I created a branch off this and merged yours into that, uncommented the test, and got a failure:
Command failed: java -cp /home/leif/git/arrow/java/tools/target/arrow-tools-0.2.1-SNAPSHOT-jar-with-dependencies.jar org.apache.arrow.tools.Integration -a /tmp/tmp4dpiwmfk/c377ba55676a45d28e09e9a5ea70b617 -j /tmp/tmpadyd70vm/628548c02e5440eba693410d754f7c31.json -c VALIDATE
With output:
--------------
12:32:16.427 [main] DEBUG i.n.u.i.l.InternalLoggerFactory - Using SLF4J as the default logging framework
12:32:16.432 [main] DEBUG io.netty.buffer.AbstractByteBuf - -Dio.netty.buffer.bytebuf.checkAccessible: true
12:32:16.434 [main] DEBUG io.netty.util.ResourceLeakDetector - -Dio.netty.leakDetection.level: simple
12:32:16.434 [main] DEBUG io.netty.util.ResourceLeakDetector - -Dio.netty.leakDetection.maxRecords: 4
12:32:16.447 [main] DEBUG i.n.util.internal.PlatformDependent0 - java.nio.Buffer.address: available
12:32:16.447 [main] DEBUG i.n.util.internal.PlatformDependent0 - sun.misc.Unsafe.theUnsafe: available
12:32:16.448 [main] DEBUG i.n.util.internal.PlatformDependent0 - sun.misc.Unsafe.copyMemory: available
12:32:16.448 [main] DEBUG i.n.util.internal.PlatformDependent0 - direct buffer constructor: available
12:32:16.449 [main] DEBUG i.n.util.internal.PlatformDependent0 - java.nio.Bits.unaligned: available, true
12:32:16.449 [main] DEBUG i.n.util.internal.PlatformDependent0 - java.nio.DirectByteBuffer.<init>(long, int): available
12:32:16.450 [main] DEBUG io.netty.util.internal.Cleaner0 - java.nio.ByteBuffer.cleaner(): available
12:32:16.451 [main] DEBUG i.n.util.internal.PlatformDependent - Java version: 8
12:32:16.451 [main] DEBUG i.n.util.internal.PlatformDependent - -Dio.netty.noUnsafe: false
12:32:16.451 [main] DEBUG i.n.util.internal.PlatformDependent - sun.misc.Unsafe: available
12:32:16.451 [main] DEBUG i.n.util.internal.PlatformDependent - -Dio.netty.noJavassist: false
12:32:16.452 [main] DEBUG i.n.util.internal.PlatformDependent - Javassist: unavailable
12:32:16.452 [main] DEBUG i.n.util.internal.PlatformDependent - You don't have Javassist in your class path or you don't have enough permission to load dynamically generated classes. Please check the configuration for better performance.
12:32:16.452 [main] DEBUG i.n.util.internal.PlatformDependent - -Dio.netty.tmpdir: /tmp (java.io.tmpdir)
12:32:16.453 [main] DEBUG i.n.util.internal.PlatformDependent - -Dio.netty.bitMode: 64 (sun.arch.data.model)
12:32:16.453 [main] DEBUG i.n.util.internal.PlatformDependent - -Dio.netty.noPreferDirect: false
12:32:16.453 [main] DEBUG i.n.util.internal.PlatformDependent - io.netty.maxDirectMemory: 1836580864 bytes
12:32:16.454 [main] DEBUG i.n.util.ResourceLeakDetectorFactory - Loaded default ResourceLeakDetector: io.netty.util.ResourceLeakDetector@1c6b6478
12:32:16.464 [main] DEBUG i.n.buffer.PooledByteBufAllocator - -Dio.netty.allocator.numHeapArenas: 8
12:32:16.464 [main] DEBUG i.n.buffer.PooledByteBufAllocator - -Dio.netty.allocator.numDirectArenas: 8
12:32:16.464 [main] DEBUG i.n.buffer.PooledByteBufAllocator - -Dio.netty.allocator.pageSize: 8192
12:32:16.465 [main] DEBUG i.n.buffer.PooledByteBufAllocator - -Dio.netty.allocator.maxOrder: 11
12:32:16.465 [main] DEBUG i.n.buffer.PooledByteBufAllocator - -Dio.netty.allocator.chunkSize: 16777216
12:32:16.465 [main] DEBUG i.n.buffer.PooledByteBufAllocator - -Dio.netty.allocator.tinyCacheSize: 512
12:32:16.465 [main] DEBUG i.n.buffer.PooledByteBufAllocator - -Dio.netty.allocator.smallCacheSize: 256
12:32:16.465 [main] DEBUG i.n.buffer.PooledByteBufAllocator - -Dio.netty.allocator.normalCacheSize: 64
12:32:16.465 [main] DEBUG i.n.buffer.PooledByteBufAllocator - -Dio.netty.allocator.maxCachedBufferCapacity: 32768
12:32:16.465 [main] DEBUG i.n.buffer.PooledByteBufAllocator - -Dio.netty.allocator.cacheTrimInterval: 8192
12:32:16.724 [main] DEBUG o.a.arrow.vector.file.ReadChannel - Reading buffer with size: 10
12:32:16.726 [main] DEBUG o.a.a.vector.file.ArrowFileReader - Footer starts at 5968, length: 1040
12:32:16.726 [main] DEBUG o.a.arrow.vector.file.ReadChannel - Reading buffer with size: 1040
12:32:16.808 [main] DEBUG org.apache.arrow.tools.Integration - Arrow Input file size: 7018
12:32:16.809 [main] DEBUG org.apache.arrow.tools.Integration - ARROW schema: Schema<f0: Date(DAY), f1: Date(MILLISECOND), f2: Time(SECOND, 0), f3: Time(MILLISECOND, 0), f4: Time(MICROSECOND, 0), f5: Time(NANOSECOND, 0), f6: Timestamp(SECOND, null), f7: Timestamp(MILLISECOND, null), f8: Timestamp(MICROSECOND, null), f9: Timestamp(NANOSECOND, null), f10: Timestamp(MILLISECOND, America/New_York)>
12:32:16.809 [main] DEBUG org.apache.arrow.tools.Integration - JSON Input file size: 15314
12:32:16.809 [main] DEBUG org.apache.arrow.tools.Integration - JSON schema: Schema<f0: Date(DAY), f1: Date(MILLISECOND), f2: Time(SECOND, 32), f3: Time(MILLISECOND, 32), f4: Time(MICROSECOND, 64), f5: Time(NANOSECOND, 64), f6: Timestamp(SECOND, null), f7: Timestamp(MILLISECOND, null), f8: Timestamp(MICROSECOND, null), f9: Timestamp(NANOSECOND, null), f10: Timestamp(MILLISECOND, America/New_York)>
Incompatible files
Different schemas:
Schema<f0: Date(DAY), f1: Date(MILLISECOND), f2: Time(SECOND, 0), f3: Time(MILLISECOND, 0), f4: Time(MICROSECOND, 0), f5: Time(NANOSECOND, 0), f6: Timestamp(SECOND, null), f7: Timestamp(MILLISECOND, null), f8: Timestamp(MICROSECOND, null), f9: Timestamp(NANOSECOND, null), f10: Timestamp(MILLISECOND, America/New_York)>
Schema<f0: Date(DAY), f1: Date(MILLISECOND), f2: Time(SECOND, 32), f3: Time(MILLISECOND, 32), f4: Time(MICROSECOND, 64), f5: Time(NANOSECOND, 64), f6: Timestamp(SECOND, null), f7: Timestamp(MILLISECOND, null), f8: Timestamp(MICROSECOND, null), f9: Timestamp(NANOSECOND, null), f10: Timestamp(MILLISECOND, America/New_York)>
12:32:16.815 [main] ERROR org.apache.arrow.tools.Integration - Incompatible files
java.lang.IllegalArgumentException: Different schemas:
Schema<f0: Date(DAY), f1: Date(MILLISECOND), f2: Time(SECOND, 0), f3: Time(MILLISECOND, 0), f4: Time(MICROSECOND, 0), f5: Time(NANOSECOND, 0), f6: Timestamp(SECOND, null), f7: Timestamp(MILLISECOND, null), f8: Timestamp(MICROSECOND, null), f9: Timestamp(NANOSECOND, null), f10: Timestamp(MILLISECOND, America/New_York)>
Schema<f0: Date(DAY), f1: Date(MILLISECOND), f2: Time(SECOND, 32), f3: Time(MILLISECOND, 32), f4: Time(MICROSECOND, 64), f5: Time(NANOSECOND, 64), f6: Timestamp(SECOND, null), f7: Timestamp(MILLISECOND, null), f8: Timestamp(MICROSECOND, null), f9: Timestamp(NANOSECOND, null), f10: Timestamp(MILLISECOND, America/New_York)>
at org.apache.arrow.vector.util.Validator.compareSchemas(Validator.java:43) ~[arrow-tools-0.2.1-SNAPSHOT-jar-with-dependencies.jar:na]
at org.apache.arrow.tools.Integration$Command$3.execute(Integration.java:177) ~[arrow-tools-0.2.1-SNAPSHOT-jar-with-dependencies.jar:na]
at org.apache.arrow.tools.Integration.run(Integration.java:101) ~[arrow-tools-0.2.1-SNAPSHOT-jar-with-dependencies.jar:na]
at org.apache.arrow.tools.Integration.main(Integration.java:62) ~[arrow-tools-0.2.1-SNAPSHOT-jar-with-dependencies.jar:na]

I'm not sure if I'm reading this right but it looks like the schema in the json file is specifying bit widths for Times but the arrow file that C++ generated sets them to 0?
@wesm
wesm
commented
Apr 1, 2017•

edited

Loading

Copy link


Copy Markdown

Member
Yeah, that looks like the case: https://github.com/apache/arrow/blob/master/cpp/src/arrow/ipc/metadata.cc#L389
Let me fix that in #458 right now and add a check to validate that the bit width is what we expect for each unit
@leifwalshleifwalsh
changed the title
ARROW-729: [Java] Added day and milli implementations for date, change Time to TimeMilliARROW-729: [Java] Added units for Time and Date typesApr 1, 2017
@wesm
wesm
commented
Apr 1, 2017

Copy link


Copy Markdown

Member
done 06c4eed
@leifwalsh
leifwalsh
commented
Apr 1, 2017

Copy link


Copy Markdown

ContributorAuthor
Okay, I implemented these types in JsonFileReader and JsonFileWriter and now I get this failure:
-- Java producing, C++ consuming
Testing file /home/leif/git/arrow/integration/data/struct_example.json
-- Creating binary inputs
-- Validating file
-- Validating stream
Testing file /home/leif/git/arrow/integration/data/simple.json
-- Creating binary inputs
-- Validating file
-- Validating stream
Testing file /tmp/tmpugxk26ua/053a211f7b8f448cb7093a44b16258c5.json
-- Creating binary inputs
-- Validating file
-- Validating stream
Testing file /tmp/tmpugxk26ua/9b7a257132e74a5f81364c88d94c1b12.json
-- Creating binary inputs
-- Validating file
-- Validating stream
Testing file /tmp/tmpugxk26ua/34173bff0e02405eb44015751d80d2cd.json
-- Creating binary inputs
-- Validating file
-- Validating stream
Testing file /tmp/tmpugxk26ua/1e0a01f57e7740a7bc715f95d6b9ea7c.json
-- Creating binary inputs
-- Validating file
Command failed: /home/leif/git/arrow/cpp/test-build/debug/json-integration-test --integration --arrow=/tmp/tmp4wae8kve/641c555737874c5e80c5f2ea5aa91179 --json=/tmp/tmpugxk26ua/1e0a01f57e7740a7bc715f95d6b9ea7c.json --mode=VALIDATE
With output:
--------------
Error message: Invalid: Record batch 0 did not match
JSON:
f0: [null, -1534653433, null, -471089378, null, null, 1726023356]
f1: [1218641917, -2011903818, -470743102, null, -1756786353, null, null]
f2: [810115012, null, null, -883898073, -1982197003, null, -1729996989]
f3: [1591774331, -314120230, null, 1196117999, -1539384497, null, 872289996]
f4: [658927759, null, null, -961381010, -1231869285, -1369355730, null]
f5: [null, 552443512, null, 2013848155, 709970719, null, -1972820288]
f6: [null, null, -102932588, 1536992262, null, null, null]
f7: [-1730827518, 1939266448, -678382409, null, null, null, null]
f8: [null, -264948505, -1250649123, 2006597095, null, 1050350308, -394676957]
f9: [33833774, null, 114513685, null, 1900471701, -669924741, 1015463225]
f10: [399170240, null, null, null, null, null, -548617532]
Arrow:
f0: [null, -1534653433, null, -471089378, null, null, 1726023356]
f1: [1218641917, -2011903818, -470743102, null, -1756786353, null, null]
f2: [810115012, null, null, -883898073, -1982197003, null, -1729996989]
f3: [1591774331, -314120230, null, 1196117999, -1539384497, null, 872289996]
f4: [658927759, null, null, -961381010, -1231869285, -1369355730, null]
f5: [null, 552443512, null, 2013848155, 709970719, null, -1972820288]
f6: [null, null, -102932588, 1536992262, null, null, null]
f7: [-1730827518, 1939266448, -678382409, null, null, null, null]
f8: [null, -264948505, -1250649123, 2006597095, null, 1050350308, -394676957]
f9: [33833774, null, 114513685, null, 1900471701, -669924741, 1015463225]
f10: [399170240, null, null, null, null, null, -548617532]

Funny thing is, those are identical according to diff.
@wesm
wesm
commented
Apr 1, 2017

Copy link


Copy Markdown

Member
Interesting. Can you pull my commits from #458 into this patch and I'll take a look to see what's up?
@leifwalsh
leifwalsh
commented
Apr 1, 2017

Copy link


Copy Markdown

ContributorAuthor
Why do you not check if right.IsNull() here? https://github.com/apache/arrow/blob/master/cpp/src/arrow/compare.cc#L472
Is the expectation that left.IsNull() iff right.IsNull()?
@leifwalsh
leifwalsh
commented
Apr 1, 2017

Copy link


Copy Markdown

ContributorAuthor
Sure.
@leifwalsh
leifwalsh
commented
Apr 1, 2017

Copy link


Copy Markdown

ContributorAuthor
Pushed
@wesm
wesm
commented
Apr 1, 2017

Copy link


Copy Markdown

Member
looking
@wesm
wesm
commented
Apr 1, 2017

Copy link


Copy Markdown

Member
Found the issue, the bit width on Date32Type in arrow/type.h was incorrect. fixed at wesm@2e5570c. integration tests pass now... awesome!
@wesmwesm
mentioned this pull request
Apr 1, 2017
 Closed

@leifwalsh
leifwalsh
commented
Apr 1, 2017
via email •

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
@wesm
wesm
commented
Apr 1, 2017

Copy link


Copy Markdown

Member
I closed my PR -- do you mind doing all the code reviews in this patch? Feel free to review the integration test / C++ stuff (the changes were pretty minimal)
@wesm
wesm
commented
Apr 1, 2017

Copy link


Copy Markdown

Member
Also, if you can add "closes #366" into your PR description then that will close that PR when this is merged
@leifwalsh
leifwalsh
commented
Apr 1, 2017
via email •

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
@leifwalshleifwalsh
changed the title
ARROW-729: [Java] Added units for Time and Date typesARROW-510 ARROW-582 ARROW-729: [Java] Added units for Time and Date types, and integration testsApr 1, 2017
@leifwalshleifwalsh
changed the title
ARROW-510 ARROW-582 ARROW-729: [Java] Added units for Time and Date types, and integration testsARROW-510 ARROW-582 ARROW-663 ARROW-729: [Java] Added units for Time and Date types, and integration testsApr 1, 2017
@icexelloss
icexelloss
commented
Apr 1, 2017

Copy link


Copy Markdown

Contributor
Wow, awesome! Looks like you guys are pretty much all set. Let me know if I can help with anything.
@leifwalsh
leifwalsh
commented
Apr 1, 2017

Copy link


Copy Markdown

ContributorAuthor
@icexelloss would appreciate review, especially of what I'm doing with micros and nanos to create joda objects here #475 (comment)
@leifwalsh
leifwalshforce-pushed
the
feature/java-date-time-types
branch
from
2e5570c to
0bfe92eCompareApril 1, 2017 17:58
@leifwalsh
leifwalsh
commented
Apr 1, 2017

Copy link


Copy Markdown

ContributorAuthor
Reorganized commits a little
@wesm
wesm
commented
Apr 1, 2017

Copy link


Copy Markdown

Member
@leifwalsh looks like you pulled in a few unintentional commits, can you rebase on apache/master?
@leifwalsh
leifwalshforce-pushed
the
feature/java-date-time-types
branch
from
27c6853 to
49392f2CompareApril 2, 2017 21:59
@wesmwesm
mentioned this pull request
Apr 3, 2017
 Closed

julienledem
julienledem
reviewed
Apr 3, 2017
View reviewed changes
@julienledemjulienledem
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall, this looks good to me.
@jacques-n ?
@wesm
wesm
commented
Apr 3, 2017

Copy link


Copy Markdown

Member
@leifwalsh there is a conflict (hopefully not too horrible to resolve) after merging #409 -- the idea is that this will make it easier to add more types that have additional metadata
@jacques-n
jacques-n
commented
Apr 3, 2017

Copy link


Copy Markdown

Contributor
@julienledem, I think this patch further propagates the issues we've with using DateTime from Joda instead of LocalDateTime. I think those should probably be addressed separately but wanted to confirm your thoughts?
@julienledem
julienledem
commented
Apr 3, 2017

Copy link


Copy Markdown

Member
@jacques-n: right. This should probably be fixed in a separate change. I'd suggest the new vectors should just use Integer and Long in getObject().
@wesm
wesm
commented
Apr 3, 2017

Copy link


Copy Markdown

Member
I just opened https://issues.apache.org/jira/browse/ARROW-768 and made it a blocker for 0.3
@leifwalsh
leifwalsh
commented
Apr 4, 2017

Copy link


Copy Markdown

ContributorAuthor
@julienledem do you want me to change getObject() to return Integer and Long or leave the way it is for now and address later?
@leifwalsh
leifwalsh
commented
Apr 4, 2017

Copy link


Copy Markdown

ContributorAuthor
Resolved merge conflicts by rebasing on master, passes java unit and c++/java integration tests.
leifwalshand others
added 3 commits
April 3, 2017 23:23
@leifwalsh
ARROW-729: [Java] Added units for date/time types
2dca474
@wesm@leifwalsh
ARROW-733: [C++/Python] Rename FixedWidthBinary to FixedSizeBinary fo…
6e86422
…r consistency with FixedSizeList
As discussed on JIRA
Author: Wes McKinney <wes.mckinney@twosigma.com>
Closesapache#473 from wesm/ARROW-733 and squashes the following commits:
0e30af3 [Wes McKinney] Rename FixedWidthBinary to FixedSizeBinary for consistency with FixedSizeList type
@wesm@leifwalsh
Integration tests for all date and time combinations
47f83a8
Change-Id: I79f5d942f64c275c87568703f9edf6a7e89467ac
@leifwalsh
leifwalshforce-pushed
the
feature/java-date-time-types
branch
from
49392f2 to
47f83a8CompareApril 4, 2017 03:26
@julienledem
julienledem
commented
Apr 4, 2017

Copy link


Copy Markdown

Member
@leifwalsh: Yes, please make getObject() return Integer and Long (as appropriate) for new vectors.
@leifwalsh
leifwalsh
commented
Apr 4, 2017

Copy link


Copy Markdown

ContributorAuthor
@julienledem should I also do that to the existing TimeStamp* types?
@leifwalsh
leifwalsh
commented
Apr 4, 2017

Copy link


Copy Markdown

ContributorAuthor
Also, ComplexReaders.java defines read${minor.boxedType} for all the timestamp types and my new types. Should I remove that as well?
@julienledem
julienledem
commented
Apr 5, 2017

Copy link


Copy Markdown

Member
@leifwalsh I think for now it's better to do it only for new vectors. Changing the behavior of existing vectors requires more work on the code using them. Which is why I'd rather have those as separate changes.
@leifwalsh
leifwalsh
commented
Apr 5, 2017

Copy link


Copy Markdown

ContributorAuthor
Okay then I think this is good to go.
@wesm
wesm
commented
Apr 5, 2017

Copy link


Copy Markdown

Member
Cool. +1 -- thanks @leifwalsh for your contribution!
@asfgitasfgit
closed this
in
f4fcb42Apr 5, 2017
julienledem
julienledem
reviewed
Apr 6, 2017
View reviewed changes
@julienledemjulienledem
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we'd better not change the behavior of dateVector.getAccessor().getObject() just yet.

Otherwise, this looks good to me.
Comment threadjava/vector/src/main/codegen/data/ValueVectorTypes.tdd
 { class: "TimeStampMilli", javaType: "long", boxedType: "Long", friendlyType: "DateTime" }
 { class: "TimeStampMicro", javaType: "long", boxedType: "Long", friendlyType: "DateTime" }
 { class: "TimeStampNano", javaType: "long", boxedType: "Long", friendlyType: "DateTime" }
 { class: "DateMilli" },
@julienledemjulienledemApr 6, 2017

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is changing an existing vector, right?
Comment threadjava/vector/src/main/codegen/templates/FixedValueVectors.java

 </#if>

 <#if minor.class == "Date">
@julienledemjulienledemApr 6, 2017

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here we're changing the behavior of Date.
@leifwalsh
leifwalsh
commented
Apr 6, 2017
via email 

Copy link


Copy Markdown

ContributorAuthor
@wesm
wesm
commented
Apr 6, 2017

Copy link


Copy Markdown

Member
@julienledem sorry for merging early, should have waited for your final review. Let's revert what needs to be reverted for 0.3?
@wesm
wesm
commented
Apr 6, 2017

Copy link


Copy Markdown

Member
I opened https://issues.apache.org/jira/browse/ARROW-777, will leave it to you all to resolve ahead of 0.3
@leifwalsh
leifwalsh
deleted the

feature/java-date-time-types

branch
April 7, 2017 01:44

This was referenced Apr 5, 2017

Closed


Closed

pribor pushed a commit
to GlobalWebIndex/arrow
that referenced
this pull request
Oct 24, 2025
@leifwalsh@kou
ARROW-510 ARROW-582 ARROW-663 ARROW-729: [Java] Added units for Time …
58dafc2
…and Date types, and integration tests
closesapache#366
Author: Leif Walsh <leif.walsh@gmail.com>
Author: Wes McKinney <wes.mckinney@twosigma.com>
Closesapache#475 from leifwalsh/feature/java-date-time-types and squashes the following commits:
2e2a4cf [Leif Walsh] ARROW-729: [Java] removed Joda DateTime getters from Date* and Time* types
47f83a8 [Wes McKinney] Integration tests for all date and time combinations
6e86422 [Wes McKinney] ARROW-733: [C++/Python] Rename FixedWidthBinary to FixedSizeBinary for consistency with FixedSizeList
2dca474 [Leif Walsh] ARROW-729: [Java] Added units for date/time types
pribor pushed a commit
to GlobalWebIndex/arrow
that referenced
this pull request
Oct 24, 2025
@lidavidm
apacheGH-447: Port fix from apache/arrow that was missed (apache#475)
5b6ee22
apache@ded148c
was merged after the migration.
Fixesapache#447.
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@wesmwesmwesm left review comments
@julienledemjulienledemjulienledem left review comments
@cpcloudcpcloudcpcloud left review comments
@icexellossicexellossicexelloss left review comments

Assignees


No one assigned


Labels


None yet


Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

6 participants

@leifwalsh@icexelloss@wesm@jacques-n@julienledem@cpcloud

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Highlight search terms from Google/DuckDuckGo/Bing referrer\n(function() {\n var ref = document.referrer;\n var terms = [];\n \n if (ref.includes('google.com') || ref.includes('duckduckgo.com') || ref.includes('bing.com')) {\n var url = new URL(ref);\n var q = url.searchParams.get('q') || url.searchParams.get('p');\n if (q) {\n terms = q.split(/\\s+/).filter(function(t) { return t.length > 2; });\n }\n }\n \n if (terms.length === 0) return;\n \n var style = document.createElement('style');\n style.textContent = '.userscript-highlight { background: #fbbf24; color: #1a1a2e; padding: 1px 3px; border-radius: 2px; }';\n document.head.appendChild(style);\n \n function highlight(node) {\n if (node.nodeType === 3) { // text node\n var text = node.textContent;\n var found = false;\n terms.forEach(function(term) {\n var regex = new RegExp('(' + term.replace(/[.*+?^${}()|[\\]\\\\]/g, '\\\\') + ')', 'gi');\n if (regex.test(text)) {\n found = true;\n var frag = document.createDocumentFragment();\n var parts = text.split(regex);\n parts.forEach(function(part, i) {\n if (i % 2 === 0) {\n frag.appendChild(document.createTextNode(part));\n } else {\n var span = document.createElement('span');\n span.className = 'userscript-highlight';\n span.textContent = part;\n frag.appendChild(span);\n }\n });\n node.parentNode.replaceChild(frag, node);\n }\n });\n } else if (node.nodeType === 1 && node.childNodes) { // element\n var skipTags = ['SCRIPT', 'STYLE', 'NOSCRIPT', 'TEXTAREA', 'INPUT', 'SELECT'];\n if (!skipTags.includes(node.tagName)) {\n Array.from(node.childNodes).forEach(highlight);\n }\n }\n }\n \n highlight(document.body);\n \n // Re-highlight on dynamic content\n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1 || node.nodeType === 3) highlight(node);\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Highlight Search Terms");
}
} catch(__e) { console.warn('[Userscript:Highlight Search Terms]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
ARROW-510 ARROW-582 ARROW-663 ARROW-729: [Java] Added units for Time and Date types, and integration tests - #475
Closed
leifwalsh wants to merge 4 commits into
apache:masterfrom 
leifwalsh:feature/java-date-time-types
Closed
ARROW-510 ARROW-582 ARROW-663 ARROW-729: [Java] Added units for Time and Date types, and integration tests#475
leifwalsh wants to merge 4 commits into
apache:masterfrom 
leifwalsh:feature/java-date-time-types
Conversation
@leifwalsh
@leifwalshleifwalsh
commented
Apr 1, 2017•

edited

Loading

Copy link


Copy Markdown

Contributor
closes#366
@leifwalshleifwalsh
changed the title
ARROW-670: [Java] Added day and milli implementations for date, change Time to TimeMilliARROW-729: [Java] Added day and milli implementations for date, change Time to TimeMilliApr 1, 2017
@leifwalsh
leifwalshforce-pushed
the
feature/java-date-time-types
branch
from
8a85ab9 to
d10713eCompareApril 1, 2017 00:13
@icexelloss
icexelloss
commented
Apr 1, 2017

Copy link


Copy Markdown

Contributor
This is how I would do it.
@leifwalsh
leifwalsh
commented
Apr 1, 2017

Copy link


Copy Markdown

ContributorAuthor
@julienledem@wesm where are the roundtrip integration tests so I can add these to them and how do I run them?
@icexelloss
icexelloss
commented
Apr 1, 2017•

edited

Loading

Copy link


Copy Markdown

Contributor
@leifwalsh , you first want to look at the java integration test:
https://github.com/apache/arrow/blob/master/java/tools/src/main/java/org/apache/arrow/tools/Integration.java
Then there is https://github.com/apache/arrow/blob/master/integration/integration_test.py#L659 which invokes the Java integration thing.
@leifwalsh
leifwalsh
commented
Apr 1, 2017

Copy link


Copy Markdown

ContributorAuthor
It seems to work but I think the test data doesn't have any timestamps, dates, or times in it. Were the json files constructed manually or programmatically?
@wesm
wesm
commented
Apr 1, 2017

Copy link


Copy Markdown

Member
The integration tests for date/times are in the patch for ARROW-510. There's a line commented out that generates the test cases
@leifwalsh
leifwalsh
commented
Apr 1, 2017

Copy link


Copy Markdown

ContributorAuthor
@wesm awesome. I created a branch off this and merged yours into that, uncommented the test, and got a failure:
Command failed: java -cp /home/leif/git/arrow/java/tools/target/arrow-tools-0.2.1-SNAPSHOT-jar-with-dependencies.jar org.apache.arrow.tools.Integration -a /tmp/tmp4dpiwmfk/c377ba55676a45d28e09e9a5ea70b617 -j /tmp/tmpadyd70vm/628548c02e5440eba693410d754f7c31.json -c VALIDATE
With output:
--------------
12:32:16.427 [main] DEBUG i.n.u.i.l.InternalLoggerFactory - Using SLF4J as the default logging framework
12:32:16.432 [main] DEBUG io.netty.buffer.AbstractByteBuf - -Dio.netty.buffer.bytebuf.checkAccessible: true
12:32:16.434 [main] DEBUG io.netty.util.ResourceLeakDetector - -Dio.netty.leakDetection.level: simple
12:32:16.434 [main] DEBUG io.netty.util.ResourceLeakDetector - -Dio.netty.leakDetection.maxRecords: 4
12:32:16.447 [main] DEBUG i.n.util.internal.PlatformDependent0 - java.nio.Buffer.address: available
12:32:16.447 [main] DEBUG i.n.util.internal.PlatformDependent0 - sun.misc.Unsafe.theUnsafe: available
12:32:16.448 [main] DEBUG i.n.util.internal.PlatformDependent0 - sun.misc.Unsafe.copyMemory: available
12:32:16.448 [main] DEBUG i.n.util.internal.PlatformDependent0 - direct buffer constructor: available
12:32:16.449 [main] DEBUG i.n.util.internal.PlatformDependent0 - java.nio.Bits.unaligned: available, true
12:32:16.449 [main] DEBUG i.n.util.internal.PlatformDependent0 - java.nio.DirectByteBuffer.<init>(long, int): available
12:32:16.450 [main] DEBUG io.netty.util.internal.Cleaner0 - java.nio.ByteBuffer.cleaner(): available
12:32:16.451 [main] DEBUG i.n.util.internal.PlatformDependent - Java version: 8
12:32:16.451 [main] DEBUG i.n.util.internal.PlatformDependent - -Dio.netty.noUnsafe: false
12:32:16.451 [main] DEBUG i.n.util.internal.PlatformDependent - sun.misc.Unsafe: available
12:32:16.451 [main] DEBUG i.n.util.internal.PlatformDependent - -Dio.netty.noJavassist: false
12:32:16.452 [main] DEBUG i.n.util.internal.PlatformDependent - Javassist: unavailable
12:32:16.452 [main] DEBUG i.n.util.internal.PlatformDependent - You don't have Javassist in your class path or you don't have enough permission to load dynamically generated classes. Please check the configuration for better performance.
12:32:16.452 [main] DEBUG i.n.util.internal.PlatformDependent - -Dio.netty.tmpdir: /tmp (java.io.tmpdir)
12:32:16.453 [main] DEBUG i.n.util.internal.PlatformDependent - -Dio.netty.bitMode: 64 (sun.arch.data.model)
12:32:16.453 [main] DEBUG i.n.util.internal.PlatformDependent - -Dio.netty.noPreferDirect: false
12:32:16.453 [main] DEBUG i.n.util.internal.PlatformDependent - io.netty.maxDirectMemory: 1836580864 bytes
12:32:16.454 [main] DEBUG i.n.util.ResourceLeakDetectorFactory - Loaded default ResourceLeakDetector: io.netty.util.ResourceLeakDetector@1c6b6478
12:32:16.464 [main] DEBUG i.n.buffer.PooledByteBufAllocator - -Dio.netty.allocator.numHeapArenas: 8
12:32:16.464 [main] DEBUG i.n.buffer.PooledByteBufAllocator - -Dio.netty.allocator.numDirectArenas: 8
12:32:16.464 [main] DEBUG i.n.buffer.PooledByteBufAllocator - -Dio.netty.allocator.pageSize: 8192
12:32:16.465 [main] DEBUG i.n.buffer.PooledByteBufAllocator - -Dio.netty.allocator.maxOrder: 11
12:32:16.465 [main] DEBUG i.n.buffer.PooledByteBufAllocator - -Dio.netty.allocator.chunkSize: 16777216
12:32:16.465 [main] DEBUG i.n.buffer.PooledByteBufAllocator - -Dio.netty.allocator.tinyCacheSize: 512
12:32:16.465 [main] DEBUG i.n.buffer.PooledByteBufAllocator - -Dio.netty.allocator.smallCacheSize: 256
12:32:16.465 [main] DEBUG i.n.buffer.PooledByteBufAllocator - -Dio.netty.allocator.normalCacheSize: 64
12:32:16.465 [main] DEBUG i.n.buffer.PooledByteBufAllocator - -Dio.netty.allocator.maxCachedBufferCapacity: 32768
12:32:16.465 [main] DEBUG i.n.buffer.PooledByteBufAllocator - -Dio.netty.allocator.cacheTrimInterval: 8192
12:32:16.724 [main] DEBUG o.a.arrow.vector.file.ReadChannel - Reading buffer with size: 10
12:32:16.726 [main] DEBUG o.a.a.vector.file.ArrowFileReader - Footer starts at 5968, length: 1040
12:32:16.726 [main] DEBUG o.a.arrow.vector.file.ReadChannel - Reading buffer with size: 1040
12:32:16.808 [main] DEBUG org.apache.arrow.tools.Integration - Arrow Input file size: 7018
12:32:16.809 [main] DEBUG org.apache.arrow.tools.Integration - ARROW schema: Schema<f0: Date(DAY), f1: Date(MILLISECOND), f2: Time(SECOND, 0), f3: Time(MILLISECOND, 0), f4: Time(MICROSECOND, 0), f5: Time(NANOSECOND, 0), f6: Timestamp(SECOND, null), f7: Timestamp(MILLISECOND, null), f8: Timestamp(MICROSECOND, null), f9: Timestamp(NANOSECOND, null), f10: Timestamp(MILLISECOND, America/New_York)>
12:32:16.809 [main] DEBUG org.apache.arrow.tools.Integration - JSON Input file size: 15314
12:32:16.809 [main] DEBUG org.apache.arrow.tools.Integration - JSON schema: Schema<f0: Date(DAY), f1: Date(MILLISECOND), f2: Time(SECOND, 32), f3: Time(MILLISECOND, 32), f4: Time(MICROSECOND, 64), f5: Time(NANOSECOND, 64), f6: Timestamp(SECOND, null), f7: Timestamp(MILLISECOND, null), f8: Timestamp(MICROSECOND, null), f9: Timestamp(NANOSECOND, null), f10: Timestamp(MILLISECOND, America/New_York)>
Incompatible files
Different schemas:
Schema<f0: Date(DAY), f1: Date(MILLISECOND), f2: Time(SECOND, 0), f3: Time(MILLISECOND, 0), f4: Time(MICROSECOND, 0), f5: Time(NANOSECOND, 0), f6: Timestamp(SECOND, null), f7: Timestamp(MILLISECOND, null), f8: Timestamp(MICROSECOND, null), f9: Timestamp(NANOSECOND, null), f10: Timestamp(MILLISECOND, America/New_York)>
Schema<f0: Date(DAY), f1: Date(MILLISECOND), f2: Time(SECOND, 32), f3: Time(MILLISECOND, 32), f4: Time(MICROSECOND, 64), f5: Time(NANOSECOND, 64), f6: Timestamp(SECOND, null), f7: Timestamp(MILLISECOND, null), f8: Timestamp(MICROSECOND, null), f9: Timestamp(NANOSECOND, null), f10: Timestamp(MILLISECOND, America/New_York)>
12:32:16.815 [main] ERROR org.apache.arrow.tools.Integration - Incompatible files
java.lang.IllegalArgumentException: Different schemas:
Schema<f0: Date(DAY), f1: Date(MILLISECOND), f2: Time(SECOND, 0), f3: Time(MILLISECOND, 0), f4: Time(MICROSECOND, 0), f5: Time(NANOSECOND, 0), f6: Timestamp(SECOND, null), f7: Timestamp(MILLISECOND, null), f8: Timestamp(MICROSECOND, null), f9: Timestamp(NANOSECOND, null), f10: Timestamp(MILLISECOND, America/New_York)>
Schema<f0: Date(DAY), f1: Date(MILLISECOND), f2: Time(SECOND, 32), f3: Time(MILLISECOND, 32), f4: Time(MICROSECOND, 64), f5: Time(NANOSECOND, 64), f6: Timestamp(SECOND, null), f7: Timestamp(MILLISECOND, null), f8: Timestamp(MICROSECOND, null), f9: Timestamp(NANOSECOND, null), f10: Timestamp(MILLISECOND, America/New_York)>
at org.apache.arrow.vector.util.Validator.compareSchemas(Validator.java:43) ~[arrow-tools-0.2.1-SNAPSHOT-jar-with-dependencies.jar:na]
at org.apache.arrow.tools.Integration$Command$3.execute(Integration.java:177) ~[arrow-tools-0.2.1-SNAPSHOT-jar-with-dependencies.jar:na]
at org.apache.arrow.tools.Integration.run(Integration.java:101) ~[arrow-tools-0.2.1-SNAPSHOT-jar-with-dependencies.jar:na]
at org.apache.arrow.tools.Integration.main(Integration.java:62) ~[arrow-tools-0.2.1-SNAPSHOT-jar-with-dependencies.jar:na]

I'm not sure if I'm reading this right but it looks like the schema in the json file is specifying bit widths for Times but the arrow file that C++ generated sets them to 0?
@wesm
wesm
commented
Apr 1, 2017•

edited

Loading

Copy link


Copy Markdown

Member
Yeah, that looks like the case: https://github.com/apache/arrow/blob/master/cpp/src/arrow/ipc/metadata.cc#L389
Let me fix that in #458 right now and add a check to validate that the bit width is what we expect for each unit
@leifwalshleifwalsh
changed the title
ARROW-729: [Java] Added day and milli implementations for date, change Time to TimeMilliARROW-729: [Java] Added units for Time and Date typesApr 1, 2017
@wesm
wesm
commented
Apr 1, 2017

Copy link


Copy Markdown

Member
done 06c4eed
@leifwalsh
leifwalsh
commented
Apr 1, 2017

Copy link


Copy Markdown

ContributorAuthor
Okay, I implemented these types in JsonFileReader and JsonFileWriter and now I get this failure:
-- Java producing, C++ consuming
Testing file /home/leif/git/arrow/integration/data/struct_example.json
-- Creating binary inputs
-- Validating file
-- Validating stream
Testing file /home/leif/git/arrow/integration/data/simple.json
-- Creating binary inputs
-- Validating file
-- Validating stream
Testing file /tmp/tmpugxk26ua/053a211f7b8f448cb7093a44b16258c5.json
-- Creating binary inputs
-- Validating file
-- Validating stream
Testing file /tmp/tmpugxk26ua/9b7a257132e74a5f81364c88d94c1b12.json
-- Creating binary inputs
-- Validating file
-- Validating stream
Testing file /tmp/tmpugxk26ua/34173bff0e02405eb44015751d80d2cd.json
-- Creating binary inputs
-- Validating file
-- Validating stream
Testing file /tmp/tmpugxk26ua/1e0a01f57e7740a7bc715f95d6b9ea7c.json
-- Creating binary inputs
-- Validating file
Command failed: /home/leif/git/arrow/cpp/test-build/debug/json-integration-test --integration --arrow=/tmp/tmp4wae8kve/641c555737874c5e80c5f2ea5aa91179 --json=/tmp/tmpugxk26ua/1e0a01f57e7740a7bc715f95d6b9ea7c.json --mode=VALIDATE
With output:
--------------
Error message: Invalid: Record batch 0 did not match
JSON:
f0: [null, -1534653433, null, -471089378, null, null, 1726023356]
f1: [1218641917, -2011903818, -470743102, null, -1756786353, null, null]
f2: [810115012, null, null, -883898073, -1982197003, null, -1729996989]
f3: [1591774331, -314120230, null, 1196117999, -1539384497, null, 872289996]
f4: [658927759, null, null, -961381010, -1231869285, -1369355730, null]
f5: [null, 552443512, null, 2013848155, 709970719, null, -1972820288]
f6: [null, null, -102932588, 1536992262, null, null, null]
f7: [-1730827518, 1939266448, -678382409, null, null, null, null]
f8: [null, -264948505, -1250649123, 2006597095, null, 1050350308, -394676957]
f9: [33833774, null, 114513685, null, 1900471701, -669924741, 1015463225]
f10: [399170240, null, null, null, null, null, -548617532]
Arrow:
f0: [null, -1534653433, null, -471089378, null, null, 1726023356]
f1: [1218641917, -2011903818, -470743102, null, -1756786353, null, null]
f2: [810115012, null, null, -883898073, -1982197003, null, -1729996989]
f3: [1591774331, -314120230, null, 1196117999, -1539384497, null, 872289996]
f4: [658927759, null, null, -961381010, -1231869285, -1369355730, null]
f5: [null, 552443512, null, 2013848155, 709970719, null, -1972820288]
f6: [null, null, -102932588, 1536992262, null, null, null]
f7: [-1730827518, 1939266448, -678382409, null, null, null, null]
f8: [null, -264948505, -1250649123, 2006597095, null, 1050350308, -394676957]
f9: [33833774, null, 114513685, null, 1900471701, -669924741, 1015463225]
f10: [399170240, null, null, null, null, null, -548617532]

Funny thing is, those are identical according to diff.
@wesm
wesm
commented
Apr 1, 2017

Copy link


Copy Markdown

Member
Interesting. Can you pull my commits from #458 into this patch and I'll take a look to see what's up?
@leifwalsh
leifwalsh
commented
Apr 1, 2017

Copy link


Copy Markdown

ContributorAuthor
Why do you not check if right.IsNull() here? https://github.com/apache/arrow/blob/master/cpp/src/arrow/compare.cc#L472
Is the expectation that left.IsNull() iff right.IsNull()?
@leifwalsh
leifwalsh
commented
Apr 1, 2017

Copy link


Copy Markdown

ContributorAuthor
Sure.
@leifwalsh
leifwalsh
commented
Apr 1, 2017

Copy link


Copy Markdown

ContributorAuthor
Pushed
@wesm
wesm
commented
Apr 1, 2017

Copy link


Copy Markdown

Member
looking
@wesm
wesm
commented
Apr 1, 2017

Copy link


Copy Markdown

Member
Found the issue, the bit width on Date32Type in arrow/type.h was incorrect. fixed at wesm@2e5570c. integration tests pass now... awesome!
@wesmwesm
mentioned this pull request
Apr 1, 2017
 Closed

@leifwalsh
leifwalsh
commented
Apr 1, 2017
via email •

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
@wesm
wesm
commented
Apr 1, 2017

Copy link


Copy Markdown

Member
I closed my PR -- do you mind doing all the code reviews in this patch? Feel free to review the integration test / C++ stuff (the changes were pretty minimal)
@wesm
wesm
commented
Apr 1, 2017

Copy link


Copy Markdown

Member
Also, if you can add "closes #366" into your PR description then that will close that PR when this is merged
@leifwalsh
leifwalsh
commented
Apr 1, 2017
via email •

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
@leifwalshleifwalsh
changed the title
ARROW-729: [Java] Added units for Time and Date typesARROW-510 ARROW-582 ARROW-729: [Java] Added units for Time and Date types, and integration testsApr 1, 2017
@leifwalshleifwalsh
changed the title
ARROW-510 ARROW-582 ARROW-729: [Java] Added units for Time and Date types, and integration testsARROW-510 ARROW-582 ARROW-663 ARROW-729: [Java] Added units for Time and Date types, and integration testsApr 1, 2017
@icexelloss
icexelloss
commented
Apr 1, 2017

Copy link


Copy Markdown

Contributor
Wow, awesome! Looks like you guys are pretty much all set. Let me know if I can help with anything.
@leifwalsh
leifwalsh
commented
Apr 1, 2017

Copy link


Copy Markdown

ContributorAuthor
@icexelloss would appreciate review, especially of what I'm doing with micros and nanos to create joda objects here #475 (comment)
@leifwalsh
leifwalshforce-pushed
the
feature/java-date-time-types
branch
from
2e5570c to
0bfe92eCompareApril 1, 2017 17:58
@leifwalsh
leifwalsh
commented
Apr 1, 2017

Copy link


Copy Markdown

ContributorAuthor
Reorganized commits a little
@wesm
wesm
commented
Apr 1, 2017

Copy link


Copy Markdown

Member
@leifwalsh looks like you pulled in a few unintentional commits, can you rebase on apache/master?
@leifwalsh
leifwalshforce-pushed
the
feature/java-date-time-types
branch
from
27c6853 to
49392f2CompareApril 2, 2017 21:59
@wesmwesm
mentioned this pull request
Apr 3, 2017
 Closed

julienledem
julienledem
reviewed
Apr 3, 2017
View reviewed changes
@julienledemjulienledem
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall, this looks good to me.
@jacques-n ?
@wesm
wesm
commented
Apr 3, 2017

Copy link


Copy Markdown

Member
@leifwalsh there is a conflict (hopefully not too horrible to resolve) after merging #409 -- the idea is that this will make it easier to add more types that have additional metadata
@jacques-n
jacques-n
commented
Apr 3, 2017

Copy link


Copy Markdown

Contributor
@julienledem, I think this patch further propagates the issues we've with using DateTime from Joda instead of LocalDateTime. I think those should probably be addressed separately but wanted to confirm your thoughts?
@julienledem
julienledem
commented
Apr 3, 2017

Copy link


Copy Markdown

Member
@jacques-n: right. This should probably be fixed in a separate change. I'd suggest the new vectors should just use Integer and Long in getObject().
@wesm
wesm
commented
Apr 3, 2017

Copy link


Copy Markdown

Member
I just opened https://issues.apache.org/jira/browse/ARROW-768 and made it a blocker for 0.3
@leifwalsh
leifwalsh
commented
Apr 4, 2017

Copy link


Copy Markdown

ContributorAuthor
@julienledem do you want me to change getObject() to return Integer and Long or leave the way it is for now and address later?
@leifwalsh
leifwalsh
commented
Apr 4, 2017

Copy link


Copy Markdown

ContributorAuthor
Resolved merge conflicts by rebasing on master, passes java unit and c++/java integration tests.
leifwalshand others
added 3 commits
April 3, 2017 23:23
@leifwalsh
ARROW-729: [Java] Added units for date/time types
2dca474
@wesm@leifwalsh
ARROW-733: [C++/Python] Rename FixedWidthBinary to FixedSizeBinary fo…
6e86422
…r consistency with FixedSizeList
As discussed on JIRA
Author: Wes McKinney <wes.mckinney@twosigma.com>
Closesapache#473 from wesm/ARROW-733 and squashes the following commits:
0e30af3 [Wes McKinney] Rename FixedWidthBinary to FixedSizeBinary for consistency with FixedSizeList type
@wesm@leifwalsh
Integration tests for all date and time combinations
47f83a8
Change-Id: I79f5d942f64c275c87568703f9edf6a7e89467ac
@leifwalsh
leifwalshforce-pushed
the
feature/java-date-time-types
branch
from
49392f2 to
47f83a8CompareApril 4, 2017 03:26
@julienledem
julienledem
commented
Apr 4, 2017

Copy link


Copy Markdown

Member
@leifwalsh: Yes, please make getObject() return Integer and Long (as appropriate) for new vectors.
@leifwalsh
leifwalsh
commented
Apr 4, 2017

Copy link


Copy Markdown

ContributorAuthor
@julienledem should I also do that to the existing TimeStamp* types?
@leifwalsh
leifwalsh
commented
Apr 4, 2017

Copy link


Copy Markdown

ContributorAuthor
Also, ComplexReaders.java defines read${minor.boxedType} for all the timestamp types and my new types. Should I remove that as well?
@julienledem
julienledem
commented
Apr 5, 2017

Copy link


Copy Markdown

Member
@leifwalsh I think for now it's better to do it only for new vectors. Changing the behavior of existing vectors requires more work on the code using them. Which is why I'd rather have those as separate changes.
@leifwalsh
leifwalsh
commented
Apr 5, 2017

Copy link


Copy Markdown

ContributorAuthor
Okay then I think this is good to go.
@wesm
wesm
commented
Apr 5, 2017

Copy link


Copy Markdown

Member
Cool. +1 -- thanks @leifwalsh for your contribution!
@asfgitasfgit
closed this
in
f4fcb42Apr 5, 2017
julienledem
julienledem
reviewed
Apr 6, 2017
View reviewed changes
@julienledemjulienledem
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we'd better not change the behavior of dateVector.getAccessor().getObject() just yet.

Otherwise, this looks good to me.
Comment threadjava/vector/src/main/codegen/data/ValueVectorTypes.tdd
 { class: "TimeStampMilli", javaType: "long", boxedType: "Long", friendlyType: "DateTime" }
 { class: "TimeStampMicro", javaType: "long", boxedType: "Long", friendlyType: "DateTime" }
 { class: "TimeStampNano", javaType: "long", boxedType: "Long", friendlyType: "DateTime" }
 { class: "DateMilli" },
@julienledemjulienledemApr 6, 2017

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is changing an existing vector, right?
Comment threadjava/vector/src/main/codegen/templates/FixedValueVectors.java

 </#if>

 <#if minor.class == "Date">
@julienledemjulienledemApr 6, 2017

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here we're changing the behavior of Date.
@leifwalsh
leifwalsh
commented
Apr 6, 2017
via email 

Copy link


Copy Markdown

ContributorAuthor
@wesm
wesm
commented
Apr 6, 2017

Copy link


Copy Markdown

Member
@julienledem sorry for merging early, should have waited for your final review. Let's revert what needs to be reverted for 0.3?
@wesm
wesm
commented
Apr 6, 2017

Copy link


Copy Markdown

Member
I opened https://issues.apache.org/jira/browse/ARROW-777, will leave it to you all to resolve ahead of 0.3
@leifwalsh
leifwalsh
deleted the

feature/java-date-time-types

branch
April 7, 2017 01:44

This was referenced Apr 5, 2017

Closed


Closed

pribor pushed a commit
to GlobalWebIndex/arrow
that referenced
this pull request
Oct 24, 2025
@leifwalsh@kou
ARROW-510 ARROW-582 ARROW-663 ARROW-729: [Java] Added units for Time …
58dafc2
…and Date types, and integration tests
closesapache#366
Author: Leif Walsh <leif.walsh@gmail.com>
Author: Wes McKinney <wes.mckinney@twosigma.com>
Closesapache#475 from leifwalsh/feature/java-date-time-types and squashes the following commits:
2e2a4cf [Leif Walsh] ARROW-729: [Java] removed Joda DateTime getters from Date* and Time* types
47f83a8 [Wes McKinney] Integration tests for all date and time combinations
6e86422 [Wes McKinney] ARROW-733: [C++/Python] Rename FixedWidthBinary to FixedSizeBinary for consistency with FixedSizeList
2dca474 [Leif Walsh] ARROW-729: [Java] Added units for date/time types
pribor pushed a commit
to GlobalWebIndex/arrow
that referenced
this pull request
Oct 24, 2025
@lidavidm
apacheGH-447: Port fix from apache/arrow that was missed (apache#475)
5b6ee22
apache@ded148c
was merged after the migration.
Fixesapache#447.
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@wesmwesmwesm left review comments
@julienledemjulienledemjulienledem left review comments
@cpcloudcpcloudcpcloud left review comments
@icexellossicexellossicexelloss left review comments

Assignees


No one assigned


Labels


None yet


Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

6 participants

@leifwalsh@icexelloss@wesm@jacques-n@julienledem@cpcloud

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Strip utm_, fbclid, gclid, etc. from all links on page\n(function() {\n var trackingParams = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content',\n 'fbclid', 'gclid', 'dclid', 'msclkid', 'yclid',\n 'ref', 'ref_src', 'source', 'medium', 'campaign'];\n \n function cleanUrl(url) {\n try {\n var u = new URL(url, window.location.origin);\n var changed = false;\n trackingParams.forEach(function(p) {\n if (u.searchParams.has(p)) {\n u.searchParams.delete(p);\n changed = true;\n }\n });\n return changed ? u.toString() : url;\n } catch (e) {\n return url;\n }\n }\n \n function cleanLinks() {\n document.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n \n cleanLinks();\n \n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1) {\n if (node.tagName === 'A') cleanLinks();\n node.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Remove Tracking Parameters from Links");
}
} catch(__e) { console.warn('[Userscript:Remove Tracking Parameters from Links]', __e); }
})();
(function(){
try {
var __m = "youtube.com";
var __re = new RegExp('^' + "youtube\\.com" + '

Skip to content
ARROW-510 ARROW-582 ARROW-663 ARROW-729: [Java] Added units for Time and Date types, and integration tests - #475
Closed
leifwalsh wants to merge 4 commits into
apache:masterfrom 
leifwalsh:feature/java-date-time-types
Closed
ARROW-510 ARROW-582 ARROW-663 ARROW-729: [Java] Added units for Time and Date types, and integration tests#475
leifwalsh wants to merge 4 commits into
apache:masterfrom 
leifwalsh:feature/java-date-time-types
Conversation
@leifwalsh
@leifwalshleifwalsh
commented
Apr 1, 2017•

edited

Loading

Copy link


Copy Markdown

Contributor
closes#366
@leifwalshleifwalsh
changed the title
ARROW-670: [Java] Added day and milli implementations for date, change Time to TimeMilliARROW-729: [Java] Added day and milli implementations for date, change Time to TimeMilliApr 1, 2017
@leifwalsh
leifwalshforce-pushed
the
feature/java-date-time-types
branch
from
8a85ab9 to
d10713eCompareApril 1, 2017 00:13
@icexelloss
icexelloss
commented
Apr 1, 2017

Copy link


Copy Markdown

Contributor
This is how I would do it.
@leifwalsh
leifwalsh
commented
Apr 1, 2017

Copy link


Copy Markdown

ContributorAuthor
@julienledem@wesm where are the roundtrip integration tests so I can add these to them and how do I run them?
@icexelloss
icexelloss
commented
Apr 1, 2017•

edited

Loading

Copy link


Copy Markdown

Contributor
@leifwalsh , you first want to look at the java integration test:
https://github.com/apache/arrow/blob/master/java/tools/src/main/java/org/apache/arrow/tools/Integration.java
Then there is https://github.com/apache/arrow/blob/master/integration/integration_test.py#L659 which invokes the Java integration thing.
@leifwalsh
leifwalsh
commented
Apr 1, 2017

Copy link


Copy Markdown

ContributorAuthor
It seems to work but I think the test data doesn't have any timestamps, dates, or times in it. Were the json files constructed manually or programmatically?
@wesm
wesm
commented
Apr 1, 2017

Copy link


Copy Markdown

Member
The integration tests for date/times are in the patch for ARROW-510. There's a line commented out that generates the test cases
@leifwalsh
leifwalsh
commented
Apr 1, 2017

Copy link


Copy Markdown

ContributorAuthor
@wesm awesome. I created a branch off this and merged yours into that, uncommented the test, and got a failure:
Command failed: java -cp /home/leif/git/arrow/java/tools/target/arrow-tools-0.2.1-SNAPSHOT-jar-with-dependencies.jar org.apache.arrow.tools.Integration -a /tmp/tmp4dpiwmfk/c377ba55676a45d28e09e9a5ea70b617 -j /tmp/tmpadyd70vm/628548c02e5440eba693410d754f7c31.json -c VALIDATE
With output:
--------------
12:32:16.427 [main] DEBUG i.n.u.i.l.InternalLoggerFactory - Using SLF4J as the default logging framework
12:32:16.432 [main] DEBUG io.netty.buffer.AbstractByteBuf - -Dio.netty.buffer.bytebuf.checkAccessible: true
12:32:16.434 [main] DEBUG io.netty.util.ResourceLeakDetector - -Dio.netty.leakDetection.level: simple
12:32:16.434 [main] DEBUG io.netty.util.ResourceLeakDetector - -Dio.netty.leakDetection.maxRecords: 4
12:32:16.447 [main] DEBUG i.n.util.internal.PlatformDependent0 - java.nio.Buffer.address: available
12:32:16.447 [main] DEBUG i.n.util.internal.PlatformDependent0 - sun.misc.Unsafe.theUnsafe: available
12:32:16.448 [main] DEBUG i.n.util.internal.PlatformDependent0 - sun.misc.Unsafe.copyMemory: available
12:32:16.448 [main] DEBUG i.n.util.internal.PlatformDependent0 - direct buffer constructor: available
12:32:16.449 [main] DEBUG i.n.util.internal.PlatformDependent0 - java.nio.Bits.unaligned: available, true
12:32:16.449 [main] DEBUG i.n.util.internal.PlatformDependent0 - java.nio.DirectByteBuffer.<init>(long, int): available
12:32:16.450 [main] DEBUG io.netty.util.internal.Cleaner0 - java.nio.ByteBuffer.cleaner(): available
12:32:16.451 [main] DEBUG i.n.util.internal.PlatformDependent - Java version: 8
12:32:16.451 [main] DEBUG i.n.util.internal.PlatformDependent - -Dio.netty.noUnsafe: false
12:32:16.451 [main] DEBUG i.n.util.internal.PlatformDependent - sun.misc.Unsafe: available
12:32:16.451 [main] DEBUG i.n.util.internal.PlatformDependent - -Dio.netty.noJavassist: false
12:32:16.452 [main] DEBUG i.n.util.internal.PlatformDependent - Javassist: unavailable
12:32:16.452 [main] DEBUG i.n.util.internal.PlatformDependent - You don't have Javassist in your class path or you don't have enough permission to load dynamically generated classes. Please check the configuration for better performance.
12:32:16.452 [main] DEBUG i.n.util.internal.PlatformDependent - -Dio.netty.tmpdir: /tmp (java.io.tmpdir)
12:32:16.453 [main] DEBUG i.n.util.internal.PlatformDependent - -Dio.netty.bitMode: 64 (sun.arch.data.model)
12:32:16.453 [main] DEBUG i.n.util.internal.PlatformDependent - -Dio.netty.noPreferDirect: false
12:32:16.453 [main] DEBUG i.n.util.internal.PlatformDependent - io.netty.maxDirectMemory: 1836580864 bytes
12:32:16.454 [main] DEBUG i.n.util.ResourceLeakDetectorFactory - Loaded default ResourceLeakDetector: io.netty.util.ResourceLeakDetector@1c6b6478
12:32:16.464 [main] DEBUG i.n.buffer.PooledByteBufAllocator - -Dio.netty.allocator.numHeapArenas: 8
12:32:16.464 [main] DEBUG i.n.buffer.PooledByteBufAllocator - -Dio.netty.allocator.numDirectArenas: 8
12:32:16.464 [main] DEBUG i.n.buffer.PooledByteBufAllocator - -Dio.netty.allocator.pageSize: 8192
12:32:16.465 [main] DEBUG i.n.buffer.PooledByteBufAllocator - -Dio.netty.allocator.maxOrder: 11
12:32:16.465 [main] DEBUG i.n.buffer.PooledByteBufAllocator - -Dio.netty.allocator.chunkSize: 16777216
12:32:16.465 [main] DEBUG i.n.buffer.PooledByteBufAllocator - -Dio.netty.allocator.tinyCacheSize: 512
12:32:16.465 [main] DEBUG i.n.buffer.PooledByteBufAllocator - -Dio.netty.allocator.smallCacheSize: 256
12:32:16.465 [main] DEBUG i.n.buffer.PooledByteBufAllocator - -Dio.netty.allocator.normalCacheSize: 64
12:32:16.465 [main] DEBUG i.n.buffer.PooledByteBufAllocator - -Dio.netty.allocator.maxCachedBufferCapacity: 32768
12:32:16.465 [main] DEBUG i.n.buffer.PooledByteBufAllocator - -Dio.netty.allocator.cacheTrimInterval: 8192
12:32:16.724 [main] DEBUG o.a.arrow.vector.file.ReadChannel - Reading buffer with size: 10
12:32:16.726 [main] DEBUG o.a.a.vector.file.ArrowFileReader - Footer starts at 5968, length: 1040
12:32:16.726 [main] DEBUG o.a.arrow.vector.file.ReadChannel - Reading buffer with size: 1040
12:32:16.808 [main] DEBUG org.apache.arrow.tools.Integration - Arrow Input file size: 7018
12:32:16.809 [main] DEBUG org.apache.arrow.tools.Integration - ARROW schema: Schema<f0: Date(DAY), f1: Date(MILLISECOND), f2: Time(SECOND, 0), f3: Time(MILLISECOND, 0), f4: Time(MICROSECOND, 0), f5: Time(NANOSECOND, 0), f6: Timestamp(SECOND, null), f7: Timestamp(MILLISECOND, null), f8: Timestamp(MICROSECOND, null), f9: Timestamp(NANOSECOND, null), f10: Timestamp(MILLISECOND, America/New_York)>
12:32:16.809 [main] DEBUG org.apache.arrow.tools.Integration - JSON Input file size: 15314
12:32:16.809 [main] DEBUG org.apache.arrow.tools.Integration - JSON schema: Schema<f0: Date(DAY), f1: Date(MILLISECOND), f2: Time(SECOND, 32), f3: Time(MILLISECOND, 32), f4: Time(MICROSECOND, 64), f5: Time(NANOSECOND, 64), f6: Timestamp(SECOND, null), f7: Timestamp(MILLISECOND, null), f8: Timestamp(MICROSECOND, null), f9: Timestamp(NANOSECOND, null), f10: Timestamp(MILLISECOND, America/New_York)>
Incompatible files
Different schemas:
Schema<f0: Date(DAY), f1: Date(MILLISECOND), f2: Time(SECOND, 0), f3: Time(MILLISECOND, 0), f4: Time(MICROSECOND, 0), f5: Time(NANOSECOND, 0), f6: Timestamp(SECOND, null), f7: Timestamp(MILLISECOND, null), f8: Timestamp(MICROSECOND, null), f9: Timestamp(NANOSECOND, null), f10: Timestamp(MILLISECOND, America/New_York)>
Schema<f0: Date(DAY), f1: Date(MILLISECOND), f2: Time(SECOND, 32), f3: Time(MILLISECOND, 32), f4: Time(MICROSECOND, 64), f5: Time(NANOSECOND, 64), f6: Timestamp(SECOND, null), f7: Timestamp(MILLISECOND, null), f8: Timestamp(MICROSECOND, null), f9: Timestamp(NANOSECOND, null), f10: Timestamp(MILLISECOND, America/New_York)>
12:32:16.815 [main] ERROR org.apache.arrow.tools.Integration - Incompatible files
java.lang.IllegalArgumentException: Different schemas:
Schema<f0: Date(DAY), f1: Date(MILLISECOND), f2: Time(SECOND, 0), f3: Time(MILLISECOND, 0), f4: Time(MICROSECOND, 0), f5: Time(NANOSECOND, 0), f6: Timestamp(SECOND, null), f7: Timestamp(MILLISECOND, null), f8: Timestamp(MICROSECOND, null), f9: Timestamp(NANOSECOND, null), f10: Timestamp(MILLISECOND, America/New_York)>
Schema<f0: Date(DAY), f1: Date(MILLISECOND), f2: Time(SECOND, 32), f3: Time(MILLISECOND, 32), f4: Time(MICROSECOND, 64), f5: Time(NANOSECOND, 64), f6: Timestamp(SECOND, null), f7: Timestamp(MILLISECOND, null), f8: Timestamp(MICROSECOND, null), f9: Timestamp(NANOSECOND, null), f10: Timestamp(MILLISECOND, America/New_York)>
at org.apache.arrow.vector.util.Validator.compareSchemas(Validator.java:43) ~[arrow-tools-0.2.1-SNAPSHOT-jar-with-dependencies.jar:na]
at org.apache.arrow.tools.Integration$Command$3.execute(Integration.java:177) ~[arrow-tools-0.2.1-SNAPSHOT-jar-with-dependencies.jar:na]
at org.apache.arrow.tools.Integration.run(Integration.java:101) ~[arrow-tools-0.2.1-SNAPSHOT-jar-with-dependencies.jar:na]
at org.apache.arrow.tools.Integration.main(Integration.java:62) ~[arrow-tools-0.2.1-SNAPSHOT-jar-with-dependencies.jar:na]

I'm not sure if I'm reading this right but it looks like the schema in the json file is specifying bit widths for Times but the arrow file that C++ generated sets them to 0?
@wesm
wesm
commented
Apr 1, 2017•

edited

Loading

Copy link


Copy Markdown

Member
Yeah, that looks like the case: https://github.com/apache/arrow/blob/master/cpp/src/arrow/ipc/metadata.cc#L389
Let me fix that in #458 right now and add a check to validate that the bit width is what we expect for each unit
@leifwalshleifwalsh
changed the title
ARROW-729: [Java] Added day and milli implementations for date, change Time to TimeMilliARROW-729: [Java] Added units for Time and Date typesApr 1, 2017
@wesm
wesm
commented
Apr 1, 2017

Copy link


Copy Markdown

Member
done 06c4eed
@leifwalsh
leifwalsh
commented
Apr 1, 2017

Copy link


Copy Markdown

ContributorAuthor
Okay, I implemented these types in JsonFileReader and JsonFileWriter and now I get this failure:
-- Java producing, C++ consuming
Testing file /home/leif/git/arrow/integration/data/struct_example.json
-- Creating binary inputs
-- Validating file
-- Validating stream
Testing file /home/leif/git/arrow/integration/data/simple.json
-- Creating binary inputs
-- Validating file
-- Validating stream
Testing file /tmp/tmpugxk26ua/053a211f7b8f448cb7093a44b16258c5.json
-- Creating binary inputs
-- Validating file
-- Validating stream
Testing file /tmp/tmpugxk26ua/9b7a257132e74a5f81364c88d94c1b12.json
-- Creating binary inputs
-- Validating file
-- Validating stream
Testing file /tmp/tmpugxk26ua/34173bff0e02405eb44015751d80d2cd.json
-- Creating binary inputs
-- Validating file
-- Validating stream
Testing file /tmp/tmpugxk26ua/1e0a01f57e7740a7bc715f95d6b9ea7c.json
-- Creating binary inputs
-- Validating file
Command failed: /home/leif/git/arrow/cpp/test-build/debug/json-integration-test --integration --arrow=/tmp/tmp4wae8kve/641c555737874c5e80c5f2ea5aa91179 --json=/tmp/tmpugxk26ua/1e0a01f57e7740a7bc715f95d6b9ea7c.json --mode=VALIDATE
With output:
--------------
Error message: Invalid: Record batch 0 did not match
JSON:
f0: [null, -1534653433, null, -471089378, null, null, 1726023356]
f1: [1218641917, -2011903818, -470743102, null, -1756786353, null, null]
f2: [810115012, null, null, -883898073, -1982197003, null, -1729996989]
f3: [1591774331, -314120230, null, 1196117999, -1539384497, null, 872289996]
f4: [658927759, null, null, -961381010, -1231869285, -1369355730, null]
f5: [null, 552443512, null, 2013848155, 709970719, null, -1972820288]
f6: [null, null, -102932588, 1536992262, null, null, null]
f7: [-1730827518, 1939266448, -678382409, null, null, null, null]
f8: [null, -264948505, -1250649123, 2006597095, null, 1050350308, -394676957]
f9: [33833774, null, 114513685, null, 1900471701, -669924741, 1015463225]
f10: [399170240, null, null, null, null, null, -548617532]
Arrow:
f0: [null, -1534653433, null, -471089378, null, null, 1726023356]
f1: [1218641917, -2011903818, -470743102, null, -1756786353, null, null]
f2: [810115012, null, null, -883898073, -1982197003, null, -1729996989]
f3: [1591774331, -314120230, null, 1196117999, -1539384497, null, 872289996]
f4: [658927759, null, null, -961381010, -1231869285, -1369355730, null]
f5: [null, 552443512, null, 2013848155, 709970719, null, -1972820288]
f6: [null, null, -102932588, 1536992262, null, null, null]
f7: [-1730827518, 1939266448, -678382409, null, null, null, null]
f8: [null, -264948505, -1250649123, 2006597095, null, 1050350308, -394676957]
f9: [33833774, null, 114513685, null, 1900471701, -669924741, 1015463225]
f10: [399170240, null, null, null, null, null, -548617532]

Funny thing is, those are identical according to diff.
@wesm
wesm
commented
Apr 1, 2017

Copy link


Copy Markdown

Member
Interesting. Can you pull my commits from #458 into this patch and I'll take a look to see what's up?
@leifwalsh
leifwalsh
commented
Apr 1, 2017

Copy link


Copy Markdown

ContributorAuthor
Why do you not check if right.IsNull() here? https://github.com/apache/arrow/blob/master/cpp/src/arrow/compare.cc#L472
Is the expectation that left.IsNull() iff right.IsNull()?
@leifwalsh
leifwalsh
commented
Apr 1, 2017

Copy link


Copy Markdown

ContributorAuthor
Sure.
@leifwalsh
leifwalsh
commented
Apr 1, 2017

Copy link


Copy Markdown

ContributorAuthor
Pushed
@wesm
wesm
commented
Apr 1, 2017

Copy link


Copy Markdown

Member
looking
@wesm
wesm
commented
Apr 1, 2017

Copy link


Copy Markdown

Member
Found the issue, the bit width on Date32Type in arrow/type.h was incorrect. fixed at wesm@2e5570c. integration tests pass now... awesome!
@wesmwesm
mentioned this pull request
Apr 1, 2017
 Closed

@leifwalsh
leifwalsh
commented
Apr 1, 2017
via email •

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
@wesm
wesm
commented
Apr 1, 2017

Copy link


Copy Markdown

Member
I closed my PR -- do you mind doing all the code reviews in this patch? Feel free to review the integration test / C++ stuff (the changes were pretty minimal)
@wesm
wesm
commented
Apr 1, 2017

Copy link


Copy Markdown

Member
Also, if you can add "closes #366" into your PR description then that will close that PR when this is merged
@leifwalsh
leifwalsh
commented
Apr 1, 2017
via email •

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
@leifwalshleifwalsh
changed the title
ARROW-729: [Java] Added units for Time and Date typesARROW-510 ARROW-582 ARROW-729: [Java] Added units for Time and Date types, and integration testsApr 1, 2017
@leifwalshleifwalsh
changed the title
ARROW-510 ARROW-582 ARROW-729: [Java] Added units for Time and Date types, and integration testsARROW-510 ARROW-582 ARROW-663 ARROW-729: [Java] Added units for Time and Date types, and integration testsApr 1, 2017
@icexelloss
icexelloss
commented
Apr 1, 2017

Copy link


Copy Markdown

Contributor
Wow, awesome! Looks like you guys are pretty much all set. Let me know if I can help with anything.
@leifwalsh
leifwalsh
commented
Apr 1, 2017

Copy link


Copy Markdown

ContributorAuthor
@icexelloss would appreciate review, especially of what I'm doing with micros and nanos to create joda objects here #475 (comment)
@leifwalsh
leifwalshforce-pushed
the
feature/java-date-time-types
branch
from
2e5570c to
0bfe92eCompareApril 1, 2017 17:58
@leifwalsh
leifwalsh
commented
Apr 1, 2017

Copy link


Copy Markdown

ContributorAuthor
Reorganized commits a little
@wesm
wesm
commented
Apr 1, 2017

Copy link


Copy Markdown

Member
@leifwalsh looks like you pulled in a few unintentional commits, can you rebase on apache/master?
@leifwalsh
leifwalshforce-pushed
the
feature/java-date-time-types
branch
from
27c6853 to
49392f2CompareApril 2, 2017 21:59
@wesmwesm
mentioned this pull request
Apr 3, 2017
 Closed

julienledem
julienledem
reviewed
Apr 3, 2017
View reviewed changes
@julienledemjulienledem
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall, this looks good to me.
@jacques-n ?
@wesm
wesm
commented
Apr 3, 2017

Copy link


Copy Markdown

Member
@leifwalsh there is a conflict (hopefully not too horrible to resolve) after merging #409 -- the idea is that this will make it easier to add more types that have additional metadata
@jacques-n
jacques-n
commented
Apr 3, 2017

Copy link


Copy Markdown

Contributor
@julienledem, I think this patch further propagates the issues we've with using DateTime from Joda instead of LocalDateTime. I think those should probably be addressed separately but wanted to confirm your thoughts?
@julienledem
julienledem
commented
Apr 3, 2017

Copy link


Copy Markdown

Member
@jacques-n: right. This should probably be fixed in a separate change. I'd suggest the new vectors should just use Integer and Long in getObject().
@wesm
wesm
commented
Apr 3, 2017

Copy link


Copy Markdown

Member
I just opened https://issues.apache.org/jira/browse/ARROW-768 and made it a blocker for 0.3
@leifwalsh
leifwalsh
commented
Apr 4, 2017

Copy link


Copy Markdown

ContributorAuthor
@julienledem do you want me to change getObject() to return Integer and Long or leave the way it is for now and address later?
@leifwalsh
leifwalsh
commented
Apr 4, 2017

Copy link


Copy Markdown

ContributorAuthor
Resolved merge conflicts by rebasing on master, passes java unit and c++/java integration tests.
leifwalshand others
added 3 commits
April 3, 2017 23:23
@leifwalsh
ARROW-729: [Java] Added units for date/time types
2dca474
@wesm@leifwalsh
ARROW-733: [C++/Python] Rename FixedWidthBinary to FixedSizeBinary fo…
6e86422
…r consistency with FixedSizeList
As discussed on JIRA
Author: Wes McKinney <wes.mckinney@twosigma.com>
Closesapache#473 from wesm/ARROW-733 and squashes the following commits:
0e30af3 [Wes McKinney] Rename FixedWidthBinary to FixedSizeBinary for consistency with FixedSizeList type
@wesm@leifwalsh
Integration tests for all date and time combinations
47f83a8
Change-Id: I79f5d942f64c275c87568703f9edf6a7e89467ac
@leifwalsh
leifwalshforce-pushed
the
feature/java-date-time-types
branch
from
49392f2 to
47f83a8CompareApril 4, 2017 03:26
@julienledem
julienledem
commented
Apr 4, 2017

Copy link


Copy Markdown

Member
@leifwalsh: Yes, please make getObject() return Integer and Long (as appropriate) for new vectors.
@leifwalsh
leifwalsh
commented
Apr 4, 2017

Copy link


Copy Markdown

ContributorAuthor
@julienledem should I also do that to the existing TimeStamp* types?
@leifwalsh
leifwalsh
commented
Apr 4, 2017

Copy link


Copy Markdown

ContributorAuthor
Also, ComplexReaders.java defines read${minor.boxedType} for all the timestamp types and my new types. Should I remove that as well?
@julienledem
julienledem
commented
Apr 5, 2017

Copy link


Copy Markdown

Member
@leifwalsh I think for now it's better to do it only for new vectors. Changing the behavior of existing vectors requires more work on the code using them. Which is why I'd rather have those as separate changes.
@leifwalsh
leifwalsh
commented
Apr 5, 2017

Copy link


Copy Markdown

ContributorAuthor
Okay then I think this is good to go.
@wesm
wesm
commented
Apr 5, 2017

Copy link


Copy Markdown

Member
Cool. +1 -- thanks @leifwalsh for your contribution!
@asfgitasfgit
closed this
in
f4fcb42Apr 5, 2017
julienledem
julienledem
reviewed
Apr 6, 2017
View reviewed changes
@julienledemjulienledem
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we'd better not change the behavior of dateVector.getAccessor().getObject() just yet.

Otherwise, this looks good to me.
Comment threadjava/vector/src/main/codegen/data/ValueVectorTypes.tdd
 { class: "TimeStampMilli", javaType: "long", boxedType: "Long", friendlyType: "DateTime" }
 { class: "TimeStampMicro", javaType: "long", boxedType: "Long", friendlyType: "DateTime" }
 { class: "TimeStampNano", javaType: "long", boxedType: "Long", friendlyType: "DateTime" }
 { class: "DateMilli" },
@julienledemjulienledemApr 6, 2017

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is changing an existing vector, right?
Comment threadjava/vector/src/main/codegen/templates/FixedValueVectors.java

 </#if>

 <#if minor.class == "Date">
@julienledemjulienledemApr 6, 2017

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here we're changing the behavior of Date.
@leifwalsh
leifwalsh
commented
Apr 6, 2017
via email 

Copy link


Copy Markdown

ContributorAuthor
@wesm
wesm
commented
Apr 6, 2017

Copy link


Copy Markdown

Member
@julienledem sorry for merging early, should have waited for your final review. Let's revert what needs to be reverted for 0.3?
@wesm
wesm
commented
Apr 6, 2017

Copy link


Copy Markdown

Member
I opened https://issues.apache.org/jira/browse/ARROW-777, will leave it to you all to resolve ahead of 0.3
@leifwalsh
leifwalsh
deleted the

feature/java-date-time-types

branch
April 7, 2017 01:44

This was referenced Apr 5, 2017

Closed


Closed

pribor pushed a commit
to GlobalWebIndex/arrow
that referenced
this pull request
Oct 24, 2025
@leifwalsh@kou
ARROW-510 ARROW-582 ARROW-663 ARROW-729: [Java] Added units for Time …
58dafc2
…and Date types, and integration tests
closesapache#366
Author: Leif Walsh <leif.walsh@gmail.com>
Author: Wes McKinney <wes.mckinney@twosigma.com>
Closesapache#475 from leifwalsh/feature/java-date-time-types and squashes the following commits:
2e2a4cf [Leif Walsh] ARROW-729: [Java] removed Joda DateTime getters from Date* and Time* types
47f83a8 [Wes McKinney] Integration tests for all date and time combinations
6e86422 [Wes McKinney] ARROW-733: [C++/Python] Rename FixedWidthBinary to FixedSizeBinary for consistency with FixedSizeList
2dca474 [Leif Walsh] ARROW-729: [Java] Added units for date/time types
pribor pushed a commit
to GlobalWebIndex/arrow
that referenced
this pull request
Oct 24, 2025
@lidavidm
apacheGH-447: Port fix from apache/arrow that was missed (apache#475)
5b6ee22
apache@ded148c
was merged after the migration.
Fixesapache#447.
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@wesmwesmwesm left review comments
@julienledemjulienledemjulienledem left review comments
@cpcloudcpcloudcpcloud left review comments
@icexellossicexellossicexelloss left review comments

Assignees


No one assigned


Labels


None yet


Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

6 participants

@leifwalsh@icexelloss@wesm@jacques-n@julienledem@cpcloud

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Auto-enable theater mode on YouTube\n(function() {\n function tryTheater() {\n var btn = document.querySelector('button[aria-label=\"Theater mode\"], ytd-player #player button[title=\"Theater mode\"]');\n if (btn && !btn.classList.contains('activated')) {\n btn.click();\n }\n }\n \n // Try immediately\n tryTheater();\n \n // Try after navigation (SPA)\n var lastUrl = location.href;\n setInterval(function() {\n if (location.href !== lastUrl) {\n lastUrl = location.href;\n setTimeout(tryTheater, 500);\n }\n }, 1000);\n \n // Also try on player load\n var observer = new MutationObserver(tryTheater);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "YouTube Theater Mode Default");
}
} catch(__e) { console.warn('[Userscript:YouTube Theater Mode Default]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
ARROW-510 ARROW-582 ARROW-663 ARROW-729: [Java] Added units for Time and Date types, and integration tests - #475
Closed
leifwalsh wants to merge 4 commits into
apache:masterfrom 
leifwalsh:feature/java-date-time-types
Closed
ARROW-510 ARROW-582 ARROW-663 ARROW-729: [Java] Added units for Time and Date types, and integration tests#475
leifwalsh wants to merge 4 commits into
apache:masterfrom 
leifwalsh:feature/java-date-time-types
Conversation
@leifwalsh
@leifwalshleifwalsh
commented
Apr 1, 2017•

edited

Loading

Copy link


Copy Markdown

Contributor
closes#366
@leifwalshleifwalsh
changed the title
ARROW-670: [Java] Added day and milli implementations for date, change Time to TimeMilliARROW-729: [Java] Added day and milli implementations for date, change Time to TimeMilliApr 1, 2017
@leifwalsh
leifwalshforce-pushed
the
feature/java-date-time-types
branch
from
8a85ab9 to
d10713eCompareApril 1, 2017 00:13
@icexelloss
icexelloss
commented
Apr 1, 2017

Copy link


Copy Markdown

Contributor
This is how I would do it.
@leifwalsh
leifwalsh
commented
Apr 1, 2017

Copy link


Copy Markdown

ContributorAuthor
@julienledem@wesm where are the roundtrip integration tests so I can add these to them and how do I run them?
@icexelloss
icexelloss
commented
Apr 1, 2017•

edited

Loading

Copy link


Copy Markdown

Contributor
@leifwalsh , you first want to look at the java integration test:
https://github.com/apache/arrow/blob/master/java/tools/src/main/java/org/apache/arrow/tools/Integration.java
Then there is https://github.com/apache/arrow/blob/master/integration/integration_test.py#L659 which invokes the Java integration thing.
@leifwalsh
leifwalsh
commented
Apr 1, 2017

Copy link


Copy Markdown

ContributorAuthor
It seems to work but I think the test data doesn't have any timestamps, dates, or times in it. Were the json files constructed manually or programmatically?
@wesm
wesm
commented
Apr 1, 2017

Copy link


Copy Markdown

Member
The integration tests for date/times are in the patch for ARROW-510. There's a line commented out that generates the test cases
@leifwalsh
leifwalsh
commented
Apr 1, 2017

Copy link


Copy Markdown

ContributorAuthor
@wesm awesome. I created a branch off this and merged yours into that, uncommented the test, and got a failure:
Command failed: java -cp /home/leif/git/arrow/java/tools/target/arrow-tools-0.2.1-SNAPSHOT-jar-with-dependencies.jar org.apache.arrow.tools.Integration -a /tmp/tmp4dpiwmfk/c377ba55676a45d28e09e9a5ea70b617 -j /tmp/tmpadyd70vm/628548c02e5440eba693410d754f7c31.json -c VALIDATE
With output:
--------------
12:32:16.427 [main] DEBUG i.n.u.i.l.InternalLoggerFactory - Using SLF4J as the default logging framework
12:32:16.432 [main] DEBUG io.netty.buffer.AbstractByteBuf - -Dio.netty.buffer.bytebuf.checkAccessible: true
12:32:16.434 [main] DEBUG io.netty.util.ResourceLeakDetector - -Dio.netty.leakDetection.level: simple
12:32:16.434 [main] DEBUG io.netty.util.ResourceLeakDetector - -Dio.netty.leakDetection.maxRecords: 4
12:32:16.447 [main] DEBUG i.n.util.internal.PlatformDependent0 - java.nio.Buffer.address: available
12:32:16.447 [main] DEBUG i.n.util.internal.PlatformDependent0 - sun.misc.Unsafe.theUnsafe: available
12:32:16.448 [main] DEBUG i.n.util.internal.PlatformDependent0 - sun.misc.Unsafe.copyMemory: available
12:32:16.448 [main] DEBUG i.n.util.internal.PlatformDependent0 - direct buffer constructor: available
12:32:16.449 [main] DEBUG i.n.util.internal.PlatformDependent0 - java.nio.Bits.unaligned: available, true
12:32:16.449 [main] DEBUG i.n.util.internal.PlatformDependent0 - java.nio.DirectByteBuffer.<init>(long, int): available
12:32:16.450 [main] DEBUG io.netty.util.internal.Cleaner0 - java.nio.ByteBuffer.cleaner(): available
12:32:16.451 [main] DEBUG i.n.util.internal.PlatformDependent - Java version: 8
12:32:16.451 [main] DEBUG i.n.util.internal.PlatformDependent - -Dio.netty.noUnsafe: false
12:32:16.451 [main] DEBUG i.n.util.internal.PlatformDependent - sun.misc.Unsafe: available
12:32:16.451 [main] DEBUG i.n.util.internal.PlatformDependent - -Dio.netty.noJavassist: false
12:32:16.452 [main] DEBUG i.n.util.internal.PlatformDependent - Javassist: unavailable
12:32:16.452 [main] DEBUG i.n.util.internal.PlatformDependent - You don't have Javassist in your class path or you don't have enough permission to load dynamically generated classes. Please check the configuration for better performance.
12:32:16.452 [main] DEBUG i.n.util.internal.PlatformDependent - -Dio.netty.tmpdir: /tmp (java.io.tmpdir)
12:32:16.453 [main] DEBUG i.n.util.internal.PlatformDependent - -Dio.netty.bitMode: 64 (sun.arch.data.model)
12:32:16.453 [main] DEBUG i.n.util.internal.PlatformDependent - -Dio.netty.noPreferDirect: false
12:32:16.453 [main] DEBUG i.n.util.internal.PlatformDependent - io.netty.maxDirectMemory: 1836580864 bytes
12:32:16.454 [main] DEBUG i.n.util.ResourceLeakDetectorFactory - Loaded default ResourceLeakDetector: io.netty.util.ResourceLeakDetector@1c6b6478
12:32:16.464 [main] DEBUG i.n.buffer.PooledByteBufAllocator - -Dio.netty.allocator.numHeapArenas: 8
12:32:16.464 [main] DEBUG i.n.buffer.PooledByteBufAllocator - -Dio.netty.allocator.numDirectArenas: 8
12:32:16.464 [main] DEBUG i.n.buffer.PooledByteBufAllocator - -Dio.netty.allocator.pageSize: 8192
12:32:16.465 [main] DEBUG i.n.buffer.PooledByteBufAllocator - -Dio.netty.allocator.maxOrder: 11
12:32:16.465 [main] DEBUG i.n.buffer.PooledByteBufAllocator - -Dio.netty.allocator.chunkSize: 16777216
12:32:16.465 [main] DEBUG i.n.buffer.PooledByteBufAllocator - -Dio.netty.allocator.tinyCacheSize: 512
12:32:16.465 [main] DEBUG i.n.buffer.PooledByteBufAllocator - -Dio.netty.allocator.smallCacheSize: 256
12:32:16.465 [main] DEBUG i.n.buffer.PooledByteBufAllocator - -Dio.netty.allocator.normalCacheSize: 64
12:32:16.465 [main] DEBUG i.n.buffer.PooledByteBufAllocator - -Dio.netty.allocator.maxCachedBufferCapacity: 32768
12:32:16.465 [main] DEBUG i.n.buffer.PooledByteBufAllocator - -Dio.netty.allocator.cacheTrimInterval: 8192
12:32:16.724 [main] DEBUG o.a.arrow.vector.file.ReadChannel - Reading buffer with size: 10
12:32:16.726 [main] DEBUG o.a.a.vector.file.ArrowFileReader - Footer starts at 5968, length: 1040
12:32:16.726 [main] DEBUG o.a.arrow.vector.file.ReadChannel - Reading buffer with size: 1040
12:32:16.808 [main] DEBUG org.apache.arrow.tools.Integration - Arrow Input file size: 7018
12:32:16.809 [main] DEBUG org.apache.arrow.tools.Integration - ARROW schema: Schema<f0: Date(DAY), f1: Date(MILLISECOND), f2: Time(SECOND, 0), f3: Time(MILLISECOND, 0), f4: Time(MICROSECOND, 0), f5: Time(NANOSECOND, 0), f6: Timestamp(SECOND, null), f7: Timestamp(MILLISECOND, null), f8: Timestamp(MICROSECOND, null), f9: Timestamp(NANOSECOND, null), f10: Timestamp(MILLISECOND, America/New_York)>
12:32:16.809 [main] DEBUG org.apache.arrow.tools.Integration - JSON Input file size: 15314
12:32:16.809 [main] DEBUG org.apache.arrow.tools.Integration - JSON schema: Schema<f0: Date(DAY), f1: Date(MILLISECOND), f2: Time(SECOND, 32), f3: Time(MILLISECOND, 32), f4: Time(MICROSECOND, 64), f5: Time(NANOSECOND, 64), f6: Timestamp(SECOND, null), f7: Timestamp(MILLISECOND, null), f8: Timestamp(MICROSECOND, null), f9: Timestamp(NANOSECOND, null), f10: Timestamp(MILLISECOND, America/New_York)>
Incompatible files
Different schemas:
Schema<f0: Date(DAY), f1: Date(MILLISECOND), f2: Time(SECOND, 0), f3: Time(MILLISECOND, 0), f4: Time(MICROSECOND, 0), f5: Time(NANOSECOND, 0), f6: Timestamp(SECOND, null), f7: Timestamp(MILLISECOND, null), f8: Timestamp(MICROSECOND, null), f9: Timestamp(NANOSECOND, null), f10: Timestamp(MILLISECOND, America/New_York)>
Schema<f0: Date(DAY), f1: Date(MILLISECOND), f2: Time(SECOND, 32), f3: Time(MILLISECOND, 32), f4: Time(MICROSECOND, 64), f5: Time(NANOSECOND, 64), f6: Timestamp(SECOND, null), f7: Timestamp(MILLISECOND, null), f8: Timestamp(MICROSECOND, null), f9: Timestamp(NANOSECOND, null), f10: Timestamp(MILLISECOND, America/New_York)>
12:32:16.815 [main] ERROR org.apache.arrow.tools.Integration - Incompatible files
java.lang.IllegalArgumentException: Different schemas:
Schema<f0: Date(DAY), f1: Date(MILLISECOND), f2: Time(SECOND, 0), f3: Time(MILLISECOND, 0), f4: Time(MICROSECOND, 0), f5: Time(NANOSECOND, 0), f6: Timestamp(SECOND, null), f7: Timestamp(MILLISECOND, null), f8: Timestamp(MICROSECOND, null), f9: Timestamp(NANOSECOND, null), f10: Timestamp(MILLISECOND, America/New_York)>
Schema<f0: Date(DAY), f1: Date(MILLISECOND), f2: Time(SECOND, 32), f3: Time(MILLISECOND, 32), f4: Time(MICROSECOND, 64), f5: Time(NANOSECOND, 64), f6: Timestamp(SECOND, null), f7: Timestamp(MILLISECOND, null), f8: Timestamp(MICROSECOND, null), f9: Timestamp(NANOSECOND, null), f10: Timestamp(MILLISECOND, America/New_York)>
at org.apache.arrow.vector.util.Validator.compareSchemas(Validator.java:43) ~[arrow-tools-0.2.1-SNAPSHOT-jar-with-dependencies.jar:na]
at org.apache.arrow.tools.Integration$Command$3.execute(Integration.java:177) ~[arrow-tools-0.2.1-SNAPSHOT-jar-with-dependencies.jar:na]
at org.apache.arrow.tools.Integration.run(Integration.java:101) ~[arrow-tools-0.2.1-SNAPSHOT-jar-with-dependencies.jar:na]
at org.apache.arrow.tools.Integration.main(Integration.java:62) ~[arrow-tools-0.2.1-SNAPSHOT-jar-with-dependencies.jar:na]

I'm not sure if I'm reading this right but it looks like the schema in the json file is specifying bit widths for Times but the arrow file that C++ generated sets them to 0?
@wesm
wesm
commented
Apr 1, 2017•

edited

Loading

Copy link


Copy Markdown

Member
Yeah, that looks like the case: https://github.com/apache/arrow/blob/master/cpp/src/arrow/ipc/metadata.cc#L389
Let me fix that in #458 right now and add a check to validate that the bit width is what we expect for each unit
@leifwalshleifwalsh
changed the title
ARROW-729: [Java] Added day and milli implementations for date, change Time to TimeMilliARROW-729: [Java] Added units for Time and Date typesApr 1, 2017
@wesm
wesm
commented
Apr 1, 2017

Copy link


Copy Markdown

Member
done 06c4eed
@leifwalsh
leifwalsh
commented
Apr 1, 2017

Copy link


Copy Markdown

ContributorAuthor
Okay, I implemented these types in JsonFileReader and JsonFileWriter and now I get this failure:
-- Java producing, C++ consuming
Testing file /home/leif/git/arrow/integration/data/struct_example.json
-- Creating binary inputs
-- Validating file
-- Validating stream
Testing file /home/leif/git/arrow/integration/data/simple.json
-- Creating binary inputs
-- Validating file
-- Validating stream
Testing file /tmp/tmpugxk26ua/053a211f7b8f448cb7093a44b16258c5.json
-- Creating binary inputs
-- Validating file
-- Validating stream
Testing file /tmp/tmpugxk26ua/9b7a257132e74a5f81364c88d94c1b12.json
-- Creating binary inputs
-- Validating file
-- Validating stream
Testing file /tmp/tmpugxk26ua/34173bff0e02405eb44015751d80d2cd.json
-- Creating binary inputs
-- Validating file
-- Validating stream
Testing file /tmp/tmpugxk26ua/1e0a01f57e7740a7bc715f95d6b9ea7c.json
-- Creating binary inputs
-- Validating file
Command failed: /home/leif/git/arrow/cpp/test-build/debug/json-integration-test --integration --arrow=/tmp/tmp4wae8kve/641c555737874c5e80c5f2ea5aa91179 --json=/tmp/tmpugxk26ua/1e0a01f57e7740a7bc715f95d6b9ea7c.json --mode=VALIDATE
With output:
--------------
Error message: Invalid: Record batch 0 did not match
JSON:
f0: [null, -1534653433, null, -471089378, null, null, 1726023356]
f1: [1218641917, -2011903818, -470743102, null, -1756786353, null, null]
f2: [810115012, null, null, -883898073, -1982197003, null, -1729996989]
f3: [1591774331, -314120230, null, 1196117999, -1539384497, null, 872289996]
f4: [658927759, null, null, -961381010, -1231869285, -1369355730, null]
f5: [null, 552443512, null, 2013848155, 709970719, null, -1972820288]
f6: [null, null, -102932588, 1536992262, null, null, null]
f7: [-1730827518, 1939266448, -678382409, null, null, null, null]
f8: [null, -264948505, -1250649123, 2006597095, null, 1050350308, -394676957]
f9: [33833774, null, 114513685, null, 1900471701, -669924741, 1015463225]
f10: [399170240, null, null, null, null, null, -548617532]
Arrow:
f0: [null, -1534653433, null, -471089378, null, null, 1726023356]
f1: [1218641917, -2011903818, -470743102, null, -1756786353, null, null]
f2: [810115012, null, null, -883898073, -1982197003, null, -1729996989]
f3: [1591774331, -314120230, null, 1196117999, -1539384497, null, 872289996]
f4: [658927759, null, null, -961381010, -1231869285, -1369355730, null]
f5: [null, 552443512, null, 2013848155, 709970719, null, -1972820288]
f6: [null, null, -102932588, 1536992262, null, null, null]
f7: [-1730827518, 1939266448, -678382409, null, null, null, null]
f8: [null, -264948505, -1250649123, 2006597095, null, 1050350308, -394676957]
f9: [33833774, null, 114513685, null, 1900471701, -669924741, 1015463225]
f10: [399170240, null, null, null, null, null, -548617532]

Funny thing is, those are identical according to diff.
@wesm
wesm
commented
Apr 1, 2017

Copy link


Copy Markdown

Member
Interesting. Can you pull my commits from #458 into this patch and I'll take a look to see what's up?
@leifwalsh
leifwalsh
commented
Apr 1, 2017

Copy link


Copy Markdown

ContributorAuthor
Why do you not check if right.IsNull() here? https://github.com/apache/arrow/blob/master/cpp/src/arrow/compare.cc#L472
Is the expectation that left.IsNull() iff right.IsNull()?
@leifwalsh
leifwalsh
commented
Apr 1, 2017

Copy link


Copy Markdown

ContributorAuthor
Sure.
@leifwalsh
leifwalsh
commented
Apr 1, 2017

Copy link


Copy Markdown

ContributorAuthor
Pushed
@wesm
wesm
commented
Apr 1, 2017

Copy link


Copy Markdown

Member
looking
@wesm
wesm
commented
Apr 1, 2017

Copy link


Copy Markdown

Member
Found the issue, the bit width on Date32Type in arrow/type.h was incorrect. fixed at wesm@2e5570c. integration tests pass now... awesome!
@wesmwesm
mentioned this pull request
Apr 1, 2017
 Closed

@leifwalsh
leifwalsh
commented
Apr 1, 2017
via email •

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
@wesm
wesm
commented
Apr 1, 2017

Copy link


Copy Markdown

Member
I closed my PR -- do you mind doing all the code reviews in this patch? Feel free to review the integration test / C++ stuff (the changes were pretty minimal)
@wesm
wesm
commented
Apr 1, 2017

Copy link


Copy Markdown

Member
Also, if you can add "closes #366" into your PR description then that will close that PR when this is merged
@leifwalsh
leifwalsh
commented
Apr 1, 2017
via email •

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
@leifwalshleifwalsh
changed the title
ARROW-729: [Java] Added units for Time and Date typesARROW-510 ARROW-582 ARROW-729: [Java] Added units for Time and Date types, and integration testsApr 1, 2017
@leifwalshleifwalsh
changed the title
ARROW-510 ARROW-582 ARROW-729: [Java] Added units for Time and Date types, and integration testsARROW-510 ARROW-582 ARROW-663 ARROW-729: [Java] Added units for Time and Date types, and integration testsApr 1, 2017
@icexelloss
icexelloss
commented
Apr 1, 2017

Copy link


Copy Markdown

Contributor
Wow, awesome! Looks like you guys are pretty much all set. Let me know if I can help with anything.
@leifwalsh
leifwalsh
commented
Apr 1, 2017

Copy link


Copy Markdown

ContributorAuthor
@icexelloss would appreciate review, especially of what I'm doing with micros and nanos to create joda objects here #475 (comment)
@leifwalsh
leifwalshforce-pushed
the
feature/java-date-time-types
branch
from
2e5570c to
0bfe92eCompareApril 1, 2017 17:58
@leifwalsh
leifwalsh
commented
Apr 1, 2017

Copy link


Copy Markdown

ContributorAuthor
Reorganized commits a little
@wesm
wesm
commented
Apr 1, 2017

Copy link


Copy Markdown

Member
@leifwalsh looks like you pulled in a few unintentional commits, can you rebase on apache/master?
@leifwalsh
leifwalshforce-pushed
the
feature/java-date-time-types
branch
from
27c6853 to
49392f2CompareApril 2, 2017 21:59
@wesmwesm
mentioned this pull request
Apr 3, 2017
 Closed

julienledem
julienledem
reviewed
Apr 3, 2017
View reviewed changes
@julienledemjulienledem
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall, this looks good to me.
@jacques-n ?
@wesm
wesm
commented
Apr 3, 2017

Copy link


Copy Markdown

Member
@leifwalsh there is a conflict (hopefully not too horrible to resolve) after merging #409 -- the idea is that this will make it easier to add more types that have additional metadata
@jacques-n
jacques-n
commented
Apr 3, 2017

Copy link


Copy Markdown

Contributor
@julienledem, I think this patch further propagates the issues we've with using DateTime from Joda instead of LocalDateTime. I think those should probably be addressed separately but wanted to confirm your thoughts?
@julienledem
julienledem
commented
Apr 3, 2017

Copy link


Copy Markdown

Member
@jacques-n: right. This should probably be fixed in a separate change. I'd suggest the new vectors should just use Integer and Long in getObject().
@wesm
wesm
commented
Apr 3, 2017

Copy link


Copy Markdown

Member
I just opened https://issues.apache.org/jira/browse/ARROW-768 and made it a blocker for 0.3
@leifwalsh
leifwalsh
commented
Apr 4, 2017

Copy link


Copy Markdown

ContributorAuthor
@julienledem do you want me to change getObject() to return Integer and Long or leave the way it is for now and address later?
@leifwalsh
leifwalsh
commented
Apr 4, 2017

Copy link


Copy Markdown

ContributorAuthor
Resolved merge conflicts by rebasing on master, passes java unit and c++/java integration tests.
leifwalshand others
added 3 commits
April 3, 2017 23:23
@leifwalsh
ARROW-729: [Java] Added units for date/time types
2dca474
@wesm@leifwalsh
ARROW-733: [C++/Python] Rename FixedWidthBinary to FixedSizeBinary fo…
6e86422
…r consistency with FixedSizeList
As discussed on JIRA
Author: Wes McKinney <wes.mckinney@twosigma.com>
Closesapache#473 from wesm/ARROW-733 and squashes the following commits:
0e30af3 [Wes McKinney] Rename FixedWidthBinary to FixedSizeBinary for consistency with FixedSizeList type
@wesm@leifwalsh
Integration tests for all date and time combinations
47f83a8
Change-Id: I79f5d942f64c275c87568703f9edf6a7e89467ac
@leifwalsh
leifwalshforce-pushed
the
feature/java-date-time-types
branch
from
49392f2 to
47f83a8CompareApril 4, 2017 03:26
@julienledem
julienledem
commented
Apr 4, 2017

Copy link


Copy Markdown

Member
@leifwalsh: Yes, please make getObject() return Integer and Long (as appropriate) for new vectors.
@leifwalsh
leifwalsh
commented
Apr 4, 2017

Copy link


Copy Markdown

ContributorAuthor
@julienledem should I also do that to the existing TimeStamp* types?
@leifwalsh
leifwalsh
commented
Apr 4, 2017

Copy link


Copy Markdown

ContributorAuthor
Also, ComplexReaders.java defines read${minor.boxedType} for all the timestamp types and my new types. Should I remove that as well?
@julienledem
julienledem
commented
Apr 5, 2017

Copy link


Copy Markdown

Member
@leifwalsh I think for now it's better to do it only for new vectors. Changing the behavior of existing vectors requires more work on the code using them. Which is why I'd rather have those as separate changes.
@leifwalsh
leifwalsh
commented
Apr 5, 2017

Copy link


Copy Markdown

ContributorAuthor
Okay then I think this is good to go.
@wesm
wesm
commented
Apr 5, 2017

Copy link


Copy Markdown

Member
Cool. +1 -- thanks @leifwalsh for your contribution!
@asfgitasfgit
closed this
in
f4fcb42Apr 5, 2017
julienledem
julienledem
reviewed
Apr 6, 2017
View reviewed changes
@julienledemjulienledem
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we'd better not change the behavior of dateVector.getAccessor().getObject() just yet.

Otherwise, this looks good to me.
Comment threadjava/vector/src/main/codegen/data/ValueVectorTypes.tdd
 { class: "TimeStampMilli", javaType: "long", boxedType: "Long", friendlyType: "DateTime" }
 { class: "TimeStampMicro", javaType: "long", boxedType: "Long", friendlyType: "DateTime" }
 { class: "TimeStampNano", javaType: "long", boxedType: "Long", friendlyType: "DateTime" }
 { class: "DateMilli" },
@julienledemjulienledemApr 6, 2017

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is changing an existing vector, right?
Comment threadjava/vector/src/main/codegen/templates/FixedValueVectors.java

 </#if>

 <#if minor.class == "Date">
@julienledemjulienledemApr 6, 2017

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here we're changing the behavior of Date.
@leifwalsh
leifwalsh
commented
Apr 6, 2017
via email 

Copy link


Copy Markdown

ContributorAuthor
@wesm
wesm
commented
Apr 6, 2017

Copy link


Copy Markdown

Member
@julienledem sorry for merging early, should have waited for your final review. Let's revert what needs to be reverted for 0.3?
@wesm
wesm
commented
Apr 6, 2017

Copy link


Copy Markdown

Member
I opened https://issues.apache.org/jira/browse/ARROW-777, will leave it to you all to resolve ahead of 0.3
@leifwalsh
leifwalsh
deleted the

feature/java-date-time-types

branch
April 7, 2017 01:44

This was referenced Apr 5, 2017

Closed


Closed

pribor pushed a commit
to GlobalWebIndex/arrow
that referenced
this pull request
Oct 24, 2025
@leifwalsh@kou
ARROW-510 ARROW-582 ARROW-663 ARROW-729: [Java] Added units for Time …
58dafc2
…and Date types, and integration tests
closesapache#366
Author: Leif Walsh <leif.walsh@gmail.com>
Author: Wes McKinney <wes.mckinney@twosigma.com>
Closesapache#475 from leifwalsh/feature/java-date-time-types and squashes the following commits:
2e2a4cf [Leif Walsh] ARROW-729: [Java] removed Joda DateTime getters from Date* and Time* types
47f83a8 [Wes McKinney] Integration tests for all date and time combinations
6e86422 [Wes McKinney] ARROW-733: [C++/Python] Rename FixedWidthBinary to FixedSizeBinary for consistency with FixedSizeList
2dca474 [Leif Walsh] ARROW-729: [Java] Added units for date/time types
pribor pushed a commit
to GlobalWebIndex/arrow
that referenced
this pull request
Oct 24, 2025
@lidavidm
apacheGH-447: Port fix from apache/arrow that was missed (apache#475)
5b6ee22
apache@ded148c
was merged after the migration.
Fixesapache#447.
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@wesmwesmwesm left review comments
@julienledemjulienledemjulienledem left review comments
@cpcloudcpcloudcpcloud left review comments
@icexellossicexellossicexelloss left review comments

Assignees


No one assigned


Labels


None yet


Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

6 participants

@leifwalsh@icexelloss@wesm@jacques-n@julienledem@cpcloud

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Remove or un-stick sticky/fixed headers that block content\n(function() {\n function unstick() {\n document.querySelectorAll('header, nav, [role=\"banner\"], .header, .navbar, .sticky, .fixed-top, [style*=\"position: fixed\"], [style*=\"position:sticky\"]').forEach(function(el) {\n if (el.style.position === 'fixed' || el.style.position === 'sticky' || \n getComputedStyle(el).position === 'fixed' || getComputedStyle(el).position === 'sticky') {\n el.style.position = 'static';\n el.style.top = 'auto';\n el.style.zIndex = 'auto';\n }\n });\n }\n \n unstick();\n \n var observer = new MutationObserver(unstick);\n observer.observe(document.body, { childList: true, subtree: true, attributes: true, attributeFilter: ['style', 'class'] });\n})();", "Kill Sticky Headers");
}
} catch(__e) { console.warn('[Userscript:Kill Sticky Headers]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
ARROW-510 ARROW-582 ARROW-663 ARROW-729: [Java] Added units for Time and Date types, and integration tests - #475
Closed
leifwalsh wants to merge 4 commits into
apache:masterfrom 
leifwalsh:feature/java-date-time-types
Closed
ARROW-510 ARROW-582 ARROW-663 ARROW-729: [Java] Added units for Time and Date types, and integration tests#475
leifwalsh wants to merge 4 commits into
apache:masterfrom 
leifwalsh:feature/java-date-time-types
Conversation
@leifwalsh
@leifwalshleifwalsh
commented
Apr 1, 2017•

edited

Loading

Copy link


Copy Markdown

Contributor
closes#366
@leifwalshleifwalsh
changed the title
ARROW-670: [Java] Added day and milli implementations for date, change Time to TimeMilliARROW-729: [Java] Added day and milli implementations for date, change Time to TimeMilliApr 1, 2017
@leifwalsh
leifwalshforce-pushed
the
feature/java-date-time-types
branch
from
8a85ab9 to
d10713eCompareApril 1, 2017 00:13
@icexelloss
icexelloss
commented
Apr 1, 2017

Copy link


Copy Markdown

Contributor
This is how I would do it.
@leifwalsh
leifwalsh
commented
Apr 1, 2017

Copy link


Copy Markdown

ContributorAuthor
@julienledem@wesm where are the roundtrip integration tests so I can add these to them and how do I run them?
@icexelloss
icexelloss
commented
Apr 1, 2017•

edited

Loading

Copy link


Copy Markdown

Contributor
@leifwalsh , you first want to look at the java integration test:
https://github.com/apache/arrow/blob/master/java/tools/src/main/java/org/apache/arrow/tools/Integration.java
Then there is https://github.com/apache/arrow/blob/master/integration/integration_test.py#L659 which invokes the Java integration thing.
@leifwalsh
leifwalsh
commented
Apr 1, 2017

Copy link


Copy Markdown

ContributorAuthor
It seems to work but I think the test data doesn't have any timestamps, dates, or times in it. Were the json files constructed manually or programmatically?
@wesm
wesm
commented
Apr 1, 2017

Copy link


Copy Markdown

Member
The integration tests for date/times are in the patch for ARROW-510. There's a line commented out that generates the test cases
@leifwalsh
leifwalsh
commented
Apr 1, 2017

Copy link


Copy Markdown

ContributorAuthor
@wesm awesome. I created a branch off this and merged yours into that, uncommented the test, and got a failure:
Command failed: java -cp /home/leif/git/arrow/java/tools/target/arrow-tools-0.2.1-SNAPSHOT-jar-with-dependencies.jar org.apache.arrow.tools.Integration -a /tmp/tmp4dpiwmfk/c377ba55676a45d28e09e9a5ea70b617 -j /tmp/tmpadyd70vm/628548c02e5440eba693410d754f7c31.json -c VALIDATE
With output:
--------------
12:32:16.427 [main] DEBUG i.n.u.i.l.InternalLoggerFactory - Using SLF4J as the default logging framework
12:32:16.432 [main] DEBUG io.netty.buffer.AbstractByteBuf - -Dio.netty.buffer.bytebuf.checkAccessible: true
12:32:16.434 [main] DEBUG io.netty.util.ResourceLeakDetector - -Dio.netty.leakDetection.level: simple
12:32:16.434 [main] DEBUG io.netty.util.ResourceLeakDetector - -Dio.netty.leakDetection.maxRecords: 4
12:32:16.447 [main] DEBUG i.n.util.internal.PlatformDependent0 - java.nio.Buffer.address: available
12:32:16.447 [main] DEBUG i.n.util.internal.PlatformDependent0 - sun.misc.Unsafe.theUnsafe: available
12:32:16.448 [main] DEBUG i.n.util.internal.PlatformDependent0 - sun.misc.Unsafe.copyMemory: available
12:32:16.448 [main] DEBUG i.n.util.internal.PlatformDependent0 - direct buffer constructor: available
12:32:16.449 [main] DEBUG i.n.util.internal.PlatformDependent0 - java.nio.Bits.unaligned: available, true
12:32:16.449 [main] DEBUG i.n.util.internal.PlatformDependent0 - java.nio.DirectByteBuffer.<init>(long, int): available
12:32:16.450 [main] DEBUG io.netty.util.internal.Cleaner0 - java.nio.ByteBuffer.cleaner(): available
12:32:16.451 [main] DEBUG i.n.util.internal.PlatformDependent - Java version: 8
12:32:16.451 [main] DEBUG i.n.util.internal.PlatformDependent - -Dio.netty.noUnsafe: false
12:32:16.451 [main] DEBUG i.n.util.internal.PlatformDependent - sun.misc.Unsafe: available
12:32:16.451 [main] DEBUG i.n.util.internal.PlatformDependent - -Dio.netty.noJavassist: false
12:32:16.452 [main] DEBUG i.n.util.internal.PlatformDependent - Javassist: unavailable
12:32:16.452 [main] DEBUG i.n.util.internal.PlatformDependent - You don't have Javassist in your class path or you don't have enough permission to load dynamically generated classes. Please check the configuration for better performance.
12:32:16.452 [main] DEBUG i.n.util.internal.PlatformDependent - -Dio.netty.tmpdir: /tmp (java.io.tmpdir)
12:32:16.453 [main] DEBUG i.n.util.internal.PlatformDependent - -Dio.netty.bitMode: 64 (sun.arch.data.model)
12:32:16.453 [main] DEBUG i.n.util.internal.PlatformDependent - -Dio.netty.noPreferDirect: false
12:32:16.453 [main] DEBUG i.n.util.internal.PlatformDependent - io.netty.maxDirectMemory: 1836580864 bytes
12:32:16.454 [main] DEBUG i.n.util.ResourceLeakDetectorFactory - Loaded default ResourceLeakDetector: io.netty.util.ResourceLeakDetector@1c6b6478
12:32:16.464 [main] DEBUG i.n.buffer.PooledByteBufAllocator - -Dio.netty.allocator.numHeapArenas: 8
12:32:16.464 [main] DEBUG i.n.buffer.PooledByteBufAllocator - -Dio.netty.allocator.numDirectArenas: 8
12:32:16.464 [main] DEBUG i.n.buffer.PooledByteBufAllocator - -Dio.netty.allocator.pageSize: 8192
12:32:16.465 [main] DEBUG i.n.buffer.PooledByteBufAllocator - -Dio.netty.allocator.maxOrder: 11
12:32:16.465 [main] DEBUG i.n.buffer.PooledByteBufAllocator - -Dio.netty.allocator.chunkSize: 16777216
12:32:16.465 [main] DEBUG i.n.buffer.PooledByteBufAllocator - -Dio.netty.allocator.tinyCacheSize: 512
12:32:16.465 [main] DEBUG i.n.buffer.PooledByteBufAllocator - -Dio.netty.allocator.smallCacheSize: 256
12:32:16.465 [main] DEBUG i.n.buffer.PooledByteBufAllocator - -Dio.netty.allocator.normalCacheSize: 64
12:32:16.465 [main] DEBUG i.n.buffer.PooledByteBufAllocator - -Dio.netty.allocator.maxCachedBufferCapacity: 32768
12:32:16.465 [main] DEBUG i.n.buffer.PooledByteBufAllocator - -Dio.netty.allocator.cacheTrimInterval: 8192
12:32:16.724 [main] DEBUG o.a.arrow.vector.file.ReadChannel - Reading buffer with size: 10
12:32:16.726 [main] DEBUG o.a.a.vector.file.ArrowFileReader - Footer starts at 5968, length: 1040
12:32:16.726 [main] DEBUG o.a.arrow.vector.file.ReadChannel - Reading buffer with size: 1040
12:32:16.808 [main] DEBUG org.apache.arrow.tools.Integration - Arrow Input file size: 7018
12:32:16.809 [main] DEBUG org.apache.arrow.tools.Integration - ARROW schema: Schema<f0: Date(DAY), f1: Date(MILLISECOND), f2: Time(SECOND, 0), f3: Time(MILLISECOND, 0), f4: Time(MICROSECOND, 0), f5: Time(NANOSECOND, 0), f6: Timestamp(SECOND, null), f7: Timestamp(MILLISECOND, null), f8: Timestamp(MICROSECOND, null), f9: Timestamp(NANOSECOND, null), f10: Timestamp(MILLISECOND, America/New_York)>
12:32:16.809 [main] DEBUG org.apache.arrow.tools.Integration - JSON Input file size: 15314
12:32:16.809 [main] DEBUG org.apache.arrow.tools.Integration - JSON schema: Schema<f0: Date(DAY), f1: Date(MILLISECOND), f2: Time(SECOND, 32), f3: Time(MILLISECOND, 32), f4: Time(MICROSECOND, 64), f5: Time(NANOSECOND, 64), f6: Timestamp(SECOND, null), f7: Timestamp(MILLISECOND, null), f8: Timestamp(MICROSECOND, null), f9: Timestamp(NANOSECOND, null), f10: Timestamp(MILLISECOND, America/New_York)>
Incompatible files
Different schemas:
Schema<f0: Date(DAY), f1: Date(MILLISECOND), f2: Time(SECOND, 0), f3: Time(MILLISECOND, 0), f4: Time(MICROSECOND, 0), f5: Time(NANOSECOND, 0), f6: Timestamp(SECOND, null), f7: Timestamp(MILLISECOND, null), f8: Timestamp(MICROSECOND, null), f9: Timestamp(NANOSECOND, null), f10: Timestamp(MILLISECOND, America/New_York)>
Schema<f0: Date(DAY), f1: Date(MILLISECOND), f2: Time(SECOND, 32), f3: Time(MILLISECOND, 32), f4: Time(MICROSECOND, 64), f5: Time(NANOSECOND, 64), f6: Timestamp(SECOND, null), f7: Timestamp(MILLISECOND, null), f8: Timestamp(MICROSECOND, null), f9: Timestamp(NANOSECOND, null), f10: Timestamp(MILLISECOND, America/New_York)>
12:32:16.815 [main] ERROR org.apache.arrow.tools.Integration - Incompatible files
java.lang.IllegalArgumentException: Different schemas:
Schema<f0: Date(DAY), f1: Date(MILLISECOND), f2: Time(SECOND, 0), f3: Time(MILLISECOND, 0), f4: Time(MICROSECOND, 0), f5: Time(NANOSECOND, 0), f6: Timestamp(SECOND, null), f7: Timestamp(MILLISECOND, null), f8: Timestamp(MICROSECOND, null), f9: Timestamp(NANOSECOND, null), f10: Timestamp(MILLISECOND, America/New_York)>
Schema<f0: Date(DAY), f1: Date(MILLISECOND), f2: Time(SECOND, 32), f3: Time(MILLISECOND, 32), f4: Time(MICROSECOND, 64), f5: Time(NANOSECOND, 64), f6: Timestamp(SECOND, null), f7: Timestamp(MILLISECOND, null), f8: Timestamp(MICROSECOND, null), f9: Timestamp(NANOSECOND, null), f10: Timestamp(MILLISECOND, America/New_York)>
at org.apache.arrow.vector.util.Validator.compareSchemas(Validator.java:43) ~[arrow-tools-0.2.1-SNAPSHOT-jar-with-dependencies.jar:na]
at org.apache.arrow.tools.Integration$Command$3.execute(Integration.java:177) ~[arrow-tools-0.2.1-SNAPSHOT-jar-with-dependencies.jar:na]
at org.apache.arrow.tools.Integration.run(Integration.java:101) ~[arrow-tools-0.2.1-SNAPSHOT-jar-with-dependencies.jar:na]
at org.apache.arrow.tools.Integration.main(Integration.java:62) ~[arrow-tools-0.2.1-SNAPSHOT-jar-with-dependencies.jar:na]

I'm not sure if I'm reading this right but it looks like the schema in the json file is specifying bit widths for Times but the arrow file that C++ generated sets them to 0?
@wesm
wesm
commented
Apr 1, 2017•

edited

Loading

Copy link


Copy Markdown

Member
Yeah, that looks like the case: https://github.com/apache/arrow/blob/master/cpp/src/arrow/ipc/metadata.cc#L389
Let me fix that in #458 right now and add a check to validate that the bit width is what we expect for each unit
@leifwalshleifwalsh
changed the title
ARROW-729: [Java] Added day and milli implementations for date, change Time to TimeMilliARROW-729: [Java] Added units for Time and Date typesApr 1, 2017
@wesm
wesm
commented
Apr 1, 2017

Copy link


Copy Markdown

Member
done 06c4eed
@leifwalsh
leifwalsh
commented
Apr 1, 2017

Copy link


Copy Markdown

ContributorAuthor
Okay, I implemented these types in JsonFileReader and JsonFileWriter and now I get this failure:
-- Java producing, C++ consuming
Testing file /home/leif/git/arrow/integration/data/struct_example.json
-- Creating binary inputs
-- Validating file
-- Validating stream
Testing file /home/leif/git/arrow/integration/data/simple.json
-- Creating binary inputs
-- Validating file
-- Validating stream
Testing file /tmp/tmpugxk26ua/053a211f7b8f448cb7093a44b16258c5.json
-- Creating binary inputs
-- Validating file
-- Validating stream
Testing file /tmp/tmpugxk26ua/9b7a257132e74a5f81364c88d94c1b12.json
-- Creating binary inputs
-- Validating file
-- Validating stream
Testing file /tmp/tmpugxk26ua/34173bff0e02405eb44015751d80d2cd.json
-- Creating binary inputs
-- Validating file
-- Validating stream
Testing file /tmp/tmpugxk26ua/1e0a01f57e7740a7bc715f95d6b9ea7c.json
-- Creating binary inputs
-- Validating file
Command failed: /home/leif/git/arrow/cpp/test-build/debug/json-integration-test --integration --arrow=/tmp/tmp4wae8kve/641c555737874c5e80c5f2ea5aa91179 --json=/tmp/tmpugxk26ua/1e0a01f57e7740a7bc715f95d6b9ea7c.json --mode=VALIDATE
With output:
--------------
Error message: Invalid: Record batch 0 did not match
JSON:
f0: [null, -1534653433, null, -471089378, null, null, 1726023356]
f1: [1218641917, -2011903818, -470743102, null, -1756786353, null, null]
f2: [810115012, null, null, -883898073, -1982197003, null, -1729996989]
f3: [1591774331, -314120230, null, 1196117999, -1539384497, null, 872289996]
f4: [658927759, null, null, -961381010, -1231869285, -1369355730, null]
f5: [null, 552443512, null, 2013848155, 709970719, null, -1972820288]
f6: [null, null, -102932588, 1536992262, null, null, null]
f7: [-1730827518, 1939266448, -678382409, null, null, null, null]
f8: [null, -264948505, -1250649123, 2006597095, null, 1050350308, -394676957]
f9: [33833774, null, 114513685, null, 1900471701, -669924741, 1015463225]
f10: [399170240, null, null, null, null, null, -548617532]
Arrow:
f0: [null, -1534653433, null, -471089378, null, null, 1726023356]
f1: [1218641917, -2011903818, -470743102, null, -1756786353, null, null]
f2: [810115012, null, null, -883898073, -1982197003, null, -1729996989]
f3: [1591774331, -314120230, null, 1196117999, -1539384497, null, 872289996]
f4: [658927759, null, null, -961381010, -1231869285, -1369355730, null]
f5: [null, 552443512, null, 2013848155, 709970719, null, -1972820288]
f6: [null, null, -102932588, 1536992262, null, null, null]
f7: [-1730827518, 1939266448, -678382409, null, null, null, null]
f8: [null, -264948505, -1250649123, 2006597095, null, 1050350308, -394676957]
f9: [33833774, null, 114513685, null, 1900471701, -669924741, 1015463225]
f10: [399170240, null, null, null, null, null, -548617532]

Funny thing is, those are identical according to diff.
@wesm
wesm
commented
Apr 1, 2017

Copy link


Copy Markdown

Member
Interesting. Can you pull my commits from #458 into this patch and I'll take a look to see what's up?
@leifwalsh
leifwalsh
commented
Apr 1, 2017

Copy link


Copy Markdown

ContributorAuthor
Why do you not check if right.IsNull() here? https://github.com/apache/arrow/blob/master/cpp/src/arrow/compare.cc#L472
Is the expectation that left.IsNull() iff right.IsNull()?
@leifwalsh
leifwalsh
commented
Apr 1, 2017

Copy link


Copy Markdown

ContributorAuthor
Sure.
@leifwalsh
leifwalsh
commented
Apr 1, 2017

Copy link


Copy Markdown

ContributorAuthor
Pushed
@wesm
wesm
commented
Apr 1, 2017

Copy link


Copy Markdown

Member
looking
@wesm
wesm
commented
Apr 1, 2017

Copy link


Copy Markdown

Member
Found the issue, the bit width on Date32Type in arrow/type.h was incorrect. fixed at wesm@2e5570c. integration tests pass now... awesome!
@wesmwesm
mentioned this pull request
Apr 1, 2017
 Closed

@leifwalsh
leifwalsh
commented
Apr 1, 2017
via email •

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
@wesm
wesm
commented
Apr 1, 2017

Copy link


Copy Markdown

Member
I closed my PR -- do you mind doing all the code reviews in this patch? Feel free to review the integration test / C++ stuff (the changes were pretty minimal)
@wesm
wesm
commented
Apr 1, 2017

Copy link


Copy Markdown

Member
Also, if you can add "closes #366" into your PR description then that will close that PR when this is merged
@leifwalsh
leifwalsh
commented
Apr 1, 2017
via email •

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
@leifwalshleifwalsh
changed the title
ARROW-729: [Java] Added units for Time and Date typesARROW-510 ARROW-582 ARROW-729: [Java] Added units for Time and Date types, and integration testsApr 1, 2017
@leifwalshleifwalsh
changed the title
ARROW-510 ARROW-582 ARROW-729: [Java] Added units for Time and Date types, and integration testsARROW-510 ARROW-582 ARROW-663 ARROW-729: [Java] Added units for Time and Date types, and integration testsApr 1, 2017
@icexelloss
icexelloss
commented
Apr 1, 2017

Copy link


Copy Markdown

Contributor
Wow, awesome! Looks like you guys are pretty much all set. Let me know if I can help with anything.
@leifwalsh
leifwalsh
commented
Apr 1, 2017

Copy link


Copy Markdown

ContributorAuthor
@icexelloss would appreciate review, especially of what I'm doing with micros and nanos to create joda objects here #475 (comment)
@leifwalsh
leifwalshforce-pushed
the
feature/java-date-time-types
branch
from
2e5570c to
0bfe92eCompareApril 1, 2017 17:58
@leifwalsh
leifwalsh
commented
Apr 1, 2017

Copy link


Copy Markdown

ContributorAuthor
Reorganized commits a little
@wesm
wesm
commented
Apr 1, 2017

Copy link


Copy Markdown

Member
@leifwalsh looks like you pulled in a few unintentional commits, can you rebase on apache/master?
@leifwalsh
leifwalshforce-pushed
the
feature/java-date-time-types
branch
from
27c6853 to
49392f2CompareApril 2, 2017 21:59
@wesmwesm
mentioned this pull request
Apr 3, 2017
 Closed

julienledem
julienledem
reviewed
Apr 3, 2017
View reviewed changes
@julienledemjulienledem
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall, this looks good to me.
@jacques-n ?
@wesm
wesm
commented
Apr 3, 2017

Copy link


Copy Markdown

Member
@leifwalsh there is a conflict (hopefully not too horrible to resolve) after merging #409 -- the idea is that this will make it easier to add more types that have additional metadata
@jacques-n
jacques-n
commented
Apr 3, 2017

Copy link


Copy Markdown

Contributor
@julienledem, I think this patch further propagates the issues we've with using DateTime from Joda instead of LocalDateTime. I think those should probably be addressed separately but wanted to confirm your thoughts?
@julienledem
julienledem
commented
Apr 3, 2017

Copy link


Copy Markdown

Member
@jacques-n: right. This should probably be fixed in a separate change. I'd suggest the new vectors should just use Integer and Long in getObject().
@wesm
wesm
commented
Apr 3, 2017

Copy link


Copy Markdown

Member
I just opened https://issues.apache.org/jira/browse/ARROW-768 and made it a blocker for 0.3
@leifwalsh
leifwalsh
commented
Apr 4, 2017

Copy link


Copy Markdown

ContributorAuthor
@julienledem do you want me to change getObject() to return Integer and Long or leave the way it is for now and address later?
@leifwalsh
leifwalsh
commented
Apr 4, 2017

Copy link


Copy Markdown

ContributorAuthor
Resolved merge conflicts by rebasing on master, passes java unit and c++/java integration tests.
leifwalshand others
added 3 commits
April 3, 2017 23:23
@leifwalsh
ARROW-729: [Java] Added units for date/time types
2dca474
@wesm@leifwalsh
ARROW-733: [C++/Python] Rename FixedWidthBinary to FixedSizeBinary fo…
6e86422
…r consistency with FixedSizeList
As discussed on JIRA
Author: Wes McKinney <wes.mckinney@twosigma.com>
Closesapache#473 from wesm/ARROW-733 and squashes the following commits:
0e30af3 [Wes McKinney] Rename FixedWidthBinary to FixedSizeBinary for consistency with FixedSizeList type
@wesm@leifwalsh
Integration tests for all date and time combinations
47f83a8
Change-Id: I79f5d942f64c275c87568703f9edf6a7e89467ac
@leifwalsh
leifwalshforce-pushed
the
feature/java-date-time-types
branch
from
49392f2 to
47f83a8CompareApril 4, 2017 03:26
@julienledem
julienledem
commented
Apr 4, 2017

Copy link


Copy Markdown

Member
@leifwalsh: Yes, please make getObject() return Integer and Long (as appropriate) for new vectors.
@leifwalsh
leifwalsh
commented
Apr 4, 2017

Copy link


Copy Markdown

ContributorAuthor
@julienledem should I also do that to the existing TimeStamp* types?
@leifwalsh
leifwalsh
commented
Apr 4, 2017

Copy link


Copy Markdown

ContributorAuthor
Also, ComplexReaders.java defines read${minor.boxedType} for all the timestamp types and my new types. Should I remove that as well?
@julienledem
julienledem
commented
Apr 5, 2017

Copy link


Copy Markdown

Member
@leifwalsh I think for now it's better to do it only for new vectors. Changing the behavior of existing vectors requires more work on the code using them. Which is why I'd rather have those as separate changes.
@leifwalsh
leifwalsh
commented
Apr 5, 2017

Copy link


Copy Markdown

ContributorAuthor
Okay then I think this is good to go.
@wesm
wesm
commented
Apr 5, 2017

Copy link


Copy Markdown

Member
Cool. +1 -- thanks @leifwalsh for your contribution!
@asfgitasfgit
closed this
in
f4fcb42Apr 5, 2017
julienledem
julienledem
reviewed
Apr 6, 2017
View reviewed changes
@julienledemjulienledem
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we'd better not change the behavior of dateVector.getAccessor().getObject() just yet.

Otherwise, this looks good to me.
Comment threadjava/vector/src/main/codegen/data/ValueVectorTypes.tdd
 { class: "TimeStampMilli", javaType: "long", boxedType: "Long", friendlyType: "DateTime" }
 { class: "TimeStampMicro", javaType: "long", boxedType: "Long", friendlyType: "DateTime" }
 { class: "TimeStampNano", javaType: "long", boxedType: "Long", friendlyType: "DateTime" }
 { class: "DateMilli" },
@julienledemjulienledemApr 6, 2017

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is changing an existing vector, right?
Comment threadjava/vector/src/main/codegen/templates/FixedValueVectors.java

 </#if>

 <#if minor.class == "Date">
@julienledemjulienledemApr 6, 2017

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here we're changing the behavior of Date.
@leifwalsh
leifwalsh
commented
Apr 6, 2017
via email 

Copy link


Copy Markdown

ContributorAuthor
@wesm
wesm
commented
Apr 6, 2017

Copy link


Copy Markdown

Member
@julienledem sorry for merging early, should have waited for your final review. Let's revert what needs to be reverted for 0.3?
@wesm
wesm
commented
Apr 6, 2017

Copy link


Copy Markdown

Member
I opened https://issues.apache.org/jira/browse/ARROW-777, will leave it to you all to resolve ahead of 0.3
@leifwalsh
leifwalsh
deleted the

feature/java-date-time-types

branch
April 7, 2017 01:44

This was referenced Apr 5, 2017

Closed


Closed

pribor pushed a commit
to GlobalWebIndex/arrow
that referenced
this pull request
Oct 24, 2025
@leifwalsh@kou
ARROW-510 ARROW-582 ARROW-663 ARROW-729: [Java] Added units for Time …
58dafc2
…and Date types, and integration tests
closesapache#366
Author: Leif Walsh <leif.walsh@gmail.com>
Author: Wes McKinney <wes.mckinney@twosigma.com>
Closesapache#475 from leifwalsh/feature/java-date-time-types and squashes the following commits:
2e2a4cf [Leif Walsh] ARROW-729: [Java] removed Joda DateTime getters from Date* and Time* types
47f83a8 [Wes McKinney] Integration tests for all date and time combinations
6e86422 [Wes McKinney] ARROW-733: [C++/Python] Rename FixedWidthBinary to FixedSizeBinary for consistency with FixedSizeList
2dca474 [Leif Walsh] ARROW-729: [Java] Added units for date/time types
pribor pushed a commit
to GlobalWebIndex/arrow
that referenced
this pull request
Oct 24, 2025
@lidavidm
apacheGH-447: Port fix from apache/arrow that was missed (apache#475)
5b6ee22
apache@ded148c
was merged after the migration.
Fixesapache#447.
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@wesmwesmwesm left review comments
@julienledemjulienledemjulienledem left review comments
@cpcloudcpcloudcpcloud left review comments
@icexellossicexellossicexelloss left review comments

Assignees


No one assigned


Labels


None yet


Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

6 participants

@leifwalsh@icexelloss@wesm@jacques-n@julienledem@cpcloud

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Universal Dark Mode - works on any site\n(function() {\n var enabled = true;\n \n function applyDarkMode() {\n if (!enabled) return;\n \n // Create style element if it doesn't exist\n var style = document.getElementById('universal-dark-mode-style');\n if (!style) {\n style = document.createElement('style');\n style.id = 'universal-dark-mode-style';\n document.head.appendChild(style);\n }\n \n // Dark mode CSS - inverts colors but preserves images/video\n style.textContent = '\n /* Invert everything except media */\n html {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #1a1a2e !important;\n }\n \n /* Restore images, videos, iframes, canvas */\n img, video, iframe, canvas, svg, picture, [style*=\"background-image\"] {\n filter: invert(1) hue-rotate(180deg) !important;\n }\n \n /* Preserve specific elements that should not be inverted */\n .no-dark-mode, .no-dark-mode *,\n [data-theme=\"light\"], [data-theme=\"light\"],\n .ace_editor, .ace_editor *,\n .CodeMirror, .CodeMirror *,\n .monaco-editor, .monaco-editor *,\n .markdown-body pre, .markdown-body pre *,\n .highlight, .highlight *,\n pre code, pre code * {\n filter: none !important;\n }\n \n /* Fix common UI elements */\n .modal, .popup, .dropdown-menu, .tooltip, .popover {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #2d2d44 !important;\n border-color: #444 !important;\n }\n \n /* Scrollbars */\n ::-webkit-scrollbar { background: #1a1a2e !important; }\n ::-webkit-scrollbar-thumb { background: #444 !important; }\n ::-webkit-scrollbar-thumb:hover { background: #555 !important; }\n \n /* Selection */\n ::selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ::-moz-selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ';\n }\n \n function removeDarkMode() {\n var style = document.getElementById('universal-dark-mode-style');\n if (style) style.remove();\n }\n \n // Toggle with Alt+Shift+D\n document.addEventListener('keydown', function(e) {\n if (e.altKey && e.shiftKey && e.key === 'D') {\n e.preventDefault();\n enabled = !enabled;\n if (enabled) {\n applyDarkMode();\n console.log('[Universal Dark Mode] Enabled');\n } else {\n removeDarkMode();\n console.log('[Universal Dark Mode] Disabled');\n }\n }\n });\n \n // Apply on load\n applyDarkMode();\n \n // Re-apply on dynamic content\n var observer = new MutationObserver(function(mutations) {\n if (enabled && !document.getElementById('universal-dark-mode-style')) {\n applyDarkMode();\n }\n });\n observer.observe(document.head, { childList: true });\n \n console.log('[Universal Dark Mode] Loaded - Press Alt+Shift+D to toggle');\n})();", "Universal Dark Mode");
}
} catch(__e) { console.warn('[Userscript:Universal Dark Mode]', __e); }
})();
})();

Skip to content
ARROW-510 ARROW-582 ARROW-663 ARROW-729: [Java] Added units for Time and Date types, and integration tests - #475
Closed
leifwalsh wants to merge 4 commits into
apache:masterfrom 
leifwalsh:feature/java-date-time-types
Closed
ARROW-510 ARROW-582 ARROW-663 ARROW-729: [Java] Added units for Time and Date types, and integration tests#475
leifwalsh wants to merge 4 commits into
apache:masterfrom 
leifwalsh:feature/java-date-time-types
Conversation
@leifwalsh
@leifwalshleifwalsh
commented
Apr 1, 2017•

edited

Loading

Copy link


Copy Markdown

Contributor
closes#366
@leifwalshleifwalsh
changed the title
ARROW-670: [Java] Added day and milli implementations for date, change Time to TimeMilliARROW-729: [Java] Added day and milli implementations for date, change Time to TimeMilliApr 1, 2017
@leifwalsh
leifwalshforce-pushed
the
feature/java-date-time-types
branch
from
8a85ab9 to
d10713eCompareApril 1, 2017 00:13
@icexelloss
icexelloss
commented
Apr 1, 2017

Copy link


Copy Markdown

Contributor
This is how I would do it.
@leifwalsh
leifwalsh
commented
Apr 1, 2017

Copy link


Copy Markdown

ContributorAuthor
@julienledem@wesm where are the roundtrip integration tests so I can add these to them and how do I run them?
@icexelloss
icexelloss
commented
Apr 1, 2017•

edited

Loading

Copy link


Copy Markdown

Contributor
@leifwalsh , you first want to look at the java integration test:
https://github.com/apache/arrow/blob/master/java/tools/src/main/java/org/apache/arrow/tools/Integration.java
Then there is https://github.com/apache/arrow/blob/master/integration/integration_test.py#L659 which invokes the Java integration thing.
@leifwalsh
leifwalsh
commented
Apr 1, 2017

Copy link


Copy Markdown

ContributorAuthor
It seems to work but I think the test data doesn't have any timestamps, dates, or times in it. Were the json files constructed manually or programmatically?
@wesm
wesm
commented
Apr 1, 2017

Copy link


Copy Markdown

Member
The integration tests for date/times are in the patch for ARROW-510. There's a line commented out that generates the test cases
@leifwalsh
leifwalsh
commented
Apr 1, 2017

Copy link


Copy Markdown

ContributorAuthor
@wesm awesome. I created a branch off this and merged yours into that, uncommented the test, and got a failure:
Command failed: java -cp /home/leif/git/arrow/java/tools/target/arrow-tools-0.2.1-SNAPSHOT-jar-with-dependencies.jar org.apache.arrow.tools.Integration -a /tmp/tmp4dpiwmfk/c377ba55676a45d28e09e9a5ea70b617 -j /tmp/tmpadyd70vm/628548c02e5440eba693410d754f7c31.json -c VALIDATE
With output:
--------------
12:32:16.427 [main] DEBUG i.n.u.i.l.InternalLoggerFactory - Using SLF4J as the default logging framework
12:32:16.432 [main] DEBUG io.netty.buffer.AbstractByteBuf - -Dio.netty.buffer.bytebuf.checkAccessible: true
12:32:16.434 [main] DEBUG io.netty.util.ResourceLeakDetector - -Dio.netty.leakDetection.level: simple
12:32:16.434 [main] DEBUG io.netty.util.ResourceLeakDetector - -Dio.netty.leakDetection.maxRecords: 4
12:32:16.447 [main] DEBUG i.n.util.internal.PlatformDependent0 - java.nio.Buffer.address: available
12:32:16.447 [main] DEBUG i.n.util.internal.PlatformDependent0 - sun.misc.Unsafe.theUnsafe: available
12:32:16.448 [main] DEBUG i.n.util.internal.PlatformDependent0 - sun.misc.Unsafe.copyMemory: available
12:32:16.448 [main] DEBUG i.n.util.internal.PlatformDependent0 - direct buffer constructor: available
12:32:16.449 [main] DEBUG i.n.util.internal.PlatformDependent0 - java.nio.Bits.unaligned: available, true
12:32:16.449 [main] DEBUG i.n.util.internal.PlatformDependent0 - java.nio.DirectByteBuffer.<init>(long, int): available
12:32:16.450 [main] DEBUG io.netty.util.internal.Cleaner0 - java.nio.ByteBuffer.cleaner(): available
12:32:16.451 [main] DEBUG i.n.util.internal.PlatformDependent - Java version: 8
12:32:16.451 [main] DEBUG i.n.util.internal.PlatformDependent - -Dio.netty.noUnsafe: false
12:32:16.451 [main] DEBUG i.n.util.internal.PlatformDependent - sun.misc.Unsafe: available
12:32:16.451 [main] DEBUG i.n.util.internal.PlatformDependent - -Dio.netty.noJavassist: false
12:32:16.452 [main] DEBUG i.n.util.internal.PlatformDependent - Javassist: unavailable
12:32:16.452 [main] DEBUG i.n.util.internal.PlatformDependent - You don't have Javassist in your class path or you don't have enough permission to load dynamically generated classes. Please check the configuration for better performance.
12:32:16.452 [main] DEBUG i.n.util.internal.PlatformDependent - -Dio.netty.tmpdir: /tmp (java.io.tmpdir)
12:32:16.453 [main] DEBUG i.n.util.internal.PlatformDependent - -Dio.netty.bitMode: 64 (sun.arch.data.model)
12:32:16.453 [main] DEBUG i.n.util.internal.PlatformDependent - -Dio.netty.noPreferDirect: false
12:32:16.453 [main] DEBUG i.n.util.internal.PlatformDependent - io.netty.maxDirectMemory: 1836580864 bytes
12:32:16.454 [main] DEBUG i.n.util.ResourceLeakDetectorFactory - Loaded default ResourceLeakDetector: io.netty.util.ResourceLeakDetector@1c6b6478
12:32:16.464 [main] DEBUG i.n.buffer.PooledByteBufAllocator - -Dio.netty.allocator.numHeapArenas: 8
12:32:16.464 [main] DEBUG i.n.buffer.PooledByteBufAllocator - -Dio.netty.allocator.numDirectArenas: 8
12:32:16.464 [main] DEBUG i.n.buffer.PooledByteBufAllocator - -Dio.netty.allocator.pageSize: 8192
12:32:16.465 [main] DEBUG i.n.buffer.PooledByteBufAllocator - -Dio.netty.allocator.maxOrder: 11
12:32:16.465 [main] DEBUG i.n.buffer.PooledByteBufAllocator - -Dio.netty.allocator.chunkSize: 16777216
12:32:16.465 [main] DEBUG i.n.buffer.PooledByteBufAllocator - -Dio.netty.allocator.tinyCacheSize: 512
12:32:16.465 [main] DEBUG i.n.buffer.PooledByteBufAllocator - -Dio.netty.allocator.smallCacheSize: 256
12:32:16.465 [main] DEBUG i.n.buffer.PooledByteBufAllocator - -Dio.netty.allocator.normalCacheSize: 64
12:32:16.465 [main] DEBUG i.n.buffer.PooledByteBufAllocator - -Dio.netty.allocator.maxCachedBufferCapacity: 32768
12:32:16.465 [main] DEBUG i.n.buffer.PooledByteBufAllocator - -Dio.netty.allocator.cacheTrimInterval: 8192
12:32:16.724 [main] DEBUG o.a.arrow.vector.file.ReadChannel - Reading buffer with size: 10
12:32:16.726 [main] DEBUG o.a.a.vector.file.ArrowFileReader - Footer starts at 5968, length: 1040
12:32:16.726 [main] DEBUG o.a.arrow.vector.file.ReadChannel - Reading buffer with size: 1040
12:32:16.808 [main] DEBUG org.apache.arrow.tools.Integration - Arrow Input file size: 7018
12:32:16.809 [main] DEBUG org.apache.arrow.tools.Integration - ARROW schema: Schema<f0: Date(DAY), f1: Date(MILLISECOND), f2: Time(SECOND, 0), f3: Time(MILLISECOND, 0), f4: Time(MICROSECOND, 0), f5: Time(NANOSECOND, 0), f6: Timestamp(SECOND, null), f7: Timestamp(MILLISECOND, null), f8: Timestamp(MICROSECOND, null), f9: Timestamp(NANOSECOND, null), f10: Timestamp(MILLISECOND, America/New_York)>
12:32:16.809 [main] DEBUG org.apache.arrow.tools.Integration - JSON Input file size: 15314
12:32:16.809 [main] DEBUG org.apache.arrow.tools.Integration - JSON schema: Schema<f0: Date(DAY), f1: Date(MILLISECOND), f2: Time(SECOND, 32), f3: Time(MILLISECOND, 32), f4: Time(MICROSECOND, 64), f5: Time(NANOSECOND, 64), f6: Timestamp(SECOND, null), f7: Timestamp(MILLISECOND, null), f8: Timestamp(MICROSECOND, null), f9: Timestamp(NANOSECOND, null), f10: Timestamp(MILLISECOND, America/New_York)>
Incompatible files
Different schemas:
Schema<f0: Date(DAY), f1: Date(MILLISECOND), f2: Time(SECOND, 0), f3: Time(MILLISECOND, 0), f4: Time(MICROSECOND, 0), f5: Time(NANOSECOND, 0), f6: Timestamp(SECOND, null), f7: Timestamp(MILLISECOND, null), f8: Timestamp(MICROSECOND, null), f9: Timestamp(NANOSECOND, null), f10: Timestamp(MILLISECOND, America/New_York)>
Schema<f0: Date(DAY), f1: Date(MILLISECOND), f2: Time(SECOND, 32), f3: Time(MILLISECOND, 32), f4: Time(MICROSECOND, 64), f5: Time(NANOSECOND, 64), f6: Timestamp(SECOND, null), f7: Timestamp(MILLISECOND, null), f8: Timestamp(MICROSECOND, null), f9: Timestamp(NANOSECOND, null), f10: Timestamp(MILLISECOND, America/New_York)>
12:32:16.815 [main] ERROR org.apache.arrow.tools.Integration - Incompatible files
java.lang.IllegalArgumentException: Different schemas:
Schema<f0: Date(DAY), f1: Date(MILLISECOND), f2: Time(SECOND, 0), f3: Time(MILLISECOND, 0), f4: Time(MICROSECOND, 0), f5: Time(NANOSECOND, 0), f6: Timestamp(SECOND, null), f7: Timestamp(MILLISECOND, null), f8: Timestamp(MICROSECOND, null), f9: Timestamp(NANOSECOND, null), f10: Timestamp(MILLISECOND, America/New_York)>
Schema<f0: Date(DAY), f1: Date(MILLISECOND), f2: Time(SECOND, 32), f3: Time(MILLISECOND, 32), f4: Time(MICROSECOND, 64), f5: Time(NANOSECOND, 64), f6: Timestamp(SECOND, null), f7: Timestamp(MILLISECOND, null), f8: Timestamp(MICROSECOND, null), f9: Timestamp(NANOSECOND, null), f10: Timestamp(MILLISECOND, America/New_York)>
at org.apache.arrow.vector.util.Validator.compareSchemas(Validator.java:43) ~[arrow-tools-0.2.1-SNAPSHOT-jar-with-dependencies.jar:na]
at org.apache.arrow.tools.Integration$Command$3.execute(Integration.java:177) ~[arrow-tools-0.2.1-SNAPSHOT-jar-with-dependencies.jar:na]
at org.apache.arrow.tools.Integration.run(Integration.java:101) ~[arrow-tools-0.2.1-SNAPSHOT-jar-with-dependencies.jar:na]
at org.apache.arrow.tools.Integration.main(Integration.java:62) ~[arrow-tools-0.2.1-SNAPSHOT-jar-with-dependencies.jar:na]

I'm not sure if I'm reading this right but it looks like the schema in the json file is specifying bit widths for Times but the arrow file that C++ generated sets them to 0?
@wesm
wesm
commented
Apr 1, 2017•

edited

Loading

Copy link


Copy Markdown

Member
Yeah, that looks like the case: https://github.com/apache/arrow/blob/master/cpp/src/arrow/ipc/metadata.cc#L389
Let me fix that in #458 right now and add a check to validate that the bit width is what we expect for each unit
@leifwalshleifwalsh
changed the title
ARROW-729: [Java] Added day and milli implementations for date, change Time to TimeMilliARROW-729: [Java] Added units for Time and Date typesApr 1, 2017
@wesm
wesm
commented
Apr 1, 2017

Copy link


Copy Markdown

Member
done 06c4eed
@leifwalsh
leifwalsh
commented
Apr 1, 2017

Copy link


Copy Markdown

ContributorAuthor
Okay, I implemented these types in JsonFileReader and JsonFileWriter and now I get this failure:
-- Java producing, C++ consuming
Testing file /home/leif/git/arrow/integration/data/struct_example.json
-- Creating binary inputs
-- Validating file
-- Validating stream
Testing file /home/leif/git/arrow/integration/data/simple.json
-- Creating binary inputs
-- Validating file
-- Validating stream
Testing file /tmp/tmpugxk26ua/053a211f7b8f448cb7093a44b16258c5.json
-- Creating binary inputs
-- Validating file
-- Validating stream
Testing file /tmp/tmpugxk26ua/9b7a257132e74a5f81364c88d94c1b12.json
-- Creating binary inputs
-- Validating file
-- Validating stream
Testing file /tmp/tmpugxk26ua/34173bff0e02405eb44015751d80d2cd.json
-- Creating binary inputs
-- Validating file
-- Validating stream
Testing file /tmp/tmpugxk26ua/1e0a01f57e7740a7bc715f95d6b9ea7c.json
-- Creating binary inputs
-- Validating file
Command failed: /home/leif/git/arrow/cpp/test-build/debug/json-integration-test --integration --arrow=/tmp/tmp4wae8kve/641c555737874c5e80c5f2ea5aa91179 --json=/tmp/tmpugxk26ua/1e0a01f57e7740a7bc715f95d6b9ea7c.json --mode=VALIDATE
With output:
--------------
Error message: Invalid: Record batch 0 did not match
JSON:
f0: [null, -1534653433, null, -471089378, null, null, 1726023356]
f1: [1218641917, -2011903818, -470743102, null, -1756786353, null, null]
f2: [810115012, null, null, -883898073, -1982197003, null, -1729996989]
f3: [1591774331, -314120230, null, 1196117999, -1539384497, null, 872289996]
f4: [658927759, null, null, -961381010, -1231869285, -1369355730, null]
f5: [null, 552443512, null, 2013848155, 709970719, null, -1972820288]
f6: [null, null, -102932588, 1536992262, null, null, null]
f7: [-1730827518, 1939266448, -678382409, null, null, null, null]
f8: [null, -264948505, -1250649123, 2006597095, null, 1050350308, -394676957]
f9: [33833774, null, 114513685, null, 1900471701, -669924741, 1015463225]
f10: [399170240, null, null, null, null, null, -548617532]
Arrow:
f0: [null, -1534653433, null, -471089378, null, null, 1726023356]
f1: [1218641917, -2011903818, -470743102, null, -1756786353, null, null]
f2: [810115012, null, null, -883898073, -1982197003, null, -1729996989]
f3: [1591774331, -314120230, null, 1196117999, -1539384497, null, 872289996]
f4: [658927759, null, null, -961381010, -1231869285, -1369355730, null]
f5: [null, 552443512, null, 2013848155, 709970719, null, -1972820288]
f6: [null, null, -102932588, 1536992262, null, null, null]
f7: [-1730827518, 1939266448, -678382409, null, null, null, null]
f8: [null, -264948505, -1250649123, 2006597095, null, 1050350308, -394676957]
f9: [33833774, null, 114513685, null, 1900471701, -669924741, 1015463225]
f10: [399170240, null, null, null, null, null, -548617532]

Funny thing is, those are identical according to diff.
@wesm
wesm
commented
Apr 1, 2017

Copy link


Copy Markdown

Member
Interesting. Can you pull my commits from #458 into this patch and I'll take a look to see what's up?
@leifwalsh
leifwalsh
commented
Apr 1, 2017

Copy link


Copy Markdown

ContributorAuthor
Why do you not check if right.IsNull() here? https://github.com/apache/arrow/blob/master/cpp/src/arrow/compare.cc#L472
Is the expectation that left.IsNull() iff right.IsNull()?
@leifwalsh
leifwalsh
commented
Apr 1, 2017

Copy link


Copy Markdown

ContributorAuthor
Sure.
@leifwalsh
leifwalsh
commented
Apr 1, 2017

Copy link


Copy Markdown

ContributorAuthor
Pushed
@wesm
wesm
commented
Apr 1, 2017

Copy link


Copy Markdown

Member
looking
@wesm
wesm
commented
Apr 1, 2017

Copy link


Copy Markdown

Member
Found the issue, the bit width on Date32Type in arrow/type.h was incorrect. fixed at wesm@2e5570c. integration tests pass now... awesome!
@wesmwesm
mentioned this pull request
Apr 1, 2017
 Closed

@leifwalsh
leifwalsh
commented
Apr 1, 2017
via email •

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
@wesm
wesm
commented
Apr 1, 2017

Copy link


Copy Markdown

Member
I closed my PR -- do you mind doing all the code reviews in this patch? Feel free to review the integration test / C++ stuff (the changes were pretty minimal)
@wesm
wesm
commented
Apr 1, 2017

Copy link


Copy Markdown

Member
Also, if you can add "closes #366" into your PR description then that will close that PR when this is merged
@leifwalsh
leifwalsh
commented
Apr 1, 2017
via email •

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
@leifwalshleifwalsh
changed the title
ARROW-729: [Java] Added units for Time and Date typesARROW-510 ARROW-582 ARROW-729: [Java] Added units for Time and Date types, and integration testsApr 1, 2017
@leifwalshleifwalsh
changed the title
ARROW-510 ARROW-582 ARROW-729: [Java] Added units for Time and Date types, and integration testsARROW-510 ARROW-582 ARROW-663 ARROW-729: [Java] Added units for Time and Date types, and integration testsApr 1, 2017
@icexelloss
icexelloss
commented
Apr 1, 2017

Copy link


Copy Markdown

Contributor
Wow, awesome! Looks like you guys are pretty much all set. Let me know if I can help with anything.
@leifwalsh
leifwalsh
commented
Apr 1, 2017

Copy link


Copy Markdown

ContributorAuthor
@icexelloss would appreciate review, especially of what I'm doing with micros and nanos to create joda objects here #475 (comment)
@leifwalsh
leifwalshforce-pushed
the
feature/java-date-time-types
branch
from
2e5570c to
0bfe92eCompareApril 1, 2017 17:58
@leifwalsh
leifwalsh
commented
Apr 1, 2017

Copy link


Copy Markdown

ContributorAuthor
Reorganized commits a little
@wesm
wesm
commented
Apr 1, 2017

Copy link


Copy Markdown

Member
@leifwalsh looks like you pulled in a few unintentional commits, can you rebase on apache/master?
@leifwalsh
leifwalshforce-pushed
the
feature/java-date-time-types
branch
from
27c6853 to
49392f2CompareApril 2, 2017 21:59
@wesmwesm
mentioned this pull request
Apr 3, 2017
 Closed

julienledem
julienledem
reviewed
Apr 3, 2017
View reviewed changes
@julienledemjulienledem
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall, this looks good to me.
@jacques-n ?
@wesm
wesm
commented
Apr 3, 2017

Copy link


Copy Markdown

Member
@leifwalsh there is a conflict (hopefully not too horrible to resolve) after merging #409 -- the idea is that this will make it easier to add more types that have additional metadata
@jacques-n
jacques-n
commented
Apr 3, 2017

Copy link


Copy Markdown

Contributor
@julienledem, I think this patch further propagates the issues we've with using DateTime from Joda instead of LocalDateTime. I think those should probably be addressed separately but wanted to confirm your thoughts?
@julienledem
julienledem
commented
Apr 3, 2017

Copy link


Copy Markdown

Member
@jacques-n: right. This should probably be fixed in a separate change. I'd suggest the new vectors should just use Integer and Long in getObject().
@wesm
wesm
commented
Apr 3, 2017

Copy link


Copy Markdown

Member
I just opened https://issues.apache.org/jira/browse/ARROW-768 and made it a blocker for 0.3
@leifwalsh
leifwalsh
commented
Apr 4, 2017

Copy link


Copy Markdown

ContributorAuthor
@julienledem do you want me to change getObject() to return Integer and Long or leave the way it is for now and address later?
@leifwalsh
leifwalsh
commented
Apr 4, 2017

Copy link


Copy Markdown

ContributorAuthor
Resolved merge conflicts by rebasing on master, passes java unit and c++/java integration tests.
leifwalshand others
added 3 commits
April 3, 2017 23:23
@leifwalsh
ARROW-729: [Java] Added units for date/time types
2dca474
@wesm@leifwalsh
ARROW-733: [C++/Python] Rename FixedWidthBinary to FixedSizeBinary fo…
6e86422
…r consistency with FixedSizeList
As discussed on JIRA
Author: Wes McKinney <wes.mckinney@twosigma.com>
Closesapache#473 from wesm/ARROW-733 and squashes the following commits:
0e30af3 [Wes McKinney] Rename FixedWidthBinary to FixedSizeBinary for consistency with FixedSizeList type
@wesm@leifwalsh
Integration tests for all date and time combinations
47f83a8
Change-Id: I79f5d942f64c275c87568703f9edf6a7e89467ac
@leifwalsh
leifwalshforce-pushed
the
feature/java-date-time-types
branch
from
49392f2 to
47f83a8CompareApril 4, 2017 03:26
@julienledem
julienledem
commented
Apr 4, 2017

Copy link


Copy Markdown

Member
@leifwalsh: Yes, please make getObject() return Integer and Long (as appropriate) for new vectors.
@leifwalsh
leifwalsh
commented
Apr 4, 2017

Copy link


Copy Markdown

ContributorAuthor
@julienledem should I also do that to the existing TimeStamp* types?
@leifwalsh
leifwalsh
commented
Apr 4, 2017

Copy link


Copy Markdown

ContributorAuthor
Also, ComplexReaders.java defines read${minor.boxedType} for all the timestamp types and my new types. Should I remove that as well?
@julienledem
julienledem
commented
Apr 5, 2017

Copy link


Copy Markdown

Member
@leifwalsh I think for now it's better to do it only for new vectors. Changing the behavior of existing vectors requires more work on the code using them. Which is why I'd rather have those as separate changes.
@leifwalsh
leifwalsh
commented
Apr 5, 2017

Copy link


Copy Markdown

ContributorAuthor
Okay then I think this is good to go.
@wesm
wesm
commented
Apr 5, 2017

Copy link


Copy Markdown

Member
Cool. +1 -- thanks @leifwalsh for your contribution!
@asfgitasfgit
closed this
in
f4fcb42Apr 5, 2017
julienledem
julienledem
reviewed
Apr 6, 2017
View reviewed changes
@julienledemjulienledem
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we'd better not change the behavior of dateVector.getAccessor().getObject() just yet.

Otherwise, this looks good to me.
Comment threadjava/vector/src/main/codegen/data/ValueVectorTypes.tdd
 { class: "TimeStampMilli", javaType: "long", boxedType: "Long", friendlyType: "DateTime" }
 { class: "TimeStampMicro", javaType: "long", boxedType: "Long", friendlyType: "DateTime" }
 { class: "TimeStampNano", javaType: "long", boxedType: "Long", friendlyType: "DateTime" }
 { class: "DateMilli" },
@julienledemjulienledemApr 6, 2017

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is changing an existing vector, right?
Comment threadjava/vector/src/main/codegen/templates/FixedValueVectors.java

 </#if>

 <#if minor.class == "Date">
@julienledemjulienledemApr 6, 2017

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here we're changing the behavior of Date.
@leifwalsh
leifwalsh
commented
Apr 6, 2017
via email 

Copy link


Copy Markdown

ContributorAuthor
@wesm
wesm
commented
Apr 6, 2017

Copy link


Copy Markdown

Member
@julienledem sorry for merging early, should have waited for your final review. Let's revert what needs to be reverted for 0.3?
@wesm
wesm
commented
Apr 6, 2017

Copy link


Copy Markdown

Member
I opened https://issues.apache.org/jira/browse/ARROW-777, will leave it to you all to resolve ahead of 0.3
@leifwalsh
leifwalsh
deleted the

feature/java-date-time-types

branch
April 7, 2017 01:44

This was referenced Apr 5, 2017

Closed


Closed

pribor pushed a commit
to GlobalWebIndex/arrow
that referenced
this pull request
Oct 24, 2025
@leifwalsh@kou
ARROW-510 ARROW-582 ARROW-663 ARROW-729: [Java] Added units for Time …
58dafc2
…and Date types, and integration tests
closesapache#366
Author: Leif Walsh <leif.walsh@gmail.com>
Author: Wes McKinney <wes.mckinney@twosigma.com>
Closesapache#475 from leifwalsh/feature/java-date-time-types and squashes the following commits:
2e2a4cf [Leif Walsh] ARROW-729: [Java] removed Joda DateTime getters from Date* and Time* types
47f83a8 [Wes McKinney] Integration tests for all date and time combinations
6e86422 [Wes McKinney] ARROW-733: [C++/Python] Rename FixedWidthBinary to FixedSizeBinary for consistency with FixedSizeList
2dca474 [Leif Walsh] ARROW-729: [Java] Added units for date/time types
pribor pushed a commit
to GlobalWebIndex/arrow
that referenced
this pull request
Oct 24, 2025
@lidavidm
apacheGH-447: Port fix from apache/arrow that was missed (apache#475)
5b6ee22
apache@ded148c
was merged after the migration.
Fixesapache#447.
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@wesmwesmwesm left review comments
@julienledemjulienledemjulienledem left review comments
@cpcloudcpcloudcpcloud left review comments
@icexellossicexellossicexelloss left review comments

Assignees


No one assigned


Labels


None yet


Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

6 participants

@leifwalsh@icexelloss@wesm@jacques-n@julienledem@cpcloud