Time type is not specified clearly

[asfimport]

2 options:

• Use 64 bits for microseconds and nanoseconds, 32 bits for other units
• Use 64 bits for everything
  The latter is simpler to implement, the former saves space.

Reporter: Julien Le Dem / @julienledem
Assignee: Wes McKinney / @wesm

Related issues:

• Add integration tests for date and time types (blocks)

_{Note: This issue was originally created as ARROW-617. Please see the migration documentation for further details.}

[asfimport]

Julien Le Dem / @julienledem:
Currently the java TimeVector supports millis only on 32 bits.
the c++ Time Vector is 64bits.

[asfimport]

Wes McKinney / @wesm:
Practically speaking, I'm OK with having both a "time64" and "time32" data type with a particular resolution/unit, e.g.

• time64 would be 64 bits per value and accommodate microseconds and nanoseconds. For compatibility, we could also allow Time64 to hold both seconds and milliseconds

• time32 would be milliseconds and seconds only

If it makes things too complicated, we could constrain 64-bit time to be only micro and nanoseconds.

Thoughts?

[asfimport]

Uwe Korn / @xhochy:
From the C++/Python side it actually looks to me that having a Time32 and a Time64 type with all 4 units would be the easiest solution, so I would be fine with that, too.

[asfimport]

Wes McKinney / @wesm:
Just to confirm this is what you meant: we would need to forbid micro/nanoseconds in time32.

[asfimport]

Uwe Korn / @xhochy:
@wesm actually I didn't mean that but after a night's sleep: yes. Time64 can be seconds/milli-/micro-/nanoseconds and Time32 seconds/milliseconds.

[asfimport]

Steven Phillips / @StevenMPhillips:
I personally strongly favor simplicity, so if it were up to me, I would choose to go with a single physical type, even though that means we would have to resolve compatibility issues with existing code.

I would actually go a step further and ask if we really need all 4 (nano/micro/milli/seconds). Couldn't we just store nanoseconds in all cases? Are we concerned about the cost of conversion?

[asfimport]

Jacques Nadeau / @jacques-n:
I propose two:

32bit micro and 64bit nano.

Agree with simplicity. However, I also think that paying for an extra 4 bytes for the rare case is suboptimal.

I'd be surprised if the base 10 shift for the other versions would be a huge problem for most systems.

[asfimport]

Julien Le Dem / @julienledem:
log2(1000 * 1000 * 3600 *24) = 36.3... so micros don't fit in 32 bits.
that'd be:

• 32 bits for second and millisecond precision
• 64 bits for microsecond and nanosecond precision
  I would rather make it hard requirement and not allow 64 bits for sec/millis for simplicity. That way we don't have to specify 2 types (Time32 and Time64). We just use the smallest precision that works for Time based on the unit.

[asfimport]

Bryan Cutler / @BryanCutler:
I am ok with having 2 time types if it helps with performance/zero copies. Will it be something that is transparent to the Arrow user and the storage size is determined automatically by the unit, or will it be unit-less and the user would pick the appropriate type, either Time32 or Time64? I think I'd prefer the latter because I would want to know what's going on behind the scenes.

[asfimport]

Julien Le Dem / @julienledem:
The layout in entirely defined by the Type definition which in this instance would include the unit. That's what TypeLayout is for on the java side. So that would be fixed and clear to the user

[asfimport]

Wes McKinney / @wesm:

Couldn't we just store nanoseconds in all cases? Are we concerned about the cost of conversion?

The trouble is that sometimes you are interacting with "someone else's memory". So if you encounter a C array that represents time, it may be anywhere from second to nanosecond resolution in practice. I've just run into this problem with dates and added a date32 type to the C++ library so that I can interact with days-since-UNIX-epoch as int32_t with zero copy.

The most compatible solution that maximizes ability to do zero-copy reference to memory is

• 64 bit type permitting any unit (second through nanosecond)
• 32 bit type permitting only seconds and milliseconds

In the format documents I think we should encourage only using 64 bits for micro and nanoseconds, and 32 bits otherwise.

[asfimport]

Wes McKinney / @wesm:
We could choose to disallow 64 bits for second and millisecond for now (for ease of implementation) and wait and see if it becomes a problem. Would that work for everyone?

[asfimport]

Julien Le Dem / @julienledem:
@wesm that would work but the type needs to fully specify the physical layout. If the unit is used to infer the value bit width (32 or 64 bits) then we can't change in the future. So the type would be something like this?

table Time {
 unit: TimeUnit;
 bitWidth: int; // restricted to 32 for sec and millis, and 64 for micros and nanos in v1
}

Possibly we can further limit to millis and nanos only in v1 since they are respectively the most precise for each bitWIdth (millis: 32bits, nanos: 64 bits)
This is a little verbose but explicit and future proof.

[asfimport]

Wes McKinney / @wesm:
This sound good to me. Let me put up a patch for this now

[asfimport]

Wes McKinney / @wesm:
PR: https://github.com/apache/arrow/pull/385/files