ARROW-9944: [Rust][DataFusion] Implement to_timestamp function

[alamb]

This PR adds:

1. An implementation of the 1 argument form of the TO_TIMESTAMP function that takes a single argument (string) and converts it into a nanosecond precision timestamp, as described in the proposal doc
2. An explicit dependence on the chrono library. Note that this does not add any actual new dependencies for datafusion because depends on arrow which already depends on chrono source link.

fyi @jorgecarleitao@andygrove@emkornfield@jhorstmann

[github-actions]

https://issues.apache.org/jira/browse/ARROW-9944

[andygrove]

LGTM!

[andygrove]

nit: Although the planner and type coercion logic should prevent errors here, we could actually return an Err result rather than panic.

[alamb]

Will do

[jorgecarleitao]

LGTM.

[jorgecarleitao]

add s that failed, or just a slice of it, here?

[alamb]

good idea -- I will do so

[jorgecarleitao]

I think that we get a massive performance boost if we build ArrayData here (learning from @nevi-me :P).

@nevi-me did this recently here, e.g.

let num_rows = args[0].len();let string_args = &args[0].as_any().downcast_ref::<StringArray>().expect("input cast to StringArray failed");let result = (0..num_rows).map(|i| string_to_timestamp_nanos(string_args.value(i))).collect::<Result<Vec<i64>>>()?;let data = ArrayData::new(DataType::Timestamp(TimeUnit::Nanosecond,None),
num_rows,Some(string_args.null_count()),
string_args.data().null_buffer().cloned(),0,vec![Buffer::from(result.to_byte_slice())],vec![],);Ok(make_array(Arc::new(data)))

[jhorstmann]

I'd be interested in a benchmark of this function or the kernel, maybe one for the happy case with T and Z and one for the last fallback. From my experience with Java, parsing timestamps can be rather slow and it might be worth writing a specialized implementation.

The behaviour regarding time zones should also be documented, does it default to a local time zone or to always to UTC?

Lastly, storing nanoseconds as i64 gives a range of about year 1677 to 2262, this might be a limitation for currently very few usecases but should also be mentioned.

[alamb]

I filed https://issues.apache.org/jira/browse/ARROW-9955 to track the (excellent) suggestion for benchmark and possible improvement based on results. I agree efficient timestamp parsing is a bit of an art form.

I will also document both the timezone behavior as well as the date range limitations

[alamb]

@andygrove -- I have rebased this PR and I believe I have implemented all suggestions. There is one more lingering thing regarding parsing local timestamps that I want to check out but I can also do that after merge if you prefer.

[alamb]

This PR adds:

1. An implementation of the 1 argument form of the TO_TIMESTAMP function that takes a single argument (string) and converts it into a nanosecond precision timestamp, as described in the proposal doc
2. An explicit dependence on the chrono library. Note that this does not add any actual new dependencies for datafusion because depends on arrow which already depends on chrono source link.

fyi @jorgecarleitao@andygrove@emkornfield@jhorstmann

[github-actions]

https://issues.apache.org/jira/browse/ARROW-9944

[andygrove]

LGTM!

[andygrove]

nit: Although the planner and type coercion logic should prevent errors here, we could actually return an Err result rather than panic.

[alamb]

Will do

[jorgecarleitao]

LGTM.

[jorgecarleitao]

add s that failed, or just a slice of it, here?

[alamb]

good idea -- I will do so

[jorgecarleitao]

I think that we get a massive performance boost if we build ArrayData here (learning from @nevi-me :P).

@nevi-me did this recently here, e.g.

let num_rows = args[0].len();let string_args = &args[0].as_any().downcast_ref::<StringArray>().expect("input cast to StringArray failed");let result = (0..num_rows).map(|i| string_to_timestamp_nanos(string_args.value(i))).collect::<Result<Vec<i64>>>()?;let data = ArrayData::new(DataType::Timestamp(TimeUnit::Nanosecond,None),
num_rows,Some(string_args.null_count()),
string_args.data().null_buffer().cloned(),0,vec![Buffer::from(result.to_byte_slice())],vec![],);Ok(make_array(Arc::new(data)))

[jhorstmann]

I'd be interested in a benchmark of this function or the kernel, maybe one for the happy case with T and Z and one for the last fallback. From my experience with Java, parsing timestamps can be rather slow and it might be worth writing a specialized implementation.

The behaviour regarding time zones should also be documented, does it default to a local time zone or to always to UTC?

Lastly, storing nanoseconds as i64 gives a range of about year 1677 to 2262, this might be a limitation for currently very few usecases but should also be mentioned.

[alamb]

I filed https://issues.apache.org/jira/browse/ARROW-9955 to track the (excellent) suggestion for benchmark and possible improvement based on results. I agree efficient timestamp parsing is a bit of an art form.

I will also document both the timezone behavior as well as the date range limitations

[alamb]

@andygrove -- I have rebased this PR and I believe I have implemented all suggestions. There is one more lingering thing regarding parsing local timestamps that I want to check out but I can also do that after merge if you prefer.

[alamb]

This PR adds:

1. An implementation of the 1 argument form of the TO_TIMESTAMP function that takes a single argument (string) and converts it into a nanosecond precision timestamp, as described in the proposal doc
2. An explicit dependence on the chrono library. Note that this does not add any actual new dependencies for datafusion because depends on arrow which already depends on chrono source link.

fyi @jorgecarleitao@andygrove@emkornfield@jhorstmann

[github-actions]

https://issues.apache.org/jira/browse/ARROW-9944

[andygrove]

LGTM!

[andygrove]

nit: Although the planner and type coercion logic should prevent errors here, we could actually return an Err result rather than panic.

[alamb]

Will do

[jorgecarleitao]

LGTM.

[jorgecarleitao]

add s that failed, or just a slice of it, here?

[alamb]

good idea -- I will do so

[jorgecarleitao]

I think that we get a massive performance boost if we build ArrayData here (learning from @nevi-me :P).

@nevi-me did this recently here, e.g.

let num_rows = args[0].len();let string_args = &args[0].as_any().downcast_ref::<StringArray>().expect("input cast to StringArray failed");let result = (0..num_rows).map(|i| string_to_timestamp_nanos(string_args.value(i))).collect::<Result<Vec<i64>>>()?;let data = ArrayData::new(DataType::Timestamp(TimeUnit::Nanosecond,None),
num_rows,Some(string_args.null_count()),
string_args.data().null_buffer().cloned(),0,vec![Buffer::from(result.to_byte_slice())],vec![],);Ok(make_array(Arc::new(data)))

[jhorstmann]

I'd be interested in a benchmark of this function or the kernel, maybe one for the happy case with T and Z and one for the last fallback. From my experience with Java, parsing timestamps can be rather slow and it might be worth writing a specialized implementation.

The behaviour regarding time zones should also be documented, does it default to a local time zone or to always to UTC?

Lastly, storing nanoseconds as i64 gives a range of about year 1677 to 2262, this might be a limitation for currently very few usecases but should also be mentioned.

[alamb]

I filed https://issues.apache.org/jira/browse/ARROW-9955 to track the (excellent) suggestion for benchmark and possible improvement based on results. I agree efficient timestamp parsing is a bit of an art form.

I will also document both the timezone behavior as well as the date range limitations

[alamb]

@andygrove -- I have rebased this PR and I believe I have implemented all suggestions. There is one more lingering thing regarding parsing local timestamps that I want to check out but I can also do that after merge if you prefer.

[alamb]

This PR adds:

1. An implementation of the 1 argument form of the TO_TIMESTAMP function that takes a single argument (string) and converts it into a nanosecond precision timestamp, as described in the proposal doc
2. An explicit dependence on the chrono library. Note that this does not add any actual new dependencies for datafusion because depends on arrow which already depends on chrono source link.

fyi @jorgecarleitao@andygrove@emkornfield@jhorstmann

[github-actions]

https://issues.apache.org/jira/browse/ARROW-9944

[andygrove]

LGTM!

[andygrove]

nit: Although the planner and type coercion logic should prevent errors here, we could actually return an Err result rather than panic.

[alamb]

Will do

[jorgecarleitao]

LGTM.

[jorgecarleitao]

add s that failed, or just a slice of it, here?

[alamb]

good idea -- I will do so

[jorgecarleitao]

I think that we get a massive performance boost if we build ArrayData here (learning from @nevi-me :P).

@nevi-me did this recently here, e.g.

let num_rows = args[0].len();let string_args = &args[0].as_any().downcast_ref::<StringArray>().expect("input cast to StringArray failed");let result = (0..num_rows).map(|i| string_to_timestamp_nanos(string_args.value(i))).collect::<Result<Vec<i64>>>()?;let data = ArrayData::new(DataType::Timestamp(TimeUnit::Nanosecond,None),
num_rows,Some(string_args.null_count()),
string_args.data().null_buffer().cloned(),0,vec![Buffer::from(result.to_byte_slice())],vec![],);Ok(make_array(Arc::new(data)))

[jhorstmann]

I'd be interested in a benchmark of this function or the kernel, maybe one for the happy case with T and Z and one for the last fallback. From my experience with Java, parsing timestamps can be rather slow and it might be worth writing a specialized implementation.

The behaviour regarding time zones should also be documented, does it default to a local time zone or to always to UTC?

Lastly, storing nanoseconds as i64 gives a range of about year 1677 to 2262, this might be a limitation for currently very few usecases but should also be mentioned.

[alamb]

I filed https://issues.apache.org/jira/browse/ARROW-9955 to track the (excellent) suggestion for benchmark and possible improvement based on results. I agree efficient timestamp parsing is a bit of an art form.

I will also document both the timezone behavior as well as the date range limitations

[alamb]

@andygrove -- I have rebased this PR and I believe I have implemented all suggestions. There is one more lingering thing regarding parsing local timestamps that I want to check out but I can also do that after merge if you prefer.

[alamb]

This PR adds:

1. An implementation of the 1 argument form of the TO_TIMESTAMP function that takes a single argument (string) and converts it into a nanosecond precision timestamp, as described in the proposal doc
2. An explicit dependence on the chrono library. Note that this does not add any actual new dependencies for datafusion because depends on arrow which already depends on chrono source link.

fyi @jorgecarleitao@andygrove@emkornfield@jhorstmann

[github-actions]

https://issues.apache.org/jira/browse/ARROW-9944

[andygrove]

LGTM!

[andygrove]

nit: Although the planner and type coercion logic should prevent errors here, we could actually return an Err result rather than panic.

[alamb]

Will do

[jorgecarleitao]

LGTM.

[jorgecarleitao]

add s that failed, or just a slice of it, here?

[alamb]

good idea -- I will do so

[jorgecarleitao]

I think that we get a massive performance boost if we build ArrayData here (learning from @nevi-me :P).

@nevi-me did this recently here, e.g.

let num_rows = args[0].len();let string_args = &args[0].as_any().downcast_ref::<StringArray>().expect("input cast to StringArray failed");let result = (0..num_rows).map(|i| string_to_timestamp_nanos(string_args.value(i))).collect::<Result<Vec<i64>>>()?;let data = ArrayData::new(DataType::Timestamp(TimeUnit::Nanosecond,None),
num_rows,Some(string_args.null_count()),
string_args.data().null_buffer().cloned(),0,vec![Buffer::from(result.to_byte_slice())],vec![],);Ok(make_array(Arc::new(data)))

[jhorstmann]

I'd be interested in a benchmark of this function or the kernel, maybe one for the happy case with T and Z and one for the last fallback. From my experience with Java, parsing timestamps can be rather slow and it might be worth writing a specialized implementation.

The behaviour regarding time zones should also be documented, does it default to a local time zone or to always to UTC?

Lastly, storing nanoseconds as i64 gives a range of about year 1677 to 2262, this might be a limitation for currently very few usecases but should also be mentioned.

[alamb]

I filed https://issues.apache.org/jira/browse/ARROW-9955 to track the (excellent) suggestion for benchmark and possible improvement based on results. I agree efficient timestamp parsing is a bit of an art form.

I will also document both the timezone behavior as well as the date range limitations

[alamb]

@andygrove -- I have rebased this PR and I believe I have implemented all suggestions. There is one more lingering thing regarding parsing local timestamps that I want to check out but I can also do that after merge if you prefer.

[alamb]

This PR adds:

1. An implementation of the 1 argument form of the TO_TIMESTAMP function that takes a single argument (string) and converts it into a nanosecond precision timestamp, as described in the proposal doc
2. An explicit dependence on the chrono library. Note that this does not add any actual new dependencies for datafusion because depends on arrow which already depends on chrono source link.

fyi @jorgecarleitao@andygrove@emkornfield@jhorstmann

[github-actions]

https://issues.apache.org/jira/browse/ARROW-9944

[andygrove]

LGTM!

[andygrove]

nit: Although the planner and type coercion logic should prevent errors here, we could actually return an Err result rather than panic.

[alamb]

Will do

[jorgecarleitao]

LGTM.

[jorgecarleitao]

add s that failed, or just a slice of it, here?

[alamb]

good idea -- I will do so

[jorgecarleitao]

I think that we get a massive performance boost if we build ArrayData here (learning from @nevi-me :P).

@nevi-me did this recently here, e.g.

let num_rows = args[0].len();let string_args = &args[0].as_any().downcast_ref::<StringArray>().expect("input cast to StringArray failed");let result = (0..num_rows).map(|i| string_to_timestamp_nanos(string_args.value(i))).collect::<Result<Vec<i64>>>()?;let data = ArrayData::new(DataType::Timestamp(TimeUnit::Nanosecond,None),
num_rows,Some(string_args.null_count()),
string_args.data().null_buffer().cloned(),0,vec![Buffer::from(result.to_byte_slice())],vec![],);Ok(make_array(Arc::new(data)))

[jhorstmann]

I'd be interested in a benchmark of this function or the kernel, maybe one for the happy case with T and Z and one for the last fallback. From my experience with Java, parsing timestamps can be rather slow and it might be worth writing a specialized implementation.

The behaviour regarding time zones should also be documented, does it default to a local time zone or to always to UTC?

Lastly, storing nanoseconds as i64 gives a range of about year 1677 to 2262, this might be a limitation for currently very few usecases but should also be mentioned.

[alamb]

I filed https://issues.apache.org/jira/browse/ARROW-9955 to track the (excellent) suggestion for benchmark and possible improvement based on results. I agree efficient timestamp parsing is a bit of an art form.

I will also document both the timezone behavior as well as the date range limitations

[alamb]

@andygrove -- I have rebased this PR and I believe I have implemented all suggestions. There is one more lingering thing regarding parsing local timestamps that I want to check out but I can also do that after merge if you prefer.

[alamb]

This PR adds:

1. An implementation of the 1 argument form of the TO_TIMESTAMP function that takes a single argument (string) and converts it into a nanosecond precision timestamp, as described in the proposal doc
2. An explicit dependence on the chrono library. Note that this does not add any actual new dependencies for datafusion because depends on arrow which already depends on chrono source link.

fyi @jorgecarleitao@andygrove@emkornfield@jhorstmann

[github-actions]

https://issues.apache.org/jira/browse/ARROW-9944

[andygrove]

LGTM!

[andygrove]

nit: Although the planner and type coercion logic should prevent errors here, we could actually return an Err result rather than panic.

[alamb]

Will do

[jorgecarleitao]

LGTM.

[jorgecarleitao]

add s that failed, or just a slice of it, here?

[alamb]

good idea -- I will do so

[jorgecarleitao]

I think that we get a massive performance boost if we build ArrayData here (learning from @nevi-me :P).

@nevi-me did this recently here, e.g.

let num_rows = args[0].len();let string_args = &args[0].as_any().downcast_ref::<StringArray>().expect("input cast to StringArray failed");let result = (0..num_rows).map(|i| string_to_timestamp_nanos(string_args.value(i))).collect::<Result<Vec<i64>>>()?;let data = ArrayData::new(DataType::Timestamp(TimeUnit::Nanosecond,None),
num_rows,Some(string_args.null_count()),
string_args.data().null_buffer().cloned(),0,vec![Buffer::from(result.to_byte_slice())],vec![],);Ok(make_array(Arc::new(data)))

[jhorstmann]

I'd be interested in a benchmark of this function or the kernel, maybe one for the happy case with T and Z and one for the last fallback. From my experience with Java, parsing timestamps can be rather slow and it might be worth writing a specialized implementation.

The behaviour regarding time zones should also be documented, does it default to a local time zone or to always to UTC?

Lastly, storing nanoseconds as i64 gives a range of about year 1677 to 2262, this might be a limitation for currently very few usecases but should also be mentioned.

[alamb]

I filed https://issues.apache.org/jira/browse/ARROW-9955 to track the (excellent) suggestion for benchmark and possible improvement based on results. I agree efficient timestamp parsing is a bit of an art form.

I will also document both the timezone behavior as well as the date range limitations

[alamb]

@andygrove -- I have rebased this PR and I believe I have implemented all suggestions. There is one more lingering thing regarding parsing local timestamps that I want to check out but I can also do that after merge if you prefer.

[alamb]

This PR adds:

1. An implementation of the 1 argument form of the TO_TIMESTAMP function that takes a single argument (string) and converts it into a nanosecond precision timestamp, as described in the proposal doc
2. An explicit dependence on the chrono library. Note that this does not add any actual new dependencies for datafusion because depends on arrow which already depends on chrono source link.

fyi @jorgecarleitao@andygrove@emkornfield@jhorstmann

[github-actions]

https://issues.apache.org/jira/browse/ARROW-9944

[andygrove]

LGTM!

[andygrove]

nit: Although the planner and type coercion logic should prevent errors here, we could actually return an Err result rather than panic.

[alamb]

Will do

[jorgecarleitao]

LGTM.

[jorgecarleitao]

add s that failed, or just a slice of it, here?

[alamb]

good idea -- I will do so

[jorgecarleitao]

I think that we get a massive performance boost if we build ArrayData here (learning from @nevi-me :P).

@nevi-me did this recently here, e.g.

let num_rows = args[0].len();let string_args = &args[0].as_any().downcast_ref::<StringArray>().expect("input cast to StringArray failed");let result = (0..num_rows).map(|i| string_to_timestamp_nanos(string_args.value(i))).collect::<Result<Vec<i64>>>()?;let data = ArrayData::new(DataType::Timestamp(TimeUnit::Nanosecond,None),
num_rows,Some(string_args.null_count()),
string_args.data().null_buffer().cloned(),0,vec![Buffer::from(result.to_byte_slice())],vec![],);Ok(make_array(Arc::new(data)))

[jhorstmann]

I'd be interested in a benchmark of this function or the kernel, maybe one for the happy case with T and Z and one for the last fallback. From my experience with Java, parsing timestamps can be rather slow and it might be worth writing a specialized implementation.

The behaviour regarding time zones should also be documented, does it default to a local time zone or to always to UTC?

Lastly, storing nanoseconds as i64 gives a range of about year 1677 to 2262, this might be a limitation for currently very few usecases but should also be mentioned.

[alamb]

I filed https://issues.apache.org/jira/browse/ARROW-9955 to track the (excellent) suggestion for benchmark and possible improvement based on results. I agree efficient timestamp parsing is a bit of an art form.

I will also document both the timezone behavior as well as the date range limitations

[alamb]

@andygrove -- I have rebased this PR and I believe I have implemented all suggestions. There is one more lingering thing regarding parsing local timestamps that I want to check out but I can also do that after merge if you prefer.