ARROW-16407: [R] Extend parse_date_time to cover hour, dates, and minutes components

[dragosmg]

This PR improves parse_date_time() by:

• adding support for orders with the hours, minutes, and seconds components
• adding support for unseparated strings (ARROW-16446)
• supporting the exact argument:
  • allows users to pass exact = TRUE in which case the orders are taken as they are (they are considered formats and passed to strptime)
  • exact = FALSE implies formats are derived from orders
• allowing the truncated argument
  • denotes number of formats that might be missing. For example, passing an order like ymd_HMS and a value of 1 for truncated will attempt parsing with both ymd_HMS and ymd_HM orders
• erroring when the user passes quiet = FALSE
• improves the utility function used to generate formats (which are then passed on to strptime) from orders
  • less hard-coding and increased ability to deal with different orders and separators

the ymd HMS orders (and variants) will parse correctly:

library(dplyr, warn.conflicts=FALSE)
library(lubridate, warn.conflicts=FALSE)
library(arrow, warn.conflicts=FALSE)
test_df<- tibble(
x= c("2011-12-31 12:59:59", "2010-01-01 12:11", "2010-01-01 12", "2010-01-01")
)
test_df %>%
mutate(
y= parse_date_time(x, "Ymd HMS", truncated=3)
)
#> # A tibble: 4 × 2#> x y #> <chr> <dttm> #> 1 2011-12-31 12:59:59 2011-12-31 12:59:59#> 2 2010-01-01 12:11 2010-01-01 12:11:00#> 3 2010-01-01 12 2010-01-01 12:00:00#> 4 2010-01-01 2010-01-01 00:00:00test_df %>%
arrow_table() %>% mutate(
y= parse_date_time(x, "Ymd HMS", truncated=3)
) %>% collect()
#> # A tibble: 4 × 2#> x y #> <chr> <dttm> #> 1 2011-12-31 12:59:59 2011-12-31 12:59:59#> 2 2010-01-01 12:11 2010-01-01 12:11:00#> 3 2010-01-01 12 2010-01-01 12:00:00#> 4 2010-01-01 2010-01-01 00:00:00

^{Created on 2022-05-19 by the reprex package (v2.0.1)}

exact = TRUE can also be used:

Details

library(arrow, warn.conflicts=FALSE)
library(lubridate, warn.conflicts=FALSE)
library(dplyr, warn.conflicts=FALSE)
test_df<- tibble(
x= c("11/23/1998 07:00:00", "6/18/1952 0135", "2/25/1974 0523", "9/07/1985 01", NA)
)
test_df %>%
mutate(
parsed_x=
parse_date_time(
x,
c("%m/%d/%Y %I:%M:%S", "%m/%d/%Y %H%M", "%m/%d/%Y %H"),
exact=TRUE
)
)
#> # A tibble: 5 × 2#> x parsed_x #> <chr> <dttm> #> 1 11/23/1998 07:00:00 1998-11-23 07:00:00#> 2 6/18/1952 0135 1952-06-18 01:35:00#> 3 2/25/1974 0523 1974-02-25 05:23:00#> 4 9/07/1985 01 1985-09-07 01:00:00#> 5 <NA> NAtest_df %>%
arrow_table() %>% mutate(
parsed_x=
parse_date_time(
x,
c("%m/%d/%Y %I:%M:%S", "%m/%d/%Y %H%M", "%m/%d/%Y %H"),
exact=TRUE
)
) %>%
collect()
#> # A tibble: 5 × 2#> x parsed_x #> <chr> <dttm> #> 1 11/23/1998 07:00:00 1998-11-23 07:00:00#> 2 6/18/1952 0135 1952-06-18 01:35:00#> 3 2/25/1974 0523 1974-02-25 05:23:00#> 4 9/07/1985 01 1985-09-07 01:00:00#> 5 <NA> NA

^{Created on 2022-05-20 by the reprex package (v2.0.1)}

[github-actions]

https://issues.apache.org/jira/browse/ARROW-16407

[jonkeane]

A few comments for you

[dragosmg]

@jonkeane I think this is ready for another look

[jonkeane]

This is moving, still a few functionality rough edges that need work. A few general notes:

• I'm still struggling a bit following some of the logic. More comments, especially with details about what's in some of the various variables would be helpful to follow what's going on. It's hard to know now what things are lists of expressions versus lists of characters like "Ym" versus lists of characters like "%Y-%m"

• Should we test build_formats() directly to detect some of the more bespoke | full set of order checks? This would make the tests a lot easier to reason about (what we're testing is more obvious + we wouldn't need to have a whole data frame + date setup in it) Take a look at the other helper functions to see if the same is true for them.

[dragosmg]

Edit: I opened ARROW-16653

Given this is a relatively large PR I propose I open a follow-up ticket to go with a fine-tooth comb and check for edge cases, formats , etc. I think we cover the vast majority of use cases, but I'd like to be able to spend some time looking at unit tests, helpers descriptions, error messages, messaging around unsupported orders (there shouldn't be any left) etc.

[dragosmg]

It looks like combining the separator (e.g. "%y-%m-%d") and non-separator formats (e.g. "%y%m%d") into a single vector (my original implementation) is faster than using them separately based on if the data contains a separator or not.

Results table

>results# A tibble: 2 × 13expressionminmedian`itr/sec`mem_alloc`gc/sec`n_itrn_gctotal_timeresultmemorytimegc<bch:expr><bch:tm><bch:tm><dbl><bch:byt><dbl><int><dbl><bch:tm><list><list><list><list>1separate6s6.03s0.16515.5MB0.02921731.72m<tibble><Rprofmem><bench_tm [20]><tibble>2combined3.36s3.37s0.29715.5MB0.03301821.01m<tibble><Rprofmem><bench_tm [20]><tibble>

Code

library(dplyr)
library(lubridate)
library(ggplot2)
library(hrbrthemes)
load_all()
test_df<-tibble::tibble(
a= rep(c("20220614", "2022-06-14"), 1e6)
)
results<-bench::mark(
separate=test_df %>% arrow_table() %>% mutate(b= parse_date_time(a, orders="ymd")) %>% collect(),
combined=test_df %>% arrow_table() %>% mutate(b= parse_date_time_combined(a, orders="ymd")) %>% collect(), min_iterations=20
)
resultsggplot2::autoplot(results) +
theme_ipsum_rc(grid="XxY") +
labs(title="Comparison of format parsing",
subtitle="separate = formats with or without separator are tried separately\ncombined = formats are combined in a single vector and all are passed to `coalesce()`")

Details

I benchmarked 3 different versions of separate (using different formats for x with and without separator):

• using gsub modify the format if x does not have a separator, and
• derive formats_with_sep and formats_without_sep, iterate over the first (with imap()) but pluck the value from the second, if x does not contain a separator.
• derive formats_with_sep and formats_without_sep, iterate over an independent index with a for loop, and use only the matching format (with or without the separator)

All 3 versions performed similarly, being slower than the version in which the formats were combined into a single vector.

[dragosmg]

I think this is ready for another look @jonkeane. What changed since our last conversation?

• I benchmarked my original implementation (formats with and without separators in a single vector) against versions in which the formats were separated (see this comment for more details). The initial implementation proved faster and so I went with that approach.
• I added roxygen2 headers for the helper functions to give more clarity what is happening where
• I added a suite of unit tests for build_format_from_order() and build_formats()
• I rebased.

[paleolimbot]

I can't speak to the completeness of testing/features or motivation since I haven't been following the discussion, but the code itself is easy to read and the approach seems reasonable.

[rok]

This looks good to me (but I don't know the R codebase well)!