Uh oh!
There was an error while loading. Please reload this page.
Added support for custom timestamp format - #755
Conversation
Thanks for your contribution! To satisfy the DCO policy in our contributing guide every commit message must include a sign-off message. One or more of your commits is missing this message. You can reword previous commit messages with an interactive rebase ( |
There was a problem hiding this comment.
Pull request overview
This PR introduces an optional, user-supplied timestamp parsing format that’s threaded from Connection config through result fetching and into the non-Arrow conversion paths, with unit tests covering the new behavior.
Changes:
- Added
parse_timestamp()helper and plumbed an optionaltimestamp_formatthrough Thrift column-based result conversion. - Added
non_arrow_timestamp_formatconnection option and passed it into both Thrift and SEA non-Arrow timestamp conversions. - Added unit tests for explicit-format parsing and fallback to
dateutilparsing.
Reviewed changes
Copilot reviewed 8 out of 8 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
src/databricks/sql/utils.py | Adds parse_timestamp() and wires timestamp_format into column-table timestamp conversion and queue-building. |
src/databricks/sql/client.py | Adds non_arrow_timestamp_format connection attribute sourced from **kwargs. |
src/databricks/sql/result_set.py | Passes connection timestamp format into Thrift queue/fetch paths. |
src/databricks/sql/backend/thrift_backend.py | Threads timestamp_format into build_queue() during fetch_results(). |
src/databricks/sql/backend/sea/utils/conversion.py | Routes SEA TIMESTAMP conversion through parse_timestamp() with optional format. |
src/databricks/sql/backend/sea/result_set.py | Passes connection timestamp format into SEA row conversion. |
tests/unit/test_util.py | Adds parse_timestamp() and column-table conversion tests with/without format. |
tests/unit/test_sea_conversion.py | Adds SEA timestamp conversion tests with format + fallback. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
You can also share your feedback on Copilot code review. Take the survey.
| return datetime.datetime.strptime(value, timestamp_format).replace( | ||
| tzinfo=pytz.UTC | ||
| ) |
| elif sql_type == SqlType.TIMESTAMP: | ||
| timestamp_format = kwargs.get("timestamp_format", None) | ||
| return parse_timestamp(value, timestamp_format) | ||
| else: | ||
| return converter_func(value) |
| self.disable_pandas = kwargs.get("_disable_pandas", False) | ||
| self.lz4_compression = kwargs.get("enable_query_result_lz4_compression", True) | ||
| self.use_cloud_fetch = kwargs.get("use_cloud_fetch", True) | ||
| self.non_arrow_timestamp_format = kwargs.get("non_arrow_timestamp_format", None) | ||
| self._cursors = [] # type: List[Cursor] |
Thanks for your contribution! To satisfy the DCO policy in our contributing guide every commit message must include a sign-off message. One or more of your commits is missing this message. You can reword previous commit messages with an interactive rebase ( |
Added support for custom timestamp format
What type of PR is this?
Description
How is this tested?
Related Tickets & Documents