Uh oh!
There was an error while loading. Please reload this page.
[SPARK-47496][SQL] Java SPI Support for dynamic JDBC dialect registering - #45626
[SPARK-47496][SQL] Java SPI Support for dynamic JDBC dialect registering#45626yaooqinn wants to merge 5 commits into
Conversation
| override def supportsOffset: Boolean = true | ||
| } | ||
| private[jdbc] object OracleDialect { |
There was a problem hiding this comment.
Ur, actually, case object OracleDialect has more features (serializable, hashCode, toString) than object OracleDialect. Do we need to shrink features like this?
There was a problem hiding this comment.
This is only a place for containing some const ints, which represent some variants from Oracle JDBC extensions. Thus, object is enough
There was a problem hiding this comment.
Opps, the errors occured
- simple scan with LIMIT *** FAILED *** (65 milliseconds)
[info] org.apache.spark.SparkException: Task not serializable
| assert(namedObservation.get === expected) | ||
| } | ||
dongjoon-hyun
left a comment
There was a problem hiding this comment.
+1, LGTM (with minor comments). It's a nice feature, @yaooqinn !
yaooqinn
commented
Mar 21, 2024
Thank you @dongjoon-hyun. I addressed the comments |
dongjoon-hyun
commented
Mar 21, 2024
#45626 (comment) seems to be missed still. Are you going to recover it, @yaooqinn ?
|
yaooqinn
commented
Mar 21, 2024
Yes @dongjoon-hyun it was caused by the unserializable object |
@yaooqinn It's really worth? as I know, users could register dialects with |
yaooqinn
commented
Mar 21, 2024
@beliefer Yes, it's worth it. Actually, there are plenty of hacky ways to register dialects just like what you mentioned. We can't anticipate those as typical API usages. |
yaooqinn
commented
Mar 21, 2024
Merged to master. Thank you @dongjoon-hyun |
smileyboy2019
commented
May 31, 2024
In which version will it be released, and can multiple libraries be joined for querying? |
elijah-pl
commented
Jul 29, 2025
@yaooqinn is it only in Spark 4 or 3 as well? |
yaooqinn
commented
Jul 29, 2025
@elijah-pl please refer to the JIRA sizde for fixed versions |
What changes were proposed in this pull request?
This PR brings the Java ServiceProvider Interface (SPI) Support for dynamic JDBC dialect registering.
A custom JDBC dialect can be registered easily instead of calling JdbcDialects.registerDialect manually.
Why are the changes needed?
For pure SQL and other non-Java API users, it's difficult to register a custom JDBC dialect to use. With this patch, this can be done when the jar containing the dialect class is visible to the spark classloader.
Does this PR introduce any user-facing change?
Yes, but mostly for third-party developers
How was this patch tested?
new tests
Was this patch authored or co-authored using generative AI tooling?
no