Core: Support set system level properties with environmental variables - #5659
Conversation
|
cc @rdblue @kbendick @szehon-ho, this is a small update, could you take a look when you are free? Thanks a lot. |
| public static final String SCAN_THREAD_POOL_ENABLED = "iceberg.scan.plan-in-worker-pool"; | ||
|
|
||
| static boolean getBoolean(String systemProperty, boolean defaultValue) { | ||
| public static final String SCAN_THREAD_POOL_ENABLED_ENV = "ICEBERG_SCAN_PLAN_IN_WORKER_POOL"; |
There was a problem hiding this comment.
Does this need to be explosed?
There was a problem hiding this comment.
It is used by core/src/main/java/org/apache/iceberg/util/ThreadPools.java in a different package. I updated other envs to the default access level.
d46074a to
0c6a8f1
Compare
|
@szehon-ho @rdblue can we take another look and merge if possible ? |
|
Hi guys, can you explain the motivation of the change? Not opposed, but reading the description but didn't quite understand it. To me, both seem to be more or less equivalent in ease-of-use (they are passed to the java process when it is spawned..) |
|
@szehon-ho In many production environments, there are some default parameter settings for spark driver/executor java options (eg: gc parameters). And those parameters are concated into one string. This makes it difficult to overwrite or modify those parameters to set iceberg system-level properties. While with ENV settings, we could set those system-level properties separately. |
szehon-ho
left a comment
There was a problem hiding this comment.
I see, yea spark seems to support env variables to set certain things. Left some comments.
0c6a8f1 to
fe5a9d9
Compare
|
Thanks @szehon-ho for the reviewing, and sorry got held up. |
szehon-ho
left a comment
There was a problem hiding this comment.
Looks good to me, left one nit. Will wait a bit if there's any more comment, and merge if not
| if (value != null) { | ||
| try { | ||
| return parseFunc.apply(value); | ||
| } catch (NumberFormatException e) { |
There was a problem hiding this comment.
Nit: how about catch Exception , as we don't know what kind of Function we get?
There was a problem hiding this comment.
Updated and add the log.
|
gentle ping @szehon-ho |
|
Merged, thanks @ConeyLiu ! |
|
Thanks @szehon-ho @zinking @rdblue |
…log message (#17812) * Core: Remove ThreadPools.WORKER_THREAD_POOL_SIZE_PROP and correct manifest spec fallback removal version Resolves the two remaining deprecation markers in iceberg-core that promised action "in 1.12.0" but were missed by the removal PRs. Remove ThreadPools.WORKER_THREAD_POOL_SIZE_PROP. The field was deprecated in #5659 in favor of SystemConfigs.WORKER_THREAD_POOL_SIZE and retargeted from 2.0.0 to 1.12.0 by #14392. It had no callers anywhere in the repository outside its own declaration and no references in the docs, and it was the only survivor of #14392's 13-file batch after #17700 removed the rest. The corresponding java.field.removed break is recorded in .palantir/revapi.yml under the 1.11.0 baseline. Correct the ManifestReader warning for the partition-spec-from-file- metadata fallback to say 2.0.0 rather than 1.12.0. The fallback cannot be honored in 1.12.0 because its only production caller is BaseSnapshot:276, which passes a null spec map while serving Snapshot.addedDeleteFiles(FileIO) and removedDeleteFiles(FileIO). Those accessors are themselves already marked "@deprecated will be removed in 2.0.0; use SnapshotChanges#builderFor(Table) instead", and because they receive only a FileIO they structurally cannot supply a spec map. The fallback must therefore outlive 1.12.0, and 2.0.0 aligns its removal with that of its last caller. * reword revapi change
This add supports setting system level properties with environmental variables. For example: we could control the number of threads for the default Iceberg thread pool with java system property:
iceberg.worker.num-threadsor env:ICEBERG_WORKER_NUM_THREADS. This is indeed useful when we want to change those properties while the java system properties have some default settings and we do not want to change them.