Uh oh!
There was an error while loading. Please reload this page.
TEZ-4479: Eagerly Init/Load FileSystem In Tez Task Containers - #274
Open
shameersss1 wants to merge 1 commit into
Open
TEZ-4479: Eagerly Init/Load FileSystem In Tez Task Containers#274shameersss1 wants to merge 1 commit into
shameersss1 wants to merge 1 commit into
Conversation
tez-yetus
commented
Mar 4, 2023
💔 -1 overall
This message was automatically generated. |
shameersss1
commented
Mar 16, 2023
ContributorAuthor
@abstractdog Could you please review the changes? |
| * String value. Comma seperated list of FileSystem paths which needs to be eagerly initialized. | ||
| * For example s3://bucket/,file://,hdfs://localhost:8020/ | ||
| */ | ||
| @ConfigurationScope(Scope.AM) |
Contributor
There was a problem hiding this comment.
- Scope is VERTEX as we're doing this in TezChild
- constant name should contain TEZ_ prefix
| public void run() { | ||
| try { | ||
| new Path(path).getFileSystem(conf); | ||
| LOG.info("Eagerly initiated FileSystem at path {}", path); |
Contributor
There was a problem hiding this comment.
it would be nice to measure the time spent with initialization and print it to logs too
| if (eagerInitFsPool == null && !eagerInitPaths.isEmpty()) { | ||
| eagerInitFsPool = Executors.newCachedThreadPool(new ThreadFactoryBuilder() | ||
| .setDaemon(true) | ||
| .setPriority(Thread.MAX_PRIORITY) |
Contributor
There was a problem hiding this comment.
in the code, I can see we tend not to set priority in thread pools...I guess we can remove this to simplify this code further
| @Override | ||
| public void run() { | ||
| try { | ||
| new Path(path).getFileSystem(conf); |
Contributor
There was a problem hiding this comment.
shouldn't this filesystem be closed? does it hold any resources when it's open?
ContributorAuthor
There was a problem hiding this comment.
Wouldn't this be gc'ed eventually? I can see any instances in the code were we are not explicitly closing the filesystem object. For example : https://github.com/apache/tez/blob/master/tez-plugins/tez-protobuf-history-plugin/src/main/java/org/apache/tez/dag/history/logging/proto/DatePartitionedLogger.java#L136
tez-yetus
commented
Mar 27, 2023
💔 -1 overall
This message was automatically generated. |
| hadoopShim); | ||
| } | ||
| private static void eagerInitFileSystemPaths(Configuration conf) { |
Contributor
There was a problem hiding this comment.
It will be good to measure the time spent on FS init (even for cloudstores) and share the details, before trying out this patch.
Reason: This get inited in TezChild, but for running container, it closes the FileSystem explicitly via "FileSystem.closeAllForUGI(childUGI);". Refer "public ContainerExecutionResult run()" method.
Even if this gets early inited, it will not have major impact in container reuse scenario. It will be good to measure and find out the timing spent in FS init.
| .build()); | ||
| } | ||
| for (String path : eagerInitPaths) { | ||
| eagerInitFsPool.execute(new Runnable() { |
There was a problem hiding this comment.
before rushing to create lots of fs instances in parallel, look at HADOOP-17313 and why we actually added semaphores to stop apps like tez creating too many at the same time. this code may cause overload problems, or the fs semaphore will hold you back for safety.
best to look at why its taking so long; if s3a bucket existence checks aren't involved, then it'll be whatever auth mechanism is plugged in. same for abfs
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Initing/Loading FileSystem such as S3 can take ~10s - ~20s when called for the first time and the time taken for subsequent calls are negligable. If we can load the FileSystem much before it is used can help us to save some time. It can be especially useful in case of pre-warm Tez containers where the Tez task containers comes up when the Application Master (AM) is launched and not on-demand which is the default behavior. It can be also useful in cases where the Mapper tasks spends considerable time consuming the upstream shuffle data and then heads to process some FileSystem operations, in all such cases we have few FileSystem load up time.