Describe the bug, including details regarding any error messages, version, and platform.
HadoopPositionOutputStream calls FSDataOutputStream.hflush() in close()
#1309 added an hflush() when closing a stream. This is needless and somewhat troublesome
- hdfs: extra overhead of a blocking write to all datanodes participating in the write (it doesn't know that close() is about to follow, after all)
- abfs: overhead of another write
- s3a: prints a message warning user that the Syncable api is unsupported. (once per process...we don't want log spam)
If anyone has a deployment with the setting fs.s3a.downgrade.syncable.exceptions=false then s3a will actually fail. This is the "fail fast if anyone tries to run algorithms expecting Syncable to persist data". it's a great way to find code which does exactly this.
And it won't be the cause of the problem. A simple flush() should suffice, if that is the underlying problem anyway.
Component(s)
parquet-hadoop
Describe the bug, including details regarding any error messages, version, and platform.
HadoopPositionOutputStream calls FSDataOutputStream.hflush() in close()
#1309 added an
hflush()when closing a stream. This is needless and somewhat troublesomeIf anyone has a deployment with the setting
fs.s3a.downgrade.syncable.exceptions=falsethen s3a will actually fail. This is the "fail fast if anyone tries to run algorithms expecting Syncable to persist data". it's a great way to find code which does exactly this.And it won't be the cause of the problem. A simple flush() should suffice, if that is the underlying problem anyway.
Component(s)
parquet-hadoop