Fix writing to local filesystem by kevinjqliu · Pull Request #301 · apache/iceberg-python

kevinjqliu · 2024-01-24T18:03:24Z

This PR changes the behavior of both PyArrow and FsSpec file systems. When writing to the local file system, parent directories will be created first before writing to the file. Previously, a FileNotFoundError error is thrown when the parent directories are missing.

Testing

Added tests to tests/io/test_fsspec.py and tests/io/test_pyarrow.py
Added test_append_table test to write to the local filesystem, to illustrate the issue. It fails without the changes in this PR.
Also added the catalog_sqlite_fsspec catalog which tests writing to the local filesystem using FSSPEC_FILE_IO

Fokko

I see the need for this, but I'm reluctant to support this. You could pre-create the directories outside of PyIceberg if possible.

Fokko · 2024-01-24T19:46:54Z

 if not overwrite and self.exists() is True:
 raise FileExistsError(f"Cannot create file, already exists: {self.location}")
+ # Parent directories must be created first in certain file systems, such as the LocalFileSystem.
+ self._filesystem.create_dir(os.path.dirname(self._path), recursive=True)


This is typically something that we try to avoid. Iceberg is designed to work with object stores, and those don't have a notion of directories. One recommendation is even to disallow moves and listing of directories. One thing is also creating a directory. I'm not sure what the behavior is for the Arrow S3 implementation. Since some of the implementations still make a call(s):
For example, a list operation to check if the directory is there
For example, they touch a small file under the prefix to indicate that the path should be created.
Another concern is that we currently do this in Arrow, we also would need to do this for other implementations to avoid discrepancies. The concept of the FileIO is that you easily can swap them out.

@Fokko thanks for the review.
I agree with the above. The Arrow FileIO implementation might not be the best place to implement this behavior. So far both of the supported FS implementations (ARROW_FILE_IO and FSSPEC_FILE_IO) are failing to write to the local file system.
I want to make writes work for the local file system.
Looking at the Java side, there is a LocalOutputFile implementation which implements the behavior for creating parent directories.
Maybe we can implement a new FileIO implementation and make that the preferred implementation for the file:// scheme.
iceberg-python/pyiceberg/io/__init__.py
Line 278 in 4cf1f35
"file": [ARROW_FILE_IO],

Hey @kevinjqliu That sounds like a reasonable alternative to me 👍

Thinking a bit more about it, maybe we should still use the ArrowFileSystem in this new FileIO to make sure that we can read the table into an Arrow pa.Table.

@Fokko I updated the PR for both PyArrow and FsSpec implementations

kevinjqliu · 2024-01-26T02:34:42Z



 SCHEME_TO_FS = {
+ "": _file,


pyarrow defaults scheme to file when no scheme is present.
iceberg-python/pyiceberg/io/pyarrow.py
Line 322 in 8464d71
return"file", uri.netloc, os.path.abspath(location)
we essentially do the same here

Fokko

Thanks @kevinjqliu for working on this, elegant solution! LGTM

HonahX

LGTM! Thanks @kevinjqliu

kevinjqliu · 2024-01-24T18:03:24Z

Issue #299

This PR changes the behavior of both PyArrow and FsSpec file systems. When writing to the local file system, parent directories will be created first before writing to the file. Previously, a FileNotFoundError error is thrown when the parent directories are missing.

Testing

Added tests to tests/io/test_fsspec.py and tests/io/test_pyarrow.py
Added test_append_table test to write to the local filesystem, to illustrate the issue. It fails without the changes in this PR.
Also added the catalog_sqlite_fsspec catalog which tests writing to the local filesystem using FSSPEC_FILE_IO

Fokko

I see the need for this, but I'm reluctant to support this. You could pre-create the directories outside of PyIceberg if possible.

Fokko · 2024-01-24T19:46:54Z

 if not overwrite and self.exists() is True:
 raise FileExistsError(f"Cannot create file, already exists: {self.location}")
+ # Parent directories must be created first in certain file systems, such as the LocalFileSystem.
+ self._filesystem.create_dir(os.path.dirname(self._path), recursive=True)


This is typically something that we try to avoid. Iceberg is designed to work with object stores, and those don't have a notion of directories. One recommendation is even to disallow moves and listing of directories. One thing is also creating a directory. I'm not sure what the behavior is for the Arrow S3 implementation. Since some of the implementations still make a call(s):
For example, a list operation to check if the directory is there
For example, they touch a small file under the prefix to indicate that the path should be created.
Another concern is that we currently do this in Arrow, we also would need to do this for other implementations to avoid discrepancies. The concept of the FileIO is that you easily can swap them out.

@Fokko thanks for the review.
I agree with the above. The Arrow FileIO implementation might not be the best place to implement this behavior. So far both of the supported FS implementations (ARROW_FILE_IO and FSSPEC_FILE_IO) are failing to write to the local file system.
I want to make writes work for the local file system.
Looking at the Java side, there is a LocalOutputFile implementation which implements the behavior for creating parent directories.
Maybe we can implement a new FileIO implementation and make that the preferred implementation for the file:// scheme.
iceberg-python/pyiceberg/io/__init__.py
Line 278 in 4cf1f35
"file": [ARROW_FILE_IO],

Hey @kevinjqliu That sounds like a reasonable alternative to me 👍

Thinking a bit more about it, maybe we should still use the ArrowFileSystem in this new FileIO to make sure that we can read the table into an Arrow pa.Table.

@Fokko I updated the PR for both PyArrow and FsSpec implementations

kevinjqliu · 2024-01-26T02:34:42Z



 SCHEME_TO_FS = {
+ "": _file,


pyarrow defaults scheme to file when no scheme is present.
iceberg-python/pyiceberg/io/pyarrow.py
Line 322 in 8464d71
return"file", uri.netloc, os.path.abspath(location)
we essentially do the same here

Fokko

Thanks @kevinjqliu for working on this, elegant solution! LGTM

HonahX

LGTM! Thanks @kevinjqliu

kevinjqliu · 2024-01-24T18:03:24Z

Issue #299

This PR changes the behavior of both PyArrow and FsSpec file systems. When writing to the local file system, parent directories will be created first before writing to the file. Previously, a FileNotFoundError error is thrown when the parent directories are missing.

Testing

Added tests to tests/io/test_fsspec.py and tests/io/test_pyarrow.py
Added test_append_table test to write to the local filesystem, to illustrate the issue. It fails without the changes in this PR.
Also added the catalog_sqlite_fsspec catalog which tests writing to the local filesystem using FSSPEC_FILE_IO

Fokko

I see the need for this, but I'm reluctant to support this. You could pre-create the directories outside of PyIceberg if possible.

Fokko · 2024-01-24T19:46:54Z

 if not overwrite and self.exists() is True:
 raise FileExistsError(f"Cannot create file, already exists: {self.location}")
+ # Parent directories must be created first in certain file systems, such as the LocalFileSystem.
+ self._filesystem.create_dir(os.path.dirname(self._path), recursive=True)


This is typically something that we try to avoid. Iceberg is designed to work with object stores, and those don't have a notion of directories. One recommendation is even to disallow moves and listing of directories. One thing is also creating a directory. I'm not sure what the behavior is for the Arrow S3 implementation. Since some of the implementations still make a call(s):
For example, a list operation to check if the directory is there
For example, they touch a small file under the prefix to indicate that the path should be created.
Another concern is that we currently do this in Arrow, we also would need to do this for other implementations to avoid discrepancies. The concept of the FileIO is that you easily can swap them out.

@Fokko thanks for the review.
I agree with the above. The Arrow FileIO implementation might not be the best place to implement this behavior. So far both of the supported FS implementations (ARROW_FILE_IO and FSSPEC_FILE_IO) are failing to write to the local file system.
I want to make writes work for the local file system.
Looking at the Java side, there is a LocalOutputFile implementation which implements the behavior for creating parent directories.
Maybe we can implement a new FileIO implementation and make that the preferred implementation for the file:// scheme.
iceberg-python/pyiceberg/io/__init__.py
Line 278 in 4cf1f35
"file": [ARROW_FILE_IO],

Hey @kevinjqliu That sounds like a reasonable alternative to me 👍

Thinking a bit more about it, maybe we should still use the ArrowFileSystem in this new FileIO to make sure that we can read the table into an Arrow pa.Table.

@Fokko I updated the PR for both PyArrow and FsSpec implementations

kevinjqliu · 2024-01-26T02:34:42Z



 SCHEME_TO_FS = {
+ "": _file,


pyarrow defaults scheme to file when no scheme is present.
iceberg-python/pyiceberg/io/pyarrow.py
Line 322 in 8464d71
return"file", uri.netloc, os.path.abspath(location)
we essentially do the same here

Fokko

Thanks @kevinjqliu for working on this, elegant solution! LGTM

HonahX

LGTM! Thanks @kevinjqliu

kevinjqliu · 2024-01-24T18:03:24Z

Issue #299

This PR changes the behavior of both PyArrow and FsSpec file systems. When writing to the local file system, parent directories will be created first before writing to the file. Previously, a FileNotFoundError error is thrown when the parent directories are missing.

Testing

Added tests to tests/io/test_fsspec.py and tests/io/test_pyarrow.py
Added test_append_table test to write to the local filesystem, to illustrate the issue. It fails without the changes in this PR.
Also added the catalog_sqlite_fsspec catalog which tests writing to the local filesystem using FSSPEC_FILE_IO

Fokko

I see the need for this, but I'm reluctant to support this. You could pre-create the directories outside of PyIceberg if possible.

Fokko · 2024-01-24T19:46:54Z

 if not overwrite and self.exists() is True:
 raise FileExistsError(f"Cannot create file, already exists: {self.location}")
+ # Parent directories must be created first in certain file systems, such as the LocalFileSystem.
+ self._filesystem.create_dir(os.path.dirname(self._path), recursive=True)


This is typically something that we try to avoid. Iceberg is designed to work with object stores, and those don't have a notion of directories. One recommendation is even to disallow moves and listing of directories. One thing is also creating a directory. I'm not sure what the behavior is for the Arrow S3 implementation. Since some of the implementations still make a call(s):
For example, a list operation to check if the directory is there
For example, they touch a small file under the prefix to indicate that the path should be created.
Another concern is that we currently do this in Arrow, we also would need to do this for other implementations to avoid discrepancies. The concept of the FileIO is that you easily can swap them out.

@Fokko thanks for the review.
I agree with the above. The Arrow FileIO implementation might not be the best place to implement this behavior. So far both of the supported FS implementations (ARROW_FILE_IO and FSSPEC_FILE_IO) are failing to write to the local file system.
I want to make writes work for the local file system.
Looking at the Java side, there is a LocalOutputFile implementation which implements the behavior for creating parent directories.
Maybe we can implement a new FileIO implementation and make that the preferred implementation for the file:// scheme.
iceberg-python/pyiceberg/io/__init__.py
Line 278 in 4cf1f35
"file": [ARROW_FILE_IO],

Hey @kevinjqliu That sounds like a reasonable alternative to me 👍

Thinking a bit more about it, maybe we should still use the ArrowFileSystem in this new FileIO to make sure that we can read the table into an Arrow pa.Table.

@Fokko I updated the PR for both PyArrow and FsSpec implementations

kevinjqliu · 2024-01-26T02:34:42Z



 SCHEME_TO_FS = {
+ "": _file,


pyarrow defaults scheme to file when no scheme is present.
iceberg-python/pyiceberg/io/pyarrow.py
Line 322 in 8464d71
return"file", uri.netloc, os.path.abspath(location)
we essentially do the same here

Fokko

Thanks @kevinjqliu for working on this, elegant solution! LGTM

HonahX

LGTM! Thanks @kevinjqliu

kevinjqliu · 2024-01-24T18:03:24Z

Issue #299

This PR changes the behavior of both PyArrow and FsSpec file systems. When writing to the local file system, parent directories will be created first before writing to the file. Previously, a FileNotFoundError error is thrown when the parent directories are missing.

Testing

Added tests to tests/io/test_fsspec.py and tests/io/test_pyarrow.py
Added test_append_table test to write to the local filesystem, to illustrate the issue. It fails without the changes in this PR.
Also added the catalog_sqlite_fsspec catalog which tests writing to the local filesystem using FSSPEC_FILE_IO

Fokko

I see the need for this, but I'm reluctant to support this. You could pre-create the directories outside of PyIceberg if possible.

Fokko · 2024-01-24T19:46:54Z

 if not overwrite and self.exists() is True:
 raise FileExistsError(f"Cannot create file, already exists: {self.location}")
+ # Parent directories must be created first in certain file systems, such as the LocalFileSystem.
+ self._filesystem.create_dir(os.path.dirname(self._path), recursive=True)


This is typically something that we try to avoid. Iceberg is designed to work with object stores, and those don't have a notion of directories. One recommendation is even to disallow moves and listing of directories. One thing is also creating a directory. I'm not sure what the behavior is for the Arrow S3 implementation. Since some of the implementations still make a call(s):
For example, a list operation to check if the directory is there
For example, they touch a small file under the prefix to indicate that the path should be created.
Another concern is that we currently do this in Arrow, we also would need to do this for other implementations to avoid discrepancies. The concept of the FileIO is that you easily can swap them out.

@Fokko thanks for the review.
I agree with the above. The Arrow FileIO implementation might not be the best place to implement this behavior. So far both of the supported FS implementations (ARROW_FILE_IO and FSSPEC_FILE_IO) are failing to write to the local file system.
I want to make writes work for the local file system.
Looking at the Java side, there is a LocalOutputFile implementation which implements the behavior for creating parent directories.
Maybe we can implement a new FileIO implementation and make that the preferred implementation for the file:// scheme.
iceberg-python/pyiceberg/io/__init__.py
Line 278 in 4cf1f35
"file": [ARROW_FILE_IO],

Hey @kevinjqliu That sounds like a reasonable alternative to me 👍

Thinking a bit more about it, maybe we should still use the ArrowFileSystem in this new FileIO to make sure that we can read the table into an Arrow pa.Table.

@Fokko I updated the PR for both PyArrow and FsSpec implementations

kevinjqliu · 2024-01-26T02:34:42Z



 SCHEME_TO_FS = {
+ "": _file,


pyarrow defaults scheme to file when no scheme is present.
iceberg-python/pyiceberg/io/pyarrow.py
Line 322 in 8464d71
return"file", uri.netloc, os.path.abspath(location)
we essentially do the same here

Fokko

Thanks @kevinjqliu for working on this, elegant solution! LGTM

HonahX

LGTM! Thanks @kevinjqliu

kevinjqliu · 2024-01-24T18:03:24Z

Issue #299

This PR changes the behavior of both PyArrow and FsSpec file systems. When writing to the local file system, parent directories will be created first before writing to the file. Previously, a FileNotFoundError error is thrown when the parent directories are missing.

Testing

Added tests to tests/io/test_fsspec.py and tests/io/test_pyarrow.py
Added test_append_table test to write to the local filesystem, to illustrate the issue. It fails without the changes in this PR.
Also added the catalog_sqlite_fsspec catalog which tests writing to the local filesystem using FSSPEC_FILE_IO

Fokko

I see the need for this, but I'm reluctant to support this. You could pre-create the directories outside of PyIceberg if possible.

Fokko · 2024-01-24T19:46:54Z

 if not overwrite and self.exists() is True:
 raise FileExistsError(f"Cannot create file, already exists: {self.location}")
+ # Parent directories must be created first in certain file systems, such as the LocalFileSystem.
+ self._filesystem.create_dir(os.path.dirname(self._path), recursive=True)


This is typically something that we try to avoid. Iceberg is designed to work with object stores, and those don't have a notion of directories. One recommendation is even to disallow moves and listing of directories. One thing is also creating a directory. I'm not sure what the behavior is for the Arrow S3 implementation. Since some of the implementations still make a call(s):
For example, a list operation to check if the directory is there
For example, they touch a small file under the prefix to indicate that the path should be created.
Another concern is that we currently do this in Arrow, we also would need to do this for other implementations to avoid discrepancies. The concept of the FileIO is that you easily can swap them out.

@Fokko thanks for the review.
I agree with the above. The Arrow FileIO implementation might not be the best place to implement this behavior. So far both of the supported FS implementations (ARROW_FILE_IO and FSSPEC_FILE_IO) are failing to write to the local file system.
I want to make writes work for the local file system.
Looking at the Java side, there is a LocalOutputFile implementation which implements the behavior for creating parent directories.
Maybe we can implement a new FileIO implementation and make that the preferred implementation for the file:// scheme.
iceberg-python/pyiceberg/io/__init__.py
Line 278 in 4cf1f35
"file": [ARROW_FILE_IO],

Hey @kevinjqliu That sounds like a reasonable alternative to me 👍

Thinking a bit more about it, maybe we should still use the ArrowFileSystem in this new FileIO to make sure that we can read the table into an Arrow pa.Table.

@Fokko I updated the PR for both PyArrow and FsSpec implementations

kevinjqliu · 2024-01-26T02:34:42Z



 SCHEME_TO_FS = {
+ "": _file,


pyarrow defaults scheme to file when no scheme is present.
iceberg-python/pyiceberg/io/pyarrow.py
Line 322 in 8464d71
return"file", uri.netloc, os.path.abspath(location)
we essentially do the same here

Fokko

Thanks @kevinjqliu for working on this, elegant solution! LGTM

HonahX

LGTM! Thanks @kevinjqliu

kevinjqliu · 2024-01-24T18:03:24Z

Issue #299

This PR changes the behavior of both PyArrow and FsSpec file systems. When writing to the local file system, parent directories will be created first before writing to the file. Previously, a FileNotFoundError error is thrown when the parent directories are missing.

Testing

Added tests to tests/io/test_fsspec.py and tests/io/test_pyarrow.py
Added test_append_table test to write to the local filesystem, to illustrate the issue. It fails without the changes in this PR.
Also added the catalog_sqlite_fsspec catalog which tests writing to the local filesystem using FSSPEC_FILE_IO

Fokko

I see the need for this, but I'm reluctant to support this. You could pre-create the directories outside of PyIceberg if possible.

Fokko · 2024-01-24T19:46:54Z

 if not overwrite and self.exists() is True:
 raise FileExistsError(f"Cannot create file, already exists: {self.location}")
+ # Parent directories must be created first in certain file systems, such as the LocalFileSystem.
+ self._filesystem.create_dir(os.path.dirname(self._path), recursive=True)


This is typically something that we try to avoid. Iceberg is designed to work with object stores, and those don't have a notion of directories. One recommendation is even to disallow moves and listing of directories. One thing is also creating a directory. I'm not sure what the behavior is for the Arrow S3 implementation. Since some of the implementations still make a call(s):
For example, a list operation to check if the directory is there
For example, they touch a small file under the prefix to indicate that the path should be created.
Another concern is that we currently do this in Arrow, we also would need to do this for other implementations to avoid discrepancies. The concept of the FileIO is that you easily can swap them out.

@Fokko thanks for the review.
I agree with the above. The Arrow FileIO implementation might not be the best place to implement this behavior. So far both of the supported FS implementations (ARROW_FILE_IO and FSSPEC_FILE_IO) are failing to write to the local file system.
I want to make writes work for the local file system.
Looking at the Java side, there is a LocalOutputFile implementation which implements the behavior for creating parent directories.
Maybe we can implement a new FileIO implementation and make that the preferred implementation for the file:// scheme.
iceberg-python/pyiceberg/io/__init__.py
Line 278 in 4cf1f35
"file": [ARROW_FILE_IO],

Hey @kevinjqliu That sounds like a reasonable alternative to me 👍

Thinking a bit more about it, maybe we should still use the ArrowFileSystem in this new FileIO to make sure that we can read the table into an Arrow pa.Table.

@Fokko I updated the PR for both PyArrow and FsSpec implementations

kevinjqliu · 2024-01-26T02:34:42Z



 SCHEME_TO_FS = {
+ "": _file,


pyarrow defaults scheme to file when no scheme is present.
iceberg-python/pyiceberg/io/pyarrow.py
Line 322 in 8464d71
return"file", uri.netloc, os.path.abspath(location)
we essentially do the same here

Fokko

Thanks @kevinjqliu for working on this, elegant solution! LGTM

HonahX

LGTM! Thanks @kevinjqliu

kevinjqliu · 2024-01-24T18:03:24Z

Issue #299

This PR changes the behavior of both PyArrow and FsSpec file systems. When writing to the local file system, parent directories will be created first before writing to the file. Previously, a FileNotFoundError error is thrown when the parent directories are missing.

Testing

Added tests to tests/io/test_fsspec.py and tests/io/test_pyarrow.py
Added test_append_table test to write to the local filesystem, to illustrate the issue. It fails without the changes in this PR.
Also added the catalog_sqlite_fsspec catalog which tests writing to the local filesystem using FSSPEC_FILE_IO

Fokko

I see the need for this, but I'm reluctant to support this. You could pre-create the directories outside of PyIceberg if possible.

Fokko · 2024-01-24T19:46:54Z

 if not overwrite and self.exists() is True:
 raise FileExistsError(f"Cannot create file, already exists: {self.location}")
+ # Parent directories must be created first in certain file systems, such as the LocalFileSystem.
+ self._filesystem.create_dir(os.path.dirname(self._path), recursive=True)


This is typically something that we try to avoid. Iceberg is designed to work with object stores, and those don't have a notion of directories. One recommendation is even to disallow moves and listing of directories. One thing is also creating a directory. I'm not sure what the behavior is for the Arrow S3 implementation. Since some of the implementations still make a call(s):
For example, a list operation to check if the directory is there
For example, they touch a small file under the prefix to indicate that the path should be created.
Another concern is that we currently do this in Arrow, we also would need to do this for other implementations to avoid discrepancies. The concept of the FileIO is that you easily can swap them out.

@Fokko thanks for the review.
I agree with the above. The Arrow FileIO implementation might not be the best place to implement this behavior. So far both of the supported FS implementations (ARROW_FILE_IO and FSSPEC_FILE_IO) are failing to write to the local file system.
I want to make writes work for the local file system.
Looking at the Java side, there is a LocalOutputFile implementation which implements the behavior for creating parent directories.
Maybe we can implement a new FileIO implementation and make that the preferred implementation for the file:// scheme.
iceberg-python/pyiceberg/io/__init__.py
Line 278 in 4cf1f35
"file": [ARROW_FILE_IO],

Hey @kevinjqliu That sounds like a reasonable alternative to me 👍

Thinking a bit more about it, maybe we should still use the ArrowFileSystem in this new FileIO to make sure that we can read the table into an Arrow pa.Table.

@Fokko I updated the PR for both PyArrow and FsSpec implementations

kevinjqliu · 2024-01-26T02:34:42Z



 SCHEME_TO_FS = {
+ "": _file,


pyarrow defaults scheme to file when no scheme is present.
iceberg-python/pyiceberg/io/pyarrow.py
Line 322 in 8464d71
return"file", uri.netloc, os.path.abspath(location)
we essentially do the same here

Fokko

Thanks @kevinjqliu for working on this, elegant solution! LGTM

HonahX

LGTM! Thanks @kevinjqliu

add a test for writing data to disk
369df6e

Fokko reviewed Jan 24, 2024
View reviewed changes

kevinjqliu force-pushed the kevinjqliu/fix-write-local-fs branch from 752b63f to 1c480a5Compare January 26, 2024 01:20

create dir first
5751b80

kevinjqliu force-pushed the kevinjqliu/fix-write-local-fs branch from 5ea2ceb to 5c77796Compare January 26, 2024 02:12

kevinjqliu added 2 commits January 25, 2024 18:23

add fsspec test
4101dd1

make lint
57d68f2

kevinjqliu force-pushed the kevinjqliu/fix-write-local-fs branch from 5c77796 to 57d68f2Compare January 26, 2024 02:23

kevinjqliu commented Jan 26, 2024
View reviewed changes

kevinjqliu requested a review from Fokko January 26, 2024 02:35

add fs tests
242e365

Fokko approved these changes Jan 29, 2024
View reviewed changes

HonahX approved these changes Jan 29, 2024
View reviewed changes

Fokko merged commit f66e365 into apache:mainJan 29, 2024

Fokko added this to the PyIceberg 0.6.0 release milestone Jan 29, 2024

kevinjqliu deleted the kevinjqliu/fix-write-local-fs branch January 30, 2024 02:34

kevinjqliu mentioned this pull request Feb 24, 2024
Cleanup conftest, remove LocalOutputFile #468
Merged

add a test for writing data to disk
369df6e

Fokko reviewed Jan 24, 2024
View reviewed changes

kevinjqliu force-pushed the kevinjqliu/fix-write-local-fs branch from 752b63f to 1c480a5Compare January 26, 2024 01:20

create dir first
5751b80

kevinjqliu force-pushed the kevinjqliu/fix-write-local-fs branch from 5ea2ceb to 5c77796Compare January 26, 2024 02:12

kevinjqliu added 2 commits January 25, 2024 18:23

add fsspec test
4101dd1

make lint
57d68f2

kevinjqliu force-pushed the kevinjqliu/fix-write-local-fs branch from 5c77796 to 57d68f2Compare January 26, 2024 02:23

kevinjqliu commented Jan 26, 2024
View reviewed changes

kevinjqliu requested a review from Fokko January 26, 2024 02:35

add fs tests
242e365

Fokko approved these changes Jan 29, 2024
View reviewed changes

HonahX approved these changes Jan 29, 2024
View reviewed changes

Fokko merged commit f66e365 into apache:mainJan 29, 2024

Fokko added this to the PyIceberg 0.6.0 release milestone Jan 29, 2024

kevinjqliu deleted the kevinjqliu/fix-write-local-fs branch January 30, 2024 02:34

kevinjqliu mentioned this pull request Feb 24, 2024
Cleanup conftest, remove LocalOutputFile #468
Merged

add a test for writing data to disk
369df6e

Fokko reviewed Jan 24, 2024
View reviewed changes

kevinjqliu force-pushed the kevinjqliu/fix-write-local-fs branch from 752b63f to 1c480a5Compare January 26, 2024 01:20

create dir first
5751b80

kevinjqliu force-pushed the kevinjqliu/fix-write-local-fs branch from 5ea2ceb to 5c77796Compare January 26, 2024 02:12

kevinjqliu added 2 commits January 25, 2024 18:23

add fsspec test
4101dd1

make lint
57d68f2

kevinjqliu force-pushed the kevinjqliu/fix-write-local-fs branch from 5c77796 to 57d68f2Compare January 26, 2024 02:23

kevinjqliu commented Jan 26, 2024
View reviewed changes

kevinjqliu requested a review from Fokko January 26, 2024 02:35

add fs tests
242e365

Fokko approved these changes Jan 29, 2024
View reviewed changes

HonahX approved these changes Jan 29, 2024
View reviewed changes

Fokko merged commit f66e365 into apache:mainJan 29, 2024

Fokko added this to the PyIceberg 0.6.0 release milestone Jan 29, 2024

kevinjqliu deleted the kevinjqliu/fix-write-local-fs branch January 30, 2024 02:34

kevinjqliu mentioned this pull request Feb 24, 2024
Cleanup conftest, remove LocalOutputFile #468
Merged

add a test for writing data to disk
369df6e

Fokko reviewed Jan 24, 2024
View reviewed changes

kevinjqliu force-pushed the kevinjqliu/fix-write-local-fs branch from 752b63f to 1c480a5Compare January 26, 2024 01:20

create dir first
5751b80

kevinjqliu force-pushed the kevinjqliu/fix-write-local-fs branch from 5ea2ceb to 5c77796Compare January 26, 2024 02:12

kevinjqliu added 2 commits January 25, 2024 18:23

add fsspec test
4101dd1

make lint
57d68f2

kevinjqliu force-pushed the kevinjqliu/fix-write-local-fs branch from 5c77796 to 57d68f2Compare January 26, 2024 02:23

kevinjqliu commented Jan 26, 2024
View reviewed changes

kevinjqliu requested a review from Fokko January 26, 2024 02:35

add fs tests
242e365

Fokko approved these changes Jan 29, 2024
View reviewed changes

HonahX approved these changes Jan 29, 2024
View reviewed changes

Fokko merged commit f66e365 into apache:mainJan 29, 2024

Fokko added this to the PyIceberg 0.6.0 release milestone Jan 29, 2024

kevinjqliu deleted the kevinjqliu/fix-write-local-fs branch January 30, 2024 02:34

kevinjqliu mentioned this pull request Feb 24, 2024
Cleanup conftest, remove LocalOutputFile #468
Merged

add a test for writing data to disk
369df6e

Fokko reviewed Jan 24, 2024
View reviewed changes

kevinjqliu force-pushed the kevinjqliu/fix-write-local-fs branch from 752b63f to 1c480a5Compare January 26, 2024 01:20

create dir first
5751b80

kevinjqliu force-pushed the kevinjqliu/fix-write-local-fs branch from 5ea2ceb to 5c77796Compare January 26, 2024 02:12

kevinjqliu added 2 commits January 25, 2024 18:23

add fsspec test
4101dd1

make lint
57d68f2

kevinjqliu force-pushed the kevinjqliu/fix-write-local-fs branch from 5c77796 to 57d68f2Compare January 26, 2024 02:23

kevinjqliu commented Jan 26, 2024
View reviewed changes

kevinjqliu requested a review from Fokko January 26, 2024 02:35

add fs tests
242e365

Fokko approved these changes Jan 29, 2024
View reviewed changes

HonahX approved these changes Jan 29, 2024
View reviewed changes

Fokko merged commit f66e365 into apache:mainJan 29, 2024

Fokko added this to the PyIceberg 0.6.0 release milestone Jan 29, 2024

kevinjqliu deleted the kevinjqliu/fix-write-local-fs branch January 30, 2024 02:34

kevinjqliu mentioned this pull request Feb 24, 2024
Cleanup conftest, remove LocalOutputFile #468
Merged

add a test for writing data to disk
369df6e

Fokko reviewed Jan 24, 2024
View reviewed changes

kevinjqliu force-pushed the kevinjqliu/fix-write-local-fs branch from 752b63f to 1c480a5Compare January 26, 2024 01:20

create dir first
5751b80

kevinjqliu force-pushed the kevinjqliu/fix-write-local-fs branch from 5ea2ceb to 5c77796Compare January 26, 2024 02:12

kevinjqliu added 2 commits January 25, 2024 18:23

add fsspec test
4101dd1

make lint
57d68f2

kevinjqliu force-pushed the kevinjqliu/fix-write-local-fs branch from 5c77796 to 57d68f2Compare January 26, 2024 02:23

kevinjqliu commented Jan 26, 2024
View reviewed changes

kevinjqliu requested a review from Fokko January 26, 2024 02:35

add fs tests
242e365

Fokko approved these changes Jan 29, 2024
View reviewed changes

HonahX approved these changes Jan 29, 2024
View reviewed changes

Fokko merged commit f66e365 into apache:mainJan 29, 2024

Fokko added this to the PyIceberg 0.6.0 release milestone Jan 29, 2024

kevinjqliu deleted the kevinjqliu/fix-write-local-fs branch January 30, 2024 02:34

kevinjqliu mentioned this pull request Feb 24, 2024
Cleanup conftest, remove LocalOutputFile #468
Merged

add a test for writing data to disk
369df6e

Fokko reviewed Jan 24, 2024
View reviewed changes

kevinjqliu force-pushed the kevinjqliu/fix-write-local-fs branch from 752b63f to 1c480a5Compare January 26, 2024 01:20

create dir first
5751b80

kevinjqliu force-pushed the kevinjqliu/fix-write-local-fs branch from 5ea2ceb to 5c77796Compare January 26, 2024 02:12

kevinjqliu added 2 commits January 25, 2024 18:23

add fsspec test
4101dd1

make lint
57d68f2

kevinjqliu force-pushed the kevinjqliu/fix-write-local-fs branch from 5c77796 to 57d68f2Compare January 26, 2024 02:23

kevinjqliu commented Jan 26, 2024
View reviewed changes

kevinjqliu requested a review from Fokko January 26, 2024 02:35

add fs tests
242e365

Fokko approved these changes Jan 29, 2024
View reviewed changes

HonahX approved these changes Jan 29, 2024
View reviewed changes

Fokko merged commit f66e365 into apache:mainJan 29, 2024

Fokko added this to the PyIceberg 0.6.0 release milestone Jan 29, 2024

kevinjqliu deleted the kevinjqliu/fix-write-local-fs branch January 30, 2024 02:34

kevinjqliu mentioned this pull request Feb 24, 2024
Cleanup conftest, remove LocalOutputFile #468
Merged

add a test for writing data to disk
369df6e

Fokko reviewed Jan 24, 2024
View reviewed changes

kevinjqliu force-pushed the kevinjqliu/fix-write-local-fs branch from 752b63f to 1c480a5Compare January 26, 2024 01:20

create dir first
5751b80

kevinjqliu force-pushed the kevinjqliu/fix-write-local-fs branch from 5ea2ceb to 5c77796Compare January 26, 2024 02:12

kevinjqliu added 2 commits January 25, 2024 18:23

add fsspec test
4101dd1

make lint
57d68f2

kevinjqliu force-pushed the kevinjqliu/fix-write-local-fs branch from 5c77796 to 57d68f2Compare January 26, 2024 02:23

kevinjqliu commented Jan 26, 2024
View reviewed changes

kevinjqliu requested a review from Fokko January 26, 2024 02:35

add fs tests
242e365

Fokko approved these changes Jan 29, 2024
View reviewed changes

HonahX approved these changes Jan 29, 2024
View reviewed changes

Fokko merged commit f66e365 into apache:mainJan 29, 2024

Fokko added this to the PyIceberg 0.6.0 release milestone Jan 29, 2024

kevinjqliu deleted the kevinjqliu/fix-write-local-fs branch January 30, 2024 02:34

kevinjqliu mentioned this pull request Feb 24, 2024
Cleanup conftest, remove LocalOutputFile #468
Merged

Uh oh!

Conversation

kevinjqliu commented Jan 24, 2024• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Testing

Uh oh!

Fokko left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

FokkoJan 25, 2024• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kevinjqliuJan 26, 2024• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Fokko left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

HonahX left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

kevinjqliu commented Jan 24, 2024• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Testing

Uh oh!

Fokko left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

FokkoJan 25, 2024• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kevinjqliuJan 26, 2024• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Fokko left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

HonahX left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

kevinjqliu commented Jan 24, 2024•
edited
Loading

FokkoJan 25, 2024•
edited
Loading

kevinjqliuJan 26, 2024•
edited
Loading

Fokko left a comment •
edited
Loading

kevinjqliu commented Jan 24, 2024•
edited
Loading

FokkoJan 25, 2024•
edited
Loading

kevinjqliuJan 26, 2024•
edited
Loading

Fokko left a comment •
edited
Loading

kevinjqliu commented Jan 24, 2024•
edited
Loading

FokkoJan 25, 2024•
edited
Loading

kevinjqliuJan 26, 2024•
edited
Loading

Fokko left a comment •
edited
Loading