Skip to content

Add ResidualVisitor to compute residuals - #1388

Merged
Fokko merged 33 commits into
apache:mainfrom
tusharchou:main
Feb 11, 2025
Merged

Add ResidualVisitor to compute residuals#1388
Fokko merged 33 commits into
apache:mainfrom
tusharchou:main

Conversation

@tusharchou

@tusharchoutusharchou commented Nov 28, 2024

Copy link
Copy Markdown
Contributor

closes issue: Count rows as a metadata-only operation #1223

kevinjqliu
kevinjqliu reviewed Nov 28, 2024
View reviewed changes
Comment threadpyiceberg/table/__init__.py
@jayceslesar

jayceslesar commented Dec 2, 2024

Copy link
Copy Markdown
Contributor

Question: Does it make sense to expose this as the __len__ dunder method because python? It would just return the self.count()

@tusharchoutusharchou mentioned this pull request Dec 3, 2024
Closed
gli-chris-hao
gli-chris-hao reviewed Dec 31, 2024
View reviewed changes
Comment threadpyiceberg/table/__init__.py
@tusharchou
Merge branch 'apache:main' into main
0019f92
@tusharchou
Merge branch 'apache:main' into main
a372a93
@tusharchou
Gh 1223 metadata only row count (#5)
ab4c000
* added residual evaluator in plan files
* tested counts with positional deletes
* merged main
* implemented batch reader in count
* breaking integration test
* fixed integration test
* git pull main
* revert
* revert
* revert test_partitioning_key.py
* revert test_parser.py
* added residual evaluator in visitor
* deleted residual_evaluator.py
* removed test count from test_sql.py
* ignored lint type
* fixed lint
* working on plan_files
* type ignored
* make lint
@tusharchou

tusharchou commented Jan 6, 2025

Copy link
Copy Markdown
ContributorAuthor

Hi @Fokko@kevinjqliu@gli-chris-hao ,

I have implemented these suggestions with my best understanding.

  • residual evaluator
  • positional deletes
  • batch processing of files larger than 512mb

It would be helpful to get fresh review

@FokkoFokko changed the title Count rows as a metadata only operationAdd ResidualVisitor to compute residualsJan 13, 2025
Fokko
Fokko reviewed Feb 6, 2025
View reviewed changes
Comment threadpyiceberg/expressions/visitors.py Outdated
Fokko
Fokko reviewed Feb 6, 2025
View reviewed changes
Comment threadpyiceberg/expressions/visitors.py Outdated
Fokko
Fokko reviewed Feb 6, 2025
View reviewed changes
Comment threadpyiceberg/expressions/visitors.py Outdated
Fokko
Fokko reviewed Feb 6, 2025
View reviewed changes
Comment threadpyiceberg/expressions/visitors.py Outdated
Fokko
Fokko reviewed Feb 6, 2025
View reviewed changes
Comment threadpyiceberg/expressions/visitors.py Outdated
Fokko
Fokko reviewed Feb 6, 2025
View reviewed changes
Comment threadpyiceberg/expressions/visitors.py Outdated
Fokko
Fokko reviewed Feb 6, 2025
View reviewed changes
Comment threadpyiceberg/expressions/visitors.py Outdated
Fokko
Fokko reviewed Feb 6, 2025
View reviewed changes
Comment threadpyiceberg/expressions/visitors.py Outdated
Fokko
Fokko reviewed Feb 6, 2025
View reviewed changes
Comment threadpyiceberg/table/__init__.py Outdated
@tusharchou
Merge branch 'apache:main' into main
c9805f2
@tusharchou
Gh 1223 metadata only row count (#8)
f46739e
* added residual evaluator in plan files
* tested counts with positional deletes
* merged main
* implemented batch reader in count
* breaking integration test
* fixed integration test
* git pull main
* revert
* revert
* revert test_partitioning_key.py
* revert test_parser.py
* added residual evaluator in visitor
* deleted residual_evaluator.py
* removed test count from test_sql.py
* ignored lint type
* fixed lint
* working on plan_files
* type ignored
* make lint
* explicit delete files len is zero
* residual eval only if manifest is true
* default residual is always true
* used projection schema
* refactored residual in plan files
* fixed lint issue with isnan
* simplified count if else conditions
* implemented refactoring comments on residual visitor
@tusharchou
tusharchou requested a review from FokkoFebruary 7, 2025 03:48
Fokko added a commit to Fokko/iceberg-python that referenced this pull request Feb 10, 2025
@Fokko
Break circular import
6d0935a
Sometime I'm seeing this:
```
ImportError while loading conftest '/home/runner/work/iceberg-python/iceberg-python/tests/conftest.py'.
tests/conftest.py:52: in <module>
from pyiceberg.catalog import Catalog, load_catalog
pyiceberg/catalog/__init__.py:51: in <module>
from pyiceberg.serializers import ToOutputFile
pyiceberg/serializers.py:25: in <module>
from pyiceberg.table.metadata import TableMetadata, TableMetadataUtil
pyiceberg/table/__init__.py:65: in <module>
from pyiceberg.io.pyarrow import ArrowScan, schema_to_pyarrow
pyiceberg/io/pyarrow.py:141: in <module>
from pyiceberg.table.locations import load_location_provider
pyiceberg/table/locations.py:25: in <module>
from pyiceberg.table import TableProperties
E ImportError: cannot import name 'TableProperties' from partially initialized module 'pyiceberg.table' (most likely due to a circular import) (/home/runner/work/iceberg-python/iceberg-python/pyiceberg/table/__init__.py)
```
Also observed in: apache#1388
I prefer the imports at the top, but I think this is a small price
to pay to avoid having circular imports.
@FokkoFokko mentioned this pull request Feb 10, 2025
Merged
kevinjqliu pushed a commit that referenced this pull request Feb 10, 2025
@Fokko
Break circular import (#1639)
c2529b9
Sometime I'm seeing this:
```
ImportError while loading conftest '/home/runner/work/iceberg-python/iceberg-python/tests/conftest.py'.
tests/conftest.py:52: in <module>
from pyiceberg.catalog import Catalog, load_catalog
pyiceberg/catalog/__init__.py:51: in <module>
from pyiceberg.serializers import ToOutputFile
pyiceberg/serializers.py:25: in <module>
from pyiceberg.table.metadata import TableMetadata, TableMetadataUtil
pyiceberg/table/__init__.py:65: in <module>
from pyiceberg.io.pyarrow import ArrowScan, schema_to_pyarrow
pyiceberg/io/pyarrow.py:141: in <module>
from pyiceberg.table.locations import load_location_provider
pyiceberg/table/locations.py:25: in <module>
from pyiceberg.table import TableProperties
E ImportError: cannot import name 'TableProperties' from partially initialized module 'pyiceberg.table' (most likely due to a circular import) (/home/runner/work/iceberg-python/iceberg-python/pyiceberg/table/__init__.py)
```
Also observed in: #1388
I prefer the imports at the top, but I think this is a small price to
pay to avoid having circular imports.
Fokko
Fokko reviewed Feb 11, 2025
View reviewed changes
Comment threadtests/expressions/test_residual_evaluator.py Outdated
Fokko
Fokko reviewed Feb 11, 2025
View reviewed changes
Comment threadtests/expressions/test_residual_evaluator.py Outdated
Fokko
Fokko approved these changes Feb 11, 2025
View reviewed changes

@FokkoFokko left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more. 

deftest_fokko(session_catalog):
importpyarrow.parquetaspqdf=pq.read_table("/Users/fokko.driesprong/Downloads/yellow_tripdata_2024-01.parquet")
session_catalog.drop_table("default.taxis")
tbl=session_catalog.create_table("default.taxis", schema=df.schema)
withtbl.update_spec() asspec:
spec.add_field("tpep_pickup_datetime", DayTransform())
tbl.append(df)
# tbl = session_catalog.load_table("default.taxis")pred=LessThan("tpep_pickup_datetime", datetime.datetime(2024, 1, 3, 12, 0, 0))
cnt=tbl.scan(row_filter=pred).count()
assertcnt==len(df.filter(expression_to_pyarrow(pred.bind(tbl.schema()))))

Works very well:

image

Fokko
Fokko reviewed Feb 11, 2025
View reviewed changes
Comment threadpyiceberg/table/__init__.py Outdated
@Fokko
Fokko merged commit 4ff1558 into apache:mainFeb 11, 2025
@Fokko

Fokko commented Feb 11, 2025

Copy link
Copy Markdown
Contributor

Thanks @tusharchou for working on this 🚀

This was referenced Feb 11, 2025
Merged
Closed
kevinjqliu pushed a commit that referenced this pull request Feb 11, 2025
@Fokko
Remove_task_to_table(#1643)
509713b
Seems not being used. Less is more!
Noticed this while reviewing
#1388
@mrutunjay-kinagi

mrutunjay-kinagi commented Feb 11, 2025

Copy link
Copy Markdown
Contributor

🚀

@tushar-choudhary-tctushar-choudhary-tc mentioned this pull request Sep 3, 2025
Open
3 tasks
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@FokkoFokkoFokko approved these changes
@gli-chris-haogli-chris-haoAwaiting requested review from gli-chris-hao
@kevinjqliukevinjqliuAwaiting requested review from kevinjqliu

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@tusharchou@jayceslesar@Fokko@mrutunjay-kinagi@kevinjqliu@gli-chris-hao