Skip to content
This repository was archived by the owner on Mar 2, 2026. It is now read-only.

feat: add primary pipeline stages - #1048

Merged
daniel-sanche merged 194 commits into
pipeline_queries_approvedfrom
pipeline_queries_3_stable_stages
Jul 16, 2025
Merged

feat: add primary pipeline stages#1048
daniel-sanche merged 194 commits into
pipeline_queries_approvedfrom
pipeline_queries_3_stable_stages

Conversation

@daniel-sanche

@daniel-sanchedaniel-sanche commented May 13, 2025•
edited
Loading

Copy link
Copy Markdown
Contributor

This PR builds on the previous to add all currently supported additional stages and expressions

There are a lot of new files to review, but review should be fairly straightforward - just adding new expression and stage subclasses as introduced previously

This PR also adds end to end testing through test_pipeline_acceptance.py, which runs test cases in pipelines_e2e.yaml

@daniel-sanche
added skeletons for pipeline expressions
1087d64
@daniel-sanche
added quick implemtation for expressions
053f55b
@daniel-sanche
added default implementation for pipeline stages
e56459c
@daniel-sanche
ran black
babafb2
@daniel-sanche
got code to run
149c61f
@daniel-sanche
moved helpers
320cea1
@daniel-sanche
added basic pipelines.py file
cd2963d
@daniel-sanche
added yaml test file
3cbc2d5
@daniel-sanche
wrote basic parser for pipeline yaml
0c2241d
@daniel-sanche
encoded extra java system tests
9c8982c
@daniel-sanche
reconstruct pipeline expr objects
863bd1d
@daniel-sanche
got yaml to run
7cd2c11
@daniel-sanche
add data loading code
c328934
@daniel-sanche
updated protos
de0a862
@daniel-sanche
added pyyaml to system test dependencies
45c83aa
@daniel-sanche
added Expr methods
fc57f04
@daniel-sanche
trying to improve accumulator api
0224f94
@daniel-sanche
improved aggregate/accumulators
cdee937
@daniel-sanche
fixed naming in yaml
5d35c54
@daniel-sanche
create objects in parsing code
11373e3
@daniel-sanche
use order enum
d2b4153
@daniel-sanche
standardize how I deal with map stages
bfdfba3
@daniel-sanche
fix broke super.__init__ calls
afa823a
@daniel-sanche
added repr for custom classes
3348127
@daniel-sanche
added repr to pipeline
e3c995d
@daniel-sanche
fixed vector formatting
2d604e1
@daniel-sanche
only treat capitalized strings as possible exprs
3bed807
@daniel-sanche
Where uses positional args in yaml
9b9eaa8
@daniel-sanche
support union stage
e35accf
@daniel-sanche
fixed testReplace
72301a1
@daniel-sanche
daniel-sanche changed the base branch from pipeline_queries_2_query_parity to pipeline_queries_approvedJune 23, 2025 21:50
@daniel-sanche
Squashed commit of the following:
3de11fa
commit ad3e3df
Author: Daniel Sanche <d.sanche14@gmail.com>
Date: Fri Jun 20 17:44:37 2025 -0700
added .pipeline() to aggregation query
commit 4ce1b91
Author: Daniel Sanche <d.sanche14@gmail.com>
Date: Fri Jun 20 15:17:20 2025 -0700
added exprs and stages needed for aggregation pipelines
commit e7d8e52
Merge: 0d1535517e71b9
Author: Daniel Sanche <d.sanche14@gmail.com>
Date: Tue Jun 17 16:57:45 2025 -0700
Merge branch 'pipeline_queries_approved' into pipeline_queries_2_query_parity
commit 17e71b9
Author: Daniel Sanche <sanche@google.com>
Date: Tue Jun 17 15:54:31 2025 -0700
feat: add pipelines structure (#1046)
commit 0d15355
Author: Daniel Sanche <d.sanche14@gmail.com>
Date: Tue Jun 17 15:52:38 2025 -0700
fixed tests
commit bc25930
Merge: 6351ae713389b8
Author: Daniel Sanche <d.sanche14@gmail.com>
Date: Tue Jun 17 15:38:05 2025 -0700
Merge branch 'pipeline_queries_1_stubs' into pipeline_queries_2_query_parity
commit 6351ae7
Author: Daniel Sanche <d.sanche14@gmail.com>
Date: Tue Jun 17 15:35:15 2025 -0700
merged PR #1
commit 13389b8
Author: Daniel Sanche <d.sanche14@gmail.com>
Date: Mon Jun 16 15:15:26 2025 -0700
fixed mypy
commit 06a2084
Author: Daniel Sanche <d.sanche14@gmail.com>
Date: Mon Jun 16 14:57:54 2025 -0700
added generic_stage method to base_pipeline
commit 8a9c3ec
Author: Daniel Sanche <d.sanche14@gmail.com>
Date: Mon Jun 16 14:55:50 2025 -0700
made stages private
commit a818f52
Author: Daniel Sanche <d.sanche14@gmail.com>
Date: Mon Jun 16 14:28:20 2025 -0700
removed converter reference
commit e74e04d
Author: Daniel Sanche <d.sanche14@gmail.com>
Date: Mon Jun 16 14:22:13 2025 -0700
added separate stream/execute methods
commit 64cd4fb
Author: Daniel Sanche <d.sanche14@gmail.com>
Date: Thu Jun 12 11:22:25 2025 -0700
fixed comment
commit 3432322
Author: Daniel Sanche <d.sanche14@gmail.com>
Date: Tue Jun 10 17:03:09 2025 -0700
fixed lint
@daniel-sanche
Merge branch 'pipeline_queries_approved' into pipeline_queries_3_stab…
50f7d7b
…le_stages
@daniel-sanche
fixed broken tests
3f675b7
@daniel-sanche
fixed coverage
b81c637
@daniel-sanche
fixed system test
1f0595a
@daniel-sanche
removed query -> pipeline conversions
4552918
@daniel-sanche
simplified event loop
1ff2053
@daniel-sanche
cleaned up test file
f732f01
@daniel-sanche
daniel-sancheforce-pushed the pipeline_queries_3_stable_stages branch from 8df814c to f732f01CompareJune 24, 2025 22:44
@daniel-sanchedaniel-sanche changed the title [DRAFT] feat: add primary pipeline stagesfeat: add primary pipeline stagesJun 24, 2025
@daniel-sanche
daniel-sanche marked this pull request as ready for review June 24, 2025 22:47
@daniel-sanche
daniel-sanche requested review from a teamJune 24, 2025 22:47
bhshkh
bhshkh reviewed Jul 1, 2025
View reviewed changes
Comment threadgoogle/cloud/firestore_v1/_pipeline_stages.py
Comment threadgoogle/cloud/firestore_v1/base_pipeline.py
Comment threadgoogle/cloud/firestore_v1/pipeline_expressions.py
class Add(Function):
"""Represents the addition function."""

def __init__(self, left: Expr, right: Expr):

@bhshkhbhshkhJul 1, 2025

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Java and node have convenience functions which accept string as first arg. Are these needed in Python too ?

@daniel-sanchedaniel-sancheJul 1, 2025

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Those convenience functions are implemented on the Expr class, so they can be used infix. Example: Field.of("cost").add(5)

It seems like that's how Java is implemented too? Unless I missed another implementation somewhere

@bhshkhbhshkhJul 1, 2025

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These

https://github.com/googleapis/java-firestore/blob/6f640d021aeeb7f79c89d3d5e3dc6f6897e92456/google-cloud-firestore/src/main/java/com/google/cloud/firestore/pipeline/expressions/Function.java#L108-L144

@daniel-sanchedaniel-sancheJul 11, 2025

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see, good catch. I added those!

bhshkh
bhshkh reviewed Jul 1, 2025
View reviewed changes
Comment threadgoogle/cloud/firestore_v1/pipeline_expressions.py
Comment threadgoogle/cloud/firestore_v1/_pipeline_stages.py

@bhshkhbhshkhJul 1, 2025

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Replace stage is missing. Something similar to https://github.com/googleapis/java-firestore/blob/6f640d021aeeb7f79c89d3d5e3dc6f6897e92456/google-cloud-firestore/src/main/java/com/google/cloud/firestore/pipeline/stages/Replace.java#L26

@daniel-sanchedaniel-sancheJul 1, 2025

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is replace working for you? I was getting errors, so I put it off for the follow up PR.

I get google.api_core.exceptions.InvalidArgument: 400 Unsupported stage: replace

Comment threadgoogle/cloud/firestore_v1/pipeline_expressions.py
Comment threadgoogle/cloud/firestore_v1/pipeline_expressions.py
Comment threadgoogle/cloud/firestore_v1/pipeline_expressions.py
@daniel-sanche
daniel-sanche merged commit 6e06336 into pipeline_queries_approvedJul 16, 2025
7 checks passed
@daniel-sanche
daniel-sanche deleted the pipeline_queries_3_stable_stages branch July 16, 2025 17:57
Sign up for freeto subscribe to this conversation on GitHub. Already have an account? Sign in.

Reviewers

1 more reviewer
@bhshkhbhshkhbhshkh approved these changes
Reviewers whose approvals may not affect merge requirements

Assignees

@bhshkhbhshkh

Labels

api: firestoreIssues related to the googleapis/python-firestore API.size: xlPull request size is extra large.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@daniel-sanche@bhshkh@kevkim-codes