Skip to content

Add Google Vertex AI Feature Store - Feature View Sync Operators, Sensor - #44891

Merged
potiuk merged 8 commits into
apache:mainfrom
CYarros10:feature-store
Dec 16, 2024
Merged

Add Google Vertex AI Feature Store - Feature View Sync Operators, Sensor#44891
potiuk merged 8 commits into
apache:mainfrom
CYarros10:feature-store

Conversation

@CYarros10

Copy link
Copy Markdown
Contributor

This PR adds support for Google Cloud Vertex AI Feature Store synchronization operations in Apache Airflow.

Vertex AI Feature Store is a managed, cloud-native feature store service that's integral to Vertex AI. It streamlines your ML feature management and online serving processes by letting you manage your feature data in a BigQuery table or view. You can then serve features online directly from the BigQuery data source.

Components Added

  1. SyncFeatureViewOperator: Triggers synchronization of a feature view, updating online serving data from the batch source
  2. GetFeatureViewSyncOperator: Retrieves status and details of a feature view sync operation
  3. FeatureViewSyncSensor: Monitors the progress of feature view sync operations
  4. FeatureStoreHook: Core integration with the Vertex AI Feature Store API

Usage Example

sync_task=SyncFeatureViewOperator(
task_id="sync_task",
project_id=PROJECT_ID,
location=REGION,
feature_online_store_id=FEATURE_ONLINE_STORE_ID,
feature_view_id=FEATURE_VIEW_ID,
)
wait_for_sync=FeatureViewSyncSensor(
task_id="wait_for_sync",
location=REGION,
feature_view_sync_name="{{ task_instance.xcom_pull(task_ids='sync_task', key='return_value')}}",
poke_interval=60, # Check every minutetimeout=600, # Timeout after 10 minutesmode="reschedule",
)
get_task=GetFeatureViewSyncOperator(
task_id="get_task",
location=REGION,
feature_view_sync_name="{{ task_instance.xcom_pull(task_ids='sync_task', key='return_value')}}",
)
sync_task>>wait_for_sync>>get_task

@boring-cyborgboring-cyborgBot added area:providers kind:documentation provider:google Google (including GCP) related issues labels Dec 12, 2024

@shahar1shahar1 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice! Got a small suggestion :-)
Could be merged IMO after resolving the above + @MaksYermak 's comments.

@MaksYermakMaksYermak left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@potiuk
potiuk merged commit 16022e0 into apache:mainDec 16, 2024
got686-yandex pushed a commit to got686-yandex/airflow that referenced this pull request Jan 30, 2025
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:providerskind:documentationprovider:googleGoogle (including GCP) related issues

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants

@CYarros10@potiuk@MaksYermak@shahar1