Skip to content

fix(js/vertexai): dedupe, improve types, fix rouge - #3592

Merged
ssbushi merged 7 commits into
genkit-ai:mainfrom
10kc-emilyk:vertex-evaluation-types-dedupe
Aug 10, 2026
Merged

fix(js/vertexai): dedupe, improve types, fix rouge#3592
ssbushi merged 7 commits into
genkit-ai:mainfrom
10kc-emilyk:vertex-evaluation-types-dedupe

Conversation

@10kc-emilyk

Copy link
Copy Markdown
Contributor

While investigating the appropriate types for vertex evaluators, I'm adding them here too, and discovered an incorrect type used for the ROUGE score evaluator. Also there was some code duplication with the types.

Checklist (if applicable):

  • PR title is following https://www.conventionalcommits.org/en/v1.0.0/
  • Tested (manually, unit tested, etc.) not yet, but i'll be doing some manual testing to verify that rouge was failing before this change
  • N/ADocs updated (updated docs or a docs bug required)

Comment threadjs/plugins/vertexai/src/evaluation/evaluation.ts
Comment threadjs/plugins/vertexai/src/evaluation/evaluation.ts
@MichaelDoyle

Copy link
Copy Markdown
Contributor

/gemini summary

@MichaelDoyle

Copy link
Copy Markdown
Contributor

/gemini review

@gemini-code-assist

Copy link
Copy Markdown
Contributor

Summary of Changes

This pull request refactors the Vertex AI evaluation plugin to improve type safety and maintainability. By leveraging official protobuf definitions and centralizing shared types, the codebase is now more robust and less prone to configuration errors. Additionally, the changes address specific implementation bugs in the ROUGE evaluator and fix minor naming inconsistencies.

Highlights

  • Type Safety Improvements: Replaced generic 'any' types with specific Google Cloud AI Platform protobuf types for metric specifications across all evaluators.
  • Code Deduplication: Centralized evaluation metric types and configurations into a shared 'types.ts' file, removing redundant definitions from the main evaluation logic.
  • Bug Fixes: Corrected the ROUGE evaluator implementation to use the expected array format for instances and fixed a typo in the 'FLUENCY' metric enum value.
Activity
  • The pull request was created by 10kc-emilyk.
  • The author identified and addressed code duplication between evaluation files.
  • The author noted a typo in the 'FLUENCY' metric enum value.
  • A summary and review were requested via automated commands.

@MichaelDoyle

Copy link
Copy Markdown
Contributor

@ssbushi can you take a peek?

@gemini-code-assistgemini-code-assistBot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request enhances type safety within the Vertex AI evaluation plugin by migrating evaluation types to a separate file and replacing any types with specific Protobuf definitions from @google-cloud/aiplatform. The evaluator factory and specific evaluator functions were updated to utilize these stricter types. Review feedback pointed out that type-casting datapoint.reference as a string in the BLEU and ROUGE evaluators could mask runtime errors if the value is undefined, suggesting the addition of explicit runtime validation checks.

Comment threadjs/plugins/vertexai/src/evaluation/evaluation.ts
Comment threadjs/plugins/vertexai/src/evaluation/evaluation.ts
@ssbushi
ssbushi enabled auto-merge (squash) August 10, 2026 18:05

@ssbushissbushi left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for your contribution. I made a few trivial tweak to get this ready for merging

@ssbushi
ssbushi merged commit 9fded1d into genkit-ai:mainAug 10, 2026
13 checks passed
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants

@10kc-emilyk@MichaelDoyle@ssbushi