Skip to content

Add initial conformance test - #48

Merged
williammartin merged 1 commit into
mainfrom
wm/conformance-test
Mar 28, 2025
Merged

Add initial conformance test#48
williammartin merged 1 commit into
mainfrom
wm/conformance-test

Conversation

@williammartin

@williammartinwilliammartin commented Mar 28, 2025

Copy link
Copy Markdown
Collaborator

Description

Given you have both github/github-mcp-server and mcp/github images available locally.

Requires -tags conformance to avoid polluting go test ./... with failures due to missing deps.

A failure looks like:

➜ github-mcp-server git:(wm/conformance-test) ✗ GITHUB_PERSONAL_ACCESS_TOKEN=foo go test -tags conformance -count 1 ./conformance
--- FAIL: TestCapabilities (0.57s)
conformance_test.go:121: waiting for anthropic server to start...
conformance_test.go:121: waiting for github server to start...
conformance_test.go:33: sending anthropic: {"jsonrpc":"2.0","id":1,"method":"initialize","params":{"protocolVersion":"2024-11-05","capabilities":{},"clientInfo":{"name":"ConformanceTest","version":"0.0.1"}}}
conformance_test.go:33: received from anthropic: {"result":{"protocolVersion":"2024-11-05","capabilities":{"tools":{}},"serverInfo":{"name":"github-mcp-server","version":"0.1.0"}},"jsonrpc":"2.0","id":1}
conformance_test.go:33: sending github: {"jsonrpc":"2.0","id":1,"method":"initialize","params":{"protocolVersion":"2024-11-05","capabilities":{},"clientInfo":{"name":"ConformanceTest","version":"0.0.1"}}}
conformance_test.go:33: received from github: {"jsonrpc":"2.0","id":1,"result":{"protocolVersion":"2024-11-05","capabilities":{"resources":{},"tools":{"listChanged":true}},"serverInfo":{"name":"github-mcp-server","version":"0.0.1"}}}
conformance_test.go:154: capabilities mismatch:
Resources
Tools.ListChanged

Notes

Lots of quality of life improvements to be had, lots of shortcuts taken, lots lots lots.

I tried to use testcontainers first to get some lifecycle management out of the box but it turns out they don't provide any way to get a handle to stdin.

I know the mark3labs module provides types for us, and probably we'll end up using them, but I was exploring the schema ground up as I was writing this, so it was exploratory for me too.

@SamMorrowDrumsSamMorrowDrums left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think that's great work to get started. I think lettings these evolve and not standing in the way is the best way to get this going, so this is a bit of a LGTM review, but I did glance over all the lines 😅

@williammartin

williammartin commented Mar 28, 2025

Copy link
Copy Markdown
CollaboratorAuthor

Well, the test currently fails so I don't want to merge it but thanks 😬 I'll probably put these behind a build tag.

@SamMorrowDrums

Copy link
Copy Markdown
Collaborator

Well, the test currently fails so I don't want to merge it but thanks 😬

Yeah I figured you'd fix it first, but whether you do that now, or in multiple PRs to me is secondary.

@williammartin
williammartinforce-pushed the wm/conformance-test branch 2 times, most recently from dbe5bd1 to 58b3a27CompareMarch 28, 2025 20:38
@williammartin
williammartin marked this pull request as ready for review March 28, 2025 20:39
}
}

func diffNonNilFields(a, b interface{}, path string) string {

Copy link
Copy Markdown
CollaboratorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Don't look too closely.

Copy link
Copy Markdown
CollaboratorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Avert your eyes.

@williammartinwilliammartinMar 28, 2025

Copy link
Copy Markdown
CollaboratorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is call of Cthulhu.

@williammartin
williammartin merged commit 75d71ad into mainMar 28, 2025
@williammartinwilliammartin mentioned this pull request Apr 22, 2025
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants

@williammartin@SamMorrowDrums