Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
41 changes: 41 additions & 0 deletions .github/CODE_OF_CONDUCT.md
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,41 @@
# Code of Conduct

OpenChainBench publishes performance numbers about real services run by real people. The conversations around those numbers can get heated — especially when a provider's reputation is tied to the data. This document is short and direct so the bar is clear.

## Expected behavior

- **Critique numbers, not people.** It's fair to question a methodology, a sample window, a regional bias. It's not fair to suggest someone is incompetent or dishonest.
- **Cite, don't assert.** If you think a published figure is wrong, file a [provider correction](https://github.com/OpenChainBench/OpenChainBench/issues/new?template=provider-correction.yml) with a reproducer. "This number is bullshit" without a reproducer is a non-issue.
- **Disclose conflicts.** If you work for a provider that's measured here, say so when you participate in discussions about that provider. It doesn't disqualify your input — it just keeps the conversation honest.
- **Respect contributors' time.** Maintainers are not obligated to debate. If a thread is going in circles, a maintainer may close it with a dated decision and that decision stands until new evidence is filed.

## Unacceptable behavior

- Harassment, personal attacks, or sustained disruption of discussion.
- Posting private communications without consent.
- Deliberately misleading benchmarks or harness submissions designed to flatter a provider you're affiliated with.
- Targeting individual contributors based on their employer, nationality, or any protected characteristic.

## Enforcement

A maintainer may, at their discretion:

- Edit, lock, or delete issues, PRs, comments, and discussions that violate this code.
- Block users from the repository.
- Revert merged PRs that contain manipulated data or undisclosed conflicts of interest.

Decisions are recorded in the affected thread.

## Reporting

For private reports — including suspected manipulation, harassment, or undisclosed conflicts of interest — email **conduct@openchainbench.xyz** or open a [private security advisory](https://github.com/OpenChainBench/OpenChainBench/security/advisories/new) (the same channel works for non-security disclosures).

We treat reporters' identities as confidential by default.

## Scope

This Code of Conduct applies to all OpenChainBench spaces — issues, PRs, discussions, the Discord/Telegram if those exist, and any in-person event held under the project's name.

---

Loosely adapted from the [Contributor Covenant 2.1](https://www.contributor-covenant.org/version/2/1/code_of_conduct/), trimmed for a small, technical project.
77 changes: 77 additions & 0 deletions .github/DISCUSSIONS_AND_PROJECT_SETUP.md
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,77 @@
# Discussions & Project board — manual setup

This file documents the GitHub UI configuration that lives outside the repo (Discussions categories, the Project board). It exists so a maintainer can recreate the setup if it gets lost or migrated.

## Discussions

**Setup path:** Repo → Discussions tab → ⚙️ "Manage categories"

| Category | Type | Description |
| --- | --- | --- |
| 📣 Announcements | Announcement (maintainers only) | New benchmarks shipped, methodology changes, infrastructure migrations. |
| 💡 Ideas | Open-ended discussion | Brainstorm new benchmarks, providers, or methodology improvements before writing them up. |
| 🙋 Q&A | Question / Answer | Ask how a spec / harness / metric works. Pick a best answer when one is given. |
| 📊 Show & tell | Open-ended discussion | Share a fork, a derived dashboard, a benchmark you ran locally. |

Default categories like "General" and "Polls" should be deleted to keep the surface focused.

## Roadmap Project

**Setup path:** Org → Projects tab → New project → "From scratch" → Board template

- **Name**: `OpenChainBench Roadmap`
- **Visibility**: Public
- **Linked repository**: `OpenChainBench/OpenChainBench`

### Columns

```
Requested → Approved → In progress → Live → Archived
```

| Column | Meaning |
| --- | --- |
| Requested | A bench has been proposed (issue with `bench-request` label or a Discussion thread). Methodology not yet validated by maintainers. |
| Approved | Methodology agreed on. Spec / harness work can start. |
| In progress | A PR is open implementing the benchmark. |
| Live | Merged + harness wired into Railway + emitting data. |
| Archived | Deprecated, retired, or rejected with a dated note. |

### Workflows

In Project Settings → Workflows, enable:

- **Auto-add to project** — for issues in `OpenChainBench/OpenChainBench` matching the filter `is:issue label:bench-request`. They land in `Requested` automatically.
- **Item closed → Archived** — when an issue is closed without a `live` label, move it to `Archived`.
- **PR merged → Live** (manual; GitHub doesn't have a native trigger, so a maintainer moves the card after merge).

### Custom fields (optional)

- `Category` (single-select): RPC / Aggregator / Bridge / Price feed / Wallet / Other.
- `Priority` (single-select): P0 / P1 / P2.
- `Owner` (text or person reference).

## Labels

These labels back the issue templates and the project workflow. Create them under Repo → Issues → Labels:

| Label | Color | Description |
| --- | --- | --- |
| `bench-request` | `#0e8a16` (green) | Used by `new-benchmark.yml` template; auto-adds to the Project board. |
| `data-quality` | `#fbca04` (yellow) | Used by `data-quality.yml`; site / data anomaly. |
| `correction` | `#d93f0b` (orange) | Provider-submitted correction. |
| `provider` | `#5319e7` (purple) | Combined with `correction` to mark provider-affiliated issues. |
| `live` | `#0366d6` (blue) | Bench is published and emitting data; survives an issue's lifetime. |
| `harness-bug` | `#b60205` (red) | A harness has stopped emitting or is reporting wrong values. |
| `methodology` | `#5319e7` (purple) | Discussion about how a benchmark is computed. |
| `good first issue` | `#7057ff` (purple) | Curated easy-entry tasks for new contributors. |
| `help wanted` | `#008672` (teal) | Maintainers explicitly want outside contribution. |

## Re-running this setup

If the Project or Discussions are wiped:

1. Re-create the Discussions categories per the table above.
2. Re-create the Project with the columns + workflows.
3. Add the labels listed in the Labels section.
4. Push a tiny change to this file documenting the date of the rebuild so the next maintainer knows when the configuration was last verified.
14 changes: 14 additions & 0 deletions .github/ISSUE_TEMPLATE/config.yml
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,14 @@
blank_issues_enabled: false
contact_links:
- name: 💡 Idea or rough proposal
url: https://github.com/OpenChainBench/OpenChainBench/discussions/new?category=ideas
about: Have an idea but not ready for a formal benchmark proposal? Start a discussion in Ideas.
- name: 🙋 Question (How do I…?)
url: https://github.com/OpenChainBench/OpenChainBench/discussions/new?category=q-a
about: Ask a question about a spec, harness, methodology, or how the site works.
- name: 📊 Show & tell
url: https://github.com/OpenChainBench/OpenChainBench/discussions/new?category=show-and-tell
about: Show a fork, a dashboard, or a benchmark you ran locally.
- name: 🔐 Security disclosure
url: https://github.com/OpenChainBench/OpenChainBench/security/advisories/new
about: Privately report a vulnerability. Do not file a public issue for security problems.
30 changes: 0 additions & 30 deletions .github/ISSUE_TEMPLATE/correction.md

This file was deleted.

62 changes: 62 additions & 0 deletions .github/ISSUE_TEMPLATE/data-quality.yml
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,62 @@
name: 🐞 Data quality issue
description: Report a number on the site that looks wrong, broken, or stale. For provider-specific corrections (you measured a different number for your own service), use the "Provider correction" template instead.
title: "Data: <bench number> · <short description>"
labels: ["data-quality"]
body:
- type: markdown
attributes:
value: |
Thanks for flagging this. Be specific so we can investigate quickly.
- type: dropdown
id: bench
attributes:
label: Which benchmark
options:
- "№ 001 · aggregator-head-lag"
- "№ 002 · bridge-quote-latency"
- "№ 003 · bridge-fee"
- "Other (specify in description)"
validations:
required: true
- type: dropdown
id: kind
attributes:
label: What kind of issue
options:
- "Number looks wrong (out of plausible range)"
- "Provider missing from the page"
- "Sparkline / chart not updating"
- "Last-run timestamp is stale"
- "Region label looks incorrect"
- "Page rendering / formatting bug"
- "Other"
validations:
required: true
- type: textarea
id: observed
attributes:
label: What you see
description: Paste the number, screenshot, or describe the symptom. Include the URL of the report.
validations:
required: true
- type: textarea
id: expected
attributes:
label: What you expected
description: Why does this look wrong to you? A reference number, a sanity check, an external dashboard.
validations:
required: false
- type: input
id: when
attributes:
label: Approximate time observed (UTC)
placeholder: e.g. "2026-04-30 14:20 UTC"
validations:
required: false
- type: textarea
id: anything
attributes:
label: Anything else
description: Context that might help narrow it down — region you're checking from, browser, recent harness changes, related issues.
validations:
required: false
33 changes: 0 additions & 33 deletions .github/ISSUE_TEMPLATE/new-benchmark.md

This file was deleted.

101 changes: 101 additions & 0 deletions .github/ISSUE_TEMPLATE/new-benchmark.yml
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,101 @@
name: 📊 Propose a benchmark
description: Suggest a new benchmark for OpenChainBench. Sketch the metric, the providers, the methodology — the goal is to align before code is written.
title: "Bench: <short description>"
labels: ["bench-request"]
body:
- type: markdown
attributes:
value: |
Thanks for proposing a benchmark.

OpenChainBench prefers to align on **what** is measured and **how** before any code is written. Fill out the sections below — fields marked required are the bare minimum needed for the maintainers to give meaningful feedback. If you're not sure on some of them, write what you have and leave a note.

Have a rough idea but not ready to fill all this? Open a thread in [Discussions → Ideas](https://github.com/OpenChainBench/OpenChainBench/discussions/categories/ideas) instead.
- type: input
id: metric
attributes:
label: What should be measured (one sentence)
description: The metric and the providers. Be specific.
placeholder: e.g. "Time-to-finality of an Ethereum block, measured by RPC providers across 3 regions."
validations:
required: true
- type: textarea
id: motivation
attributes:
label: Why it matters
description: Who cares about this number, and what decision does it inform?
placeholder: e.g. "Wallet UX teams choose RPCs based on confirmation speed; today no one publishes a like-for-like number."
validations:
required: true
- type: textarea
id: providers
attributes:
label: Providers in scope
description: One per line. Mark "(?)" next to any you want feedback on including.
placeholder: |
Alchemy
QuickNode
Infura
Ankr
Public RPCs (?)
validations:
required: true
- type: textarea
id: methodology
attributes:
label: Proposed methodology
description: How will the harness run? Cadence, timeout, success criteria, regions.
value: |
- Workload / inputs:
- Cadence:
- Timeout:
- Regions:
- What counts as success:
- What's excluded from latency aggregates:
validations:
required: true
- type: dropdown
id: hosting
attributes:
label: Where will the harness run?
description: Light = single HTTP loop, no wallets, no signing. Heavy = wallets, on-chain transactions, capital-at-risk.
options:
- "Light — happy to have OpenChainBench Railway host it"
- "Heavy — I'll host on my own infra and push to a public Prom endpoint"
- "Not sure yet"
validations:
required: true
- type: input
id: language
attributes:
label: Harness language / runtime (planned)
placeholder: e.g. Go 1.24, Bun, Python 3.13, Rust
validations:
required: false
- type: textarea
id: existing
attributes:
label: Existing data sources
description: Are you already running a similar harness? Got Prometheus metrics ready? Link/paste anything relevant.
placeholder: |
- Existing harness repo (private or public):
- Existing Prometheus instance:
- Existing metric names:
validations:
required: false
- type: textarea
id: anything
attributes:
label: Anything else
description: Caveats, prior art, related benchmarks, conflicts of interest.
validations:
required: false
- type: checkboxes
id: terms
attributes:
label: Confirm
options:
- label: I understand that the harness code, methodology, and any corrections will be published under the project's MIT/CC-BY-4.0 license.
required: true
- label: I have searched [existing benchmarks](https://github.com/OpenChainBench/OpenChainBench/tree/main/benchmarks) and [open issues](https://github.com/OpenChainBench/OpenChainBench/issues?q=is%3Aissue+label%3Abench-request) for duplicates.
required: true
Loading
Loading