Uh oh!
There was an error while loading. Please reload this page.
feat(site): enrich prism board with detail, eras, and gpt-2 refs - #126
Conversation
Expose recipeEra, evalGroups, and public benchmarks on Prism board rows, plus submission detail and literature-baseline reference endpoints for the marketing FE modal.
📝 WalkthroughWalkthroughPrism now exposes enriched leaderboard and submission data, public submission details, telemetry, and frozen GPT-2 reference baselines. The change adds response types, payload enrichment, bounded detail fetching, diff-based pin recovery, data mapping, tests, and API documentation. ChangesPrism enrichment and public API
Estimated code review effort: 4 (Complex) | ~45 minutes Sequence Diagram(s)sequenceDiagram
participant Client
participant SiteAPI
participant Prism
participant Enrichment
Client->>SiteAPI: Request Prism submission detail
SiteAPI->>Prism: Fetch submission and optional diff
Prism-->>SiteAPI: Return payloads
SiteAPI->>Enrichment: Map and enrich public detail
Enrichment-->>SiteAPI: Return PrismSubmissionDetail
SiteAPI-->>Client: Return public response
Possibly related PRs
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
There was a problem hiding this comment.
Actionable comments posted: 5
🧹 Nitpick comments (1)
crates/site-api/src/handlers.rs (1)
740-750: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winExtract the shared
pin_idrecovery block.Lines 740-750 duplicate Lines 479-489 of
fetch_prism_detailsexactly: checkpin_id_from_payload, fetch/v1/submissions/{id}/diff, and insertpin_idinto the detail object. The two copies define the same recovery contract in two places, so a later change to the pin source has to be applied twice.Extract one helper and call it from both sites.
♻️ Proposed refactor
/// Recover `pin_id` from the submission diff when the detail payload lacks it.asyncfnbackfill_pin_id(st:&SiteState,id:&str,detail:&mutValue){ifpin_id_from_payload(detail).is_some(){return;}letSome(diff) = upstream::get_json_opt(st,PRISM,&format!("/v1/submissions/{id}/diff")).awaitelse{return;};ifletSome(pin) = diff.get("pin_id").and_then(Value::as_str){ifletSome(obj) = detail.as_object_mut(){ obj.insert("pin_id".into(),json!(pin));}}}Then call
backfill_pin_id(&st, &id, &mut detail).await;inget_prism_submission_detailandbackfill_pin_id(st, id, &mut detail).await;infetch_prism_details.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@crates/site-api/src/handlers.rs` around lines 740 - 750, Extract the duplicated pin_id recovery logic from get_prism_submission_detail and fetch_prism_details into a shared async backfill_pin_id helper. The helper should leave details unchanged when pin_id_from_payload already finds a value, otherwise fetch the submission diff and insert its string pin_id into the mutable detail object; call it from both existing sites with their respective state and submission ID references.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@crates/site-api/src/handlers.rs`:
- Around line 726-739: Validate the decoded id at the start of
get_prism_submission_detail before constructing the upstream URL, accepting only
plain submission identifiers and rejecting values containing path separators,
traversal segments, or other invalid characters with an appropriate client
error. Keep the existing upstream fetch and not-found behavior unchanged for
valid identifiers.
- Around line 468-494: Update fetch_prism_details to fetch IDs with bounded
concurrency and apply a per-request timeout to both submission-detail and
optional diff requests. Reuse the existing GatewayState.client and established
concurrency/timeout utilities or constants, ensuring at most the configured
number of Prism requests run simultaneously while preserving the current result
and pin_id enrichment behavior.
In `@crates/site-api/src/prism_enrich.rs`:
- Around line 81-105: Restrict the fallback in pin_id_from_payload to the same
AutoModel-pin contract as the preceding pointer loop. After reading
sub.get("pin_id"), trim the value and accept it only when non-empty and prefixed
with "automodel@", or remove this redundant fallback while preserving valid
AutoModel extraction.
- Around line 23-32: Update the GPT-2 baseline constants and GPT2_SOURCE_URL to
use a citable results artifact that explicitly identifies GPT-2, each metric,
and its evaluation configuration. Verify and correct the HellaSwag,
ARC-Challenge, PIQA, and WinoGrande values against that source, and remove or
withhold GPT2_BOOLQ until a source directly establishes the GPT-2 BoolQ result.
In `@docs/SITE_API.md`:
- Around line 42-63: The Prism miner documentation must reflect the new public
site API. Update docs/external-miner/prism.md with the detail and references
routes plus the new champion/leaderboard response fields described around
PrismSubmissionDetail and PrismReferenceBaseline, then apply the same
documentation changes to the public miner repository.
---
Nitpick comments:
In `@crates/site-api/src/handlers.rs`:
- Around line 740-750: Extract the duplicated pin_id recovery logic from
get_prism_submission_detail and fetch_prism_details into a shared async
backfill_pin_id helper. The helper should leave details unchanged when
pin_id_from_payload already finds a value, otherwise fetch the submission diff
and insert its string pin_id into the mutable detail object; call it from both
existing sites with their respective state and submission ID references.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: 02df525a-15bd-48b0-9ef4-f1d55d94d105
📒 Files selected for processing (6)
crates/site-api/src/handlers.rscrates/site-api/src/lib.rscrates/site-api/src/prism_enrich.rscrates/site-data/src/map.rscrates/site-types/src/types.rsdocs/SITE_API.md
| /// Fetch bounded challenge detail payloads keyed by submission id. | ||
| async fn fetch_prism_details(st: &SiteState, ids: &[String]) -> HashMap<String, Value> { | ||
| let mut out = HashMap::new(); | ||
| for id in ids { | ||
| if id.is_empty() { | ||
| continue; | ||
| } | ||
| if let Some(mut detail) = | ||
| upstream::get_json_opt(st, PRISM, &format!("/v1/submissions/{id}")).await | ||
| { | ||
| // Optional /diff for pin_id when detail alone lacks era signals. | ||
| if pin_id_from_payload(&detail).is_none() { | ||
| if let Some(diff) = | ||
| upstream::get_json_opt(st, PRISM, &format!("/v1/submissions/{id}/diff")).await | ||
| { | ||
| if let Some(pin) = diff.get("pin_id").and_then(Value::as_str) { | ||
| if let Some(obj) = detail.as_object_mut() { | ||
| obj.insert("pin_id".into(), json!(pin)); | ||
| } | ||
| } | ||
| } | ||
| } | ||
| out.insert(id.clone(), detail); | ||
| } | ||
| } | ||
| out | ||
| } |
There was a problem hiding this comment.
🚀 Performance & Scalability | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash# Description: Inspect the upstream helper for timeout and concurrency handling.set -euo pipefail
fd -t f 'upstream.rs' crates --exec cat -n {}
echo"=== reqwest client construction (timeout settings) ==="
rg -n -C 5 'Client::builder|\.timeout\(' --type=rustRepository: BaseIntelligence/base
Length of output: 3028
🏁 Script executed:
#!/bin/bashset -euo pipefail
echo"=== candidate files ==="
fd -t f . crates/site-api crates | rg '(^|/)(handlers|state|main|lib|upstream)\.rs$|Cargo\.toml$'echo"=== fetch_prism_details and callers ==="
rg -n -C 12 'fetch_prism_details|PRISM_CHAMPION_DETAIL_FANOUT|arenas/prism/(leaderboard|submissions)' crates
echo"=== SiteState and client construction ==="
rg -n -C 12 'struct SiteState|client:|reqwest::Client|Client::builder|Client::new|timeout\(' crates
echo"=== relevant handler structure ==="
sed -n '400,530p' crates/site-api/src/handlers.rsRepository: BaseIntelligence/base
Length of output: 50378
🏁 Script executed:
#!/bin/bashset -euo pipefail
echo"=== site-api files ==="
git ls-files crates/site-api
echo"=== Prism fan-out symbols and callers ==="
rg -n -C 8 'fetch_prism_details|PRISM_CHAMPION_DETAIL_FANOUT|arenas/prism/(leaderboard|submissions)' crates/site-api
echo"=== site-api state/client construction ==="
rg -n -C 10 'struct SiteState|reqwest::Client|Client::builder|Client::new|timeout\(' crates/site-api
echo"=== handlers relevant range ==="
sed -n '420,520p' crates/site-api/src/handlers.rs
echo"=== site-api dependency/runtime configuration ==="
rg -n -C 6 'site-api|SiteState|with_parts|reqwest' crates/site-api/Cargo.toml crates/*/src/main.rs crates/*/src/lib.rsRepository: BaseIntelligence/base
Length of output: 38783
🏁 Script executed:
#!/bin/bashset -euo pipefail
echo"=== all SiteState construction and site-router wiring ==="
rg -n -C 12 'SiteState::new|site_router\(|site-api|site_api|SiteState' crates/gateway crates/site-api --glob '*.rs'echo"=== all relevant async request calls in upstream helper ==="
cat -n crates/site-api/src/upstream.rs
echo"=== exact fan-out and request-count verifier ==="
python3 - <<'PY'from pathlib import Pathimport rehandlers = Path("crates/site-api/src/handlers.rs").read_text()upstream = Path("crates/site-api/src/upstream.rs").read_text()fanout = int(re.search( r'const\s+PRISM_CHAMPION_DETAIL_FANOUT:\s+usize\s*=\s*(\d+)', handlers,).group(1))detail_requests = handlers.count('"/v1/submissions/{id}"')diff_requests = handlers.count('"/v1/submissions/{id}/diff"')assert detail_requests == 1 and diff_requests == 1print({ "champion_fanout": fanout, "requests_per_id": detail_requests + diff_requests, "max_sequential_requests": fanout * (detail_requests + diff_requests), "helper_awaits_get_json_opt": upstream.count( 'get_json(st, challenge_id, path).await' ),})# Restrict timeout inspection to the site-api client construction and helper.site_state = Path("crates/site-api/src/state.rs").read_text()print({ "site_state_accepts_client": "client: reqwest::Client" in site_state, "site_api_timeout_configured": bool(re.search(r'\.timeout\s*\(', site_state)), "upstream_timeout_configured": bool(re.search(r'\.timeout\s*\(', upstream)),})PYRepository: BaseIntelligence/base
Length of output: 50377
🏁 Script executed:
#!/bin/bashset -euo pipefail
echo"=== production client construction ==="
sed -n '70,105p' crates/gateway/src/api.rs
sed -n '28,48p' crates/gateway/src/app.rs
echo"=== deterministic source verifier ==="
python3 - <<'PY'from pathlib import Pathimport rehandlers = Path("crates/site-api/src/handlers.rs").read_text()upstream = Path("crates/site-api/src/upstream.rs").read_text()gateway_api = Path("crates/gateway/src/api.rs").read_text()fanout = int(re.search( r'const\s+PRISM_CHAMPION_DETAIL_FANOUT:\s+usize\s*=\s*(\d+)', handlers,).group(1))per_id = sum(handlers.count(x) for x in ( '"/v1/submissions/{id}"', '"/v1/submissions/{id}/diff"',))assert per_id == 2assert "for id in ids" in handlersassert "get_json_opt(st, PRISM" in handlersassert ".timeout(" not in gateway_api[gateway_api.index("pub fn with_parts_seal"):gateway_api.index("pub fn with_parts_seal") + 1200]print(f"fanout={fanout}; requests_per_id={per_id}; max_sequential_requests={fanout * per_id}")print("fetch_loop_is_sequential=True")print("production_gateway_client_timeout=False")PYRepository: BaseIntelligence/base
Length of output: 2163
🏁 Script executed:
#!/bin/bashset -euo pipefail
echo"=== Prism submissions handler ==="
sed -n '518,575p' crates/site-api/src/handlers.rs
echo"=== isolated detail helper ==="
sed -n '468,494p' crates/site-api/src/handlers.rs
echo"=== isolated source verifier ==="
python3 - <<'PY'from pathlib import Pathimport resource = Path("crates/site-api/src/handlers.rs").read_text()helper = source[source.index("async fn fetch_prism_details"):source.index("\nasync fn get_leaderboard", source.index("async fn fetch_prism_details"))]fanout = int(re.search(r'PRISM_CHAMPION_DETAIL_FANOUT:\s*usize\s*=\s*(\d+)', source).group(1))detail = helper.count('"/v1/submissions/{id}"')diff = helper.count('"/v1/submissions/{id}/diff"')assert "for id in ids" in helperassert detail == 1 and diff == 1assert ".await" in helperprint(f"fanout={fanout}")print(f"detail_requests_per_id={detail}")print(f"diff_requests_per_id={diff}")print(f"max_detail_fanout_requests={fanout * (detail + diff)}")print("helper_awaits_each_request_in_loop=True")PYRepository: BaseIntelligence/base
Length of output: 3813
Run Prism detail fetches with bounded concurrency and request timeouts.
Each Prism leaderboard or submissions request can fetch up to 24 IDs. The detail fan-out can issue up to 48 sequential upstream requests, in addition to the submissions request. The production GatewayState.client has no timeout.
Bound the detail-fetch concurrency and configure a per-request timeout so a slow Prism backend cannot hold either endpoint open indefinitely.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@crates/site-api/src/handlers.rs` around lines 468 - 494, Update
fetch_prism_details to fetch IDs with bounded concurrency and apply a
per-request timeout to both submission-detail and optional diff requests. Reuse
the existing GatewayState.client and established concurrency/timeout utilities
or constants, ensuring at most the configured number of Prism requests run
simultaneously while preserving the current result and pin_id enrichment
behavior.
| /// Public Prism submission detail (era, eval groups, benches, telemetry). | ||
| async fn get_prism_submission_detail( | ||
| State(st): State<SiteState>, | ||
| Path(id): Path<String>, | ||
| ) -> Response { | ||
| let Some(mut detail) = | ||
| upstream::get_json_opt(&st, PRISM, &format!("/v1/submissions/{id}")).await | ||
| else { | ||
| return json_err( | ||
| StatusCode::NOT_FOUND, | ||
| "not_found", | ||
| "unknown prism submission", | ||
| ); | ||
| }; |
There was a problem hiding this comment.
🔒 Security & Privacy | 🟠 Major | ⚡ Quick win
Validate id before you interpolate it into the upstream path.
Path(id) is URL-decoded by axum. The handler inserts the value directly into format!("/v1/submissions/{id}"). A caller can therefore control the upstream path. A value such as sub1/diff reaches GET /v1/submissions/sub1/diff on the prism backend, and a value containing .. segments is normalized by the URL parser into a different upstream route.
The response body is filtered by prism_submission_detail, which limits what a caller can read back. The unauthenticated public route still drives arbitrary GET requests against the challenge API.
Reject any id that is not a plain submission identifier before the fetch.
🔒 Proposed fix
async fn get_prism_submission_detail(
State(st): State<SiteState>,
Path(id): Path<String>,
) -> Response {
+ if id.is_empty() || !id.chars().all(|c| c.is_ascii_alphanumeric() || c == '-' || c == '_') {+ return json_err(+ StatusCode::NOT_FOUND,+ "not_found",+ "unknown prism submission",+ );+ }
let Some(mut detail) =
upstream::get_json_opt(&st, PRISM, &format!("/v1/submissions/{id}")).awaitAs per coding guidelines: "Do not conflate wallet roles: gateway signing keys seal bundles, admin tokens authenticate /v1/admin/* … and gateway owner identity checks are distinct."
📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| /// Public Prism submission detail (era, eval groups, benches, telemetry). | |
| asyncfn get_prism_submission_detail( | |
| State(st):State<SiteState>, | |
| Path(id):Path<String>, | |
| ) -> Response{ | |
| letSome(mut detail) = | |
| upstream::get_json_opt(&st,PRISM,&format!("/v1/submissions/{id}")).await | |
| else { | |
| return json_err( | |
| StatusCode::NOT_FOUND, | |
| "not_found", | |
| "unknown prism submission", | |
| ); | |
| }; | |
| /// Public Prism submission detail (era, eval groups, benches, telemetry). | |
| asyncfn get_prism_submission_detail( | |
| State(st):State<SiteState>, | |
| Path(id):Path<String>, | |
| ) -> Response{ | |
| if id.is_empty() || !id.chars().all(|c| c.is_ascii_alphanumeric() || c == '-' || c == '_'){ | |
| return json_err( | |
| StatusCode::NOT_FOUND, | |
| "not_found", | |
| "unknown prism submission", | |
| ); | |
| } | |
| letSome(mut detail) = | |
| upstream::get_json_opt(&st,PRISM,&format!("/v1/submissions/{id}")).await | |
| else { | |
| return json_err( | |
| StatusCode::NOT_FOUND, | |
| "not_found", | |
| "unknown prism submission", | |
| ); | |
| }; |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@crates/site-api/src/handlers.rs` around lines 726 - 739, Validate the decoded
id at the start of get_prism_submission_detail before constructing the upstream
URL, accepting only plain submission identifiers and rejecting values containing
path separators, traversal segments, or other invalid characters with an
appropriate client error. Keep the existing upstream fetch and not-found
behavior unchanged for valid identifiers.
Source: Coding guidelines
| /// HellaSwag `acc_norm` for gpt2 (Eleuther harness; commonly cited 31.14%). | ||
| pub const GPT2_HELLASWAG: f64 = 0.3114; | ||
| /// ARC-Challenge `acc_norm` (zero-shot Eleuther-style gpt2). | ||
| pub const GPT2_ARC_CHALLENGE: f64 = 0.2270; | ||
| /// PIQA `acc_norm` (zero-shot Eleuther-style gpt2). | ||
| pub const GPT2_PIQA: f64 = 0.6251; | ||
| /// WinoGrande `acc` (zero-shot Eleuther-style gpt2). | ||
| pub const GPT2_WINOGRANDE: f64 = 0.5162; | ||
| /// BoolQ `acc` (zero-shot Eleuther-style gpt2). | ||
| pub const GPT2_BOOLQ: f64 = 0.6012; |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟡 Minor | ⚡ Quick win
🧩 Analysis chain
🌐 Web query:
EleutherAI lm-evaluation-harness published GPT-2 small 124M zero-shot results hellaswag arc_challenge piqa winogrande boolq
💡 Result:
The EleutherAI LM Evaluation Harness is a widely used, open-source framework designed to provide standardized evaluation for language models, including GPT-2 [1][2][3]. While the harness provides the infrastructure to run evaluations, it does not maintain a static, canonical "published" table of zero-shot results for specific models like GPT-2 Small (124M) as part of its core documentation [1][4]. However, zero-shot performance metrics for GPT-2 Small (124M) on the requested benchmarks have been documented in various technical contexts and repositories utilizing the harness [5][6][7]. The following table summarizes representative zero-shot accuracy results for GPT-2 Small (124M) found in these sources: | Benchmark | Zero-Shot Accuracy (%) | |:--- |:--- | | HellaSwag | 30.9 - 31.5% [5][6][7] | | ARC Challenge | 22.2% [5] | | PIQA | 62.3% [5] | | WinoGrande | 50.4% [6][7] | | BoolQ | ~60% (varies by implementation) [6] | Note: Evaluation results can vary based on specific implementation details, such as prompt formatting, normalization strategies, and the specific version of the LM Evaluation Harness used [4][8]. The LM Evaluation Harness itself supports evaluating models via command-line interface, allowing users to reproduce these results by targeting the GPT-2 124M model via the Hugging Face transformers integration [1][2]. For exact replication, users are encouraged to run the harness locally using the desired tasks and the appropriate pretrained model path [1][9].
Citations:
- 1: https://github.com/EleutherAI/lm-evaluation-harness/
- 2: https://github.com/EleutherAI/lm-evaluation-harness/tree/v0.4.11
- 3: https://www.eleuther.ai/projects/large-language-model-evaluation
- 4: https://doi.org/10.48550/arxiv.2405.14782
- 5: https://github.com/karpathy/nanochat/blob/master/dev/estimate_gpt3_core.ipynb
- 6: https://huggingface.co/JulianKrgd/julian-600m-40b-instruct-sft100k
- 7: https://huggingface.co/JulianKrgd/julian-600m-40b-instruct-sft30k/commit/1ae78cc87c9d0968b58a89eb98f292f8007205cd
- 8: https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard/discussions/60
- 9: https://github.com/EleutherAI/lm-evaluation-harness/blob/v0.4.7/docs/interface.md
🏁 Script executed:
#!/bin/bashset -eu
file="$(git ls-files | rg '(^|/)prism_enrich\.rs$'| head -n 1)"printf'%s\n'"$file"
cat -n "$file"| sed -n '1,55p'printf'\nReferences to the constants and source URL:\n'
rg -n 'GPT2_(HELLASWAG|ARC_CHALLENGE|PIQA|WINOGRANDE|BOOLQ|SOURCE_URL)'.Repository: BaseIntelligence/base
Length of output: 4010
🌐 Web query:
"GPT2_HELLASWAG" 0.3114 GPT-2
💡 Result:
The value 0.3114 refers to the HellaSwag "acc_norm" (normalized accuracy) score achieved by the original GPT-2 model when evaluated using the EleutherAI language model evaluation harness [1][2]. In the context of the HellaSwag benchmark, which is a multiple-choice commonsense reasoning dataset, models are often evaluated using two primary metrics: 1. Accuracy (acc): The raw accuracy of predicting the correct completion [1][2]. 2. Normalized Accuracy (acc_norm): A metric that adjusts for the length of the completions by comparing the likelihood of the correct ending against the other options [1][2]. The EleutherAI harness specifically reports a score of approximately 0.2892 for accuracy and 0.3114 for normalized accuracy for the original GPT-2 model [1][2]. This 0.3114 figure is frequently cited in technical implementations and documentation (such as in various "nanoGPT" repositories) to provide a standard baseline for verifying that a reproduction of GPT-2 is performing correctly [1][3][2].
Citations:
- 1: https://github.com/karpathy/build-nanogpt/blob/master/hellaswag.py
- 2: https://huggingface.co/olm/olm-gpt2-latest
- 3: https://github.com/denma98/GPT2-124M-NanoGPT/blob/main/hellaswag.py
🌐 Web query:
"GPT2_ARC_CHALLENGE" 0.2270 OR "GPT2_PIQA" 0.6251 OR "GPT2_WINOGRANDE" 0.5162 OR "GPT2_BOOLQ" 0.6012
💡 Result:
The values you provided—GPT2_ARC_CHALLENGE 0.2270, GPT2_PIQA 0.6251, GPT2_WINOGRANDE 0.5162, and GPT2_BOOLQ 0.6012—represent zero-shot performance metrics for the GPT-2 language model on common natural language processing benchmarks [1]. These scores are consistent with standard evaluations of GPT-2 (specifically the base 124M parameter model) found in academic literature and technical reports concerning zero-shot multitask learning [1][2]. The benchmarks cited evaluate different aspects of model capability: * ARC-CHALLENGE (0.2270): The AI2 Reasoning Challenge, a grade-school science question-answering benchmark [3][4]. A score of 0.2270 is near the random baseline of 0.25, reflecting the difficulty of this task for the original GPT-2 [3][4]. * PIQA (0.6251): Physical Interaction Question Answering, which tests commonsense reasoning about physical tasks [5][6]. * WINOGRANDE (0.5162): A dataset designed to test commonsense reasoning and ambiguity resolution, based on the Winograd Schema Challenge [7]. * BOOLQ (0.6012): A question-answering benchmark requiring yes/no answers to questions about provided passages [1]. These metrics are frequently used to demonstrate the zero-shot generalization capabilities of GPT-2 compared to later, more specialized, or larger language models [1]. Note that while these figures characterize the base GPT-2 model, many modern models have significantly surpassed these scores on these benchmarks [8][4][6].
Citations:
- 1: https://openreview.net/attachment?id=smzz8anRkl&name=pdf
- 2: https://huggingface.co/openai-community/gpt2
- 3: https://ar5iv.labs.arxiv.org/html/1803.05457
- 4: https://benchmarks.darvinyi.com/benchmarks/arc
- 5: https://github.com/Hypogenic-AI/mechanistic-llm-failures-69ca-claude/blob/main/REPORT.md
- 6: https://genailist.net/benchmark/piqa
- 7: https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf
- 8: https://genailist.net/benchmark/arc-challenge
🌐 Web query:
OpenReview smzz8anRkl GPT-2 0.2270 0.6251 0.5162 0.6012 table
💡 Result:
The OpenReview ID "smzz8anRkl" corresponds to an attached PDF document titled "Updated Zero-Shot Performance on SuperGLUE" [1]. This document contains a table (Table 7) that reports the zero-shot performance of various GPT-2 model variants across several SuperGLUE benchmarks, including BoolQ, RTE, CB, COPA, ReCoRD, WIC, and WSC [1]. The specific numerical values provided in your query—0.2270, 0.6251, 0.5162, and 0.6012—appear to be performance metrics (likely scores or coefficients) associated with the model evaluations in that table [1]. As indicated in the referenced document, ReCoRD uses the F1 score as its evaluation metric, while all other benchmarks in the table utilize accuracy [1]. The document provides comparative performance data for GPT-2 and GPT-2 Large across these metrics [1].
Citations:
🌐 Web query:
"0.2270" "0.6251" GPT-2 ARC PIQA
💡 Result:
The values 0.2270 and 0.6251 correspond to specific performance metrics for the original GPT-2 model (often referred to as the standard or base GPT-2) when evaluated on the ARC and PIQA benchmarks, respectively [1][2][3][4]. Specifically: - 0.2270: This is the normalized accuracy (acc_norm) achieved by the original GPT-2 model on the ARC-Challenge (AI2 Reasoning Challenge) benchmark [1][2][3][4]. - 0.6251: This is the normalized accuracy (acc_norm) achieved by the original GPT-2 model on the PIQA (Physical Interaction Question Answering) benchmark [1][2][3][4]. These figures are widely used as a baseline for comparing the performance of other language models within the EleutherAI LM Evaluation Harness and similar benchmarking frameworks [1][2]. In some technical contexts, such as experimental logs, 0.2270 may also appear as an accuracy result for other models depending on the specific training run or architecture being documented [5], but within the common tables comparing "Original GPT-2" to modern reproductions, these specific values are the established benchmarks for the original GPT-2 model [2][3][4].
Citations:
- 1: [Refactor] Benchmark scripts EleutherAI/lm-evaluation-harness#612
- 2: https://huggingface.co/olm/olm-gpt2-latest
- 3: https://huggingface.co/olm/olm-gpt2-dec-2022
- 4: https://huggingface.co/olm/olm-gpt2-oct-2022
- 5: https://medium.com/@gupta.aman/pre-training-logs-entry-1-training-a-smol-llama-from-scratch-04e4b5d4c5f7
🌐 Web query:
"0.5162" "0.6012" GPT-2 WinoGrande BoolQ
💡 Result:
The values 0.5162 and 0.6012 appear in evaluation logs and research datasets, but they do not correspond to a specific, unique standard benchmark result for GPT-2 on WinoGrande and BoolQ respectively [1][2]. The value 0.5162 is explicitly recorded as the WinoGrande accuracy for the GPT-2 (124M) model in certain configurations [1][2]. However, 0.6012 is not a standard GPT-2 accuracy score for BoolQ; rather, it appears in various unrelated contexts, such as machine translation evaluation metrics (e.g., WMT 2014) [3], NIST OpenMT12 results [4], or as an accuracy/F1 score for different models (e.g., RoBERTa, MAE-ViT) in independent training logs [5][6]. In summary, 0.5162 is associated with GPT-2's performance on WinoGrande [1][2], while 0.6012 is a number commonly found in miscellaneous machine learning experiment logs and historical evaluations unrelated to GPT-2's performance on BoolQ [5][6][3][4].
Citations:
- 1: https://huggingface.co/olm/olm-gpt2-latest
- 2: https://github.com/AMD-AGI/AMD-LLM/blob/main/README.md
- 3: https://mt-archive.net/10/WMT-2014-Beck.pdf
- 4: https://www.nist.gov/itl/iad/mig/openmt12-evaluation-results
- 5: https://huggingface.co/aiface/roberta-large-mnli_nli
- 6: https://huggingface.co/fpsandnoob/mae-vit-base-patch32-224-ct
🏁 Script executed:
#!/bin/bashset -eu
forurlin \
'https://github.com/EleutherAI/lm-evaluation-harness/pull/612.patch' \
'https://huggingface.co/olm/olm-gpt2-latest/raw/main/README.md' \
'https://raw.githubusercontent.com/AMD-AGI/AMD-LLM/main/README.md'doprintf'\n===== %s =====\n'"$url"
curl -L --fail --silent --show-error "$url" \
| rg -n -C 3 '0\.3114|0\.2270|0\.6251|0\.5162|0\.6012|HellaSwag|ARC|PIQA|WinoGrande|BoolQ|gpt2' \
| head -n 220 ||truedoneRepository: BaseIntelligence/base
Length of output: 50378
🌐 Web query:
site:github.com/EleutherAI/lm-evaluation-harness/pull/612 GPT-2 results BoolQ 0.6012
💡 Result:
There is no verifiable evidence or documentation in the provided search results to support the specific figure of 0.6012 as a BoolQ benchmark result for GPT-2 [1]. The available technical data regarding LLM evaluation benchmarks focuses on different tasks and models, and no specific performance metrics connecting GPT-2 to the BoolQ dataset with that exact value were identified [1].
Citations:
🌐 Web query:
site:huggingface.co/olm/olm-gpt2-latest "BoolQ" "0.6012"
💡 Result:
There is no publicly documented information or authoritative source linking the specific value 0.6012 to the BoolQ benchmark. Searches of technical repositories and benchmark datasets do not associate this figure with the BoolQ (Boolean Questions) dataset, which is a question-answering benchmark for yes/no questions [1][2][3][4]. It is possible this number refers to a local evaluation metric for a specific model instance or a misidentified value from an unrelated technical context.
Citations:
- 1: https://huggingface.co/olm/olm-gpt2-latest/discussions/1
- 2: https://huggingface.co/olm/olm-gpt2-latest/tensorboard
- 3: https://huggingface.co/olm/olm-gpt2-latest/tree/main
- 4: https://huggingface.co/olm/olm-gpt2-latest/discussions
Correct and directly cite the GPT-2 baseline values.
GPT2_SOURCE_URL points to the repository root, not to a results table. Link a citable artifact that identifies the model, metrics, and evaluation configuration. Do not publish GPT2_BOOLQ = 0.6012 until a source establishes it as a GPT-2 BoolQ result.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@crates/site-api/src/prism_enrich.rs` around lines 23 - 32, Update the GPT-2
baseline constants and GPT2_SOURCE_URL to use a citable results artifact that
explicitly identifies GPT-2, each metric, and its evaluation configuration.
Verify and correct the HellaSwag, ARC-Challenge, PIQA, and WinoGrande values
against that source, and remove or withhold GPT2_BOOLQ until a source directly
establishes the GPT-2 BoolQ result.
| pub fn pin_id_from_payload(payload: &Value) -> Option<String> { | ||
| let sub = payload.get("submission").unwrap_or(payload); | ||
| for path in [ | ||
| "/pin_id", | ||
| "/submission/pin_id", | ||
| "/metrics/pod_manifest/automodel_base", | ||
| "/metrics/pod_manifest/pin_id", | ||
| "/submission/metrics/pod_manifest/automodel_base", | ||
| "/submission/metrics/pod_manifest/pin_id", | ||
| ] { | ||
| if let Some(p) = payload | ||
| .pointer(path) | ||
| .and_then(Value::as_str) | ||
| .map(str::trim) | ||
| .filter(|s| !s.is_empty() && s.starts_with("automodel@")) | ||
| { | ||
| return Some(p.to_owned()); | ||
| } | ||
| } | ||
| sub.get("pin_id") | ||
| .and_then(Value::as_str) | ||
| .map(str::trim) | ||
| .filter(|s| !s.is_empty()) | ||
| .map(str::to_owned) | ||
| } |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟡 Minor | ⚡ Quick win
Restrict the pin_id fallback to AutoModel pins.
The loop at Lines 83-99 already reads /pin_id and /submission/pin_id and accepts a value only when it starts with automodel@. The fallback at Lines 100-104 reads the same pin_id field again but accepts any non-empty string. The fallback therefore only adds values that failed the automodel@ check.
Two consequences follow. infer_recipe_era returns RecipeEra::Automodel for a legacy submission that carries any pin_id string. enrich_submission_from_detail then publishes that string as pinId. docs/SITE_API.md Line 49 documents pinId as an AutoModel pin (automodel@…).
Apply the same prefix filter to the fallback, or remove the fallback.
🔧 Proposed fix
sub.get("pin_id")
.and_then(Value::as_str)
.map(str::trim)
- .filter(|s| !s.is_empty())+ .filter(|s| s.starts_with("automodel@"))
.map(str::to_owned)
}📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| pubfn pin_id_from_payload(payload:&Value) -> Option<String>{ | |
| let sub = payload.get("submission").unwrap_or(payload); | |
| for path in[ | |
| "/pin_id", | |
| "/submission/pin_id", | |
| "/metrics/pod_manifest/automodel_base", | |
| "/metrics/pod_manifest/pin_id", | |
| "/submission/metrics/pod_manifest/automodel_base", | |
| "/submission/metrics/pod_manifest/pin_id", | |
| ]{ | |
| ifletSome(p) = payload | |
| .pointer(path) | |
| .and_then(Value::as_str) | |
| .map(str::trim) | |
| .filter(|s| !s.is_empty() && s.starts_with("automodel@")) | |
| { | |
| returnSome(p.to_owned()); | |
| } | |
| } | |
| sub.get("pin_id") | |
| .and_then(Value::as_str) | |
| .map(str::trim) | |
| .filter(|s| !s.is_empty()) | |
| .map(str::to_owned) | |
| } | |
| pubfn pin_id_from_payload(payload:&Value) -> Option<String>{ | |
| let sub = payload.get("submission").unwrap_or(payload); | |
| for path in[ | |
| "/pin_id", | |
| "/submission/pin_id", | |
| "/metrics/pod_manifest/automodel_base", | |
| "/metrics/pod_manifest/pin_id", | |
| "/submission/metrics/pod_manifest/automodel_base", | |
| "/submission/metrics/pod_manifest/pin_id", | |
| ]{ | |
| ifletSome(p) = payload | |
| .pointer(path) | |
| .and_then(Value::as_str) | |
| .map(str::trim) | |
| .filter(|s| !s.is_empty() && s.starts_with("automodel@")) | |
| { | |
| returnSome(p.to_owned()); | |
| } | |
| } | |
| sub.get("pin_id") | |
| .and_then(Value::as_str) | |
| .map(str::trim) | |
| .filter(|s| s.starts_with("automodel@")) | |
| .map(str::to_owned) | |
| } |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@crates/site-api/src/prism_enrich.rs` around lines 81 - 105, Restrict the
fallback in pin_id_from_payload to the same AutoModel-pin contract as the
preceding pointer loop. After reading sub.get("pin_id"), trim the value and
accept it only when non-empty and prefixed with "automodel@", or remove this
redundant fallback while preserving valid AutoModel extraction.
| ### Prism era, benches, detail, GPT-2 references | ||
| Champion list / leaderboard rows may carry (when detail fan-out succeeds): | ||
| | Field | Meaning | | ||
| |-------|---------| | ||
| | `recipeEra` | `"automodel"` \| `"legacy"` — AutoModel if `pin_id` / recipe major ≥ 2 / diff shape; else legacy | | ||
| | `pinId` | AutoModel pin (`automodel@…`) when era is automodel | | ||
| | `evalGroups` | `{ group: "g1"…"g8", g }` from composite `eval.groups` | | ||
| | `benchmarks` | Public G2 subset: `hellaswag`, `arcChallenge`, `piqa`, `winogrande`, `boolq` from `org.g2.*` / battery keys | | ||
| | `submissionId` | (leaderboard) best-BPB champion id for the detail modal | | ||
| Additional Prism routes: | ||
| | Path | Response | | ||
| |------|----------| | ||
| | `GET /v1/site/arenas/prism/submissions/{id}` | `PrismSubmissionDetail` — list fields + `eval` summary (status, groups, gates, composite) + telemetry + public `review` / `similarity` (quality/kind only). **No** raw patch text. | | ||
| | `GET /v1/site/arenas/prism/references` | `PrismReferenceBaseline[]` — frozen **published** GPT-2 Small (124M) Eleuther-style accuracies + `sourceUrl` / `disclaimer`. **`bpb` omitted** (not Prism-protocol BPB). | | ||
| | `GET /v1/site/arenas/prism/submissions/{id}/telemetry` | Existing loss-curve payload (also embedded on detail). | | ||
| GPT-2 constants live in `crates/site-api` (`prism_enrich`) so API and FE stay aligned; they are literature / Eleuther harness numbers, not a Prism battery run. List/leaderboard row shells still map in `crates/site-data`. | ||
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟡 Minor | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash# Description: Check whether external-miner docs reference the site API surfaces.set -euo pipefail
fd -t d 'external-miner' docs ||echo"docs/external-miner not found"echo"=== site API references in external miner docs ==="
rg -n 'v1/site|recipeEra|pinId|evalGroups|references' docs/external-miner ||echo"no references"echo"=== external-docs-check task definition ==="
fd -t f -e rs . xtask --exec rg -n -C 5 'external.docs.check|external_docs_check' {} \;Repository: BaseIntelligence/base
Length of output: 2273
🏁 Script executed:
#!/bin/bashset -euo pipefail
echo"=== relevant repository files ==="
git ls-files 'docs/SITE_API.md''docs/external-miner/*''xtask/src/external_docs_check.rs''xtask/src/main.rs'echo"=== SITE_API.md relevant sections ==="
sed -n '1,95p' docs/SITE_API.md
echo"=== external miner Prism documentation ==="
sed -n '230,295p' docs/external-miner/prism.md
echo"=== external-docs-check implementation ==="
sed -n '1,260p' xtask/src/external_docs_check.rs
echo"=== recent file changes ==="
git diff --stat
git status --shortRepository: BaseIntelligence/base
Length of output: 18150
🏁 Script executed:
#!/bin/bashset -euo pipefail
echo"=== all site-route references in external-miner docs ==="
rg -n -C 3 '/v1/site|site API|marketing|leaderboard|references|recipeEra|pinId|evalGroups|benchmarks|submissionId' docs/external-miner
echo"=== site routes in repository source ==="
rg -n -C 2 'arenas/prism/(submissions|references)|PrismSubmissionDetail|PrismReferenceBaseline|recipeEra|evalGroups|benchmarks|submissionId' crates docs
echo"=== external-docs-check coverage of site routes ==="
rg -n 'site|SITE_API|v1/site|external-miner' xtask/src/external_docs_check.rs docs/AGENTS.md AGENTS.mdRepository: BaseIntelligence/base
Length of output: 50377
Update the Prism miner documentation for the new public site API. Add the Prism detail and references routes and the new response fields to docs/external-miner/prism.md, then mirror the update in the public miner repository. external-docs-check validates protocol pins and D19; it does not detect site API documentation drift.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@docs/SITE_API.md` around lines 42 - 63, The Prism miner documentation must
reflect the new public site API. Update docs/external-miner/prism.md with the
detail and references routes plus the new champion/leaderboard response fields
described around PrismSubmissionDetail and PrismReferenceBaseline, then apply
the same documentation changes to the public miner repository.
Source: Coding guidelines
Uh oh!
There was an error while loading. Please reload this page.
Summary
recipeEra,evalGroups, public G2benchmarks, and championsubmissionId/pinIdon board rowsGET /v1/site/arenas/prism/submissions/{id}(safe public detail + telemetry/eval/review) andGET /v1/site/arenas/prism/references(frozen GPT-2 Small literature baseline)docs/SITE_API.mdand cover mappers/enrich helpers with unit testsTest plan
cargo fmt --allcargo clippy -p site-types -p site-data -p site-api --all-targets -- -D warningscargo test -p site-types -p site-data -p site-apicargo run -p xtask -- loc-capGET /v1/site/arenas/prism/references→ 200 with GPT-2 baselineGET /v1/site/arenas/prism/leaderboardrows includerecipeEra/ optional benchesGET /v1/site/arenas/prism/submissions/{id}for a champion idPairs with frontend PR (Prism modal / era tabs / GPT-2 REF row). Do not merge until FE is ready to consume, or FE will soft-miss new fields until deploy.
Summary by CodeRabbit
New Features
Bug Fixes
Documentation