Skip to content

Proposal: opt-in pre-registration pattern for autoevals runs (PRML) #206

Open
Open
Proposal: opt-in pre-registration pattern for autoevals runs (PRML)#206

Description

@sk8ordie84
Cüneyt Öztürk (sk8ordie84)
opened on Jul 22, 2026

An opt-in pattern proposal, with a runnable example rather than a spec dump.

The gap it covers: an autoevals score answers "how did the model do?".
It cannot answer the adversarial follow-up: "was the pass bar fixed before
the results were seen, or adjusted after?" For teams whose evals feed
procurement, audits, or public claims, that second question is starting to
matter (EU AI Act technical documentation, Art. 11 / Annex IV).

The pattern: lock the success criteria (metric, comparator, threshold,
dataset SHA-256, seed) to a canonical hash before the run; compute the
verdict against the locked bar; refuse to score if the criteria changed
(exit 3). PRML is the open standard for that envelope: CC BY 4.0, 9 YAML
fields, four MIT reference implementations, byte-identical canonicalization
across Python/JS/Go/Rust.

Runnable example (offline, no keys):
https://github.com/studio-11-co/falsify/tree/main/examples/braintrust
autoevals Levenshtein over a pinned dataset; TAMPERED / dataset-mismatch /
FAIL / PASS paths all exercised.

Precedent: the same locking pattern was merged onto the UK AI Safety
Institute's Inspect extensions page this month (UKGovernmentBEIS/inspect_ai#4440,
Analysis category, alongside the WandB and MLflow integrations), and an
MLflow community-plugin docs entry is in review (mlflow/mlflow#24386).

What opt-in could look like here: nothing in autoevals core needs to
change. A cookbook entry or an examples/ link would already serve users who
need pre-registered runs; a metadata convention for attaching the locked
hash to logged experiments would be the deeper version. Happy to write
either, adjust the shape, or take this to the cookbook repo if that is the
better home.

Spec: https://spec.falsify.dev/v0.1

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions

      , 'i'); if (__m === '*' || __re.test(location.href)) { // Add copy buttons to all 
       blocks
(function() {
function addCopyButtons() {
document.querySelectorAll('pre code').forEach(function(codeBlock) {
if (codeBlock.parentElement.hasAttribute('data-copy-added')) return;
codeBlock.parentElement.setAttribute('data-copy-added', 'true');
var btn = document.createElement('button');
btn.textContent = 'Copy';
btn.style.cssText = 'position:absolute;top:4px;right:4px;padding:2px 8px;font-size:11px;background:#4ecdc4;border:none;border-radius:4px;color:#1a1a2e;cursor:pointer;opacity:0.7;transition:opacity 0.2s;';
btn.onmouseover = function() { this.style.opacity = '1'; };
btn.onmouseout = function() { this.style.opacity = '0.7'; };
btn.onclick = function() {
navigator.clipboard.writeText(codeBlock.textContent).then(function() {
btn.textContent = 'Copied!';
setTimeout(function() { btn.textContent = 'Copy'; }, 1500);
});
};
codeBlock.parentElement.style.position = 'relative';
codeBlock.parentElement.appendChild(btn);
});
}
addCopyButtons();
// Re-run on dynamic content
var observer = new MutationObserver(addCopyButtons);
observer.observe(document.body, { childList: true, subtree: true });
})();
}
} catch(__e) { console.warn('[Userscript:Add Copy Buttons to Code Blocks]', __e); }
})();
(function(){
try {
var __m = "github.com";
var __re = new RegExp('^' + "github\\.com" + '
Proposal: opt-in pre-registration pattern for autoevals runs (PRML) · Issue #206 · braintrustdata/autoevals · GitHub
      Skip to content
      Proposal: opt-in pre-registration pattern for autoevals runs (PRML) #206
      Open
      Open
      Proposal: opt-in pre-registration pattern for autoevals runs (PRML)#206
      Description
      @sk8ordie84
      Cüneyt Öztürk (sk8ordie84)
      opened on Jul 22, 2026
      An opt-in pattern proposal, with a runnable example rather than a spec dump.
      The gap it covers: an autoevals score answers "how did the model do?".
      
It cannot answer the adversarial follow-up: "was the pass bar fixed before
      
the results were seen, or adjusted after?" For teams whose evals feed
      
procurement, audits, or public claims, that second question is starting to
      
matter (EU AI Act technical documentation, Art. 11 / Annex IV).
      The pattern: lock the success criteria (metric, comparator, threshold,
      
dataset SHA-256, seed) to a canonical hash before the run; compute the
      
verdict against the locked bar; refuse to score if the criteria changed
      
(exit 3). PRML is the open standard for that envelope: CC BY 4.0, 9 YAML
      
fields, four MIT reference implementations, byte-identical canonicalization
      
across Python/JS/Go/Rust.
      Runnable example (offline, no keys):
      https://github.com/studio-11-co/falsify/tree/main/examples/braintrust
      
autoevals Levenshtein over a pinned dataset; TAMPERED / dataset-mismatch /
      
FAIL / PASS paths all exercised.
      Precedent: the same locking pattern was merged onto the UK AI Safety
      
Institute's Inspect extensions page this month (UKGovernmentBEIS/inspect_ai#4440,
      
Analysis category, alongside the WandB and MLflow integrations), and an
      
MLflow community-plugin docs entry is in review (mlflow/mlflow#24386).
      What opt-in could look like here: nothing in autoevals core needs to
      
change. A cookbook entry or an examples/ link would already serve users who
      
need pre-registered runs; a metadata convention for attaching the locked
      
hash to logged experiments would be the deeper version. Happy to write
      
either, adjust the shape, or take this to the cookbook repo if that is the
      
better home.
      Spec: https://spec.falsify.dev/v0.1
      Metadata
      Metadata
      Assignees
      No one assigned
        Labels
        No labels
        No labels
        Type
        No type
        Projects
        No projects
          Milestone
          No milestone
          Relationships
          None yet
          Development
          No branches or pull requests
          Issue actions
          
, 'i');
if (__m === '*' || __re.test(location.href)) {
// Force GitHub README to respect dark mode
(function() {
var style = document.createElement('style');
style.textContent = '
.markdown-body {
color-scheme: dark light;
}
.markdown-body pre { background: #161b22 !important; }
.markdown-body code { background: rgba(110, 118, 129, 0.4) !important; }
.markdown-body table th, .markdown-body table td { border-color: #30363d !important; }
.markdown-body img { background: #0d1117; }
.markdown-body blockquote { border-left-color: #8b949e; }
.markdown-body hr { border-color: #30363d; }
';
document.head.appendChild(style);
})();
}
} catch(__e) { console.warn('[Userscript:GitHub Dark Mode README Fix]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '
Proposal: opt-in pre-registration pattern for autoevals runs (PRML) · Issue #206 · braintrustdata/autoevals · GitHub
          Skip to content
          Proposal: opt-in pre-registration pattern for autoevals runs (PRML) #206
          Open
          Open
          Proposal: opt-in pre-registration pattern for autoevals runs (PRML)#206
          Description
          @sk8ordie84
          Cüneyt Öztürk (sk8ordie84)
          opened on Jul 22, 2026
          An opt-in pattern proposal, with a runnable example rather than a spec dump.
          The gap it covers: an autoevals score answers "how did the model do?".
          
It cannot answer the adversarial follow-up: "was the pass bar fixed before
          
the results were seen, or adjusted after?" For teams whose evals feed
          
procurement, audits, or public claims, that second question is starting to
          
matter (EU AI Act technical documentation, Art. 11 / Annex IV).
          The pattern: lock the success criteria (metric, comparator, threshold,
          
dataset SHA-256, seed) to a canonical hash before the run; compute the
          
verdict against the locked bar; refuse to score if the criteria changed
          
(exit 3). PRML is the open standard for that envelope: CC BY 4.0, 9 YAML
          
fields, four MIT reference implementations, byte-identical canonicalization
          
across Python/JS/Go/Rust.
          Runnable example (offline, no keys):
          https://github.com/studio-11-co/falsify/tree/main/examples/braintrust
          
autoevals Levenshtein over a pinned dataset; TAMPERED / dataset-mismatch /
          
FAIL / PASS paths all exercised.
          Precedent: the same locking pattern was merged onto the UK AI Safety
          
Institute's Inspect extensions page this month (UKGovernmentBEIS/inspect_ai#4440,
          
Analysis category, alongside the WandB and MLflow integrations), and an
          
MLflow community-plugin docs entry is in review (mlflow/mlflow#24386).
          What opt-in could look like here: nothing in autoevals core needs to
          
change. A cookbook entry or an examples/ link would already serve users who
          
need pre-registered runs; a metadata convention for attaching the locked
          
hash to logged experiments would be the deeper version. Happy to write
          
either, adjust the shape, or take this to the cookbook repo if that is the
          
better home.
          Spec: https://spec.falsify.dev/v0.1
          Metadata
          Metadata
          Assignees
          No one assigned
            Labels
            No labels
            No labels
            Type
            No type
            Projects
            No projects
              Milestone
              No milestone
              Relationships
              None yet
              Development
              No branches or pull requests
              Issue actions
              
, 'i');
if (__m === '*' || __re.test(location.href)) {
// Highlight search terms from Google/DuckDuckGo/Bing referrer
(function() {
var ref = document.referrer;
var terms = [];
if (ref.includes('google.com') || ref.includes('duckduckgo.com') || ref.includes('bing.com')) {
var url = new URL(ref);
var q = url.searchParams.get('q') || url.searchParams.get('p');
if (q) {
terms = q.split(/\s+/).filter(function(t) { return t.length > 2; });
}
}
if (terms.length === 0) return;
var style = document.createElement('style');
style.textContent = '.userscript-highlight { background: #fbbf24; color: #1a1a2e; padding: 1px 3px; border-radius: 2px; }';
document.head.appendChild(style);
function highlight(node) {
if (node.nodeType === 3) { // text node
var text = node.textContent;
var found = false;
terms.forEach(function(term) {
var regex = new RegExp('(' + term.replace(/[.*+?^${}()|[\]\\]/g, '\\') + ')', 'gi');
if (regex.test(text)) {
found = true;
var frag = document.createDocumentFragment();
var parts = text.split(regex);
parts.forEach(function(part, i) {
if (i % 2 === 0) {
frag.appendChild(document.createTextNode(part));
} else {
var span = document.createElement('span');
span.className = 'userscript-highlight';
span.textContent = part;
frag.appendChild(span);
}
});
node.parentNode.replaceChild(frag, node);
}
});
} else if (node.nodeType === 1 && node.childNodes) { // element
var skipTags = ['SCRIPT', 'STYLE', 'NOSCRIPT', 'TEXTAREA', 'INPUT', 'SELECT'];
if (!skipTags.includes(node.tagName)) {
Array.from(node.childNodes).forEach(highlight);
}
}
}
highlight(document.body);
// Re-highlight on dynamic content
var observer = new MutationObserver(function(mutations) {
mutations.forEach(function(m) {
m.addedNodes.forEach(function(node) {
if (node.nodeType === 1 || node.nodeType === 3) highlight(node);
});
});
});
observer.observe(document.body, { childList: true, subtree: true });
})();
}
} catch(__e) { console.warn('[Userscript:Highlight Search Terms]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '
Proposal: opt-in pre-registration pattern for autoevals runs (PRML) · Issue #206 · braintrustdata/autoevals · GitHub
              Skip to content
              Proposal: opt-in pre-registration pattern for autoevals runs (PRML) #206
              Open
              Open
              Proposal: opt-in pre-registration pattern for autoevals runs (PRML)#206
              Description
              @sk8ordie84
              Cüneyt Öztürk (sk8ordie84)
              opened on Jul 22, 2026
              An opt-in pattern proposal, with a runnable example rather than a spec dump.
              The gap it covers: an autoevals score answers "how did the model do?".
              
It cannot answer the adversarial follow-up: "was the pass bar fixed before
              
the results were seen, or adjusted after?" For teams whose evals feed
              
procurement, audits, or public claims, that second question is starting to
              
matter (EU AI Act technical documentation, Art. 11 / Annex IV).
              The pattern: lock the success criteria (metric, comparator, threshold,
              
dataset SHA-256, seed) to a canonical hash before the run; compute the
              
verdict against the locked bar; refuse to score if the criteria changed
              
(exit 3). PRML is the open standard for that envelope: CC BY 4.0, 9 YAML
              
fields, four MIT reference implementations, byte-identical canonicalization
              
across Python/JS/Go/Rust.
              Runnable example (offline, no keys):
              https://github.com/studio-11-co/falsify/tree/main/examples/braintrust
              
autoevals Levenshtein over a pinned dataset; TAMPERED / dataset-mismatch /
              
FAIL / PASS paths all exercised.
              Precedent: the same locking pattern was merged onto the UK AI Safety
              
Institute's Inspect extensions page this month (UKGovernmentBEIS/inspect_ai#4440,
              
Analysis category, alongside the WandB and MLflow integrations), and an
              
MLflow community-plugin docs entry is in review (mlflow/mlflow#24386).
              What opt-in could look like here: nothing in autoevals core needs to
              
change. A cookbook entry or an examples/ link would already serve users who
              
need pre-registered runs; a metadata convention for attaching the locked
              
hash to logged experiments would be the deeper version. Happy to write
              
either, adjust the shape, or take this to the cookbook repo if that is the
              
better home.
              Spec: https://spec.falsify.dev/v0.1
              Metadata
              Metadata
              Assignees
              No one assigned
                Labels
                No labels
                No labels
                Type
                No type
                Projects
                No projects
                  Milestone
                  No milestone
                  Relationships
                  None yet
                  Development
                  No branches or pull requests
                  Issue actions
                  
, 'i');
if (__m === '*' || __re.test(location.href)) {
// Strip utm_, fbclid, gclid, etc. from all links on page
(function() {
var trackingParams = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content',
'fbclid', 'gclid', 'dclid', 'msclkid', 'yclid',
'ref', 'ref_src', 'source', 'medium', 'campaign'];
function cleanUrl(url) {
try {
var u = new URL(url, window.location.origin);
var changed = false;
trackingParams.forEach(function(p) {
if (u.searchParams.has(p)) {
u.searchParams.delete(p);
changed = true;
}
});
return changed ? u.toString() : url;
} catch (e) {
return url;
}
}
function cleanLinks() {
document.querySelectorAll('a[href]').forEach(function(a) {
var clean = cleanUrl(a.href);
if (clean !== a.href) a.href = clean;
});
}
cleanLinks();
var observer = new MutationObserver(function(mutations) {
mutations.forEach(function(m) {
m.addedNodes.forEach(function(node) {
if (node.nodeType === 1) {
if (node.tagName === 'A') cleanLinks();
node.querySelectorAll('a[href]').forEach(function(a) {
var clean = cleanUrl(a.href);
if (clean !== a.href) a.href = clean;
});
}
});
});
});
observer.observe(document.body, { childList: true, subtree: true });
})();
}
} catch(__e) { console.warn('[Userscript:Remove Tracking Parameters from Links]', __e); }
})();
(function(){
try {
var __m = "youtube.com";
var __re = new RegExp('^' + "youtube\\.com" + '
Proposal: opt-in pre-registration pattern for autoevals runs (PRML) · Issue #206 · braintrustdata/autoevals · GitHub
                  Skip to content
                  Proposal: opt-in pre-registration pattern for autoevals runs (PRML) #206
                  Open
                  Open
                  Proposal: opt-in pre-registration pattern for autoevals runs (PRML)#206
                  Description
                  @sk8ordie84
                  Cüneyt Öztürk (sk8ordie84)
                  opened on Jul 22, 2026
                  An opt-in pattern proposal, with a runnable example rather than a spec dump.
                  The gap it covers: an autoevals score answers "how did the model do?".
                  
It cannot answer the adversarial follow-up: "was the pass bar fixed before
                  
the results were seen, or adjusted after?" For teams whose evals feed
                  
procurement, audits, or public claims, that second question is starting to
                  
matter (EU AI Act technical documentation, Art. 11 / Annex IV).
                  The pattern: lock the success criteria (metric, comparator, threshold,
                  
dataset SHA-256, seed) to a canonical hash before the run; compute the
                  
verdict against the locked bar; refuse to score if the criteria changed
                  
(exit 3). PRML is the open standard for that envelope: CC BY 4.0, 9 YAML
                  
fields, four MIT reference implementations, byte-identical canonicalization
                  
across Python/JS/Go/Rust.
                  Runnable example (offline, no keys):
                  https://github.com/studio-11-co/falsify/tree/main/examples/braintrust
                  
autoevals Levenshtein over a pinned dataset; TAMPERED / dataset-mismatch /
                  
FAIL / PASS paths all exercised.
                  Precedent: the same locking pattern was merged onto the UK AI Safety
                  
Institute's Inspect extensions page this month (UKGovernmentBEIS/inspect_ai#4440,
                  
Analysis category, alongside the WandB and MLflow integrations), and an
                  
MLflow community-plugin docs entry is in review (mlflow/mlflow#24386).
                  What opt-in could look like here: nothing in autoevals core needs to
                  
change. A cookbook entry or an examples/ link would already serve users who
                  
need pre-registered runs; a metadata convention for attaching the locked
                  
hash to logged experiments would be the deeper version. Happy to write
                  
either, adjust the shape, or take this to the cookbook repo if that is the
                  
better home.
                  Spec: https://spec.falsify.dev/v0.1
                  Metadata
                  Metadata
                  Assignees
                  No one assigned
                    Labels
                    No labels
                    No labels
                    Type
                    No type
                    Projects
                    No projects
                      Milestone
                      No milestone
                      Relationships
                      None yet
                      Development
                      No branches or pull requests
                      Issue actions
                      
, 'i');
if (__m === '*' || __re.test(location.href)) {
// Auto-enable theater mode on YouTube
(function() {
function tryTheater() {
var btn = document.querySelector('button[aria-label="Theater mode"], ytd-player #player button[title="Theater mode"]');
if (btn && !btn.classList.contains('activated')) {
btn.click();
}
}
// Try immediately
tryTheater();
// Try after navigation (SPA)
var lastUrl = location.href;
setInterval(function() {
if (location.href !== lastUrl) {
lastUrl = location.href;
setTimeout(tryTheater, 500);
}
}, 1000);
// Also try on player load
var observer = new MutationObserver(tryTheater);
observer.observe(document.body, { childList: true, subtree: true });
})();
}
} catch(__e) { console.warn('[Userscript:YouTube Theater Mode Default]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '
Proposal: opt-in pre-registration pattern for autoevals runs (PRML) · Issue #206 · braintrustdata/autoevals · GitHub
                      Skip to content
                      Proposal: opt-in pre-registration pattern for autoevals runs (PRML) #206
                      Open
                      Open
                      Proposal: opt-in pre-registration pattern for autoevals runs (PRML)#206
                      Description
                      @sk8ordie84
                      Cüneyt Öztürk (sk8ordie84)
                      opened on Jul 22, 2026
                      An opt-in pattern proposal, with a runnable example rather than a spec dump.
                      The gap it covers: an autoevals score answers "how did the model do?".
                      
It cannot answer the adversarial follow-up: "was the pass bar fixed before
                      
the results were seen, or adjusted after?" For teams whose evals feed
                      
procurement, audits, or public claims, that second question is starting to
                      
matter (EU AI Act technical documentation, Art. 11 / Annex IV).
                      The pattern: lock the success criteria (metric, comparator, threshold,
                      
dataset SHA-256, seed) to a canonical hash before the run; compute the
                      
verdict against the locked bar; refuse to score if the criteria changed
                      
(exit 3). PRML is the open standard for that envelope: CC BY 4.0, 9 YAML
                      
fields, four MIT reference implementations, byte-identical canonicalization
                      
across Python/JS/Go/Rust.
                      Runnable example (offline, no keys):
                      https://github.com/studio-11-co/falsify/tree/main/examples/braintrust
                      
autoevals Levenshtein over a pinned dataset; TAMPERED / dataset-mismatch /
                      
FAIL / PASS paths all exercised.
                      Precedent: the same locking pattern was merged onto the UK AI Safety
                      
Institute's Inspect extensions page this month (UKGovernmentBEIS/inspect_ai#4440,
                      
Analysis category, alongside the WandB and MLflow integrations), and an
                      
MLflow community-plugin docs entry is in review (mlflow/mlflow#24386).
                      What opt-in could look like here: nothing in autoevals core needs to
                      
change. A cookbook entry or an examples/ link would already serve users who
                      
need pre-registered runs; a metadata convention for attaching the locked
                      
hash to logged experiments would be the deeper version. Happy to write
                      
either, adjust the shape, or take this to the cookbook repo if that is the
                      
better home.
                      Spec: https://spec.falsify.dev/v0.1
                      Metadata
                      Metadata
                      Assignees
                      No one assigned
                        Labels
                        No labels
                        No labels
                        Type
                        No type
                        Projects
                        No projects
                          Milestone
                          No milestone
                          Relationships
                          None yet
                          Development
                          No branches or pull requests
                          Issue actions
                          
, 'i');
if (__m === '*' || __re.test(location.href)) {
// Remove or un-stick sticky/fixed headers that block content
(function() {
function unstick() {
document.querySelectorAll('header, nav, [role="banner"], .header, .navbar, .sticky, .fixed-top, [style*="position: fixed"], [style*="position:sticky"]').forEach(function(el) {
if (el.style.position === 'fixed' || el.style.position === 'sticky' || getComputedStyle(el).position === 'fixed' || getComputedStyle(el).position === 'sticky') {
el.style.position = 'static';
el.style.top = 'auto';
el.style.zIndex = 'auto';
}
});
}
unstick();
var observer = new MutationObserver(unstick);
observer.observe(document.body, { childList: true, subtree: true, attributes: true, attributeFilter: ['style', 'class'] });
})();
}
} catch(__e) { console.warn('[Userscript:Kill Sticky Headers]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '
Proposal: opt-in pre-registration pattern for autoevals runs (PRML) · Issue #206 · braintrustdata/autoevals · GitHub
                          Skip to content
                          Proposal: opt-in pre-registration pattern for autoevals runs (PRML) #206
                          Open
                          Open
                          Proposal: opt-in pre-registration pattern for autoevals runs (PRML)#206
                          Description
                          @sk8ordie84
                          Cüneyt Öztürk (sk8ordie84)
                          opened on Jul 22, 2026
                          An opt-in pattern proposal, with a runnable example rather than a spec dump.
                          The gap it covers: an autoevals score answers "how did the model do?".
                          
It cannot answer the adversarial follow-up: "was the pass bar fixed before
                          
the results were seen, or adjusted after?" For teams whose evals feed
                          
procurement, audits, or public claims, that second question is starting to
                          
matter (EU AI Act technical documentation, Art. 11 / Annex IV).
                          The pattern: lock the success criteria (metric, comparator, threshold,
                          
dataset SHA-256, seed) to a canonical hash before the run; compute the
                          
verdict against the locked bar; refuse to score if the criteria changed
                          
(exit 3). PRML is the open standard for that envelope: CC BY 4.0, 9 YAML
                          
fields, four MIT reference implementations, byte-identical canonicalization
                          
across Python/JS/Go/Rust.
                          Runnable example (offline, no keys):
                          https://github.com/studio-11-co/falsify/tree/main/examples/braintrust
                          
autoevals Levenshtein over a pinned dataset; TAMPERED / dataset-mismatch /
                          
FAIL / PASS paths all exercised.
                          Precedent: the same locking pattern was merged onto the UK AI Safety
                          
Institute's Inspect extensions page this month (UKGovernmentBEIS/inspect_ai#4440,
                          
Analysis category, alongside the WandB and MLflow integrations), and an
                          
MLflow community-plugin docs entry is in review (mlflow/mlflow#24386).
                          What opt-in could look like here: nothing in autoevals core needs to
                          
change. A cookbook entry or an examples/ link would already serve users who
                          
need pre-registered runs; a metadata convention for attaching the locked
                          
hash to logged experiments would be the deeper version. Happy to write
                          
either, adjust the shape, or take this to the cookbook repo if that is the
                          
better home.
                          Spec: https://spec.falsify.dev/v0.1
                          Metadata
                          Metadata
                          Assignees
                          No one assigned
                            Labels
                            No labels
                            No labels
                            Type
                            No type
                            Projects
                            No projects
                              Milestone
                              No milestone
                              Relationships
                              None yet
                              Development
                              No branches or pull requests
                              Issue actions
                              
, 'i');
if (__m === '*' || __re.test(location.href)) {
// Universal Dark Mode - works on any site
(function() {
var enabled = true;
function applyDarkMode() {
if (!enabled) return;
// Create style element if it doesn't exist
var style = document.getElementById('universal-dark-mode-style');
if (!style) {
style = document.createElement('style');
style.id = 'universal-dark-mode-style';
document.head.appendChild(style);
}
// Dark mode CSS - inverts colors but preserves images/video
style.textContent = '
/* Invert everything except media */
html {
filter: invert(1) hue-rotate(180deg) !important;
background: #1a1a2e !important;
}
/* Restore images, videos, iframes, canvas */
img, video, iframe, canvas, svg, picture, [style*="background-image"] {
filter: invert(1) hue-rotate(180deg) !important;
}
/* Preserve specific elements that should not be inverted */
.no-dark-mode, .no-dark-mode *,
[data-theme="light"], [data-theme="light"],
.ace_editor, .ace_editor *,
.CodeMirror, .CodeMirror *,
.monaco-editor, .monaco-editor *,
.markdown-body pre, .markdown-body pre *,
.highlight, .highlight *,
pre code, pre code * {
filter: none !important;
}
/* Fix common UI elements */
.modal, .popup, .dropdown-menu, .tooltip, .popover {
filter: invert(1) hue-rotate(180deg) !important;
background: #2d2d44 !important;
border-color: #444 !important;
}
/* Scrollbars */
::-webkit-scrollbar { background: #1a1a2e !important; }
::-webkit-scrollbar-thumb { background: #444 !important; }
::-webkit-scrollbar-thumb:hover { background: #555 !important; }
/* Selection */
::selection { background: #4ecdc4 !important; color: #1a1a2e !important; }
::-moz-selection { background: #4ecdc4 !important; color: #1a1a2e !important; }
';
}
function removeDarkMode() {
var style = document.getElementById('universal-dark-mode-style');
if (style) style.remove();
}
// Toggle with Alt+Shift+D
document.addEventListener('keydown', function(e) {
if (e.altKey && e.shiftKey && e.key === 'D') {
e.preventDefault();
enabled = !enabled;
if (enabled) {
applyDarkMode();
console.log('[Universal Dark Mode] Enabled');
} else {
removeDarkMode();
console.log('[Universal Dark Mode] Disabled');
}
}
});
// Apply on load
applyDarkMode();
// Re-apply on dynamic content
var observer = new MutationObserver(function(mutations) {
if (enabled && !document.getElementById('universal-dark-mode-style')) {
applyDarkMode();
}
});
observer.observe(document.head, { childList: true });
console.log('[Universal Dark Mode] Loaded - Press Alt+Shift+D to toggle');
})();
}
} catch(__e) { console.warn('[Userscript:Universal Dark Mode]', __e); }
})();
})();
Proposal: opt-in pre-registration pattern for autoevals runs (PRML) · Issue #206 · braintrustdata/autoevals · GitHub
                              Skip to content
                              Proposal: opt-in pre-registration pattern for autoevals runs (PRML) #206
                              Open
                              Open
                              Proposal: opt-in pre-registration pattern for autoevals runs (PRML)#206
                              Description
                              @sk8ordie84
                              Cüneyt Öztürk (sk8ordie84)
                              opened on Jul 22, 2026
                              An opt-in pattern proposal, with a runnable example rather than a spec dump.
                              The gap it covers: an autoevals score answers "how did the model do?".
                              
It cannot answer the adversarial follow-up: "was the pass bar fixed before
                              
the results were seen, or adjusted after?" For teams whose evals feed
                              
procurement, audits, or public claims, that second question is starting to
                              
matter (EU AI Act technical documentation, Art. 11 / Annex IV).
                              The pattern: lock the success criteria (metric, comparator, threshold,
                              
dataset SHA-256, seed) to a canonical hash before the run; compute the
                              
verdict against the locked bar; refuse to score if the criteria changed
                              
(exit 3). PRML is the open standard for that envelope: CC BY 4.0, 9 YAML
                              
fields, four MIT reference implementations, byte-identical canonicalization
                              
across Python/JS/Go/Rust.
                              Runnable example (offline, no keys):
                              https://github.com/studio-11-co/falsify/tree/main/examples/braintrust
                              
autoevals Levenshtein over a pinned dataset; TAMPERED / dataset-mismatch /
                              
FAIL / PASS paths all exercised.
                              Precedent: the same locking pattern was merged onto the UK AI Safety
                              
Institute's Inspect extensions page this month (UKGovernmentBEIS/inspect_ai#4440,
                              
Analysis category, alongside the WandB and MLflow integrations), and an
                              
MLflow community-plugin docs entry is in review (mlflow/mlflow#24386).
                              What opt-in could look like here: nothing in autoevals core needs to
                              
change. A cookbook entry or an examples/ link would already serve users who
                              
need pre-registered runs; a metadata convention for attaching the locked
                              
hash to logged experiments would be the deeper version. Happy to write
                              
either, adjust the shape, or take this to the cookbook repo if that is the
                              
better home.
                              Spec: https://spec.falsify.dev/v0.1
                              Metadata
                              Metadata
                              Assignees
                              No one assigned
                                Labels
                                No labels
                                No labels
                                Type
                                No type
                                Projects
                                No projects
                                  Milestone
                                  No milestone
                                  Relationships
                                  None yet
                                  Development
                                  No branches or pull requests
                                  Issue actions