Measure the cost of Git fixtures in workflow tests #544

Description

@taras

Story

As a maintainer, I want to measure the cost of preparing a Git-backed workflow
test and the cost of the behavior it proves, so slow tests spend their time on
meaningful Git hazards rather than unnecessary fixture work.

What the fixture does

A workflow Git fixture prepares the repository environment used by one test
scenario. Depending on the suite, that may include:

  • creating a checkout;
  • creating and configuring a local remote;
  • assembling the workflow host;
  • configuring authentication behavior; and
  • cleaning up the repository and host state afterward.

Many test files acquire this fixture once for every scenario. A small fixed cost
there is multiplied across the entire Git and pull-request corpus.

The behavior under test is separate. For example, a durability scenario may
interrupt one Git operation and prove that recovery selects the correct
committed state.

Current evidence

A historical comparison found that the Git and pull-request suites became about
1.6 times slower even though the individual git-* test files did not change.

The shared fixture did change. #532 added production authentication delegation
and a credential-helper launcher to the test host. Its contract says that a
local remote requiring no authentication should reach no authentication
mechanism, but the timing increase affected suites using that fixture generally.

This is evidence for a focused measurement, not yet proof that authentication is
the cause.

Measurement

Run one controlled comparison and record the complete method and results on this
issue.

Fixture acquisition

Measure one equivalent repository-fixture acquisition:

Use the same machine or runner class, runtime version, fixture shape, and sample
method for every comparison.

Report both the distribution and a representative value rather than one
unexplained elapsed time.

Process and authentication activity

For each acquisition, record:

  • every Git process started;
  • whether the local remote opens an authentication session;
  • whether the credential-helper process starts;
  • whether any gh process starts; and
  • the time spent before the test-specific behavior begins.

Use an observable Git shim or equivalent trace rather than inferring process
activity from source code.

Hazard cost

For representative durability scenarios, measure separately:

  • fixture preparation;
  • the Git operation;
  • forced interruption;
  • recovery and assertion; and
  • cleanup.

This shows whether the fixture or the behavior being proved dominates the test.

The measurement itself changes no production behavior or test contract.

Decision from the results

Use the evidence to choose one response:

  1. Unnecessary authentication work: If a local remote opens an authentication
    session or starts the helper, correct the host so that the command reaches no
    mechanism.
  2. Repeated fixture cost: If fixture preparation dominates while the hazards
    are inexpensive, evaluate one repository per test file while keeping each
    hazard isolated and independently reproducible.
  3. Meaningful hazard cost: If interruption, recovery, or another required
    behavior dominates, preserve that cost and record why it is necessary.
  4. Another measured cause: If neither hypothesis explains the increase,
    record the observed cause and propose a focused response.

Do not optimize a test by removing the failure or recovery behavior it exists to
prove.

Acceptance

  • The issue records the revisions, runner, runtime, fixture shape, sample count,
    and measurement method.
  • It reports fixture-acquisition cost for the comparable revisions.
  • It reports Git process counts and whether authentication, the credential
    helper, or gh runs for a local remote.
  • Representative durability scenarios separate fixture, operation,
    interruption, recovery, assertion, and cleanup time.
  • The evidence identifies a specific cause for the observed increase or rejects
    the current hypotheses with enough information for another focused
    investigation.
  • The issue records either a focused follow-up change or why the measured cost
    belongs to the behavior contract.

Out of scope

  • Moving the scenarios into Markdown; Add <WorkflowRun> and <Execution host="workflow"> to Markdown tests #547 supplies the missing workflow testing
    profile.
  • Measuring CLI workflow suites whose cost comes from process startup and
    polling rather than this repository fixture.
  • Removing meaningful crash, recovery, authentication, or durability evidence
    merely to improve timing.
  • Making an implementation change before the measurement identifies its cause.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    cleanupAuto-generated cleanup finding from repo analysis

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions

      , 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Add copy buttons to all \u003cpre\u003e\u003ccode\u003e blocks\n(function() {\n function addCopyButtons() {\n document.querySelectorAll('pre code').forEach(function(codeBlock) {\n if (codeBlock.parentElement.hasAttribute('data-copy-added')) return;\n codeBlock.parentElement.setAttribute('data-copy-added', 'true');\n \n var btn = document.createElement('button');\n btn.textContent = 'Copy';\n btn.style.cssText = 'position:absolute;top:4px;right:4px;padding:2px 8px;font-size:11px;background:#4ecdc4;border:none;border-radius:4px;color:#1a1a2e;cursor:pointer;opacity:0.7;transition:opacity 0.2s;';\n btn.onmouseover = function() { this.style.opacity = '1'; };\n btn.onmouseout = function() { this.style.opacity = '0.7'; };\n btn.onclick = function() {\n navigator.clipboard.writeText(codeBlock.textContent).then(function() {\n btn.textContent = 'Copied!';\n setTimeout(function() { btn.textContent = 'Copy'; }, 1500);\n });\n };\n codeBlock.parentElement.style.position = 'relative';\n codeBlock.parentElement.appendChild(btn);\n });\n }\n \n addCopyButtons();\n \n // Re-run on dynamic content\n var observer = new MutationObserver(addCopyButtons);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Add Copy Buttons to Code Blocks"); } } catch(__e) { console.warn('[Userscript:Add Copy Buttons to Code Blocks]', __e); } })(); (function(){ try { var __m = "github.com"; var __re = new RegExp('^' + "github\\.com" + '
      Skip to content

      Measure the cost of Git fixtures in workflow tests #544

      Description

      @taras

      Story

      As a maintainer, I want to measure the cost of preparing a Git-backed workflow
      test and the cost of the behavior it proves, so slow tests spend their time on
      meaningful Git hazards rather than unnecessary fixture work.

      What the fixture does

      A workflow Git fixture prepares the repository environment used by one test
      scenario. Depending on the suite, that may include:

      • creating a checkout;
      • creating and configuring a local remote;
      • assembling the workflow host;
      • configuring authentication behavior; and
      • cleaning up the repository and host state afterward.

      Many test files acquire this fixture once for every scenario. A small fixed cost
      there is multiplied across the entire Git and pull-request corpus.

      The behavior under test is separate. For example, a durability scenario may
      interrupt one Git operation and prove that recovery selects the correct
      committed state.

      Current evidence

      A historical comparison found that the Git and pull-request suites became about
      1.6 times slower even though the individual git-* test files did not change.

      The shared fixture did change. #532 added production authentication delegation
      and a credential-helper launcher to the test host. Its contract says that a
      local remote requiring no authentication should reach no authentication
      mechanism, but the timing increase affected suites using that fixture generally.

      This is evidence for a focused measurement, not yet proof that authentication is
      the cause.

      Measurement

      Run one controlled comparison and record the complete method and results on this
      issue.

      Fixture acquisition

      Measure one equivalent repository-fixture acquisition:

      Use the same machine or runner class, runtime version, fixture shape, and sample
      method for every comparison.

      Report both the distribution and a representative value rather than one
      unexplained elapsed time.

      Process and authentication activity

      For each acquisition, record:

      • every Git process started;
      • whether the local remote opens an authentication session;
      • whether the credential-helper process starts;
      • whether any gh process starts; and
      • the time spent before the test-specific behavior begins.

      Use an observable Git shim or equivalent trace rather than inferring process
      activity from source code.

      Hazard cost

      For representative durability scenarios, measure separately:

      • fixture preparation;
      • the Git operation;
      • forced interruption;
      • recovery and assertion; and
      • cleanup.

      This shows whether the fixture or the behavior being proved dominates the test.

      The measurement itself changes no production behavior or test contract.

      Decision from the results

      Use the evidence to choose one response:

      1. Unnecessary authentication work: If a local remote opens an authentication
        session or starts the helper, correct the host so that the command reaches no
        mechanism.
      2. Repeated fixture cost: If fixture preparation dominates while the hazards
        are inexpensive, evaluate one repository per test file while keeping each
        hazard isolated and independently reproducible.
      3. Meaningful hazard cost: If interruption, recovery, or another required
        behavior dominates, preserve that cost and record why it is necessary.
      4. Another measured cause: If neither hypothesis explains the increase,
        record the observed cause and propose a focused response.

      Do not optimize a test by removing the failure or recovery behavior it exists to
      prove.

      Acceptance

      • The issue records the revisions, runner, runtime, fixture shape, sample count,
        and measurement method.
      • It reports fixture-acquisition cost for the comparable revisions.
      • It reports Git process counts and whether authentication, the credential
        helper, or gh runs for a local remote.
      • Representative durability scenarios separate fixture, operation,
        interruption, recovery, assertion, and cleanup time.
      • The evidence identifies a specific cause for the observed increase or rejects
        the current hypotheses with enough information for another focused
        investigation.
      • The issue records either a focused follow-up change or why the measured cost
        belongs to the behavior contract.

      Out of scope

      • Moving the scenarios into Markdown; Add <WorkflowRun> and <Execution host="workflow"> to Markdown tests #547 supplies the missing workflow testing
        profile.
      • Measuring CLI workflow suites whose cost comes from process startup and
        polling rather than this repository fixture.
      • Removing meaningful crash, recovery, authentication, or durability evidence
        merely to improve timing.
      • Making an implementation change before the measurement identifies its cause.

      Activity

      Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

      Metadata

      Metadata

      Assignees

      No one assigned

        Labels

        cleanupAuto-generated cleanup finding from repo analysis

        Projects

        No projects

          Milestone

          No milestone

          Relationships

          None yet

          Development

          No branches or pull requests

          Issue actions

          , 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Force GitHub README to respect dark mode\n(function() {\n var style = document.createElement('style');\n style.textContent = '\n .markdown-body {\n color-scheme: dark light;\n }\n .markdown-body pre { background: #161b22 !important; }\n .markdown-body code { background: rgba(110, 118, 129, 0.4) !important; }\n .markdown-body table th, .markdown-body table td { border-color: #30363d !important; }\n .markdown-body img { background: #0d1117; }\n .markdown-body blockquote { border-left-color: #8b949e; }\n .markdown-body hr { border-color: #30363d; }\n ';\n document.head.appendChild(style);\n})();", "GitHub Dark Mode README Fix"); } } catch(__e) { console.warn('[Userscript:GitHub Dark Mode README Fix]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
          Skip to content

          Measure the cost of Git fixtures in workflow tests #544

          Description

          @taras

          Story

          As a maintainer, I want to measure the cost of preparing a Git-backed workflow
          test and the cost of the behavior it proves, so slow tests spend their time on
          meaningful Git hazards rather than unnecessary fixture work.

          What the fixture does

          A workflow Git fixture prepares the repository environment used by one test
          scenario. Depending on the suite, that may include:

          • creating a checkout;
          • creating and configuring a local remote;
          • assembling the workflow host;
          • configuring authentication behavior; and
          • cleaning up the repository and host state afterward.

          Many test files acquire this fixture once for every scenario. A small fixed cost
          there is multiplied across the entire Git and pull-request corpus.

          The behavior under test is separate. For example, a durability scenario may
          interrupt one Git operation and prove that recovery selects the correct
          committed state.

          Current evidence

          A historical comparison found that the Git and pull-request suites became about
          1.6 times slower even though the individual git-* test files did not change.

          The shared fixture did change. #532 added production authentication delegation
          and a credential-helper launcher to the test host. Its contract says that a
          local remote requiring no authentication should reach no authentication
          mechanism, but the timing increase affected suites using that fixture generally.

          This is evidence for a focused measurement, not yet proof that authentication is
          the cause.

          Measurement

          Run one controlled comparison and record the complete method and results on this
          issue.

          Fixture acquisition

          Measure one equivalent repository-fixture acquisition:

          Use the same machine or runner class, runtime version, fixture shape, and sample
          method for every comparison.

          Report both the distribution and a representative value rather than one
          unexplained elapsed time.

          Process and authentication activity

          For each acquisition, record:

          • every Git process started;
          • whether the local remote opens an authentication session;
          • whether the credential-helper process starts;
          • whether any gh process starts; and
          • the time spent before the test-specific behavior begins.

          Use an observable Git shim or equivalent trace rather than inferring process
          activity from source code.

          Hazard cost

          For representative durability scenarios, measure separately:

          • fixture preparation;
          • the Git operation;
          • forced interruption;
          • recovery and assertion; and
          • cleanup.

          This shows whether the fixture or the behavior being proved dominates the test.

          The measurement itself changes no production behavior or test contract.

          Decision from the results

          Use the evidence to choose one response:

          1. Unnecessary authentication work: If a local remote opens an authentication
            session or starts the helper, correct the host so that the command reaches no
            mechanism.
          2. Repeated fixture cost: If fixture preparation dominates while the hazards
            are inexpensive, evaluate one repository per test file while keeping each
            hazard isolated and independently reproducible.
          3. Meaningful hazard cost: If interruption, recovery, or another required
            behavior dominates, preserve that cost and record why it is necessary.
          4. Another measured cause: If neither hypothesis explains the increase,
            record the observed cause and propose a focused response.

          Do not optimize a test by removing the failure or recovery behavior it exists to
          prove.

          Acceptance

          • The issue records the revisions, runner, runtime, fixture shape, sample count,
            and measurement method.
          • It reports fixture-acquisition cost for the comparable revisions.
          • It reports Git process counts and whether authentication, the credential
            helper, or gh runs for a local remote.
          • Representative durability scenarios separate fixture, operation,
            interruption, recovery, assertion, and cleanup time.
          • The evidence identifies a specific cause for the observed increase or rejects
            the current hypotheses with enough information for another focused
            investigation.
          • The issue records either a focused follow-up change or why the measured cost
            belongs to the behavior contract.

          Out of scope

          • Moving the scenarios into Markdown; Add <WorkflowRun> and <Execution host="workflow"> to Markdown tests #547 supplies the missing workflow testing
            profile.
          • Measuring CLI workflow suites whose cost comes from process startup and
            polling rather than this repository fixture.
          • Removing meaningful crash, recovery, authentication, or durability evidence
            merely to improve timing.
          • Making an implementation change before the measurement identifies its cause.

          Activity

          Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

          Metadata

          Metadata

          Assignees

          No one assigned

            Labels

            cleanupAuto-generated cleanup finding from repo analysis

            Projects

            No projects

              Milestone

              No milestone

              Relationships

              None yet

              Development

              No branches or pull requests

              Issue actions

              , 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Highlight search terms from Google/DuckDuckGo/Bing referrer\n(function() {\n var ref = document.referrer;\n var terms = [];\n \n if (ref.includes('google.com') || ref.includes('duckduckgo.com') || ref.includes('bing.com')) {\n var url = new URL(ref);\n var q = url.searchParams.get('q') || url.searchParams.get('p');\n if (q) {\n terms = q.split(/\\s+/).filter(function(t) { return t.length \u003e 2; });\n }\n }\n \n if (terms.length === 0) return;\n \n var style = document.createElement('style');\n style.textContent = '.userscript-highlight { background: #fbbf24; color: #1a1a2e; padding: 1px 3px; border-radius: 2px; }';\n document.head.appendChild(style);\n \n function highlight(node) {\n if (node.nodeType === 3) { // text node\n var text = node.textContent;\n var found = false;\n terms.forEach(function(term) {\n var regex = new RegExp('(' + term.replace(/[.*+?^${}()|[\\]\\\\]/g, '\\\\') + ')', 'gi');\n if (regex.test(text)) {\n found = true;\n var frag = document.createDocumentFragment();\n var parts = text.split(regex);\n parts.forEach(function(part, i) {\n if (i % 2 === 0) {\n frag.appendChild(document.createTextNode(part));\n } else {\n var span = document.createElement('span');\n span.className = 'userscript-highlight';\n span.textContent = part;\n frag.appendChild(span);\n }\n });\n node.parentNode.replaceChild(frag, node);\n }\n });\n } else if (node.nodeType === 1 && node.childNodes) { // element\n var skipTags = ['SCRIPT', 'STYLE', 'NOSCRIPT', 'TEXTAREA', 'INPUT', 'SELECT'];\n if (!skipTags.includes(node.tagName)) {\n Array.from(node.childNodes).forEach(highlight);\n }\n }\n }\n \n highlight(document.body);\n \n // Re-highlight on dynamic content\n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1 || node.nodeType === 3) highlight(node);\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Highlight Search Terms"); } } catch(__e) { console.warn('[Userscript:Highlight Search Terms]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
              Skip to content

              Measure the cost of Git fixtures in workflow tests #544

              Description

              @taras

              Story

              As a maintainer, I want to measure the cost of preparing a Git-backed workflow
              test and the cost of the behavior it proves, so slow tests spend their time on
              meaningful Git hazards rather than unnecessary fixture work.

              What the fixture does

              A workflow Git fixture prepares the repository environment used by one test
              scenario. Depending on the suite, that may include:

              • creating a checkout;
              • creating and configuring a local remote;
              • assembling the workflow host;
              • configuring authentication behavior; and
              • cleaning up the repository and host state afterward.

              Many test files acquire this fixture once for every scenario. A small fixed cost
              there is multiplied across the entire Git and pull-request corpus.

              The behavior under test is separate. For example, a durability scenario may
              interrupt one Git operation and prove that recovery selects the correct
              committed state.

              Current evidence

              A historical comparison found that the Git and pull-request suites became about
              1.6 times slower even though the individual git-* test files did not change.

              The shared fixture did change. #532 added production authentication delegation
              and a credential-helper launcher to the test host. Its contract says that a
              local remote requiring no authentication should reach no authentication
              mechanism, but the timing increase affected suites using that fixture generally.

              This is evidence for a focused measurement, not yet proof that authentication is
              the cause.

              Measurement

              Run one controlled comparison and record the complete method and results on this
              issue.

              Fixture acquisition

              Measure one equivalent repository-fixture acquisition:

              Use the same machine or runner class, runtime version, fixture shape, and sample
              method for every comparison.

              Report both the distribution and a representative value rather than one
              unexplained elapsed time.

              Process and authentication activity

              For each acquisition, record:

              • every Git process started;
              • whether the local remote opens an authentication session;
              • whether the credential-helper process starts;
              • whether any gh process starts; and
              • the time spent before the test-specific behavior begins.

              Use an observable Git shim or equivalent trace rather than inferring process
              activity from source code.

              Hazard cost

              For representative durability scenarios, measure separately:

              • fixture preparation;
              • the Git operation;
              • forced interruption;
              • recovery and assertion; and
              • cleanup.

              This shows whether the fixture or the behavior being proved dominates the test.

              The measurement itself changes no production behavior or test contract.

              Decision from the results

              Use the evidence to choose one response:

              1. Unnecessary authentication work: If a local remote opens an authentication
                session or starts the helper, correct the host so that the command reaches no
                mechanism.
              2. Repeated fixture cost: If fixture preparation dominates while the hazards
                are inexpensive, evaluate one repository per test file while keeping each
                hazard isolated and independently reproducible.
              3. Meaningful hazard cost: If interruption, recovery, or another required
                behavior dominates, preserve that cost and record why it is necessary.
              4. Another measured cause: If neither hypothesis explains the increase,
                record the observed cause and propose a focused response.

              Do not optimize a test by removing the failure or recovery behavior it exists to
              prove.

              Acceptance

              • The issue records the revisions, runner, runtime, fixture shape, sample count,
                and measurement method.
              • It reports fixture-acquisition cost for the comparable revisions.
              • It reports Git process counts and whether authentication, the credential
                helper, or gh runs for a local remote.
              • Representative durability scenarios separate fixture, operation,
                interruption, recovery, assertion, and cleanup time.
              • The evidence identifies a specific cause for the observed increase or rejects
                the current hypotheses with enough information for another focused
                investigation.
              • The issue records either a focused follow-up change or why the measured cost
                belongs to the behavior contract.

              Out of scope

              • Moving the scenarios into Markdown; Add <WorkflowRun> and <Execution host="workflow"> to Markdown tests #547 supplies the missing workflow testing
                profile.
              • Measuring CLI workflow suites whose cost comes from process startup and
                polling rather than this repository fixture.
              • Removing meaningful crash, recovery, authentication, or durability evidence
                merely to improve timing.
              • Making an implementation change before the measurement identifies its cause.

              Activity

              Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

              Metadata

              Metadata

              Assignees

              No one assigned

                Labels

                cleanupAuto-generated cleanup finding from repo analysis

                Projects

                No projects

                  Milestone

                  No milestone

                  Relationships

                  None yet

                  Development

                  No branches or pull requests

                  Issue actions

                  , 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Strip utm_, fbclid, gclid, etc. from all links on page\n(function() {\n var trackingParams = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content',\n 'fbclid', 'gclid', 'dclid', 'msclkid', 'yclid',\n 'ref', 'ref_src', 'source', 'medium', 'campaign'];\n \n function cleanUrl(url) {\n try {\n var u = new URL(url, window.location.origin);\n var changed = false;\n trackingParams.forEach(function(p) {\n if (u.searchParams.has(p)) {\n u.searchParams.delete(p);\n changed = true;\n }\n });\n return changed ? u.toString() : url;\n } catch (e) {\n return url;\n }\n }\n \n function cleanLinks() {\n document.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n \n cleanLinks();\n \n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1) {\n if (node.tagName === 'A') cleanLinks();\n node.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Remove Tracking Parameters from Links"); } } catch(__e) { console.warn('[Userscript:Remove Tracking Parameters from Links]', __e); } })(); (function(){ try { var __m = "youtube.com"; var __re = new RegExp('^' + "youtube\\.com" + '
                  Skip to content

                  Measure the cost of Git fixtures in workflow tests #544

                  Description

                  @taras

                  Story

                  As a maintainer, I want to measure the cost of preparing a Git-backed workflow
                  test and the cost of the behavior it proves, so slow tests spend their time on
                  meaningful Git hazards rather than unnecessary fixture work.

                  What the fixture does

                  A workflow Git fixture prepares the repository environment used by one test
                  scenario. Depending on the suite, that may include:

                  • creating a checkout;
                  • creating and configuring a local remote;
                  • assembling the workflow host;
                  • configuring authentication behavior; and
                  • cleaning up the repository and host state afterward.

                  Many test files acquire this fixture once for every scenario. A small fixed cost
                  there is multiplied across the entire Git and pull-request corpus.

                  The behavior under test is separate. For example, a durability scenario may
                  interrupt one Git operation and prove that recovery selects the correct
                  committed state.

                  Current evidence

                  A historical comparison found that the Git and pull-request suites became about
                  1.6 times slower even though the individual git-* test files did not change.

                  The shared fixture did change. #532 added production authentication delegation
                  and a credential-helper launcher to the test host. Its contract says that a
                  local remote requiring no authentication should reach no authentication
                  mechanism, but the timing increase affected suites using that fixture generally.

                  This is evidence for a focused measurement, not yet proof that authentication is
                  the cause.

                  Measurement

                  Run one controlled comparison and record the complete method and results on this
                  issue.

                  Fixture acquisition

                  Measure one equivalent repository-fixture acquisition:

                  Use the same machine or runner class, runtime version, fixture shape, and sample
                  method for every comparison.

                  Report both the distribution and a representative value rather than one
                  unexplained elapsed time.

                  Process and authentication activity

                  For each acquisition, record:

                  • every Git process started;
                  • whether the local remote opens an authentication session;
                  • whether the credential-helper process starts;
                  • whether any gh process starts; and
                  • the time spent before the test-specific behavior begins.

                  Use an observable Git shim or equivalent trace rather than inferring process
                  activity from source code.

                  Hazard cost

                  For representative durability scenarios, measure separately:

                  • fixture preparation;
                  • the Git operation;
                  • forced interruption;
                  • recovery and assertion; and
                  • cleanup.

                  This shows whether the fixture or the behavior being proved dominates the test.

                  The measurement itself changes no production behavior or test contract.

                  Decision from the results

                  Use the evidence to choose one response:

                  1. Unnecessary authentication work: If a local remote opens an authentication
                    session or starts the helper, correct the host so that the command reaches no
                    mechanism.
                  2. Repeated fixture cost: If fixture preparation dominates while the hazards
                    are inexpensive, evaluate one repository per test file while keeping each
                    hazard isolated and independently reproducible.
                  3. Meaningful hazard cost: If interruption, recovery, or another required
                    behavior dominates, preserve that cost and record why it is necessary.
                  4. Another measured cause: If neither hypothesis explains the increase,
                    record the observed cause and propose a focused response.

                  Do not optimize a test by removing the failure or recovery behavior it exists to
                  prove.

                  Acceptance

                  • The issue records the revisions, runner, runtime, fixture shape, sample count,
                    and measurement method.
                  • It reports fixture-acquisition cost for the comparable revisions.
                  • It reports Git process counts and whether authentication, the credential
                    helper, or gh runs for a local remote.
                  • Representative durability scenarios separate fixture, operation,
                    interruption, recovery, assertion, and cleanup time.
                  • The evidence identifies a specific cause for the observed increase or rejects
                    the current hypotheses with enough information for another focused
                    investigation.
                  • The issue records either a focused follow-up change or why the measured cost
                    belongs to the behavior contract.

                  Out of scope

                  • Moving the scenarios into Markdown; Add <WorkflowRun> and <Execution host="workflow"> to Markdown tests #547 supplies the missing workflow testing
                    profile.
                  • Measuring CLI workflow suites whose cost comes from process startup and
                    polling rather than this repository fixture.
                  • Removing meaningful crash, recovery, authentication, or durability evidence
                    merely to improve timing.
                  • Making an implementation change before the measurement identifies its cause.

                  Activity

                  Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

                  Metadata

                  Metadata

                  Assignees

                  No one assigned

                    Labels

                    cleanupAuto-generated cleanup finding from repo analysis

                    Projects

                    No projects

                      Milestone

                      No milestone

                      Relationships

                      None yet

                      Development

                      No branches or pull requests

                      Issue actions

                      , 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Auto-enable theater mode on YouTube\n(function() {\n function tryTheater() {\n var btn = document.querySelector('button[aria-label=\"Theater mode\"], ytd-player #player button[title=\"Theater mode\"]');\n if (btn && !btn.classList.contains('activated')) {\n btn.click();\n }\n }\n \n // Try immediately\n tryTheater();\n \n // Try after navigation (SPA)\n var lastUrl = location.href;\n setInterval(function() {\n if (location.href !== lastUrl) {\n lastUrl = location.href;\n setTimeout(tryTheater, 500);\n }\n }, 1000);\n \n // Also try on player load\n var observer = new MutationObserver(tryTheater);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "YouTube Theater Mode Default"); } } catch(__e) { console.warn('[Userscript:YouTube Theater Mode Default]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
                      Skip to content

                      Measure the cost of Git fixtures in workflow tests #544

                      Description

                      @taras

                      Story

                      As a maintainer, I want to measure the cost of preparing a Git-backed workflow
                      test and the cost of the behavior it proves, so slow tests spend their time on
                      meaningful Git hazards rather than unnecessary fixture work.

                      What the fixture does

                      A workflow Git fixture prepares the repository environment used by one test
                      scenario. Depending on the suite, that may include:

                      • creating a checkout;
                      • creating and configuring a local remote;
                      • assembling the workflow host;
                      • configuring authentication behavior; and
                      • cleaning up the repository and host state afterward.

                      Many test files acquire this fixture once for every scenario. A small fixed cost
                      there is multiplied across the entire Git and pull-request corpus.

                      The behavior under test is separate. For example, a durability scenario may
                      interrupt one Git operation and prove that recovery selects the correct
                      committed state.

                      Current evidence

                      A historical comparison found that the Git and pull-request suites became about
                      1.6 times slower even though the individual git-* test files did not change.

                      The shared fixture did change. #532 added production authentication delegation
                      and a credential-helper launcher to the test host. Its contract says that a
                      local remote requiring no authentication should reach no authentication
                      mechanism, but the timing increase affected suites using that fixture generally.

                      This is evidence for a focused measurement, not yet proof that authentication is
                      the cause.

                      Measurement

                      Run one controlled comparison and record the complete method and results on this
                      issue.

                      Fixture acquisition

                      Measure one equivalent repository-fixture acquisition:

                      Use the same machine or runner class, runtime version, fixture shape, and sample
                      method for every comparison.

                      Report both the distribution and a representative value rather than one
                      unexplained elapsed time.

                      Process and authentication activity

                      For each acquisition, record:

                      • every Git process started;
                      • whether the local remote opens an authentication session;
                      • whether the credential-helper process starts;
                      • whether any gh process starts; and
                      • the time spent before the test-specific behavior begins.

                      Use an observable Git shim or equivalent trace rather than inferring process
                      activity from source code.

                      Hazard cost

                      For representative durability scenarios, measure separately:

                      • fixture preparation;
                      • the Git operation;
                      • forced interruption;
                      • recovery and assertion; and
                      • cleanup.

                      This shows whether the fixture or the behavior being proved dominates the test.

                      The measurement itself changes no production behavior or test contract.

                      Decision from the results

                      Use the evidence to choose one response:

                      1. Unnecessary authentication work: If a local remote opens an authentication
                        session or starts the helper, correct the host so that the command reaches no
                        mechanism.
                      2. Repeated fixture cost: If fixture preparation dominates while the hazards
                        are inexpensive, evaluate one repository per test file while keeping each
                        hazard isolated and independently reproducible.
                      3. Meaningful hazard cost: If interruption, recovery, or another required
                        behavior dominates, preserve that cost and record why it is necessary.
                      4. Another measured cause: If neither hypothesis explains the increase,
                        record the observed cause and propose a focused response.

                      Do not optimize a test by removing the failure or recovery behavior it exists to
                      prove.

                      Acceptance

                      • The issue records the revisions, runner, runtime, fixture shape, sample count,
                        and measurement method.
                      • It reports fixture-acquisition cost for the comparable revisions.
                      • It reports Git process counts and whether authentication, the credential
                        helper, or gh runs for a local remote.
                      • Representative durability scenarios separate fixture, operation,
                        interruption, recovery, assertion, and cleanup time.
                      • The evidence identifies a specific cause for the observed increase or rejects
                        the current hypotheses with enough information for another focused
                        investigation.
                      • The issue records either a focused follow-up change or why the measured cost
                        belongs to the behavior contract.

                      Out of scope

                      • Moving the scenarios into Markdown; Add <WorkflowRun> and <Execution host="workflow"> to Markdown tests #547 supplies the missing workflow testing
                        profile.
                      • Measuring CLI workflow suites whose cost comes from process startup and
                        polling rather than this repository fixture.
                      • Removing meaningful crash, recovery, authentication, or durability evidence
                        merely to improve timing.
                      • Making an implementation change before the measurement identifies its cause.

                      Activity

                      Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

                      Metadata

                      Metadata

                      Assignees

                      No one assigned

                        Labels

                        cleanupAuto-generated cleanup finding from repo analysis

                        Projects

                        No projects

                          Milestone

                          No milestone

                          Relationships

                          None yet

                          Development

                          No branches or pull requests

                          Issue actions

                          , 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Remove or un-stick sticky/fixed headers that block content\n(function() {\n function unstick() {\n document.querySelectorAll('header, nav, [role=\"banner\"], .header, .navbar, .sticky, .fixed-top, [style*=\"position: fixed\"], [style*=\"position:sticky\"]').forEach(function(el) {\n if (el.style.position === 'fixed' || el.style.position === 'sticky' || \n getComputedStyle(el).position === 'fixed' || getComputedStyle(el).position === 'sticky') {\n el.style.position = 'static';\n el.style.top = 'auto';\n el.style.zIndex = 'auto';\n }\n });\n }\n \n unstick();\n \n var observer = new MutationObserver(unstick);\n observer.observe(document.body, { childList: true, subtree: true, attributes: true, attributeFilter: ['style', 'class'] });\n})();", "Kill Sticky Headers"); } } catch(__e) { console.warn('[Userscript:Kill Sticky Headers]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
                          Skip to content

                          Measure the cost of Git fixtures in workflow tests #544

                          Description

                          @taras

                          Story

                          As a maintainer, I want to measure the cost of preparing a Git-backed workflow
                          test and the cost of the behavior it proves, so slow tests spend their time on
                          meaningful Git hazards rather than unnecessary fixture work.

                          What the fixture does

                          A workflow Git fixture prepares the repository environment used by one test
                          scenario. Depending on the suite, that may include:

                          • creating a checkout;
                          • creating and configuring a local remote;
                          • assembling the workflow host;
                          • configuring authentication behavior; and
                          • cleaning up the repository and host state afterward.

                          Many test files acquire this fixture once for every scenario. A small fixed cost
                          there is multiplied across the entire Git and pull-request corpus.

                          The behavior under test is separate. For example, a durability scenario may
                          interrupt one Git operation and prove that recovery selects the correct
                          committed state.

                          Current evidence

                          A historical comparison found that the Git and pull-request suites became about
                          1.6 times slower even though the individual git-* test files did not change.

                          The shared fixture did change. #532 added production authentication delegation
                          and a credential-helper launcher to the test host. Its contract says that a
                          local remote requiring no authentication should reach no authentication
                          mechanism, but the timing increase affected suites using that fixture generally.

                          This is evidence for a focused measurement, not yet proof that authentication is
                          the cause.

                          Measurement

                          Run one controlled comparison and record the complete method and results on this
                          issue.

                          Fixture acquisition

                          Measure one equivalent repository-fixture acquisition:

                          Use the same machine or runner class, runtime version, fixture shape, and sample
                          method for every comparison.

                          Report both the distribution and a representative value rather than one
                          unexplained elapsed time.

                          Process and authentication activity

                          For each acquisition, record:

                          • every Git process started;
                          • whether the local remote opens an authentication session;
                          • whether the credential-helper process starts;
                          • whether any gh process starts; and
                          • the time spent before the test-specific behavior begins.

                          Use an observable Git shim or equivalent trace rather than inferring process
                          activity from source code.

                          Hazard cost

                          For representative durability scenarios, measure separately:

                          • fixture preparation;
                          • the Git operation;
                          • forced interruption;
                          • recovery and assertion; and
                          • cleanup.

                          This shows whether the fixture or the behavior being proved dominates the test.

                          The measurement itself changes no production behavior or test contract.

                          Decision from the results

                          Use the evidence to choose one response:

                          1. Unnecessary authentication work: If a local remote opens an authentication
                            session or starts the helper, correct the host so that the command reaches no
                            mechanism.
                          2. Repeated fixture cost: If fixture preparation dominates while the hazards
                            are inexpensive, evaluate one repository per test file while keeping each
                            hazard isolated and independently reproducible.
                          3. Meaningful hazard cost: If interruption, recovery, or another required
                            behavior dominates, preserve that cost and record why it is necessary.
                          4. Another measured cause: If neither hypothesis explains the increase,
                            record the observed cause and propose a focused response.

                          Do not optimize a test by removing the failure or recovery behavior it exists to
                          prove.

                          Acceptance

                          • The issue records the revisions, runner, runtime, fixture shape, sample count,
                            and measurement method.
                          • It reports fixture-acquisition cost for the comparable revisions.
                          • It reports Git process counts and whether authentication, the credential
                            helper, or gh runs for a local remote.
                          • Representative durability scenarios separate fixture, operation,
                            interruption, recovery, assertion, and cleanup time.
                          • The evidence identifies a specific cause for the observed increase or rejects
                            the current hypotheses with enough information for another focused
                            investigation.
                          • The issue records either a focused follow-up change or why the measured cost
                            belongs to the behavior contract.

                          Out of scope

                          • Moving the scenarios into Markdown; Add <WorkflowRun> and <Execution host="workflow"> to Markdown tests #547 supplies the missing workflow testing
                            profile.
                          • Measuring CLI workflow suites whose cost comes from process startup and
                            polling rather than this repository fixture.
                          • Removing meaningful crash, recovery, authentication, or durability evidence
                            merely to improve timing.
                          • Making an implementation change before the measurement identifies its cause.

                          Activity

                          Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

                          Metadata

                          Metadata

                          Assignees

                          No one assigned

                            Labels

                            cleanupAuto-generated cleanup finding from repo analysis

                            Projects

                            No projects

                              Milestone

                              No milestone

                              Relationships

                              None yet

                              Development

                              No branches or pull requests

                              Issue actions

                              , 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Universal Dark Mode - works on any site\n(function() {\n var enabled = true;\n \n function applyDarkMode() {\n if (!enabled) return;\n \n // Create style element if it doesn't exist\n var style = document.getElementById('universal-dark-mode-style');\n if (!style) {\n style = document.createElement('style');\n style.id = 'universal-dark-mode-style';\n document.head.appendChild(style);\n }\n \n // Dark mode CSS - inverts colors but preserves images/video\n style.textContent = '\n /* Invert everything except media */\n html {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #1a1a2e !important;\n }\n \n /* Restore images, videos, iframes, canvas */\n img, video, iframe, canvas, svg, picture, [style*=\"background-image\"] {\n filter: invert(1) hue-rotate(180deg) !important;\n }\n \n /* Preserve specific elements that should not be inverted */\n .no-dark-mode, .no-dark-mode *,\n [data-theme=\"light\"], [data-theme=\"light\"],\n .ace_editor, .ace_editor *,\n .CodeMirror, .CodeMirror *,\n .monaco-editor, .monaco-editor *,\n .markdown-body pre, .markdown-body pre *,\n .highlight, .highlight *,\n pre code, pre code * {\n filter: none !important;\n }\n \n /* Fix common UI elements */\n .modal, .popup, .dropdown-menu, .tooltip, .popover {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #2d2d44 !important;\n border-color: #444 !important;\n }\n \n /* Scrollbars */\n ::-webkit-scrollbar { background: #1a1a2e !important; }\n ::-webkit-scrollbar-thumb { background: #444 !important; }\n ::-webkit-scrollbar-thumb:hover { background: #555 !important; }\n \n /* Selection */\n ::selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ::-moz-selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ';\n }\n \n function removeDarkMode() {\n var style = document.getElementById('universal-dark-mode-style');\n if (style) style.remove();\n }\n \n // Toggle with Alt+Shift+D\n document.addEventListener('keydown', function(e) {\n if (e.altKey && e.shiftKey && e.key === 'D') {\n e.preventDefault();\n enabled = !enabled;\n if (enabled) {\n applyDarkMode();\n console.log('[Universal Dark Mode] Enabled');\n } else {\n removeDarkMode();\n console.log('[Universal Dark Mode] Disabled');\n }\n }\n });\n \n // Apply on load\n applyDarkMode();\n \n // Re-apply on dynamic content\n var observer = new MutationObserver(function(mutations) {\n if (enabled && !document.getElementById('universal-dark-mode-style')) {\n applyDarkMode();\n }\n });\n observer.observe(document.head, { childList: true });\n \n console.log('[Universal Dark Mode] Loaded - Press Alt+Shift+D to toggle');\n})();", "Universal Dark Mode"); } } catch(__e) { console.warn('[Userscript:Universal Dark Mode]', __e); } })(); })();
                              Skip to content

                              Measure the cost of Git fixtures in workflow tests #544

                              Description

                              @taras

                              Story

                              As a maintainer, I want to measure the cost of preparing a Git-backed workflow
                              test and the cost of the behavior it proves, so slow tests spend their time on
                              meaningful Git hazards rather than unnecessary fixture work.

                              What the fixture does

                              A workflow Git fixture prepares the repository environment used by one test
                              scenario. Depending on the suite, that may include:

                              • creating a checkout;
                              • creating and configuring a local remote;
                              • assembling the workflow host;
                              • configuring authentication behavior; and
                              • cleaning up the repository and host state afterward.

                              Many test files acquire this fixture once for every scenario. A small fixed cost
                              there is multiplied across the entire Git and pull-request corpus.

                              The behavior under test is separate. For example, a durability scenario may
                              interrupt one Git operation and prove that recovery selects the correct
                              committed state.

                              Current evidence

                              A historical comparison found that the Git and pull-request suites became about
                              1.6 times slower even though the individual git-* test files did not change.

                              The shared fixture did change. #532 added production authentication delegation
                              and a credential-helper launcher to the test host. Its contract says that a
                              local remote requiring no authentication should reach no authentication
                              mechanism, but the timing increase affected suites using that fixture generally.

                              This is evidence for a focused measurement, not yet proof that authentication is
                              the cause.

                              Measurement

                              Run one controlled comparison and record the complete method and results on this
                              issue.

                              Fixture acquisition

                              Measure one equivalent repository-fixture acquisition:

                              Use the same machine or runner class, runtime version, fixture shape, and sample
                              method for every comparison.

                              Report both the distribution and a representative value rather than one
                              unexplained elapsed time.

                              Process and authentication activity

                              For each acquisition, record:

                              • every Git process started;
                              • whether the local remote opens an authentication session;
                              • whether the credential-helper process starts;
                              • whether any gh process starts; and
                              • the time spent before the test-specific behavior begins.

                              Use an observable Git shim or equivalent trace rather than inferring process
                              activity from source code.

                              Hazard cost

                              For representative durability scenarios, measure separately:

                              • fixture preparation;
                              • the Git operation;
                              • forced interruption;
                              • recovery and assertion; and
                              • cleanup.

                              This shows whether the fixture or the behavior being proved dominates the test.

                              The measurement itself changes no production behavior or test contract.

                              Decision from the results

                              Use the evidence to choose one response:

                              1. Unnecessary authentication work: If a local remote opens an authentication
                                session or starts the helper, correct the host so that the command reaches no
                                mechanism.
                              2. Repeated fixture cost: If fixture preparation dominates while the hazards
                                are inexpensive, evaluate one repository per test file while keeping each
                                hazard isolated and independently reproducible.
                              3. Meaningful hazard cost: If interruption, recovery, or another required
                                behavior dominates, preserve that cost and record why it is necessary.
                              4. Another measured cause: If neither hypothesis explains the increase,
                                record the observed cause and propose a focused response.

                              Do not optimize a test by removing the failure or recovery behavior it exists to
                              prove.

                              Acceptance

                              • The issue records the revisions, runner, runtime, fixture shape, sample count,
                                and measurement method.
                              • It reports fixture-acquisition cost for the comparable revisions.
                              • It reports Git process counts and whether authentication, the credential
                                helper, or gh runs for a local remote.
                              • Representative durability scenarios separate fixture, operation,
                                interruption, recovery, assertion, and cleanup time.
                              • The evidence identifies a specific cause for the observed increase or rejects
                                the current hypotheses with enough information for another focused
                                investigation.
                              • The issue records either a focused follow-up change or why the measured cost
                                belongs to the behavior contract.

                              Out of scope

                              • Moving the scenarios into Markdown; Add <WorkflowRun> and <Execution host="workflow"> to Markdown tests #547 supplies the missing workflow testing
                                profile.
                              • Measuring CLI workflow suites whose cost comes from process startup and
                                polling rather than this repository fixture.
                              • Removing meaningful crash, recovery, authentication, or durability evidence
                                merely to improve timing.
                              • Making an implementation change before the measurement identifies its cause.

                              Activity

                              Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

                              Metadata

                              Metadata

                              Assignees

                              No one assigned

                                Labels

                                cleanupAuto-generated cleanup finding from repo analysis

                                Projects

                                No projects

                                  Milestone

                                  No milestone

                                  Relationships

                                  None yet

                                  Development

                                  No branches or pull requests

                                  Issue actions