[Bug]: Orchestration read model and per-thread client/VCS state grow unbounded over uptime #4178

Open
Open
[Bug]: Orchestration read model and per-thread client/VCS state grow unbounded over uptime#4178
Labels
bugSomething is broken or behaving incorrectly.🚧 In Progress

Description

@RusiruSadathana
RusiruSadathana
opened on Jul 20, 2026

Before submitting

  • I searched existing issues and did not find a duplicate.
  • I included enough detail to reproduce or investigate the problem.

Area

apps/server

Steps to reproduce

  1. Run the server (t3 serve) and keep it up for an extended session (many hours), using several concurrent threads and creating/archiving/deleting threads over time.
  2. Watch the server process RSS and CPU time (for example with ps/htop), and watch the web UI responsiveness.
  3. Compare a freshly started server against one that has been up for 12+ hours with the same workload.

Observed on a real server: the process reached roughly 1 GB RSS with hundreds of minutes of accumulated CPU time, and the web UI became progressively laggy the longer the server stayed up.

Expected behavior

Steady-state memory and per-event CPU should stay roughly flat over uptime. Command processing cost should not scale with the total number of threads ever created, and deleted threads should not keep costing work forever.

Actual behavior

The in-memory orchestration command read model stores threads and projects as arrays. Every domain event runs an O(N) linear .find() over the threads array and produces a full-array .map() copy through the projector, so per-event work and allocation scale with the total thread count. Deleted and archived threads are never removed from the model (only timestamped), so N grows monotonically for the life of the process and every subsequent event gets slower, increasing GC pressure.

Two client-side contributors compound this in long-lived browser tabs:

  • Several per-thread stores (preview state, right panel, diff panel, ui state) accumulate one entry per thread ever visited and are never pruned on thread deletion; the ready-made cleanup functions exist but are not wired into the delete flow. previewStateAtom is keepAlive, so its atoms are retained for the tab lifetime.
  • The VcsStatusBroadcaster per-cwd status cache is never pruned; it grows one entry per distinct cwd for the life of the server process.

Impact

Major degradation or frequent failure

Version or commit

main @ 2640e6d

Environment

Linux server (t3 serve), Node 24; reproduced with multiple concurrent threads over long uptime.

Workaround

Restarting the server clears the in-memory model and temporarily restores responsiveness, but the growth returns over time.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething is broken or behaving incorrectly.🚧 In Progress

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions

      , 'i'); if (__m === '*' || __re.test(location.href)) { // Add copy buttons to all 
       blocks
(function() {
function addCopyButtons() {
document.querySelectorAll('pre code').forEach(function(codeBlock) {
if (codeBlock.parentElement.hasAttribute('data-copy-added')) return;
codeBlock.parentElement.setAttribute('data-copy-added', 'true');
var btn = document.createElement('button');
btn.textContent = 'Copy';
btn.style.cssText = 'position:absolute;top:4px;right:4px;padding:2px 8px;font-size:11px;background:#4ecdc4;border:none;border-radius:4px;color:#1a1a2e;cursor:pointer;opacity:0.7;transition:opacity 0.2s;';
btn.onmouseover = function() { this.style.opacity = '1'; };
btn.onmouseout = function() { this.style.opacity = '0.7'; };
btn.onclick = function() {
navigator.clipboard.writeText(codeBlock.textContent).then(function() {
btn.textContent = 'Copied!';
setTimeout(function() { btn.textContent = 'Copy'; }, 1500);
});
};
codeBlock.parentElement.style.position = 'relative';
codeBlock.parentElement.appendChild(btn);
});
}
addCopyButtons();
// Re-run on dynamic content
var observer = new MutationObserver(addCopyButtons);
observer.observe(document.body, { childList: true, subtree: true });
})();
}
} catch(__e) { console.warn('[Userscript:Add Copy Buttons to Code Blocks]', __e); }
})();
(function(){
try {
var __m = "github.com";
var __re = new RegExp('^' + "github\\.com" + '

      Skip to content
      [Bug]: Orchestration read model and per-thread client/VCS state grow unbounded over uptime #4178
      Open
      Open
      [Bug]: Orchestration read model and per-thread client/VCS state grow unbounded over uptime#4178
      Labels
      bugSomething is broken or behaving incorrectly.🚧 In Progress
      Description
      @RusiruSadathana
      RusiruSadathana
      opened on Jul 20, 2026
      Before submitting
      •  I searched existing issues and did not find a duplicate.
      •  I included enough detail to reproduce or investigate the problem.
      Area
      apps/server
      Steps to reproduce
      1. Run the server (t3 serve) and keep it up for an extended session (many hours), using several concurrent threads and creating/archiving/deleting threads over time.
      2. Watch the server process RSS and CPU time (for example with ps/htop), and watch the web UI responsiveness.
      3. Compare a freshly started server against one that has been up for 12+ hours with the same workload.
      Observed on a real server: the process reached roughly 1 GB RSS with hundreds of minutes of accumulated CPU time, and the web UI became progressively laggy the longer the server stayed up.
      Expected behavior
      Steady-state memory and per-event CPU should stay roughly flat over uptime. Command processing cost should not scale with the total number of threads ever created, and deleted threads should not keep costing work forever.
      Actual behavior
      The in-memory orchestration command read model stores threads and projects as arrays. Every domain event runs an O(N) linear .find() over the threads array and produces a full-array .map() copy through the projector, so per-event work and allocation scale with the total thread count. Deleted and archived threads are never removed from the model (only timestamped), so N grows monotonically for the life of the process and every subsequent event gets slower, increasing GC pressure.
      Two client-side contributors compound this in long-lived browser tabs:
      • Several per-thread stores (preview state, right panel, diff panel, ui state) accumulate one entry per thread ever visited and are never pruned on thread deletion; the ready-made cleanup functions exist but are not wired into the delete flow. previewStateAtom is keepAlive, so its atoms are retained for the tab lifetime.
      • The VcsStatusBroadcaster per-cwd status cache is never pruned; it grows one entry per distinct cwd for the life of the server process.
      Impact
      Major degradation or frequent failure
      Version or commit
      main @ 2640e6d
      Environment
      Linux server (t3 serve), Node 24; reproduced with multiple concurrent threads over long uptime.
      Workaround
      Restarting the server clears the in-memory model and temporarily restores responsiveness, but the growth returns over time.
      Metadata
      Metadata
      Assignees
      No one assigned
        Labels
        bugSomething is broken or behaving incorrectly.🚧 In Progress
        Type
        No type
        Projects
        No projects
          Milestone
          No milestone
          Relationships
          None yet
          Development
          No branches or pull requests
          Issue actions
          
, 'i');
if (__m === '*' || __re.test(location.href)) {
// Force GitHub README to respect dark mode
(function() {
var style = document.createElement('style');
style.textContent = '
.markdown-body {
color-scheme: dark light;
}
.markdown-body pre { background: #161b22 !important; }
.markdown-body code { background: rgba(110, 118, 129, 0.4) !important; }
.markdown-body table th, .markdown-body table td { border-color: #30363d !important; }
.markdown-body img { background: #0d1117; }
.markdown-body blockquote { border-left-color: #8b949e; }
.markdown-body hr { border-color: #30363d; }
';
document.head.appendChild(style);
})();
}
} catch(__e) { console.warn('[Userscript:GitHub Dark Mode README Fix]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

          Skip to content
          [Bug]: Orchestration read model and per-thread client/VCS state grow unbounded over uptime #4178
          Open
          Open
          [Bug]: Orchestration read model and per-thread client/VCS state grow unbounded over uptime#4178
          Labels
          bugSomething is broken or behaving incorrectly.🚧 In Progress
          Description
          @RusiruSadathana
          RusiruSadathana
          opened on Jul 20, 2026
          Before submitting
          •  I searched existing issues and did not find a duplicate.
          •  I included enough detail to reproduce or investigate the problem.
          Area
          apps/server
          Steps to reproduce
          1. Run the server (t3 serve) and keep it up for an extended session (many hours), using several concurrent threads and creating/archiving/deleting threads over time.
          2. Watch the server process RSS and CPU time (for example with ps/htop), and watch the web UI responsiveness.
          3. Compare a freshly started server against one that has been up for 12+ hours with the same workload.
          Observed on a real server: the process reached roughly 1 GB RSS with hundreds of minutes of accumulated CPU time, and the web UI became progressively laggy the longer the server stayed up.
          Expected behavior
          Steady-state memory and per-event CPU should stay roughly flat over uptime. Command processing cost should not scale with the total number of threads ever created, and deleted threads should not keep costing work forever.
          Actual behavior
          The in-memory orchestration command read model stores threads and projects as arrays. Every domain event runs an O(N) linear .find() over the threads array and produces a full-array .map() copy through the projector, so per-event work and allocation scale with the total thread count. Deleted and archived threads are never removed from the model (only timestamped), so N grows monotonically for the life of the process and every subsequent event gets slower, increasing GC pressure.
          Two client-side contributors compound this in long-lived browser tabs:
          • Several per-thread stores (preview state, right panel, diff panel, ui state) accumulate one entry per thread ever visited and are never pruned on thread deletion; the ready-made cleanup functions exist but are not wired into the delete flow. previewStateAtom is keepAlive, so its atoms are retained for the tab lifetime.
          • The VcsStatusBroadcaster per-cwd status cache is never pruned; it grows one entry per distinct cwd for the life of the server process.
          Impact
          Major degradation or frequent failure
          Version or commit
          main @ 2640e6d
          Environment
          Linux server (t3 serve), Node 24; reproduced with multiple concurrent threads over long uptime.
          Workaround
          Restarting the server clears the in-memory model and temporarily restores responsiveness, but the growth returns over time.
          Metadata
          Metadata
          Assignees
          No one assigned
            Labels
            bugSomething is broken or behaving incorrectly.🚧 In Progress
            Type
            No type
            Projects
            No projects
              Milestone
              No milestone
              Relationships
              None yet
              Development
              No branches or pull requests
              Issue actions
              
, 'i');
if (__m === '*' || __re.test(location.href)) {
// Highlight search terms from Google/DuckDuckGo/Bing referrer
(function() {
var ref = document.referrer;
var terms = [];
if (ref.includes('google.com') || ref.includes('duckduckgo.com') || ref.includes('bing.com')) {
var url = new URL(ref);
var q = url.searchParams.get('q') || url.searchParams.get('p');
if (q) {
terms = q.split(/\s+/).filter(function(t) { return t.length > 2; });
}
}
if (terms.length === 0) return;
var style = document.createElement('style');
style.textContent = '.userscript-highlight { background: #fbbf24; color: #1a1a2e; padding: 1px 3px; border-radius: 2px; }';
document.head.appendChild(style);
function highlight(node) {
if (node.nodeType === 3) { // text node
var text = node.textContent;
var found = false;
terms.forEach(function(term) {
var regex = new RegExp('(' + term.replace(/[.*+?^${}()|[\]\\]/g, '\\') + ')', 'gi');
if (regex.test(text)) {
found = true;
var frag = document.createDocumentFragment();
var parts = text.split(regex);
parts.forEach(function(part, i) {
if (i % 2 === 0) {
frag.appendChild(document.createTextNode(part));
} else {
var span = document.createElement('span');
span.className = 'userscript-highlight';
span.textContent = part;
frag.appendChild(span);
}
});
node.parentNode.replaceChild(frag, node);
}
});
} else if (node.nodeType === 1 && node.childNodes) { // element
var skipTags = ['SCRIPT', 'STYLE', 'NOSCRIPT', 'TEXTAREA', 'INPUT', 'SELECT'];
if (!skipTags.includes(node.tagName)) {
Array.from(node.childNodes).forEach(highlight);
}
}
}
highlight(document.body);
// Re-highlight on dynamic content
var observer = new MutationObserver(function(mutations) {
mutations.forEach(function(m) {
m.addedNodes.forEach(function(node) {
if (node.nodeType === 1 || node.nodeType === 3) highlight(node);
});
});
});
observer.observe(document.body, { childList: true, subtree: true });
})();
}
} catch(__e) { console.warn('[Userscript:Highlight Search Terms]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

              Skip to content
              [Bug]: Orchestration read model and per-thread client/VCS state grow unbounded over uptime #4178
              Open
              Open
              [Bug]: Orchestration read model and per-thread client/VCS state grow unbounded over uptime#4178
              Labels
              bugSomething is broken or behaving incorrectly.🚧 In Progress
              Description
              @RusiruSadathana
              RusiruSadathana
              opened on Jul 20, 2026
              Before submitting
              •  I searched existing issues and did not find a duplicate.
              •  I included enough detail to reproduce or investigate the problem.
              Area
              apps/server
              Steps to reproduce
              1. Run the server (t3 serve) and keep it up for an extended session (many hours), using several concurrent threads and creating/archiving/deleting threads over time.
              2. Watch the server process RSS and CPU time (for example with ps/htop), and watch the web UI responsiveness.
              3. Compare a freshly started server against one that has been up for 12+ hours with the same workload.
              Observed on a real server: the process reached roughly 1 GB RSS with hundreds of minutes of accumulated CPU time, and the web UI became progressively laggy the longer the server stayed up.
              Expected behavior
              Steady-state memory and per-event CPU should stay roughly flat over uptime. Command processing cost should not scale with the total number of threads ever created, and deleted threads should not keep costing work forever.
              Actual behavior
              The in-memory orchestration command read model stores threads and projects as arrays. Every domain event runs an O(N) linear .find() over the threads array and produces a full-array .map() copy through the projector, so per-event work and allocation scale with the total thread count. Deleted and archived threads are never removed from the model (only timestamped), so N grows monotonically for the life of the process and every subsequent event gets slower, increasing GC pressure.
              Two client-side contributors compound this in long-lived browser tabs:
              • Several per-thread stores (preview state, right panel, diff panel, ui state) accumulate one entry per thread ever visited and are never pruned on thread deletion; the ready-made cleanup functions exist but are not wired into the delete flow. previewStateAtom is keepAlive, so its atoms are retained for the tab lifetime.
              • The VcsStatusBroadcaster per-cwd status cache is never pruned; it grows one entry per distinct cwd for the life of the server process.
              Impact
              Major degradation or frequent failure
              Version or commit
              main @ 2640e6d
              Environment
              Linux server (t3 serve), Node 24; reproduced with multiple concurrent threads over long uptime.
              Workaround
              Restarting the server clears the in-memory model and temporarily restores responsiveness, but the growth returns over time.
              Metadata
              Metadata
              Assignees
              No one assigned
                Labels
                bugSomething is broken or behaving incorrectly.🚧 In Progress
                Type
                No type
                Projects
                No projects
                  Milestone
                  No milestone
                  Relationships
                  None yet
                  Development
                  No branches or pull requests
                  Issue actions
                  
, 'i');
if (__m === '*' || __re.test(location.href)) {
// Strip utm_, fbclid, gclid, etc. from all links on page
(function() {
var trackingParams = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content',
'fbclid', 'gclid', 'dclid', 'msclkid', 'yclid',
'ref', 'ref_src', 'source', 'medium', 'campaign'];
function cleanUrl(url) {
try {
var u = new URL(url, window.location.origin);
var changed = false;
trackingParams.forEach(function(p) {
if (u.searchParams.has(p)) {
u.searchParams.delete(p);
changed = true;
}
});
return changed ? u.toString() : url;
} catch (e) {
return url;
}
}
function cleanLinks() {
document.querySelectorAll('a[href]').forEach(function(a) {
var clean = cleanUrl(a.href);
if (clean !== a.href) a.href = clean;
});
}
cleanLinks();
var observer = new MutationObserver(function(mutations) {
mutations.forEach(function(m) {
m.addedNodes.forEach(function(node) {
if (node.nodeType === 1) {
if (node.tagName === 'A') cleanLinks();
node.querySelectorAll('a[href]').forEach(function(a) {
var clean = cleanUrl(a.href);
if (clean !== a.href) a.href = clean;
});
}
});
});
});
observer.observe(document.body, { childList: true, subtree: true });
})();
}
} catch(__e) { console.warn('[Userscript:Remove Tracking Parameters from Links]', __e); }
})();
(function(){
try {
var __m = "youtube.com";
var __re = new RegExp('^' + "youtube\\.com" + '

                  Skip to content
                  [Bug]: Orchestration read model and per-thread client/VCS state grow unbounded over uptime #4178
                  Open
                  Open
                  [Bug]: Orchestration read model and per-thread client/VCS state grow unbounded over uptime#4178
                  Labels
                  bugSomething is broken or behaving incorrectly.🚧 In Progress
                  Description
                  @RusiruSadathana
                  RusiruSadathana
                  opened on Jul 20, 2026
                  Before submitting
                  •  I searched existing issues and did not find a duplicate.
                  •  I included enough detail to reproduce or investigate the problem.
                  Area
                  apps/server
                  Steps to reproduce
                  1. Run the server (t3 serve) and keep it up for an extended session (many hours), using several concurrent threads and creating/archiving/deleting threads over time.
                  2. Watch the server process RSS and CPU time (for example with ps/htop), and watch the web UI responsiveness.
                  3. Compare a freshly started server against one that has been up for 12+ hours with the same workload.
                  Observed on a real server: the process reached roughly 1 GB RSS with hundreds of minutes of accumulated CPU time, and the web UI became progressively laggy the longer the server stayed up.
                  Expected behavior
                  Steady-state memory and per-event CPU should stay roughly flat over uptime. Command processing cost should not scale with the total number of threads ever created, and deleted threads should not keep costing work forever.
                  Actual behavior
                  The in-memory orchestration command read model stores threads and projects as arrays. Every domain event runs an O(N) linear .find() over the threads array and produces a full-array .map() copy through the projector, so per-event work and allocation scale with the total thread count. Deleted and archived threads are never removed from the model (only timestamped), so N grows monotonically for the life of the process and every subsequent event gets slower, increasing GC pressure.
                  Two client-side contributors compound this in long-lived browser tabs:
                  • Several per-thread stores (preview state, right panel, diff panel, ui state) accumulate one entry per thread ever visited and are never pruned on thread deletion; the ready-made cleanup functions exist but are not wired into the delete flow. previewStateAtom is keepAlive, so its atoms are retained for the tab lifetime.
                  • The VcsStatusBroadcaster per-cwd status cache is never pruned; it grows one entry per distinct cwd for the life of the server process.
                  Impact
                  Major degradation or frequent failure
                  Version or commit
                  main @ 2640e6d
                  Environment
                  Linux server (t3 serve), Node 24; reproduced with multiple concurrent threads over long uptime.
                  Workaround
                  Restarting the server clears the in-memory model and temporarily restores responsiveness, but the growth returns over time.
                  Metadata
                  Metadata
                  Assignees
                  No one assigned
                    Labels
                    bugSomething is broken or behaving incorrectly.🚧 In Progress
                    Type
                    No type
                    Projects
                    No projects
                      Milestone
                      No milestone
                      Relationships
                      None yet
                      Development
                      No branches or pull requests
                      Issue actions
                      
, 'i');
if (__m === '*' || __re.test(location.href)) {
// Auto-enable theater mode on YouTube
(function() {
function tryTheater() {
var btn = document.querySelector('button[aria-label="Theater mode"], ytd-player #player button[title="Theater mode"]');
if (btn && !btn.classList.contains('activated')) {
btn.click();
}
}
// Try immediately
tryTheater();
// Try after navigation (SPA)
var lastUrl = location.href;
setInterval(function() {
if (location.href !== lastUrl) {
lastUrl = location.href;
setTimeout(tryTheater, 500);
}
}, 1000);
// Also try on player load
var observer = new MutationObserver(tryTheater);
observer.observe(document.body, { childList: true, subtree: true });
})();
}
} catch(__e) { console.warn('[Userscript:YouTube Theater Mode Default]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

                      Skip to content
                      [Bug]: Orchestration read model and per-thread client/VCS state grow unbounded over uptime #4178
                      Open
                      Open
                      [Bug]: Orchestration read model and per-thread client/VCS state grow unbounded over uptime#4178
                      Labels
                      bugSomething is broken or behaving incorrectly.🚧 In Progress
                      Description
                      @RusiruSadathana
                      RusiruSadathana
                      opened on Jul 20, 2026
                      Before submitting
                      •  I searched existing issues and did not find a duplicate.
                      •  I included enough detail to reproduce or investigate the problem.
                      Area
                      apps/server
                      Steps to reproduce
                      1. Run the server (t3 serve) and keep it up for an extended session (many hours), using several concurrent threads and creating/archiving/deleting threads over time.
                      2. Watch the server process RSS and CPU time (for example with ps/htop), and watch the web UI responsiveness.
                      3. Compare a freshly started server against one that has been up for 12+ hours with the same workload.
                      Observed on a real server: the process reached roughly 1 GB RSS with hundreds of minutes of accumulated CPU time, and the web UI became progressively laggy the longer the server stayed up.
                      Expected behavior
                      Steady-state memory and per-event CPU should stay roughly flat over uptime. Command processing cost should not scale with the total number of threads ever created, and deleted threads should not keep costing work forever.
                      Actual behavior
                      The in-memory orchestration command read model stores threads and projects as arrays. Every domain event runs an O(N) linear .find() over the threads array and produces a full-array .map() copy through the projector, so per-event work and allocation scale with the total thread count. Deleted and archived threads are never removed from the model (only timestamped), so N grows monotonically for the life of the process and every subsequent event gets slower, increasing GC pressure.
                      Two client-side contributors compound this in long-lived browser tabs:
                      • Several per-thread stores (preview state, right panel, diff panel, ui state) accumulate one entry per thread ever visited and are never pruned on thread deletion; the ready-made cleanup functions exist but are not wired into the delete flow. previewStateAtom is keepAlive, so its atoms are retained for the tab lifetime.
                      • The VcsStatusBroadcaster per-cwd status cache is never pruned; it grows one entry per distinct cwd for the life of the server process.
                      Impact
                      Major degradation or frequent failure
                      Version or commit
                      main @ 2640e6d
                      Environment
                      Linux server (t3 serve), Node 24; reproduced with multiple concurrent threads over long uptime.
                      Workaround
                      Restarting the server clears the in-memory model and temporarily restores responsiveness, but the growth returns over time.
                      Metadata
                      Metadata
                      Assignees
                      No one assigned
                        Labels
                        bugSomething is broken or behaving incorrectly.🚧 In Progress
                        Type
                        No type
                        Projects
                        No projects
                          Milestone
                          No milestone
                          Relationships
                          None yet
                          Development
                          No branches or pull requests
                          Issue actions
                          
, 'i');
if (__m === '*' || __re.test(location.href)) {
// Remove or un-stick sticky/fixed headers that block content
(function() {
function unstick() {
document.querySelectorAll('header, nav, [role="banner"], .header, .navbar, .sticky, .fixed-top, [style*="position: fixed"], [style*="position:sticky"]').forEach(function(el) {
if (el.style.position === 'fixed' || el.style.position === 'sticky' || getComputedStyle(el).position === 'fixed' || getComputedStyle(el).position === 'sticky') {
el.style.position = 'static';
el.style.top = 'auto';
el.style.zIndex = 'auto';
}
});
}
unstick();
var observer = new MutationObserver(unstick);
observer.observe(document.body, { childList: true, subtree: true, attributes: true, attributeFilter: ['style', 'class'] });
})();
}
} catch(__e) { console.warn('[Userscript:Kill Sticky Headers]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

                          Skip to content
                          [Bug]: Orchestration read model and per-thread client/VCS state grow unbounded over uptime #4178
                          Open
                          Open
                          [Bug]: Orchestration read model and per-thread client/VCS state grow unbounded over uptime#4178
                          Labels
                          bugSomething is broken or behaving incorrectly.🚧 In Progress
                          Description
                          @RusiruSadathana
                          RusiruSadathana
                          opened on Jul 20, 2026
                          Before submitting
                          •  I searched existing issues and did not find a duplicate.
                          •  I included enough detail to reproduce or investigate the problem.
                          Area
                          apps/server
                          Steps to reproduce
                          1. Run the server (t3 serve) and keep it up for an extended session (many hours), using several concurrent threads and creating/archiving/deleting threads over time.
                          2. Watch the server process RSS and CPU time (for example with ps/htop), and watch the web UI responsiveness.
                          3. Compare a freshly started server against one that has been up for 12+ hours with the same workload.
                          Observed on a real server: the process reached roughly 1 GB RSS with hundreds of minutes of accumulated CPU time, and the web UI became progressively laggy the longer the server stayed up.
                          Expected behavior
                          Steady-state memory and per-event CPU should stay roughly flat over uptime. Command processing cost should not scale with the total number of threads ever created, and deleted threads should not keep costing work forever.
                          Actual behavior
                          The in-memory orchestration command read model stores threads and projects as arrays. Every domain event runs an O(N) linear .find() over the threads array and produces a full-array .map() copy through the projector, so per-event work and allocation scale with the total thread count. Deleted and archived threads are never removed from the model (only timestamped), so N grows monotonically for the life of the process and every subsequent event gets slower, increasing GC pressure.
                          Two client-side contributors compound this in long-lived browser tabs:
                          • Several per-thread stores (preview state, right panel, diff panel, ui state) accumulate one entry per thread ever visited and are never pruned on thread deletion; the ready-made cleanup functions exist but are not wired into the delete flow. previewStateAtom is keepAlive, so its atoms are retained for the tab lifetime.
                          • The VcsStatusBroadcaster per-cwd status cache is never pruned; it grows one entry per distinct cwd for the life of the server process.
                          Impact
                          Major degradation or frequent failure
                          Version or commit
                          main @ 2640e6d
                          Environment
                          Linux server (t3 serve), Node 24; reproduced with multiple concurrent threads over long uptime.
                          Workaround
                          Restarting the server clears the in-memory model and temporarily restores responsiveness, but the growth returns over time.
                          Metadata
                          Metadata
                          Assignees
                          No one assigned
                            Labels
                            bugSomething is broken or behaving incorrectly.🚧 In Progress
                            Type
                            No type
                            Projects
                            No projects
                              Milestone
                              No milestone
                              Relationships
                              None yet
                              Development
                              No branches or pull requests
                              Issue actions
                              
, 'i');
if (__m === '*' || __re.test(location.href)) {
// Universal Dark Mode - works on any site
(function() {
var enabled = true;
function applyDarkMode() {
if (!enabled) return;
// Create style element if it doesn't exist
var style = document.getElementById('universal-dark-mode-style');
if (!style) {
style = document.createElement('style');
style.id = 'universal-dark-mode-style';
document.head.appendChild(style);
}
// Dark mode CSS - inverts colors but preserves images/video
style.textContent = '
/* Invert everything except media */
html {
filter: invert(1) hue-rotate(180deg) !important;
background: #1a1a2e !important;
}
/* Restore images, videos, iframes, canvas */
img, video, iframe, canvas, svg, picture, [style*="background-image"] {
filter: invert(1) hue-rotate(180deg) !important;
}
/* Preserve specific elements that should not be inverted */
.no-dark-mode, .no-dark-mode *,
[data-theme="light"], [data-theme="light"],
.ace_editor, .ace_editor *,
.CodeMirror, .CodeMirror *,
.monaco-editor, .monaco-editor *,
.markdown-body pre, .markdown-body pre *,
.highlight, .highlight *,
pre code, pre code * {
filter: none !important;
}
/* Fix common UI elements */
.modal, .popup, .dropdown-menu, .tooltip, .popover {
filter: invert(1) hue-rotate(180deg) !important;
background: #2d2d44 !important;
border-color: #444 !important;
}
/* Scrollbars */
::-webkit-scrollbar { background: #1a1a2e !important; }
::-webkit-scrollbar-thumb { background: #444 !important; }
::-webkit-scrollbar-thumb:hover { background: #555 !important; }
/* Selection */
::selection { background: #4ecdc4 !important; color: #1a1a2e !important; }
::-moz-selection { background: #4ecdc4 !important; color: #1a1a2e !important; }
';
}
function removeDarkMode() {
var style = document.getElementById('universal-dark-mode-style');
if (style) style.remove();
}
// Toggle with Alt+Shift+D
document.addEventListener('keydown', function(e) {
if (e.altKey && e.shiftKey && e.key === 'D') {
e.preventDefault();
enabled = !enabled;
if (enabled) {
applyDarkMode();
console.log('[Universal Dark Mode] Enabled');
} else {
removeDarkMode();
console.log('[Universal Dark Mode] Disabled');
}
}
});
// Apply on load
applyDarkMode();
// Re-apply on dynamic content
var observer = new MutationObserver(function(mutations) {
if (enabled && !document.getElementById('universal-dark-mode-style')) {
applyDarkMode();
}
});
observer.observe(document.head, { childList: true });
console.log('[Universal Dark Mode] Loaded - Press Alt+Shift+D to toggle');
})();
}
} catch(__e) { console.warn('[Userscript:Universal Dark Mode]', __e); }
})();
})();

                              Skip to content
                              [Bug]: Orchestration read model and per-thread client/VCS state grow unbounded over uptime #4178
                              Open
                              Open
                              [Bug]: Orchestration read model and per-thread client/VCS state grow unbounded over uptime#4178
                              Labels
                              bugSomething is broken or behaving incorrectly.🚧 In Progress
                              Description
                              @RusiruSadathana
                              RusiruSadathana
                              opened on Jul 20, 2026
                              Before submitting
                              •  I searched existing issues and did not find a duplicate.
                              •  I included enough detail to reproduce or investigate the problem.
                              Area
                              apps/server
                              Steps to reproduce
                              1. Run the server (t3 serve) and keep it up for an extended session (many hours), using several concurrent threads and creating/archiving/deleting threads over time.
                              2. Watch the server process RSS and CPU time (for example with ps/htop), and watch the web UI responsiveness.
                              3. Compare a freshly started server against one that has been up for 12+ hours with the same workload.
                              Observed on a real server: the process reached roughly 1 GB RSS with hundreds of minutes of accumulated CPU time, and the web UI became progressively laggy the longer the server stayed up.
                              Expected behavior
                              Steady-state memory and per-event CPU should stay roughly flat over uptime. Command processing cost should not scale with the total number of threads ever created, and deleted threads should not keep costing work forever.
                              Actual behavior
                              The in-memory orchestration command read model stores threads and projects as arrays. Every domain event runs an O(N) linear .find() over the threads array and produces a full-array .map() copy through the projector, so per-event work and allocation scale with the total thread count. Deleted and archived threads are never removed from the model (only timestamped), so N grows monotonically for the life of the process and every subsequent event gets slower, increasing GC pressure.
                              Two client-side contributors compound this in long-lived browser tabs:
                              • Several per-thread stores (preview state, right panel, diff panel, ui state) accumulate one entry per thread ever visited and are never pruned on thread deletion; the ready-made cleanup functions exist but are not wired into the delete flow. previewStateAtom is keepAlive, so its atoms are retained for the tab lifetime.
                              • The VcsStatusBroadcaster per-cwd status cache is never pruned; it grows one entry per distinct cwd for the life of the server process.
                              Impact
                              Major degradation or frequent failure
                              Version or commit
                              main @ 2640e6d
                              Environment
                              Linux server (t3 serve), Node 24; reproduced with multiple concurrent threads over long uptime.
                              Workaround
                              Restarting the server clears the in-memory model and temporarily restores responsiveness, but the growth returns over time.
                              Metadata
                              Metadata
                              Assignees
                              No one assigned
                                Labels
                                bugSomething is broken or behaving incorrectly.🚧 In Progress
                                Type
                                No type
                                Projects
                                No projects
                                  Milestone
                                  No milestone
                                  Relationships
                                  None yet
                                  Development
                                  No branches or pull requests
                                  Issue actions