Observability - #10

Merged
gotha merged 5 commits into
mainfrom
observability
Aug 31, 2026
Merged

Observability#10
gotha merged 5 commits into
mainfrom
observability

Conversation

@gotha

@gothagotha commented Aug 28, 2026

Copy link
Copy Markdown
Owner

No description provided.

@gotha
lucie: add grafana, prometheus, loki and tempo
a0a3903
Metrics, logs and traces, as native NixOS services rather than containers -
the modules exist and handle users, state directories and unit hardening, so a
compose file would only add moving parts.
Ports: grafana 3000, prometheus 9090, loki 3100, tempo 3200, and tempo's OTLP
receivers on 4317 (gRPC) and 4318 (HTTP). Loki's push API and the OTLP
receivers bind 0.0.0.0 and the firewall admits 172.16.0.0/12, because the point
is for Docker services on this host to reach them: a container talking to
host.docker.internal - which litellm.nix already maps to host-gateway - arrives
from the Docker bridge, and a 127.0.0.1 bind would be unreachable. Everything
else follows litellm.nix's policy of localhost, LAN and WireGuard only. Loki's
gRPC port stays closed; nothing off-host speaks it.
Grafana's datasources are provisioned rather than clicked in, so a rebuild is
enough to get a working instance. Its secret_key lost its default in NixOS
26.05 and is now required, so it is generated and encrypted in
secrets/grafana.enc.json and read through $__file{} to keep it out of the
world-readable store copy of grafana.ini. That file needs lucie's host age key
as well as the PGP key for sops-nix to decrypt at boot, so it joins litellm
under the .sops.yaml rule that carries both.
Prometheus only scrapes itself so far - the Docker services that will expose
/metrics get added as further jobs later.
Entire-Checkpoint: e7f8764a74b0
@gotha
lucie: fix tempo crash loop and move observability off default ports
70e63f9
Tempo restarted every 30s (848 times in one boot) with:
module=live-store err="failed to create shutdown marker directory:
mkdir /var/tempo: read-only file system"
DynamicUser=true implies ProtectSystem=strict, so StateDirectory=tempo
is the only writable path. storage.trace already pointed under
/var/lib/tempo, but Tempo 3.0 added modules whose paths still default
under /var/tempo:
live-store.shutdown_marker_dir /var/tempo/live-store/shutdown-marker
live-store.wal.path /var/tempo/live-store/traces
block-builder.wal.path /var/tempo/block-builder/traces
backend-scheduler.local-work-path /var/tempo
live-store failed on the first, distributor depends on live-store, so
trace ingestion never came up. block_builder is inactive in
single-binary mode but is set too - it fails identically if it ever
activates.
Also moves every port into a 320xx range. Grafana on 3000 and
Prometheus on 9090 collide with anything else that wants the usual
defaults, and lokiPort and tempoPort were briefly both 32030.
Verified: tempo NRestarts=0, /ready returns 200, /var/tempo is never
created, and /var/lib/tempo holds live-store/ traces/ wal/.
Entire-Checkpoint: baef4153a49b
@gotha
lucie: enable aarch64-linux emulation
32368aa
Registers the binfmt handlers and adds aarch64-linux to nix.conf's
extra-platforms so this x86_64 box can build the devbox-arm image for
the mac.
Entire-Checkpoint: 197d69023cbc
@gotha
lucie: expire loki logs after 30 days
ccd51c4
Loki keeps everything forever by default: retention_period was 0s and
compactor.retention_enabled false, so nothing ever deleted a chunk.
Since /var/lib/loki sits on the root filesystem - already at 96% - an
unbounded log store is a slow-motion outage.
Deletion needs both halves. retention_period on its own is inert
without retention_enabled, and the compactor refuses to start unless
delete_request_store is set.
Verified against loki 3.7.6 with this config: compactor reaches ACTIVE
in the ring and logs "this instance has been chosen to run the
compactor", with no config errors.
Entire-Checkpoint: fd6175e529d5
@gotha
lucie: provision the Dissona logs dashboard
3b97e0a
Adds a file-backed dashboard provider pointing at ./dashboards, plus a
first dashboard: error/warning/info/unclassified counts, log volume by
level, warnings and errors by container, nginx edge HTTP status, and
the matching log lines.
allowUiUpdates is false, so the JSON in ./dashboards is the only source
of truth and the UI serves it read-only. Editing in Grafana means
exporting the JSON model back into the repo and rebuilding.
Panels bind to the "loki" datasource uid already provisioned above.
Entire-Checkpoint: a61237526802
@gotha
gotha merged commit 0bf1cc8 into mainAug 31, 2026
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@gotha
, 'i'); if (__m === '*' || __re.test(location.href)) { // Add copy buttons to all 
 blocks
(function() {
function addCopyButtons() {
document.querySelectorAll('pre code').forEach(function(codeBlock) {
if (codeBlock.parentElement.hasAttribute('data-copy-added')) return;
codeBlock.parentElement.setAttribute('data-copy-added', 'true');
var btn = document.createElement('button');
btn.textContent = 'Copy';
btn.style.cssText = 'position:absolute;top:4px;right:4px;padding:2px 8px;font-size:11px;background:#4ecdc4;border:none;border-radius:4px;color:#1a1a2e;cursor:pointer;opacity:0.7;transition:opacity 0.2s;';
btn.onmouseover = function() { this.style.opacity = '1'; };
btn.onmouseout = function() { this.style.opacity = '0.7'; };
btn.onclick = function() {
navigator.clipboard.writeText(codeBlock.textContent).then(function() {
btn.textContent = 'Copied!';
setTimeout(function() { btn.textContent = 'Copy'; }, 1500);
});
};
codeBlock.parentElement.style.position = 'relative';
codeBlock.parentElement.appendChild(btn);
});
}
addCopyButtons();
// Re-run on dynamic content
var observer = new MutationObserver(addCopyButtons);
observer.observe(document.body, { childList: true, subtree: true });
})();
}
} catch(__e) { console.warn('[Userscript:Add Copy Buttons to Code Blocks]', __e); }
})();
(function(){
try {
var __m = "github.com";
var __re = new RegExp('^' + "github\\.com" + '

Skip to content
Observability - #10
Merged
gotha merged 5 commits into
mainfrom 
observability
Aug 31, 2026
Merged
Observability#10
gotha merged 5 commits into
mainfrom 
observability
Conversation
@gotha
@gothagotha
commented
Aug 28, 2026

Copy link


Copy Markdown

Owner
 No description provided. 
@gotha
lucie: add grafana, prometheus, loki and tempo
a0a3903
Metrics, logs and traces, as native NixOS services rather than containers -
the modules exist and handle users, state directories and unit hardening, so a
compose file would only add moving parts.
Ports: grafana 3000, prometheus 9090, loki 3100, tempo 3200, and tempo's OTLP
receivers on 4317 (gRPC) and 4318 (HTTP). Loki's push API and the OTLP
receivers bind 0.0.0.0 and the firewall admits 172.16.0.0/12, because the point
is for Docker services on this host to reach them: a container talking to
host.docker.internal - which litellm.nix already maps to host-gateway - arrives
from the Docker bridge, and a 127.0.0.1 bind would be unreachable. Everything
else follows litellm.nix's policy of localhost, LAN and WireGuard only. Loki's
gRPC port stays closed; nothing off-host speaks it.
Grafana's datasources are provisioned rather than clicked in, so a rebuild is
enough to get a working instance. Its secret_key lost its default in NixOS
26.05 and is now required, so it is generated and encrypted in
secrets/grafana.enc.json and read through $__file{} to keep it out of the
world-readable store copy of grafana.ini. That file needs lucie's host age key
as well as the PGP key for sops-nix to decrypt at boot, so it joins litellm
under the .sops.yaml rule that carries both.
Prometheus only scrapes itself so far - the Docker services that will expose
/metrics get added as further jobs later.
Entire-Checkpoint: e7f8764a74b0
@gotha
lucie: fix tempo crash loop and move observability off default ports
70e63f9
Tempo restarted every 30s (848 times in one boot) with:
module=live-store err="failed to create shutdown marker directory:
mkdir /var/tempo: read-only file system"
DynamicUser=true implies ProtectSystem=strict, so StateDirectory=tempo
is the only writable path. storage.trace already pointed under
/var/lib/tempo, but Tempo 3.0 added modules whose paths still default
under /var/tempo:
live-store.shutdown_marker_dir /var/tempo/live-store/shutdown-marker
live-store.wal.path /var/tempo/live-store/traces
block-builder.wal.path /var/tempo/block-builder/traces
backend-scheduler.local-work-path /var/tempo
live-store failed on the first, distributor depends on live-store, so
trace ingestion never came up. block_builder is inactive in
single-binary mode but is set too - it fails identically if it ever
activates.
Also moves every port into a 320xx range. Grafana on 3000 and
Prometheus on 9090 collide with anything else that wants the usual
defaults, and lokiPort and tempoPort were briefly both 32030.
Verified: tempo NRestarts=0, /ready returns 200, /var/tempo is never
created, and /var/lib/tempo holds live-store/ traces/ wal/.
Entire-Checkpoint: baef4153a49b
@gotha
lucie: enable aarch64-linux emulation
32368aa
Registers the binfmt handlers and adds aarch64-linux to nix.conf's
extra-platforms so this x86_64 box can build the devbox-arm image for
the mac.
Entire-Checkpoint: 197d69023cbc
@gotha
lucie: expire loki logs after 30 days
ccd51c4
Loki keeps everything forever by default: retention_period was 0s and
compactor.retention_enabled false, so nothing ever deleted a chunk.
Since /var/lib/loki sits on the root filesystem - already at 96% - an
unbounded log store is a slow-motion outage.
Deletion needs both halves. retention_period on its own is inert
without retention_enabled, and the compactor refuses to start unless
delete_request_store is set.
Verified against loki 3.7.6 with this config: compactor reaches ACTIVE
in the ring and logs "this instance has been chosen to run the
compactor", with no config errors.
Entire-Checkpoint: fd6175e529d5
@gotha
lucie: provision the Dissona logs dashboard
3b97e0a
Adds a file-backed dashboard provider pointing at ./dashboards, plus a
first dashboard: error/warning/info/unclassified counts, log volume by
level, warnings and errors by container, nginx edge HTTP status, and
the matching log lines.
allowUiUpdates is false, so the JSON in ./dashboards is the only source
of truth and the UI serves it read-only. Editing in Grafana means
exporting the JSON model back into the repo and rebuilding.
Panels bind to the "loki" datasource uid already provisioned above.
Entire-Checkpoint: a61237526802
@gotha
gotha
merged commit 0bf1cc8
into
mainAug 31, 2026
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers


No reviews


Assignees


No one assigned


Labels


None yet


Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

1 participant

@gotha

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Force GitHub README to respect dark mode
(function() {
var style = document.createElement('style');
style.textContent = '
.markdown-body {
color-scheme: dark light;
}
.markdown-body pre { background: #161b22 !important; }
.markdown-body code { background: rgba(110, 118, 129, 0.4) !important; }
.markdown-body table th, .markdown-body table td { border-color: #30363d !important; }
.markdown-body img { background: #0d1117; }
.markdown-body blockquote { border-left-color: #8b949e; }
.markdown-body hr { border-color: #30363d; }
';
document.head.appendChild(style);
})();
}
} catch(__e) { console.warn('[Userscript:GitHub Dark Mode README Fix]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
Observability - #10
Merged
gotha merged 5 commits into
mainfrom 
observability
Aug 31, 2026
Merged
Observability#10
gotha merged 5 commits into
mainfrom 
observability
Conversation
@gotha
@gothagotha
commented
Aug 28, 2026

Copy link


Copy Markdown

Owner
 No description provided. 
@gotha
lucie: add grafana, prometheus, loki and tempo
a0a3903
Metrics, logs and traces, as native NixOS services rather than containers -
the modules exist and handle users, state directories and unit hardening, so a
compose file would only add moving parts.
Ports: grafana 3000, prometheus 9090, loki 3100, tempo 3200, and tempo's OTLP
receivers on 4317 (gRPC) and 4318 (HTTP). Loki's push API and the OTLP
receivers bind 0.0.0.0 and the firewall admits 172.16.0.0/12, because the point
is for Docker services on this host to reach them: a container talking to
host.docker.internal - which litellm.nix already maps to host-gateway - arrives
from the Docker bridge, and a 127.0.0.1 bind would be unreachable. Everything
else follows litellm.nix's policy of localhost, LAN and WireGuard only. Loki's
gRPC port stays closed; nothing off-host speaks it.
Grafana's datasources are provisioned rather than clicked in, so a rebuild is
enough to get a working instance. Its secret_key lost its default in NixOS
26.05 and is now required, so it is generated and encrypted in
secrets/grafana.enc.json and read through $__file{} to keep it out of the
world-readable store copy of grafana.ini. That file needs lucie's host age key
as well as the PGP key for sops-nix to decrypt at boot, so it joins litellm
under the .sops.yaml rule that carries both.
Prometheus only scrapes itself so far - the Docker services that will expose
/metrics get added as further jobs later.
Entire-Checkpoint: e7f8764a74b0
@gotha
lucie: fix tempo crash loop and move observability off default ports
70e63f9
Tempo restarted every 30s (848 times in one boot) with:
module=live-store err="failed to create shutdown marker directory:
mkdir /var/tempo: read-only file system"
DynamicUser=true implies ProtectSystem=strict, so StateDirectory=tempo
is the only writable path. storage.trace already pointed under
/var/lib/tempo, but Tempo 3.0 added modules whose paths still default
under /var/tempo:
live-store.shutdown_marker_dir /var/tempo/live-store/shutdown-marker
live-store.wal.path /var/tempo/live-store/traces
block-builder.wal.path /var/tempo/block-builder/traces
backend-scheduler.local-work-path /var/tempo
live-store failed on the first, distributor depends on live-store, so
trace ingestion never came up. block_builder is inactive in
single-binary mode but is set too - it fails identically if it ever
activates.
Also moves every port into a 320xx range. Grafana on 3000 and
Prometheus on 9090 collide with anything else that wants the usual
defaults, and lokiPort and tempoPort were briefly both 32030.
Verified: tempo NRestarts=0, /ready returns 200, /var/tempo is never
created, and /var/lib/tempo holds live-store/ traces/ wal/.
Entire-Checkpoint: baef4153a49b
@gotha
lucie: enable aarch64-linux emulation
32368aa
Registers the binfmt handlers and adds aarch64-linux to nix.conf's
extra-platforms so this x86_64 box can build the devbox-arm image for
the mac.
Entire-Checkpoint: 197d69023cbc
@gotha
lucie: expire loki logs after 30 days
ccd51c4
Loki keeps everything forever by default: retention_period was 0s and
compactor.retention_enabled false, so nothing ever deleted a chunk.
Since /var/lib/loki sits on the root filesystem - already at 96% - an
unbounded log store is a slow-motion outage.
Deletion needs both halves. retention_period on its own is inert
without retention_enabled, and the compactor refuses to start unless
delete_request_store is set.
Verified against loki 3.7.6 with this config: compactor reaches ACTIVE
in the ring and logs "this instance has been chosen to run the
compactor", with no config errors.
Entire-Checkpoint: fd6175e529d5
@gotha
lucie: provision the Dissona logs dashboard
3b97e0a
Adds a file-backed dashboard provider pointing at ./dashboards, plus a
first dashboard: error/warning/info/unclassified counts, log volume by
level, warnings and errors by container, nginx edge HTTP status, and
the matching log lines.
allowUiUpdates is false, so the JSON in ./dashboards is the only source
of truth and the UI serves it read-only. Editing in Grafana means
exporting the JSON model back into the repo and rebuilding.
Panels bind to the "loki" datasource uid already provisioned above.
Entire-Checkpoint: a61237526802
@gotha
gotha
merged commit 0bf1cc8
into
mainAug 31, 2026
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers


No reviews


Assignees


No one assigned


Labels


None yet


Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

1 participant

@gotha

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Highlight search terms from Google/DuckDuckGo/Bing referrer
(function() {
var ref = document.referrer;
var terms = [];
if (ref.includes('google.com') || ref.includes('duckduckgo.com') || ref.includes('bing.com')) {
var url = new URL(ref);
var q = url.searchParams.get('q') || url.searchParams.get('p');
if (q) {
terms = q.split(/\s+/).filter(function(t) { return t.length > 2; });
}
}
if (terms.length === 0) return;
var style = document.createElement('style');
style.textContent = '.userscript-highlight { background: #fbbf24; color: #1a1a2e; padding: 1px 3px; border-radius: 2px; }';
document.head.appendChild(style);
function highlight(node) {
if (node.nodeType === 3) { // text node
var text = node.textContent;
var found = false;
terms.forEach(function(term) {
var regex = new RegExp('(' + term.replace(/[.*+?^${}()|[\]\\]/g, '\\') + ')', 'gi');
if (regex.test(text)) {
found = true;
var frag = document.createDocumentFragment();
var parts = text.split(regex);
parts.forEach(function(part, i) {
if (i % 2 === 0) {
frag.appendChild(document.createTextNode(part));
} else {
var span = document.createElement('span');
span.className = 'userscript-highlight';
span.textContent = part;
frag.appendChild(span);
}
});
node.parentNode.replaceChild(frag, node);
}
});
} else if (node.nodeType === 1 && node.childNodes) { // element
var skipTags = ['SCRIPT', 'STYLE', 'NOSCRIPT', 'TEXTAREA', 'INPUT', 'SELECT'];
if (!skipTags.includes(node.tagName)) {
Array.from(node.childNodes).forEach(highlight);
}
}
}
highlight(document.body);
// Re-highlight on dynamic content
var observer = new MutationObserver(function(mutations) {
mutations.forEach(function(m) {
m.addedNodes.forEach(function(node) {
if (node.nodeType === 1 || node.nodeType === 3) highlight(node);
});
});
});
observer.observe(document.body, { childList: true, subtree: true });
})();
}
} catch(__e) { console.warn('[Userscript:Highlight Search Terms]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
Observability - #10
Merged
gotha merged 5 commits into
mainfrom 
observability
Aug 31, 2026
Merged
Observability#10
gotha merged 5 commits into
mainfrom 
observability
Conversation
@gotha
@gothagotha
commented
Aug 28, 2026

Copy link


Copy Markdown

Owner
 No description provided. 
@gotha
lucie: add grafana, prometheus, loki and tempo
a0a3903
Metrics, logs and traces, as native NixOS services rather than containers -
the modules exist and handle users, state directories and unit hardening, so a
compose file would only add moving parts.
Ports: grafana 3000, prometheus 9090, loki 3100, tempo 3200, and tempo's OTLP
receivers on 4317 (gRPC) and 4318 (HTTP). Loki's push API and the OTLP
receivers bind 0.0.0.0 and the firewall admits 172.16.0.0/12, because the point
is for Docker services on this host to reach them: a container talking to
host.docker.internal - which litellm.nix already maps to host-gateway - arrives
from the Docker bridge, and a 127.0.0.1 bind would be unreachable. Everything
else follows litellm.nix's policy of localhost, LAN and WireGuard only. Loki's
gRPC port stays closed; nothing off-host speaks it.
Grafana's datasources are provisioned rather than clicked in, so a rebuild is
enough to get a working instance. Its secret_key lost its default in NixOS
26.05 and is now required, so it is generated and encrypted in
secrets/grafana.enc.json and read through $__file{} to keep it out of the
world-readable store copy of grafana.ini. That file needs lucie's host age key
as well as the PGP key for sops-nix to decrypt at boot, so it joins litellm
under the .sops.yaml rule that carries both.
Prometheus only scrapes itself so far - the Docker services that will expose
/metrics get added as further jobs later.
Entire-Checkpoint: e7f8764a74b0
@gotha
lucie: fix tempo crash loop and move observability off default ports
70e63f9
Tempo restarted every 30s (848 times in one boot) with:
module=live-store err="failed to create shutdown marker directory:
mkdir /var/tempo: read-only file system"
DynamicUser=true implies ProtectSystem=strict, so StateDirectory=tempo
is the only writable path. storage.trace already pointed under
/var/lib/tempo, but Tempo 3.0 added modules whose paths still default
under /var/tempo:
live-store.shutdown_marker_dir /var/tempo/live-store/shutdown-marker
live-store.wal.path /var/tempo/live-store/traces
block-builder.wal.path /var/tempo/block-builder/traces
backend-scheduler.local-work-path /var/tempo
live-store failed on the first, distributor depends on live-store, so
trace ingestion never came up. block_builder is inactive in
single-binary mode but is set too - it fails identically if it ever
activates.
Also moves every port into a 320xx range. Grafana on 3000 and
Prometheus on 9090 collide with anything else that wants the usual
defaults, and lokiPort and tempoPort were briefly both 32030.
Verified: tempo NRestarts=0, /ready returns 200, /var/tempo is never
created, and /var/lib/tempo holds live-store/ traces/ wal/.
Entire-Checkpoint: baef4153a49b
@gotha
lucie: enable aarch64-linux emulation
32368aa
Registers the binfmt handlers and adds aarch64-linux to nix.conf's
extra-platforms so this x86_64 box can build the devbox-arm image for
the mac.
Entire-Checkpoint: 197d69023cbc
@gotha
lucie: expire loki logs after 30 days
ccd51c4
Loki keeps everything forever by default: retention_period was 0s and
compactor.retention_enabled false, so nothing ever deleted a chunk.
Since /var/lib/loki sits on the root filesystem - already at 96% - an
unbounded log store is a slow-motion outage.
Deletion needs both halves. retention_period on its own is inert
without retention_enabled, and the compactor refuses to start unless
delete_request_store is set.
Verified against loki 3.7.6 with this config: compactor reaches ACTIVE
in the ring and logs "this instance has been chosen to run the
compactor", with no config errors.
Entire-Checkpoint: fd6175e529d5
@gotha
lucie: provision the Dissona logs dashboard
3b97e0a
Adds a file-backed dashboard provider pointing at ./dashboards, plus a
first dashboard: error/warning/info/unclassified counts, log volume by
level, warnings and errors by container, nginx edge HTTP status, and
the matching log lines.
allowUiUpdates is false, so the JSON in ./dashboards is the only source
of truth and the UI serves it read-only. Editing in Grafana means
exporting the JSON model back into the repo and rebuilding.
Panels bind to the "loki" datasource uid already provisioned above.
Entire-Checkpoint: a61237526802
@gotha
gotha
merged commit 0bf1cc8
into
mainAug 31, 2026
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers


No reviews


Assignees


No one assigned


Labels


None yet


Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

1 participant

@gotha

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Strip utm_, fbclid, gclid, etc. from all links on page
(function() {
var trackingParams = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content',
'fbclid', 'gclid', 'dclid', 'msclkid', 'yclid',
'ref', 'ref_src', 'source', 'medium', 'campaign'];
function cleanUrl(url) {
try {
var u = new URL(url, window.location.origin);
var changed = false;
trackingParams.forEach(function(p) {
if (u.searchParams.has(p)) {
u.searchParams.delete(p);
changed = true;
}
});
return changed ? u.toString() : url;
} catch (e) {
return url;
}
}
function cleanLinks() {
document.querySelectorAll('a[href]').forEach(function(a) {
var clean = cleanUrl(a.href);
if (clean !== a.href) a.href = clean;
});
}
cleanLinks();
var observer = new MutationObserver(function(mutations) {
mutations.forEach(function(m) {
m.addedNodes.forEach(function(node) {
if (node.nodeType === 1) {
if (node.tagName === 'A') cleanLinks();
node.querySelectorAll('a[href]').forEach(function(a) {
var clean = cleanUrl(a.href);
if (clean !== a.href) a.href = clean;
});
}
});
});
});
observer.observe(document.body, { childList: true, subtree: true });
})();
}
} catch(__e) { console.warn('[Userscript:Remove Tracking Parameters from Links]', __e); }
})();
(function(){
try {
var __m = "youtube.com";
var __re = new RegExp('^' + "youtube\\.com" + '

Skip to content
Observability - #10
Merged
gotha merged 5 commits into
mainfrom 
observability
Aug 31, 2026
Merged
Observability#10
gotha merged 5 commits into
mainfrom 
observability
Conversation
@gotha
@gothagotha
commented
Aug 28, 2026

Copy link


Copy Markdown

Owner
 No description provided. 
@gotha
lucie: add grafana, prometheus, loki and tempo
a0a3903
Metrics, logs and traces, as native NixOS services rather than containers -
the modules exist and handle users, state directories and unit hardening, so a
compose file would only add moving parts.
Ports: grafana 3000, prometheus 9090, loki 3100, tempo 3200, and tempo's OTLP
receivers on 4317 (gRPC) and 4318 (HTTP). Loki's push API and the OTLP
receivers bind 0.0.0.0 and the firewall admits 172.16.0.0/12, because the point
is for Docker services on this host to reach them: a container talking to
host.docker.internal - which litellm.nix already maps to host-gateway - arrives
from the Docker bridge, and a 127.0.0.1 bind would be unreachable. Everything
else follows litellm.nix's policy of localhost, LAN and WireGuard only. Loki's
gRPC port stays closed; nothing off-host speaks it.
Grafana's datasources are provisioned rather than clicked in, so a rebuild is
enough to get a working instance. Its secret_key lost its default in NixOS
26.05 and is now required, so it is generated and encrypted in
secrets/grafana.enc.json and read through $__file{} to keep it out of the
world-readable store copy of grafana.ini. That file needs lucie's host age key
as well as the PGP key for sops-nix to decrypt at boot, so it joins litellm
under the .sops.yaml rule that carries both.
Prometheus only scrapes itself so far - the Docker services that will expose
/metrics get added as further jobs later.
Entire-Checkpoint: e7f8764a74b0
@gotha
lucie: fix tempo crash loop and move observability off default ports
70e63f9
Tempo restarted every 30s (848 times in one boot) with:
module=live-store err="failed to create shutdown marker directory:
mkdir /var/tempo: read-only file system"
DynamicUser=true implies ProtectSystem=strict, so StateDirectory=tempo
is the only writable path. storage.trace already pointed under
/var/lib/tempo, but Tempo 3.0 added modules whose paths still default
under /var/tempo:
live-store.shutdown_marker_dir /var/tempo/live-store/shutdown-marker
live-store.wal.path /var/tempo/live-store/traces
block-builder.wal.path /var/tempo/block-builder/traces
backend-scheduler.local-work-path /var/tempo
live-store failed on the first, distributor depends on live-store, so
trace ingestion never came up. block_builder is inactive in
single-binary mode but is set too - it fails identically if it ever
activates.
Also moves every port into a 320xx range. Grafana on 3000 and
Prometheus on 9090 collide with anything else that wants the usual
defaults, and lokiPort and tempoPort were briefly both 32030.
Verified: tempo NRestarts=0, /ready returns 200, /var/tempo is never
created, and /var/lib/tempo holds live-store/ traces/ wal/.
Entire-Checkpoint: baef4153a49b
@gotha
lucie: enable aarch64-linux emulation
32368aa
Registers the binfmt handlers and adds aarch64-linux to nix.conf's
extra-platforms so this x86_64 box can build the devbox-arm image for
the mac.
Entire-Checkpoint: 197d69023cbc
@gotha
lucie: expire loki logs after 30 days
ccd51c4
Loki keeps everything forever by default: retention_period was 0s and
compactor.retention_enabled false, so nothing ever deleted a chunk.
Since /var/lib/loki sits on the root filesystem - already at 96% - an
unbounded log store is a slow-motion outage.
Deletion needs both halves. retention_period on its own is inert
without retention_enabled, and the compactor refuses to start unless
delete_request_store is set.
Verified against loki 3.7.6 with this config: compactor reaches ACTIVE
in the ring and logs "this instance has been chosen to run the
compactor", with no config errors.
Entire-Checkpoint: fd6175e529d5
@gotha
lucie: provision the Dissona logs dashboard
3b97e0a
Adds a file-backed dashboard provider pointing at ./dashboards, plus a
first dashboard: error/warning/info/unclassified counts, log volume by
level, warnings and errors by container, nginx edge HTTP status, and
the matching log lines.
allowUiUpdates is false, so the JSON in ./dashboards is the only source
of truth and the UI serves it read-only. Editing in Grafana means
exporting the JSON model back into the repo and rebuilding.
Panels bind to the "loki" datasource uid already provisioned above.
Entire-Checkpoint: a61237526802
@gotha
gotha
merged commit 0bf1cc8
into
mainAug 31, 2026
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers


No reviews


Assignees


No one assigned


Labels


None yet


Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

1 participant

@gotha

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Auto-enable theater mode on YouTube
(function() {
function tryTheater() {
var btn = document.querySelector('button[aria-label="Theater mode"], ytd-player #player button[title="Theater mode"]');
if (btn && !btn.classList.contains('activated')) {
btn.click();
}
}
// Try immediately
tryTheater();
// Try after navigation (SPA)
var lastUrl = location.href;
setInterval(function() {
if (location.href !== lastUrl) {
lastUrl = location.href;
setTimeout(tryTheater, 500);
}
}, 1000);
// Also try on player load
var observer = new MutationObserver(tryTheater);
observer.observe(document.body, { childList: true, subtree: true });
})();
}
} catch(__e) { console.warn('[Userscript:YouTube Theater Mode Default]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
Observability - #10
Merged
gotha merged 5 commits into
mainfrom 
observability
Aug 31, 2026
Merged
Observability#10
gotha merged 5 commits into
mainfrom 
observability
Conversation
@gotha
@gothagotha
commented
Aug 28, 2026

Copy link


Copy Markdown

Owner
 No description provided. 
@gotha
lucie: add grafana, prometheus, loki and tempo
a0a3903
Metrics, logs and traces, as native NixOS services rather than containers -
the modules exist and handle users, state directories and unit hardening, so a
compose file would only add moving parts.
Ports: grafana 3000, prometheus 9090, loki 3100, tempo 3200, and tempo's OTLP
receivers on 4317 (gRPC) and 4318 (HTTP). Loki's push API and the OTLP
receivers bind 0.0.0.0 and the firewall admits 172.16.0.0/12, because the point
is for Docker services on this host to reach them: a container talking to
host.docker.internal - which litellm.nix already maps to host-gateway - arrives
from the Docker bridge, and a 127.0.0.1 bind would be unreachable. Everything
else follows litellm.nix's policy of localhost, LAN and WireGuard only. Loki's
gRPC port stays closed; nothing off-host speaks it.
Grafana's datasources are provisioned rather than clicked in, so a rebuild is
enough to get a working instance. Its secret_key lost its default in NixOS
26.05 and is now required, so it is generated and encrypted in
secrets/grafana.enc.json and read through $__file{} to keep it out of the
world-readable store copy of grafana.ini. That file needs lucie's host age key
as well as the PGP key for sops-nix to decrypt at boot, so it joins litellm
under the .sops.yaml rule that carries both.
Prometheus only scrapes itself so far - the Docker services that will expose
/metrics get added as further jobs later.
Entire-Checkpoint: e7f8764a74b0
@gotha
lucie: fix tempo crash loop and move observability off default ports
70e63f9
Tempo restarted every 30s (848 times in one boot) with:
module=live-store err="failed to create shutdown marker directory:
mkdir /var/tempo: read-only file system"
DynamicUser=true implies ProtectSystem=strict, so StateDirectory=tempo
is the only writable path. storage.trace already pointed under
/var/lib/tempo, but Tempo 3.0 added modules whose paths still default
under /var/tempo:
live-store.shutdown_marker_dir /var/tempo/live-store/shutdown-marker
live-store.wal.path /var/tempo/live-store/traces
block-builder.wal.path /var/tempo/block-builder/traces
backend-scheduler.local-work-path /var/tempo
live-store failed on the first, distributor depends on live-store, so
trace ingestion never came up. block_builder is inactive in
single-binary mode but is set too - it fails identically if it ever
activates.
Also moves every port into a 320xx range. Grafana on 3000 and
Prometheus on 9090 collide with anything else that wants the usual
defaults, and lokiPort and tempoPort were briefly both 32030.
Verified: tempo NRestarts=0, /ready returns 200, /var/tempo is never
created, and /var/lib/tempo holds live-store/ traces/ wal/.
Entire-Checkpoint: baef4153a49b
@gotha
lucie: enable aarch64-linux emulation
32368aa
Registers the binfmt handlers and adds aarch64-linux to nix.conf's
extra-platforms so this x86_64 box can build the devbox-arm image for
the mac.
Entire-Checkpoint: 197d69023cbc
@gotha
lucie: expire loki logs after 30 days
ccd51c4
Loki keeps everything forever by default: retention_period was 0s and
compactor.retention_enabled false, so nothing ever deleted a chunk.
Since /var/lib/loki sits on the root filesystem - already at 96% - an
unbounded log store is a slow-motion outage.
Deletion needs both halves. retention_period on its own is inert
without retention_enabled, and the compactor refuses to start unless
delete_request_store is set.
Verified against loki 3.7.6 with this config: compactor reaches ACTIVE
in the ring and logs "this instance has been chosen to run the
compactor", with no config errors.
Entire-Checkpoint: fd6175e529d5
@gotha
lucie: provision the Dissona logs dashboard
3b97e0a
Adds a file-backed dashboard provider pointing at ./dashboards, plus a
first dashboard: error/warning/info/unclassified counts, log volume by
level, warnings and errors by container, nginx edge HTTP status, and
the matching log lines.
allowUiUpdates is false, so the JSON in ./dashboards is the only source
of truth and the UI serves it read-only. Editing in Grafana means
exporting the JSON model back into the repo and rebuilding.
Panels bind to the "loki" datasource uid already provisioned above.
Entire-Checkpoint: a61237526802
@gotha
gotha
merged commit 0bf1cc8
into
mainAug 31, 2026
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers


No reviews


Assignees


No one assigned


Labels


None yet


Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

1 participant

@gotha

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Remove or un-stick sticky/fixed headers that block content
(function() {
function unstick() {
document.querySelectorAll('header, nav, [role="banner"], .header, .navbar, .sticky, .fixed-top, [style*="position: fixed"], [style*="position:sticky"]').forEach(function(el) {
if (el.style.position === 'fixed' || el.style.position === 'sticky' || getComputedStyle(el).position === 'fixed' || getComputedStyle(el).position === 'sticky') {
el.style.position = 'static';
el.style.top = 'auto';
el.style.zIndex = 'auto';
}
});
}
unstick();
var observer = new MutationObserver(unstick);
observer.observe(document.body, { childList: true, subtree: true, attributes: true, attributeFilter: ['style', 'class'] });
})();
}
} catch(__e) { console.warn('[Userscript:Kill Sticky Headers]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
Observability - #10
Merged
gotha merged 5 commits into
mainfrom 
observability
Aug 31, 2026
Merged
Observability#10
gotha merged 5 commits into
mainfrom 
observability
Conversation
@gotha
@gothagotha
commented
Aug 28, 2026

Copy link


Copy Markdown

Owner
 No description provided. 
@gotha
lucie: add grafana, prometheus, loki and tempo
a0a3903
Metrics, logs and traces, as native NixOS services rather than containers -
the modules exist and handle users, state directories and unit hardening, so a
compose file would only add moving parts.
Ports: grafana 3000, prometheus 9090, loki 3100, tempo 3200, and tempo's OTLP
receivers on 4317 (gRPC) and 4318 (HTTP). Loki's push API and the OTLP
receivers bind 0.0.0.0 and the firewall admits 172.16.0.0/12, because the point
is for Docker services on this host to reach them: a container talking to
host.docker.internal - which litellm.nix already maps to host-gateway - arrives
from the Docker bridge, and a 127.0.0.1 bind would be unreachable. Everything
else follows litellm.nix's policy of localhost, LAN and WireGuard only. Loki's
gRPC port stays closed; nothing off-host speaks it.
Grafana's datasources are provisioned rather than clicked in, so a rebuild is
enough to get a working instance. Its secret_key lost its default in NixOS
26.05 and is now required, so it is generated and encrypted in
secrets/grafana.enc.json and read through $__file{} to keep it out of the
world-readable store copy of grafana.ini. That file needs lucie's host age key
as well as the PGP key for sops-nix to decrypt at boot, so it joins litellm
under the .sops.yaml rule that carries both.
Prometheus only scrapes itself so far - the Docker services that will expose
/metrics get added as further jobs later.
Entire-Checkpoint: e7f8764a74b0
@gotha
lucie: fix tempo crash loop and move observability off default ports
70e63f9
Tempo restarted every 30s (848 times in one boot) with:
module=live-store err="failed to create shutdown marker directory:
mkdir /var/tempo: read-only file system"
DynamicUser=true implies ProtectSystem=strict, so StateDirectory=tempo
is the only writable path. storage.trace already pointed under
/var/lib/tempo, but Tempo 3.0 added modules whose paths still default
under /var/tempo:
live-store.shutdown_marker_dir /var/tempo/live-store/shutdown-marker
live-store.wal.path /var/tempo/live-store/traces
block-builder.wal.path /var/tempo/block-builder/traces
backend-scheduler.local-work-path /var/tempo
live-store failed on the first, distributor depends on live-store, so
trace ingestion never came up. block_builder is inactive in
single-binary mode but is set too - it fails identically if it ever
activates.
Also moves every port into a 320xx range. Grafana on 3000 and
Prometheus on 9090 collide with anything else that wants the usual
defaults, and lokiPort and tempoPort were briefly both 32030.
Verified: tempo NRestarts=0, /ready returns 200, /var/tempo is never
created, and /var/lib/tempo holds live-store/ traces/ wal/.
Entire-Checkpoint: baef4153a49b
@gotha
lucie: enable aarch64-linux emulation
32368aa
Registers the binfmt handlers and adds aarch64-linux to nix.conf's
extra-platforms so this x86_64 box can build the devbox-arm image for
the mac.
Entire-Checkpoint: 197d69023cbc
@gotha
lucie: expire loki logs after 30 days
ccd51c4
Loki keeps everything forever by default: retention_period was 0s and
compactor.retention_enabled false, so nothing ever deleted a chunk.
Since /var/lib/loki sits on the root filesystem - already at 96% - an
unbounded log store is a slow-motion outage.
Deletion needs both halves. retention_period on its own is inert
without retention_enabled, and the compactor refuses to start unless
delete_request_store is set.
Verified against loki 3.7.6 with this config: compactor reaches ACTIVE
in the ring and logs "this instance has been chosen to run the
compactor", with no config errors.
Entire-Checkpoint: fd6175e529d5
@gotha
lucie: provision the Dissona logs dashboard
3b97e0a
Adds a file-backed dashboard provider pointing at ./dashboards, plus a
first dashboard: error/warning/info/unclassified counts, log volume by
level, warnings and errors by container, nginx edge HTTP status, and
the matching log lines.
allowUiUpdates is false, so the JSON in ./dashboards is the only source
of truth and the UI serves it read-only. Editing in Grafana means
exporting the JSON model back into the repo and rebuilding.
Panels bind to the "loki" datasource uid already provisioned above.
Entire-Checkpoint: a61237526802
@gotha
gotha
merged commit 0bf1cc8
into
mainAug 31, 2026
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers


No reviews


Assignees


No one assigned


Labels


None yet


Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

1 participant

@gotha

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Universal Dark Mode - works on any site
(function() {
var enabled = true;
function applyDarkMode() {
if (!enabled) return;
// Create style element if it doesn't exist
var style = document.getElementById('universal-dark-mode-style');
if (!style) {
style = document.createElement('style');
style.id = 'universal-dark-mode-style';
document.head.appendChild(style);
}
// Dark mode CSS - inverts colors but preserves images/video
style.textContent = '
/* Invert everything except media */
html {
filter: invert(1) hue-rotate(180deg) !important;
background: #1a1a2e !important;
}
/* Restore images, videos, iframes, canvas */
img, video, iframe, canvas, svg, picture, [style*="background-image"] {
filter: invert(1) hue-rotate(180deg) !important;
}
/* Preserve specific elements that should not be inverted */
.no-dark-mode, .no-dark-mode *,
[data-theme="light"], [data-theme="light"],
.ace_editor, .ace_editor *,
.CodeMirror, .CodeMirror *,
.monaco-editor, .monaco-editor *,
.markdown-body pre, .markdown-body pre *,
.highlight, .highlight *,
pre code, pre code * {
filter: none !important;
}
/* Fix common UI elements */
.modal, .popup, .dropdown-menu, .tooltip, .popover {
filter: invert(1) hue-rotate(180deg) !important;
background: #2d2d44 !important;
border-color: #444 !important;
}
/* Scrollbars */
::-webkit-scrollbar { background: #1a1a2e !important; }
::-webkit-scrollbar-thumb { background: #444 !important; }
::-webkit-scrollbar-thumb:hover { background: #555 !important; }
/* Selection */
::selection { background: #4ecdc4 !important; color: #1a1a2e !important; }
::-moz-selection { background: #4ecdc4 !important; color: #1a1a2e !important; }
';
}
function removeDarkMode() {
var style = document.getElementById('universal-dark-mode-style');
if (style) style.remove();
}
// Toggle with Alt+Shift+D
document.addEventListener('keydown', function(e) {
if (e.altKey && e.shiftKey && e.key === 'D') {
e.preventDefault();
enabled = !enabled;
if (enabled) {
applyDarkMode();
console.log('[Universal Dark Mode] Enabled');
} else {
removeDarkMode();
console.log('[Universal Dark Mode] Disabled');
}
}
});
// Apply on load
applyDarkMode();
// Re-apply on dynamic content
var observer = new MutationObserver(function(mutations) {
if (enabled && !document.getElementById('universal-dark-mode-style')) {
applyDarkMode();
}
});
observer.observe(document.head, { childList: true });
console.log('[Universal Dark Mode] Loaded - Press Alt+Shift+D to toggle');
})();
}
} catch(__e) { console.warn('[Userscript:Universal Dark Mode]', __e); }
})();
})();

Skip to content
Observability - #10
Merged
gotha merged 5 commits into
mainfrom 
observability
Aug 31, 2026
Merged
Observability#10
gotha merged 5 commits into
mainfrom 
observability
Conversation
@gotha
@gothagotha
commented
Aug 28, 2026

Copy link


Copy Markdown

Owner
 No description provided. 
@gotha
lucie: add grafana, prometheus, loki and tempo
a0a3903
Metrics, logs and traces, as native NixOS services rather than containers -
the modules exist and handle users, state directories and unit hardening, so a
compose file would only add moving parts.
Ports: grafana 3000, prometheus 9090, loki 3100, tempo 3200, and tempo's OTLP
receivers on 4317 (gRPC) and 4318 (HTTP). Loki's push API and the OTLP
receivers bind 0.0.0.0 and the firewall admits 172.16.0.0/12, because the point
is for Docker services on this host to reach them: a container talking to
host.docker.internal - which litellm.nix already maps to host-gateway - arrives
from the Docker bridge, and a 127.0.0.1 bind would be unreachable. Everything
else follows litellm.nix's policy of localhost, LAN and WireGuard only. Loki's
gRPC port stays closed; nothing off-host speaks it.
Grafana's datasources are provisioned rather than clicked in, so a rebuild is
enough to get a working instance. Its secret_key lost its default in NixOS
26.05 and is now required, so it is generated and encrypted in
secrets/grafana.enc.json and read through $__file{} to keep it out of the
world-readable store copy of grafana.ini. That file needs lucie's host age key
as well as the PGP key for sops-nix to decrypt at boot, so it joins litellm
under the .sops.yaml rule that carries both.
Prometheus only scrapes itself so far - the Docker services that will expose
/metrics get added as further jobs later.
Entire-Checkpoint: e7f8764a74b0
@gotha
lucie: fix tempo crash loop and move observability off default ports
70e63f9
Tempo restarted every 30s (848 times in one boot) with:
module=live-store err="failed to create shutdown marker directory:
mkdir /var/tempo: read-only file system"
DynamicUser=true implies ProtectSystem=strict, so StateDirectory=tempo
is the only writable path. storage.trace already pointed under
/var/lib/tempo, but Tempo 3.0 added modules whose paths still default
under /var/tempo:
live-store.shutdown_marker_dir /var/tempo/live-store/shutdown-marker
live-store.wal.path /var/tempo/live-store/traces
block-builder.wal.path /var/tempo/block-builder/traces
backend-scheduler.local-work-path /var/tempo
live-store failed on the first, distributor depends on live-store, so
trace ingestion never came up. block_builder is inactive in
single-binary mode but is set too - it fails identically if it ever
activates.
Also moves every port into a 320xx range. Grafana on 3000 and
Prometheus on 9090 collide with anything else that wants the usual
defaults, and lokiPort and tempoPort were briefly both 32030.
Verified: tempo NRestarts=0, /ready returns 200, /var/tempo is never
created, and /var/lib/tempo holds live-store/ traces/ wal/.
Entire-Checkpoint: baef4153a49b
@gotha
lucie: enable aarch64-linux emulation
32368aa
Registers the binfmt handlers and adds aarch64-linux to nix.conf's
extra-platforms so this x86_64 box can build the devbox-arm image for
the mac.
Entire-Checkpoint: 197d69023cbc
@gotha
lucie: expire loki logs after 30 days
ccd51c4
Loki keeps everything forever by default: retention_period was 0s and
compactor.retention_enabled false, so nothing ever deleted a chunk.
Since /var/lib/loki sits on the root filesystem - already at 96% - an
unbounded log store is a slow-motion outage.
Deletion needs both halves. retention_period on its own is inert
without retention_enabled, and the compactor refuses to start unless
delete_request_store is set.
Verified against loki 3.7.6 with this config: compactor reaches ACTIVE
in the ring and logs "this instance has been chosen to run the
compactor", with no config errors.
Entire-Checkpoint: fd6175e529d5
@gotha
lucie: provision the Dissona logs dashboard
3b97e0a
Adds a file-backed dashboard provider pointing at ./dashboards, plus a
first dashboard: error/warning/info/unclassified counts, log volume by
level, warnings and errors by container, nginx edge HTTP status, and
the matching log lines.
allowUiUpdates is false, so the JSON in ./dashboards is the only source
of truth and the UI serves it read-only. Editing in Grafana means
exporting the JSON model back into the repo and rebuilding.
Panels bind to the "loki" datasource uid already provisioned above.
Entire-Checkpoint: a61237526802
@gotha
gotha
merged commit 0bf1cc8
into
mainAug 31, 2026
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers


No reviews


Assignees


No one assigned


Labels


None yet


Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

1 participant

@gotha