Transition to ndjson export format - #10

Merged
hargoniX merged 8 commits into
leanprover:masterfrom
ammkrn:json_output
Jan 23, 2026
Merged

Transition to ndjson export format#10
hargoniX merged 8 commits into
leanprover:masterfrom
ammkrn:json_output

Conversation

@ammkrn

Copy link
Copy Markdown
Contributor

When ready, will close#3 (also see that issue for relevant discussion).

The format is described in format_ndjson and the README has been updated accordingly, along with the recommended command for invocation; the old README recommended invoking lake exe, which will not correctly set up the lake env in the most common use case, something like exporting mathlib.

Comment threadformat_ndjson.md Outdated
Expr.lit (Literal.strVal)
```
{
"lit": {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This does not seem to be what your program currently actually prints?

| .lit (.natVal i) => return .mkObj [("natVal", s!"{i}")]
| .lit (.strVal s) => return .mkObj [("strVal", s)]

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, I've pushed a correction for the format specification.

Comment threadformat_ndjson.md Outdated
"levelParams": Array<integer>,
"type": integer,
"value": integer,
"hints": Array<"opaque" | "abbrev" | integer>

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is also incorrectly documented afaict?

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ditto.

Corrects two errors in the format specification for string literals and
reducibility hints.
Comment threadformat_ndjson.md Outdated
@hargoniXhargoniX mentioned this pull request Jan 7, 2026
Correct a typo of `deBruijnIndex`
Comment threadExport.lean Outdated
Comment threadExport.lean Outdated
@ammkrn

Copy link
Copy Markdown
ContributorAuthor

The open comments should be resolved, and I believe I addressed the remaining points in #10

+ Namespaces backreferences from a generic "i" to "in", "il", and "ie" for name, level, and expr.
+ Abbreviates some json attribute names
@nomeata

nomeata commented Jan 9, 2026

Copy link
Copy Markdown
Contributor

Thanks!

(Seeing how slow lean4export is I’m inclined to rewrite the exporter to use simple string interpolation into hardcoded JSON lines, at least for everything that only contains numbers, but that can wait until other dust has settled.)

Comment threadExport.lean Outdated
Modifies the export procedure and format to export the components of an
inductive declaration (the inductive specification(s), constructors,
and recursors) together.
Consumers are generally going to want to find these related declarations
anyway, and with the introduction of the auxiliary `.rec_<N>` recursors
motivated by nested inductive support, (in my opinion) this is a more
comprehensible format.
@ammkrn

Copy link
Copy Markdown
ContributorAuthor

I've added one commit 50db814 to implement the change mentioned elsewhere, packaging related elements of an inductive declaration.

@nomeata

nomeata commented Jan 13, 2026

Copy link
Copy Markdown
Contributor

Great! Are declarations now following the Lean.Declaration type consistently, so that a lean-using importer can parse to that type directly?

Hmm, it seems we now have a somewhat odd mix of Lean.Declaration and Lean.ConstantInfo. But if its a “best of both worlds” mix it’s maybe fine?

@ammkrn

Copy link
Copy Markdown
ContributorAuthor

Great! Are declarations now following the Lean.Declaration type consistently, so that a lean-using importer can parse to that type directly?

Hmm, it seems we now have a somewhat odd mix of Lean.Declaration and Lean.ConstantInfo. But if its a “best of both worlds” mix it’s maybe fine?

I'm not sure how one would be able to parse anything directly regardless of format since (a) we're using integer pointers and (b) it was already suggested that we make simplifying cuts that would prevent direct parsing, like removing "deBruijnIndex" and changing "binderName".

Lean.Declaration only includes what's strictly necessary for a "full" kernel implementation to do the work of type checking; for example it expects the consumer to both construct and check recursors. The very first exporter implementation followed that pattern.

In the previous round of format changes, it was decided that the exporter should include additional information (like "targets" for the recursors and rec rules, Eq/Quot declarations, etc.). With this additional info it's much easier to bootstrap new checkers, implement certain memory optimizations, and have things like parallel checkers.

Looking at Lean.Declaration, maybe it's worth it to similarly group mutual definitions as well.

@nomeata

Copy link
Copy Markdown
Contributor

I didn’t mean “parse generically”, but I meant “parse export to List Lean.Declaration” (so field name changes or indirections are not a blocker).

But thanks for the historical perspective, agreed on that count.

Comment threadExport.lean
@nomeatanomeata mentioned this pull request Jan 15, 2026
@nomeata

Copy link
Copy Markdown
Contributor

Looking at Lean.Declaration, maybe it's worth it to similarly group mutual definitions as well.

These are always ever unsafe, so not of high importance here? But in principle you are right.

Exports elements of mutual theorem and def/opaque blocks together.
@ammkrn

Copy link
Copy Markdown
ContributorAuthor

Looking at Lean.Declaration, maybe it's worth it to similarly group mutual definitions as well.

These are always ever unsafe, so not of high importance here? But in principle you are right.

Should be done in 7188ec9 modulo style issues.

@nomeata

Copy link
Copy Markdown
Contributor

Am I reading this right that it doesn't distinguish between a normal definition and a singly-recursive definition? It seems that of that's the case, import information is lost. I assume that's why Lean.Declaration is explicit in where the mutual groups are. But maybe lean forgets that information so it's hard to reproduce the declaration here?

@nomeata

Copy link
Copy Markdown
Contributor

Hmm, the Kernel certainly doesn’t seem to remember whether a declaration came from a .mutualDefnDecl or not, so by the time we have a ConstantInfo this information is lost.

Ah! But the kernel treats any unsafe declaration as recursive, even if not in a mutual block. Odd. But not partial declarations.

I find it confusing.

In any case, I think I’m leaning towards an export format that more closely matches Lean.Declaration here, and with a dedicated .mutual tag that is only used when Lean.Declaration.mutualDefnDecl would be used (namely when its unsafe or partial and all is not a singleton). This way parsers for checkers who do not support this can just skip the whole line. But I don't feel strongly about it.

@ammkrn

Copy link
Copy Markdown
ContributorAuthor

In any case, I think I’m leaning towards an export format that more closely matches Lean.Declaration here, and with a dedicated .mutual tag that is only used when Lean.Declaration.mutualDefnDecl would be used (namely when its unsafe or partial and all is not a singleton). This way parsers for checkers who do not support this can just skip the whole line. But I don't feel strongly about it.

I have no issue with that. To the earlier question, I think examination of all (specifically whether there's more than one name in there) would be the context clue.

Adds a separate metadata field for the format version, since it's not
necessarily the same as the exporter version. For example a performance
fix in the exporter might justify an exporter version bump, but not a
format version bump.
Tries to export `Nat` on the appearance of a nat literal, and Char.ofNat
and String.ofList on the appearance of a string literal.
@hargoniX
hargoniX merged commit 4f1e55e into leanprover:masterJan 23, 2026
1 check passed
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[RFC] Switch to JSON

3 participants

@ammkrn@nomeata@hargoniX
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Add copy buttons to all
 blocks\n(function() {\n function addCopyButtons() {\n document.querySelectorAll('pre code').forEach(function(codeBlock) {\n if (codeBlock.parentElement.hasAttribute('data-copy-added')) return;\n codeBlock.parentElement.setAttribute('data-copy-added', 'true');\n \n var btn = document.createElement('button');\n btn.textContent = 'Copy';\n btn.style.cssText = 'position:absolute;top:4px;right:4px;padding:2px 8px;font-size:11px;background:#4ecdc4;border:none;border-radius:4px;color:#1a1a2e;cursor:pointer;opacity:0.7;transition:opacity 0.2s;';\n btn.onmouseover = function() { this.style.opacity = '1'; };\n btn.onmouseout = function() { this.style.opacity = '0.7'; };\n btn.onclick = function() {\n navigator.clipboard.writeText(codeBlock.textContent).then(function() {\n btn.textContent = 'Copied!';\n setTimeout(function() { btn.textContent = 'Copy'; }, 1500);\n });\n };\n codeBlock.parentElement.style.position = 'relative';\n codeBlock.parentElement.appendChild(btn);\n });\n }\n \n addCopyButtons();\n \n // Re-run on dynamic content\n var observer = new MutationObserver(addCopyButtons);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Add Copy Buttons to Code Blocks");
}
} catch(__e) { console.warn('[Userscript:Add Copy Buttons to Code Blocks]', __e); }
})();
(function(){
try {
var __m = "github.com";
var __re = new RegExp('^' + "github\\.com" + '
Skip to content

Transition to ndjson export format - #10

Merged
hargoniX merged 8 commits into
leanprover:masterfrom
ammkrn:json_output
Jan 23, 2026
Merged

Transition to ndjson export format#10
hargoniX merged 8 commits into
leanprover:masterfrom
ammkrn:json_output

Conversation

@ammkrn

Copy link
Copy Markdown
Contributor

When ready, will close#3 (also see that issue for relevant discussion).

The format is described in format_ndjson and the README has been updated accordingly, along with the recommended command for invocation; the old README recommended invoking lake exe, which will not correctly set up the lake env in the most common use case, something like exporting mathlib.

Comment threadformat_ndjson.md Outdated
Expr.lit (Literal.strVal)
```
{
"lit": {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This does not seem to be what your program currently actually prints?

| .lit (.natVal i) => return .mkObj [("natVal", s!"{i}")]
| .lit (.strVal s) => return .mkObj [("strVal", s)]

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, I've pushed a correction for the format specification.

Comment threadformat_ndjson.md Outdated
"levelParams": Array<integer>,
"type": integer,
"value": integer,
"hints": Array<"opaque" | "abbrev" | integer>

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is also incorrectly documented afaict?

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ditto.

Corrects two errors in the format specification for string literals and
reducibility hints.
Comment threadformat_ndjson.md Outdated
@hargoniXhargoniX mentioned this pull request Jan 7, 2026
Correct a typo of `deBruijnIndex`
Comment threadExport.lean Outdated
Comment threadExport.lean Outdated
@ammkrn

Copy link
Copy Markdown
ContributorAuthor

The open comments should be resolved, and I believe I addressed the remaining points in #10

+ Namespaces backreferences from a generic "i" to "in", "il", and "ie" for name, level, and expr.
+ Abbreviates some json attribute names
@nomeata

nomeata commented Jan 9, 2026

Copy link
Copy Markdown
Contributor

Thanks!

(Seeing how slow lean4export is I’m inclined to rewrite the exporter to use simple string interpolation into hardcoded JSON lines, at least for everything that only contains numbers, but that can wait until other dust has settled.)

Comment threadExport.lean Outdated
Modifies the export procedure and format to export the components of an
inductive declaration (the inductive specification(s), constructors,
and recursors) together.
Consumers are generally going to want to find these related declarations
anyway, and with the introduction of the auxiliary `.rec_<N>` recursors
motivated by nested inductive support, (in my opinion) this is a more
comprehensible format.
@ammkrn

Copy link
Copy Markdown
ContributorAuthor

I've added one commit 50db814 to implement the change mentioned elsewhere, packaging related elements of an inductive declaration.

@nomeata

nomeata commented Jan 13, 2026

Copy link
Copy Markdown
Contributor

Great! Are declarations now following the Lean.Declaration type consistently, so that a lean-using importer can parse to that type directly?

Hmm, it seems we now have a somewhat odd mix of Lean.Declaration and Lean.ConstantInfo. But if its a “best of both worlds” mix it’s maybe fine?

@ammkrn

Copy link
Copy Markdown
ContributorAuthor

Great! Are declarations now following the Lean.Declaration type consistently, so that a lean-using importer can parse to that type directly?

Hmm, it seems we now have a somewhat odd mix of Lean.Declaration and Lean.ConstantInfo. But if its a “best of both worlds” mix it’s maybe fine?

I'm not sure how one would be able to parse anything directly regardless of format since (a) we're using integer pointers and (b) it was already suggested that we make simplifying cuts that would prevent direct parsing, like removing "deBruijnIndex" and changing "binderName".

Lean.Declaration only includes what's strictly necessary for a "full" kernel implementation to do the work of type checking; for example it expects the consumer to both construct and check recursors. The very first exporter implementation followed that pattern.

In the previous round of format changes, it was decided that the exporter should include additional information (like "targets" for the recursors and rec rules, Eq/Quot declarations, etc.). With this additional info it's much easier to bootstrap new checkers, implement certain memory optimizations, and have things like parallel checkers.

Looking at Lean.Declaration, maybe it's worth it to similarly group mutual definitions as well.

@nomeata

Copy link
Copy Markdown
Contributor

I didn’t mean “parse generically”, but I meant “parse export to List Lean.Declaration” (so field name changes or indirections are not a blocker).

But thanks for the historical perspective, agreed on that count.

Comment threadExport.lean
@nomeatanomeata mentioned this pull request Jan 15, 2026
@nomeata

Copy link
Copy Markdown
Contributor

Looking at Lean.Declaration, maybe it's worth it to similarly group mutual definitions as well.

These are always ever unsafe, so not of high importance here? But in principle you are right.

Exports elements of mutual theorem and def/opaque blocks together.
@ammkrn

Copy link
Copy Markdown
ContributorAuthor

Looking at Lean.Declaration, maybe it's worth it to similarly group mutual definitions as well.

These are always ever unsafe, so not of high importance here? But in principle you are right.

Should be done in 7188ec9 modulo style issues.

@nomeata

Copy link
Copy Markdown
Contributor

Am I reading this right that it doesn't distinguish between a normal definition and a singly-recursive definition? It seems that of that's the case, import information is lost. I assume that's why Lean.Declaration is explicit in where the mutual groups are. But maybe lean forgets that information so it's hard to reproduce the declaration here?

@nomeata

Copy link
Copy Markdown
Contributor

Hmm, the Kernel certainly doesn’t seem to remember whether a declaration came from a .mutualDefnDecl or not, so by the time we have a ConstantInfo this information is lost.

Ah! But the kernel treats any unsafe declaration as recursive, even if not in a mutual block. Odd. But not partial declarations.

I find it confusing.

In any case, I think I’m leaning towards an export format that more closely matches Lean.Declaration here, and with a dedicated .mutual tag that is only used when Lean.Declaration.mutualDefnDecl would be used (namely when its unsafe or partial and all is not a singleton). This way parsers for checkers who do not support this can just skip the whole line. But I don't feel strongly about it.

@ammkrn

Copy link
Copy Markdown
ContributorAuthor

In any case, I think I’m leaning towards an export format that more closely matches Lean.Declaration here, and with a dedicated .mutual tag that is only used when Lean.Declaration.mutualDefnDecl would be used (namely when its unsafe or partial and all is not a singleton). This way parsers for checkers who do not support this can just skip the whole line. But I don't feel strongly about it.

I have no issue with that. To the earlier question, I think examination of all (specifically whether there's more than one name in there) would be the context clue.

Adds a separate metadata field for the format version, since it's not
necessarily the same as the exporter version. For example a performance
fix in the exporter might justify an exporter version bump, but not a
format version bump.
Tries to export `Nat` on the appearance of a nat literal, and Char.ofNat
and String.ofList on the appearance of a string literal.
@hargoniX
hargoniX merged commit 4f1e55e into leanprover:masterJan 23, 2026
1 check passed
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[RFC] Switch to JSON

3 participants

@ammkrn@nomeata@hargoniX
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Force GitHub README to respect dark mode\n(function() {\n var style = document.createElement('style');\n style.textContent = '\n .markdown-body {\n color-scheme: dark light;\n }\n .markdown-body pre { background: #161b22 !important; }\n .markdown-body code { background: rgba(110, 118, 129, 0.4) !important; }\n .markdown-body table th, .markdown-body table td { border-color: #30363d !important; }\n .markdown-body img { background: #0d1117; }\n .markdown-body blockquote { border-left-color: #8b949e; }\n .markdown-body hr { border-color: #30363d; }\n ';\n document.head.appendChild(style);\n})();", "GitHub Dark Mode README Fix"); } } catch(__e) { console.warn('[Userscript:GitHub Dark Mode README Fix]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
Skip to content

Transition to ndjson export format - #10

Merged
hargoniX merged 8 commits into
leanprover:masterfrom
ammkrn:json_output
Jan 23, 2026
Merged

Transition to ndjson export format#10
hargoniX merged 8 commits into
leanprover:masterfrom
ammkrn:json_output

Conversation

@ammkrn

Copy link
Copy Markdown
Contributor

When ready, will close#3 (also see that issue for relevant discussion).

The format is described in format_ndjson and the README has been updated accordingly, along with the recommended command for invocation; the old README recommended invoking lake exe, which will not correctly set up the lake env in the most common use case, something like exporting mathlib.

Comment threadformat_ndjson.md Outdated
Expr.lit (Literal.strVal)
```
{
"lit": {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This does not seem to be what your program currently actually prints?

| .lit (.natVal i) => return .mkObj [("natVal", s!"{i}")]
| .lit (.strVal s) => return .mkObj [("strVal", s)]

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, I've pushed a correction for the format specification.

Comment threadformat_ndjson.md Outdated
"levelParams": Array<integer>,
"type": integer,
"value": integer,
"hints": Array<"opaque" | "abbrev" | integer>

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is also incorrectly documented afaict?

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ditto.

Corrects two errors in the format specification for string literals and
reducibility hints.
Comment threadformat_ndjson.md Outdated
@hargoniXhargoniX mentioned this pull request Jan 7, 2026
Correct a typo of `deBruijnIndex`
Comment threadExport.lean Outdated
Comment threadExport.lean Outdated
@ammkrn

Copy link
Copy Markdown
ContributorAuthor

The open comments should be resolved, and I believe I addressed the remaining points in #10

+ Namespaces backreferences from a generic "i" to "in", "il", and "ie" for name, level, and expr.
+ Abbreviates some json attribute names
@nomeata

nomeata commented Jan 9, 2026

Copy link
Copy Markdown
Contributor

Thanks!

(Seeing how slow lean4export is I’m inclined to rewrite the exporter to use simple string interpolation into hardcoded JSON lines, at least for everything that only contains numbers, but that can wait until other dust has settled.)

Comment threadExport.lean Outdated
Modifies the export procedure and format to export the components of an
inductive declaration (the inductive specification(s), constructors,
and recursors) together.
Consumers are generally going to want to find these related declarations
anyway, and with the introduction of the auxiliary `.rec_<N>` recursors
motivated by nested inductive support, (in my opinion) this is a more
comprehensible format.
@ammkrn

Copy link
Copy Markdown
ContributorAuthor

I've added one commit 50db814 to implement the change mentioned elsewhere, packaging related elements of an inductive declaration.

@nomeata

nomeata commented Jan 13, 2026

Copy link
Copy Markdown
Contributor

Great! Are declarations now following the Lean.Declaration type consistently, so that a lean-using importer can parse to that type directly?

Hmm, it seems we now have a somewhat odd mix of Lean.Declaration and Lean.ConstantInfo. But if its a “best of both worlds” mix it’s maybe fine?

@ammkrn

Copy link
Copy Markdown
ContributorAuthor

Great! Are declarations now following the Lean.Declaration type consistently, so that a lean-using importer can parse to that type directly?

Hmm, it seems we now have a somewhat odd mix of Lean.Declaration and Lean.ConstantInfo. But if its a “best of both worlds” mix it’s maybe fine?

I'm not sure how one would be able to parse anything directly regardless of format since (a) we're using integer pointers and (b) it was already suggested that we make simplifying cuts that would prevent direct parsing, like removing "deBruijnIndex" and changing "binderName".

Lean.Declaration only includes what's strictly necessary for a "full" kernel implementation to do the work of type checking; for example it expects the consumer to both construct and check recursors. The very first exporter implementation followed that pattern.

In the previous round of format changes, it was decided that the exporter should include additional information (like "targets" for the recursors and rec rules, Eq/Quot declarations, etc.). With this additional info it's much easier to bootstrap new checkers, implement certain memory optimizations, and have things like parallel checkers.

Looking at Lean.Declaration, maybe it's worth it to similarly group mutual definitions as well.

@nomeata

Copy link
Copy Markdown
Contributor

I didn’t mean “parse generically”, but I meant “parse export to List Lean.Declaration” (so field name changes or indirections are not a blocker).

But thanks for the historical perspective, agreed on that count.

Comment threadExport.lean
@nomeatanomeata mentioned this pull request Jan 15, 2026
@nomeata

Copy link
Copy Markdown
Contributor

Looking at Lean.Declaration, maybe it's worth it to similarly group mutual definitions as well.

These are always ever unsafe, so not of high importance here? But in principle you are right.

Exports elements of mutual theorem and def/opaque blocks together.
@ammkrn

Copy link
Copy Markdown
ContributorAuthor

Looking at Lean.Declaration, maybe it's worth it to similarly group mutual definitions as well.

These are always ever unsafe, so not of high importance here? But in principle you are right.

Should be done in 7188ec9 modulo style issues.

@nomeata

Copy link
Copy Markdown
Contributor

Am I reading this right that it doesn't distinguish between a normal definition and a singly-recursive definition? It seems that of that's the case, import information is lost. I assume that's why Lean.Declaration is explicit in where the mutual groups are. But maybe lean forgets that information so it's hard to reproduce the declaration here?

@nomeata

Copy link
Copy Markdown
Contributor

Hmm, the Kernel certainly doesn’t seem to remember whether a declaration came from a .mutualDefnDecl or not, so by the time we have a ConstantInfo this information is lost.

Ah! But the kernel treats any unsafe declaration as recursive, even if not in a mutual block. Odd. But not partial declarations.

I find it confusing.

In any case, I think I’m leaning towards an export format that more closely matches Lean.Declaration here, and with a dedicated .mutual tag that is only used when Lean.Declaration.mutualDefnDecl would be used (namely when its unsafe or partial and all is not a singleton). This way parsers for checkers who do not support this can just skip the whole line. But I don't feel strongly about it.

@ammkrn

Copy link
Copy Markdown
ContributorAuthor

In any case, I think I’m leaning towards an export format that more closely matches Lean.Declaration here, and with a dedicated .mutual tag that is only used when Lean.Declaration.mutualDefnDecl would be used (namely when its unsafe or partial and all is not a singleton). This way parsers for checkers who do not support this can just skip the whole line. But I don't feel strongly about it.

I have no issue with that. To the earlier question, I think examination of all (specifically whether there's more than one name in there) would be the context clue.

Adds a separate metadata field for the format version, since it's not
necessarily the same as the exporter version. For example a performance
fix in the exporter might justify an exporter version bump, but not a
format version bump.
Tries to export `Nat` on the appearance of a nat literal, and Char.ofNat
and String.ofList on the appearance of a string literal.
@hargoniX
hargoniX merged commit 4f1e55e into leanprover:masterJan 23, 2026
1 check passed
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[RFC] Switch to JSON

3 participants

@ammkrn@nomeata@hargoniX
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Highlight search terms from Google/DuckDuckGo/Bing referrer\n(function() {\n var ref = document.referrer;\n var terms = [];\n \n if (ref.includes('google.com') || ref.includes('duckduckgo.com') || ref.includes('bing.com')) {\n var url = new URL(ref);\n var q = url.searchParams.get('q') || url.searchParams.get('p');\n if (q) {\n terms = q.split(/\\s+/).filter(function(t) { return t.length > 2; });\n }\n }\n \n if (terms.length === 0) return;\n \n var style = document.createElement('style');\n style.textContent = '.userscript-highlight { background: #fbbf24; color: #1a1a2e; padding: 1px 3px; border-radius: 2px; }';\n document.head.appendChild(style);\n \n function highlight(node) {\n if (node.nodeType === 3) { // text node\n var text = node.textContent;\n var found = false;\n terms.forEach(function(term) {\n var regex = new RegExp('(' + term.replace(/[.*+?^${}()|[\\]\\\\]/g, '\\\\') + ')', 'gi');\n if (regex.test(text)) {\n found = true;\n var frag = document.createDocumentFragment();\n var parts = text.split(regex);\n parts.forEach(function(part, i) {\n if (i % 2 === 0) {\n frag.appendChild(document.createTextNode(part));\n } else {\n var span = document.createElement('span');\n span.className = 'userscript-highlight';\n span.textContent = part;\n frag.appendChild(span);\n }\n });\n node.parentNode.replaceChild(frag, node);\n }\n });\n } else if (node.nodeType === 1 && node.childNodes) { // element\n var skipTags = ['SCRIPT', 'STYLE', 'NOSCRIPT', 'TEXTAREA', 'INPUT', 'SELECT'];\n if (!skipTags.includes(node.tagName)) {\n Array.from(node.childNodes).forEach(highlight);\n }\n }\n }\n \n highlight(document.body);\n \n // Re-highlight on dynamic content\n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1 || node.nodeType === 3) highlight(node);\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Highlight Search Terms"); } } catch(__e) { console.warn('[Userscript:Highlight Search Terms]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
Skip to content

Transition to ndjson export format - #10

Merged
hargoniX merged 8 commits into
leanprover:masterfrom
ammkrn:json_output
Jan 23, 2026
Merged

Transition to ndjson export format#10
hargoniX merged 8 commits into
leanprover:masterfrom
ammkrn:json_output

Conversation

@ammkrn

Copy link
Copy Markdown
Contributor

When ready, will close#3 (also see that issue for relevant discussion).

The format is described in format_ndjson and the README has been updated accordingly, along with the recommended command for invocation; the old README recommended invoking lake exe, which will not correctly set up the lake env in the most common use case, something like exporting mathlib.

Comment threadformat_ndjson.md Outdated
Expr.lit (Literal.strVal)
```
{
"lit": {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This does not seem to be what your program currently actually prints?

| .lit (.natVal i) => return .mkObj [("natVal", s!"{i}")]
| .lit (.strVal s) => return .mkObj [("strVal", s)]

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, I've pushed a correction for the format specification.

Comment threadformat_ndjson.md Outdated
"levelParams": Array<integer>,
"type": integer,
"value": integer,
"hints": Array<"opaque" | "abbrev" | integer>

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is also incorrectly documented afaict?

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ditto.

Corrects two errors in the format specification for string literals and
reducibility hints.
Comment threadformat_ndjson.md Outdated
@hargoniXhargoniX mentioned this pull request Jan 7, 2026
Correct a typo of `deBruijnIndex`
Comment threadExport.lean Outdated
Comment threadExport.lean Outdated
@ammkrn

Copy link
Copy Markdown
ContributorAuthor

The open comments should be resolved, and I believe I addressed the remaining points in #10

+ Namespaces backreferences from a generic "i" to "in", "il", and "ie" for name, level, and expr.
+ Abbreviates some json attribute names
@nomeata

nomeata commented Jan 9, 2026

Copy link
Copy Markdown
Contributor

Thanks!

(Seeing how slow lean4export is I’m inclined to rewrite the exporter to use simple string interpolation into hardcoded JSON lines, at least for everything that only contains numbers, but that can wait until other dust has settled.)

Comment threadExport.lean Outdated
Modifies the export procedure and format to export the components of an
inductive declaration (the inductive specification(s), constructors,
and recursors) together.
Consumers are generally going to want to find these related declarations
anyway, and with the introduction of the auxiliary `.rec_<N>` recursors
motivated by nested inductive support, (in my opinion) this is a more
comprehensible format.
@ammkrn

Copy link
Copy Markdown
ContributorAuthor

I've added one commit 50db814 to implement the change mentioned elsewhere, packaging related elements of an inductive declaration.

@nomeata

nomeata commented Jan 13, 2026

Copy link
Copy Markdown
Contributor

Great! Are declarations now following the Lean.Declaration type consistently, so that a lean-using importer can parse to that type directly?

Hmm, it seems we now have a somewhat odd mix of Lean.Declaration and Lean.ConstantInfo. But if its a “best of both worlds” mix it’s maybe fine?

@ammkrn

Copy link
Copy Markdown
ContributorAuthor

Great! Are declarations now following the Lean.Declaration type consistently, so that a lean-using importer can parse to that type directly?

Hmm, it seems we now have a somewhat odd mix of Lean.Declaration and Lean.ConstantInfo. But if its a “best of both worlds” mix it’s maybe fine?

I'm not sure how one would be able to parse anything directly regardless of format since (a) we're using integer pointers and (b) it was already suggested that we make simplifying cuts that would prevent direct parsing, like removing "deBruijnIndex" and changing "binderName".

Lean.Declaration only includes what's strictly necessary for a "full" kernel implementation to do the work of type checking; for example it expects the consumer to both construct and check recursors. The very first exporter implementation followed that pattern.

In the previous round of format changes, it was decided that the exporter should include additional information (like "targets" for the recursors and rec rules, Eq/Quot declarations, etc.). With this additional info it's much easier to bootstrap new checkers, implement certain memory optimizations, and have things like parallel checkers.

Looking at Lean.Declaration, maybe it's worth it to similarly group mutual definitions as well.

@nomeata

Copy link
Copy Markdown
Contributor

I didn’t mean “parse generically”, but I meant “parse export to List Lean.Declaration” (so field name changes or indirections are not a blocker).

But thanks for the historical perspective, agreed on that count.

Comment threadExport.lean
@nomeatanomeata mentioned this pull request Jan 15, 2026
@nomeata

Copy link
Copy Markdown
Contributor

Looking at Lean.Declaration, maybe it's worth it to similarly group mutual definitions as well.

These are always ever unsafe, so not of high importance here? But in principle you are right.

Exports elements of mutual theorem and def/opaque blocks together.
@ammkrn

Copy link
Copy Markdown
ContributorAuthor

Looking at Lean.Declaration, maybe it's worth it to similarly group mutual definitions as well.

These are always ever unsafe, so not of high importance here? But in principle you are right.

Should be done in 7188ec9 modulo style issues.

@nomeata

Copy link
Copy Markdown
Contributor

Am I reading this right that it doesn't distinguish between a normal definition and a singly-recursive definition? It seems that of that's the case, import information is lost. I assume that's why Lean.Declaration is explicit in where the mutual groups are. But maybe lean forgets that information so it's hard to reproduce the declaration here?

@nomeata

Copy link
Copy Markdown
Contributor

Hmm, the Kernel certainly doesn’t seem to remember whether a declaration came from a .mutualDefnDecl or not, so by the time we have a ConstantInfo this information is lost.

Ah! But the kernel treats any unsafe declaration as recursive, even if not in a mutual block. Odd. But not partial declarations.

I find it confusing.

In any case, I think I’m leaning towards an export format that more closely matches Lean.Declaration here, and with a dedicated .mutual tag that is only used when Lean.Declaration.mutualDefnDecl would be used (namely when its unsafe or partial and all is not a singleton). This way parsers for checkers who do not support this can just skip the whole line. But I don't feel strongly about it.

@ammkrn

Copy link
Copy Markdown
ContributorAuthor

In any case, I think I’m leaning towards an export format that more closely matches Lean.Declaration here, and with a dedicated .mutual tag that is only used when Lean.Declaration.mutualDefnDecl would be used (namely when its unsafe or partial and all is not a singleton). This way parsers for checkers who do not support this can just skip the whole line. But I don't feel strongly about it.

I have no issue with that. To the earlier question, I think examination of all (specifically whether there's more than one name in there) would be the context clue.

Adds a separate metadata field for the format version, since it's not
necessarily the same as the exporter version. For example a performance
fix in the exporter might justify an exporter version bump, but not a
format version bump.
Tries to export `Nat` on the appearance of a nat literal, and Char.ofNat
and String.ofList on the appearance of a string literal.
@hargoniX
hargoniX merged commit 4f1e55e into leanprover:masterJan 23, 2026
1 check passed
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[RFC] Switch to JSON

3 participants

@ammkrn@nomeata@hargoniX
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Strip utm_, fbclid, gclid, etc. from all links on page\n(function() {\n var trackingParams = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content',\n 'fbclid', 'gclid', 'dclid', 'msclkid', 'yclid',\n 'ref', 'ref_src', 'source', 'medium', 'campaign'];\n \n function cleanUrl(url) {\n try {\n var u = new URL(url, window.location.origin);\n var changed = false;\n trackingParams.forEach(function(p) {\n if (u.searchParams.has(p)) {\n u.searchParams.delete(p);\n changed = true;\n }\n });\n return changed ? u.toString() : url;\n } catch (e) {\n return url;\n }\n }\n \n function cleanLinks() {\n document.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n \n cleanLinks();\n \n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1) {\n if (node.tagName === 'A') cleanLinks();\n node.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Remove Tracking Parameters from Links"); } } catch(__e) { console.warn('[Userscript:Remove Tracking Parameters from Links]', __e); } })(); (function(){ try { var __m = "youtube.com"; var __re = new RegExp('^' + "youtube\\.com" + '
Skip to content

Transition to ndjson export format - #10

Merged
hargoniX merged 8 commits into
leanprover:masterfrom
ammkrn:json_output
Jan 23, 2026
Merged

Transition to ndjson export format#10
hargoniX merged 8 commits into
leanprover:masterfrom
ammkrn:json_output

Conversation

@ammkrn

Copy link
Copy Markdown
Contributor

When ready, will close#3 (also see that issue for relevant discussion).

The format is described in format_ndjson and the README has been updated accordingly, along with the recommended command for invocation; the old README recommended invoking lake exe, which will not correctly set up the lake env in the most common use case, something like exporting mathlib.

Comment threadformat_ndjson.md Outdated
Expr.lit (Literal.strVal)
```
{
"lit": {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This does not seem to be what your program currently actually prints?

| .lit (.natVal i) => return .mkObj [("natVal", s!"{i}")]
| .lit (.strVal s) => return .mkObj [("strVal", s)]

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, I've pushed a correction for the format specification.

Comment threadformat_ndjson.md Outdated
"levelParams": Array<integer>,
"type": integer,
"value": integer,
"hints": Array<"opaque" | "abbrev" | integer>

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is also incorrectly documented afaict?

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ditto.

Corrects two errors in the format specification for string literals and
reducibility hints.
Comment threadformat_ndjson.md Outdated
@hargoniXhargoniX mentioned this pull request Jan 7, 2026
Correct a typo of `deBruijnIndex`
Comment threadExport.lean Outdated
Comment threadExport.lean Outdated
@ammkrn

Copy link
Copy Markdown
ContributorAuthor

The open comments should be resolved, and I believe I addressed the remaining points in #10

+ Namespaces backreferences from a generic "i" to "in", "il", and "ie" for name, level, and expr.
+ Abbreviates some json attribute names
@nomeata

nomeata commented Jan 9, 2026

Copy link
Copy Markdown
Contributor

Thanks!

(Seeing how slow lean4export is I’m inclined to rewrite the exporter to use simple string interpolation into hardcoded JSON lines, at least for everything that only contains numbers, but that can wait until other dust has settled.)

Comment threadExport.lean Outdated
Modifies the export procedure and format to export the components of an
inductive declaration (the inductive specification(s), constructors,
and recursors) together.
Consumers are generally going to want to find these related declarations
anyway, and with the introduction of the auxiliary `.rec_<N>` recursors
motivated by nested inductive support, (in my opinion) this is a more
comprehensible format.
@ammkrn

Copy link
Copy Markdown
ContributorAuthor

I've added one commit 50db814 to implement the change mentioned elsewhere, packaging related elements of an inductive declaration.

@nomeata

nomeata commented Jan 13, 2026

Copy link
Copy Markdown
Contributor

Great! Are declarations now following the Lean.Declaration type consistently, so that a lean-using importer can parse to that type directly?

Hmm, it seems we now have a somewhat odd mix of Lean.Declaration and Lean.ConstantInfo. But if its a “best of both worlds” mix it’s maybe fine?

@ammkrn

Copy link
Copy Markdown
ContributorAuthor

Great! Are declarations now following the Lean.Declaration type consistently, so that a lean-using importer can parse to that type directly?

Hmm, it seems we now have a somewhat odd mix of Lean.Declaration and Lean.ConstantInfo. But if its a “best of both worlds” mix it’s maybe fine?

I'm not sure how one would be able to parse anything directly regardless of format since (a) we're using integer pointers and (b) it was already suggested that we make simplifying cuts that would prevent direct parsing, like removing "deBruijnIndex" and changing "binderName".

Lean.Declaration only includes what's strictly necessary for a "full" kernel implementation to do the work of type checking; for example it expects the consumer to both construct and check recursors. The very first exporter implementation followed that pattern.

In the previous round of format changes, it was decided that the exporter should include additional information (like "targets" for the recursors and rec rules, Eq/Quot declarations, etc.). With this additional info it's much easier to bootstrap new checkers, implement certain memory optimizations, and have things like parallel checkers.

Looking at Lean.Declaration, maybe it's worth it to similarly group mutual definitions as well.

@nomeata

Copy link
Copy Markdown
Contributor

I didn’t mean “parse generically”, but I meant “parse export to List Lean.Declaration” (so field name changes or indirections are not a blocker).

But thanks for the historical perspective, agreed on that count.

Comment threadExport.lean
@nomeatanomeata mentioned this pull request Jan 15, 2026
@nomeata

Copy link
Copy Markdown
Contributor

Looking at Lean.Declaration, maybe it's worth it to similarly group mutual definitions as well.

These are always ever unsafe, so not of high importance here? But in principle you are right.

Exports elements of mutual theorem and def/opaque blocks together.
@ammkrn

Copy link
Copy Markdown
ContributorAuthor

Looking at Lean.Declaration, maybe it's worth it to similarly group mutual definitions as well.

These are always ever unsafe, so not of high importance here? But in principle you are right.

Should be done in 7188ec9 modulo style issues.

@nomeata

Copy link
Copy Markdown
Contributor

Am I reading this right that it doesn't distinguish between a normal definition and a singly-recursive definition? It seems that of that's the case, import information is lost. I assume that's why Lean.Declaration is explicit in where the mutual groups are. But maybe lean forgets that information so it's hard to reproduce the declaration here?

@nomeata

Copy link
Copy Markdown
Contributor

Hmm, the Kernel certainly doesn’t seem to remember whether a declaration came from a .mutualDefnDecl or not, so by the time we have a ConstantInfo this information is lost.

Ah! But the kernel treats any unsafe declaration as recursive, even if not in a mutual block. Odd. But not partial declarations.

I find it confusing.

In any case, I think I’m leaning towards an export format that more closely matches Lean.Declaration here, and with a dedicated .mutual tag that is only used when Lean.Declaration.mutualDefnDecl would be used (namely when its unsafe or partial and all is not a singleton). This way parsers for checkers who do not support this can just skip the whole line. But I don't feel strongly about it.

@ammkrn

Copy link
Copy Markdown
ContributorAuthor

In any case, I think I’m leaning towards an export format that more closely matches Lean.Declaration here, and with a dedicated .mutual tag that is only used when Lean.Declaration.mutualDefnDecl would be used (namely when its unsafe or partial and all is not a singleton). This way parsers for checkers who do not support this can just skip the whole line. But I don't feel strongly about it.

I have no issue with that. To the earlier question, I think examination of all (specifically whether there's more than one name in there) would be the context clue.

Adds a separate metadata field for the format version, since it's not
necessarily the same as the exporter version. For example a performance
fix in the exporter might justify an exporter version bump, but not a
format version bump.
Tries to export `Nat` on the appearance of a nat literal, and Char.ofNat
and String.ofList on the appearance of a string literal.
@hargoniX
hargoniX merged commit 4f1e55e into leanprover:masterJan 23, 2026
1 check passed
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[RFC] Switch to JSON

3 participants

@ammkrn@nomeata@hargoniX
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Auto-enable theater mode on YouTube\n(function() {\n function tryTheater() {\n var btn = document.querySelector('button[aria-label=\"Theater mode\"], ytd-player #player button[title=\"Theater mode\"]');\n if (btn && !btn.classList.contains('activated')) {\n btn.click();\n }\n }\n \n // Try immediately\n tryTheater();\n \n // Try after navigation (SPA)\n var lastUrl = location.href;\n setInterval(function() {\n if (location.href !== lastUrl) {\n lastUrl = location.href;\n setTimeout(tryTheater, 500);\n }\n }, 1000);\n \n // Also try on player load\n var observer = new MutationObserver(tryTheater);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "YouTube Theater Mode Default"); } } catch(__e) { console.warn('[Userscript:YouTube Theater Mode Default]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
Skip to content

Transition to ndjson export format - #10

Merged
hargoniX merged 8 commits into
leanprover:masterfrom
ammkrn:json_output
Jan 23, 2026
Merged

Transition to ndjson export format#10
hargoniX merged 8 commits into
leanprover:masterfrom
ammkrn:json_output

Conversation

@ammkrn

Copy link
Copy Markdown
Contributor

When ready, will close#3 (also see that issue for relevant discussion).

The format is described in format_ndjson and the README has been updated accordingly, along with the recommended command for invocation; the old README recommended invoking lake exe, which will not correctly set up the lake env in the most common use case, something like exporting mathlib.

Comment threadformat_ndjson.md Outdated
Expr.lit (Literal.strVal)
```
{
"lit": {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This does not seem to be what your program currently actually prints?

| .lit (.natVal i) => return .mkObj [("natVal", s!"{i}")]
| .lit (.strVal s) => return .mkObj [("strVal", s)]

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, I've pushed a correction for the format specification.

Comment threadformat_ndjson.md Outdated
"levelParams": Array<integer>,
"type": integer,
"value": integer,
"hints": Array<"opaque" | "abbrev" | integer>

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is also incorrectly documented afaict?

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ditto.

Corrects two errors in the format specification for string literals and
reducibility hints.
Comment threadformat_ndjson.md Outdated
@hargoniXhargoniX mentioned this pull request Jan 7, 2026
Correct a typo of `deBruijnIndex`
Comment threadExport.lean Outdated
Comment threadExport.lean Outdated
@ammkrn

Copy link
Copy Markdown
ContributorAuthor

The open comments should be resolved, and I believe I addressed the remaining points in #10

+ Namespaces backreferences from a generic "i" to "in", "il", and "ie" for name, level, and expr.
+ Abbreviates some json attribute names
@nomeata

nomeata commented Jan 9, 2026

Copy link
Copy Markdown
Contributor

Thanks!

(Seeing how slow lean4export is I’m inclined to rewrite the exporter to use simple string interpolation into hardcoded JSON lines, at least for everything that only contains numbers, but that can wait until other dust has settled.)

Comment threadExport.lean Outdated
Modifies the export procedure and format to export the components of an
inductive declaration (the inductive specification(s), constructors,
and recursors) together.
Consumers are generally going to want to find these related declarations
anyway, and with the introduction of the auxiliary `.rec_<N>` recursors
motivated by nested inductive support, (in my opinion) this is a more
comprehensible format.
@ammkrn

Copy link
Copy Markdown
ContributorAuthor

I've added one commit 50db814 to implement the change mentioned elsewhere, packaging related elements of an inductive declaration.

@nomeata

nomeata commented Jan 13, 2026

Copy link
Copy Markdown
Contributor

Great! Are declarations now following the Lean.Declaration type consistently, so that a lean-using importer can parse to that type directly?

Hmm, it seems we now have a somewhat odd mix of Lean.Declaration and Lean.ConstantInfo. But if its a “best of both worlds” mix it’s maybe fine?

@ammkrn

Copy link
Copy Markdown
ContributorAuthor

Great! Are declarations now following the Lean.Declaration type consistently, so that a lean-using importer can parse to that type directly?

Hmm, it seems we now have a somewhat odd mix of Lean.Declaration and Lean.ConstantInfo. But if its a “best of both worlds” mix it’s maybe fine?

I'm not sure how one would be able to parse anything directly regardless of format since (a) we're using integer pointers and (b) it was already suggested that we make simplifying cuts that would prevent direct parsing, like removing "deBruijnIndex" and changing "binderName".

Lean.Declaration only includes what's strictly necessary for a "full" kernel implementation to do the work of type checking; for example it expects the consumer to both construct and check recursors. The very first exporter implementation followed that pattern.

In the previous round of format changes, it was decided that the exporter should include additional information (like "targets" for the recursors and rec rules, Eq/Quot declarations, etc.). With this additional info it's much easier to bootstrap new checkers, implement certain memory optimizations, and have things like parallel checkers.

Looking at Lean.Declaration, maybe it's worth it to similarly group mutual definitions as well.

@nomeata

Copy link
Copy Markdown
Contributor

I didn’t mean “parse generically”, but I meant “parse export to List Lean.Declaration” (so field name changes or indirections are not a blocker).

But thanks for the historical perspective, agreed on that count.

Comment threadExport.lean
@nomeatanomeata mentioned this pull request Jan 15, 2026
@nomeata

Copy link
Copy Markdown
Contributor

Looking at Lean.Declaration, maybe it's worth it to similarly group mutual definitions as well.

These are always ever unsafe, so not of high importance here? But in principle you are right.

Exports elements of mutual theorem and def/opaque blocks together.
@ammkrn

Copy link
Copy Markdown
ContributorAuthor

Looking at Lean.Declaration, maybe it's worth it to similarly group mutual definitions as well.

These are always ever unsafe, so not of high importance here? But in principle you are right.

Should be done in 7188ec9 modulo style issues.

@nomeata

Copy link
Copy Markdown
Contributor

Am I reading this right that it doesn't distinguish between a normal definition and a singly-recursive definition? It seems that of that's the case, import information is lost. I assume that's why Lean.Declaration is explicit in where the mutual groups are. But maybe lean forgets that information so it's hard to reproduce the declaration here?

@nomeata

Copy link
Copy Markdown
Contributor

Hmm, the Kernel certainly doesn’t seem to remember whether a declaration came from a .mutualDefnDecl or not, so by the time we have a ConstantInfo this information is lost.

Ah! But the kernel treats any unsafe declaration as recursive, even if not in a mutual block. Odd. But not partial declarations.

I find it confusing.

In any case, I think I’m leaning towards an export format that more closely matches Lean.Declaration here, and with a dedicated .mutual tag that is only used when Lean.Declaration.mutualDefnDecl would be used (namely when its unsafe or partial and all is not a singleton). This way parsers for checkers who do not support this can just skip the whole line. But I don't feel strongly about it.

@ammkrn

Copy link
Copy Markdown
ContributorAuthor

In any case, I think I’m leaning towards an export format that more closely matches Lean.Declaration here, and with a dedicated .mutual tag that is only used when Lean.Declaration.mutualDefnDecl would be used (namely when its unsafe or partial and all is not a singleton). This way parsers for checkers who do not support this can just skip the whole line. But I don't feel strongly about it.

I have no issue with that. To the earlier question, I think examination of all (specifically whether there's more than one name in there) would be the context clue.

Adds a separate metadata field for the format version, since it's not
necessarily the same as the exporter version. For example a performance
fix in the exporter might justify an exporter version bump, but not a
format version bump.
Tries to export `Nat` on the appearance of a nat literal, and Char.ofNat
and String.ofList on the appearance of a string literal.
@hargoniX
hargoniX merged commit 4f1e55e into leanprover:masterJan 23, 2026
1 check passed
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[RFC] Switch to JSON

3 participants

@ammkrn@nomeata@hargoniX
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Remove or un-stick sticky/fixed headers that block content\n(function() {\n function unstick() {\n document.querySelectorAll('header, nav, [role=\"banner\"], .header, .navbar, .sticky, .fixed-top, [style*=\"position: fixed\"], [style*=\"position:sticky\"]').forEach(function(el) {\n if (el.style.position === 'fixed' || el.style.position === 'sticky' || \n getComputedStyle(el).position === 'fixed' || getComputedStyle(el).position === 'sticky') {\n el.style.position = 'static';\n el.style.top = 'auto';\n el.style.zIndex = 'auto';\n }\n });\n }\n \n unstick();\n \n var observer = new MutationObserver(unstick);\n observer.observe(document.body, { childList: true, subtree: true, attributes: true, attributeFilter: ['style', 'class'] });\n})();", "Kill Sticky Headers"); } } catch(__e) { console.warn('[Userscript:Kill Sticky Headers]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
Skip to content

Transition to ndjson export format - #10

Merged
hargoniX merged 8 commits into
leanprover:masterfrom
ammkrn:json_output
Jan 23, 2026
Merged

Transition to ndjson export format#10
hargoniX merged 8 commits into
leanprover:masterfrom
ammkrn:json_output

Conversation

@ammkrn

Copy link
Copy Markdown
Contributor

When ready, will close#3 (also see that issue for relevant discussion).

The format is described in format_ndjson and the README has been updated accordingly, along with the recommended command for invocation; the old README recommended invoking lake exe, which will not correctly set up the lake env in the most common use case, something like exporting mathlib.

Comment threadformat_ndjson.md Outdated
Expr.lit (Literal.strVal)
```
{
"lit": {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This does not seem to be what your program currently actually prints?

| .lit (.natVal i) => return .mkObj [("natVal", s!"{i}")]
| .lit (.strVal s) => return .mkObj [("strVal", s)]

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, I've pushed a correction for the format specification.

Comment threadformat_ndjson.md Outdated
"levelParams": Array<integer>,
"type": integer,
"value": integer,
"hints": Array<"opaque" | "abbrev" | integer>

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is also incorrectly documented afaict?

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ditto.

Corrects two errors in the format specification for string literals and
reducibility hints.
Comment threadformat_ndjson.md Outdated
@hargoniXhargoniX mentioned this pull request Jan 7, 2026
Correct a typo of `deBruijnIndex`
Comment threadExport.lean Outdated
Comment threadExport.lean Outdated
@ammkrn

Copy link
Copy Markdown
ContributorAuthor

The open comments should be resolved, and I believe I addressed the remaining points in #10

+ Namespaces backreferences from a generic "i" to "in", "il", and "ie" for name, level, and expr.
+ Abbreviates some json attribute names
@nomeata

nomeata commented Jan 9, 2026

Copy link
Copy Markdown
Contributor

Thanks!

(Seeing how slow lean4export is I’m inclined to rewrite the exporter to use simple string interpolation into hardcoded JSON lines, at least for everything that only contains numbers, but that can wait until other dust has settled.)

Comment threadExport.lean Outdated
Modifies the export procedure and format to export the components of an
inductive declaration (the inductive specification(s), constructors,
and recursors) together.
Consumers are generally going to want to find these related declarations
anyway, and with the introduction of the auxiliary `.rec_<N>` recursors
motivated by nested inductive support, (in my opinion) this is a more
comprehensible format.
@ammkrn

Copy link
Copy Markdown
ContributorAuthor

I've added one commit 50db814 to implement the change mentioned elsewhere, packaging related elements of an inductive declaration.

@nomeata

nomeata commented Jan 13, 2026

Copy link
Copy Markdown
Contributor

Great! Are declarations now following the Lean.Declaration type consistently, so that a lean-using importer can parse to that type directly?

Hmm, it seems we now have a somewhat odd mix of Lean.Declaration and Lean.ConstantInfo. But if its a “best of both worlds” mix it’s maybe fine?

@ammkrn

Copy link
Copy Markdown
ContributorAuthor

Great! Are declarations now following the Lean.Declaration type consistently, so that a lean-using importer can parse to that type directly?

Hmm, it seems we now have a somewhat odd mix of Lean.Declaration and Lean.ConstantInfo. But if its a “best of both worlds” mix it’s maybe fine?

I'm not sure how one would be able to parse anything directly regardless of format since (a) we're using integer pointers and (b) it was already suggested that we make simplifying cuts that would prevent direct parsing, like removing "deBruijnIndex" and changing "binderName".

Lean.Declaration only includes what's strictly necessary for a "full" kernel implementation to do the work of type checking; for example it expects the consumer to both construct and check recursors. The very first exporter implementation followed that pattern.

In the previous round of format changes, it was decided that the exporter should include additional information (like "targets" for the recursors and rec rules, Eq/Quot declarations, etc.). With this additional info it's much easier to bootstrap new checkers, implement certain memory optimizations, and have things like parallel checkers.

Looking at Lean.Declaration, maybe it's worth it to similarly group mutual definitions as well.

@nomeata

Copy link
Copy Markdown
Contributor

I didn’t mean “parse generically”, but I meant “parse export to List Lean.Declaration” (so field name changes or indirections are not a blocker).

But thanks for the historical perspective, agreed on that count.

Comment threadExport.lean
@nomeatanomeata mentioned this pull request Jan 15, 2026
@nomeata

Copy link
Copy Markdown
Contributor

Looking at Lean.Declaration, maybe it's worth it to similarly group mutual definitions as well.

These are always ever unsafe, so not of high importance here? But in principle you are right.

Exports elements of mutual theorem and def/opaque blocks together.
@ammkrn

Copy link
Copy Markdown
ContributorAuthor

Looking at Lean.Declaration, maybe it's worth it to similarly group mutual definitions as well.

These are always ever unsafe, so not of high importance here? But in principle you are right.

Should be done in 7188ec9 modulo style issues.

@nomeata

Copy link
Copy Markdown
Contributor

Am I reading this right that it doesn't distinguish between a normal definition and a singly-recursive definition? It seems that of that's the case, import information is lost. I assume that's why Lean.Declaration is explicit in where the mutual groups are. But maybe lean forgets that information so it's hard to reproduce the declaration here?

@nomeata

Copy link
Copy Markdown
Contributor

Hmm, the Kernel certainly doesn’t seem to remember whether a declaration came from a .mutualDefnDecl or not, so by the time we have a ConstantInfo this information is lost.

Ah! But the kernel treats any unsafe declaration as recursive, even if not in a mutual block. Odd. But not partial declarations.

I find it confusing.

In any case, I think I’m leaning towards an export format that more closely matches Lean.Declaration here, and with a dedicated .mutual tag that is only used when Lean.Declaration.mutualDefnDecl would be used (namely when its unsafe or partial and all is not a singleton). This way parsers for checkers who do not support this can just skip the whole line. But I don't feel strongly about it.

@ammkrn

Copy link
Copy Markdown
ContributorAuthor

In any case, I think I’m leaning towards an export format that more closely matches Lean.Declaration here, and with a dedicated .mutual tag that is only used when Lean.Declaration.mutualDefnDecl would be used (namely when its unsafe or partial and all is not a singleton). This way parsers for checkers who do not support this can just skip the whole line. But I don't feel strongly about it.

I have no issue with that. To the earlier question, I think examination of all (specifically whether there's more than one name in there) would be the context clue.

Adds a separate metadata field for the format version, since it's not
necessarily the same as the exporter version. For example a performance
fix in the exporter might justify an exporter version bump, but not a
format version bump.
Tries to export `Nat` on the appearance of a nat literal, and Char.ofNat
and String.ofList on the appearance of a string literal.
@hargoniX
hargoniX merged commit 4f1e55e into leanprover:masterJan 23, 2026
1 check passed
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[RFC] Switch to JSON

3 participants

@ammkrn@nomeata@hargoniX
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Universal Dark Mode - works on any site\n(function() {\n var enabled = true;\n \n function applyDarkMode() {\n if (!enabled) return;\n \n // Create style element if it doesn't exist\n var style = document.getElementById('universal-dark-mode-style');\n if (!style) {\n style = document.createElement('style');\n style.id = 'universal-dark-mode-style';\n document.head.appendChild(style);\n }\n \n // Dark mode CSS - inverts colors but preserves images/video\n style.textContent = '\n /* Invert everything except media */\n html {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #1a1a2e !important;\n }\n \n /* Restore images, videos, iframes, canvas */\n img, video, iframe, canvas, svg, picture, [style*=\"background-image\"] {\n filter: invert(1) hue-rotate(180deg) !important;\n }\n \n /* Preserve specific elements that should not be inverted */\n .no-dark-mode, .no-dark-mode *,\n [data-theme=\"light\"], [data-theme=\"light\"],\n .ace_editor, .ace_editor *,\n .CodeMirror, .CodeMirror *,\n .monaco-editor, .monaco-editor *,\n .markdown-body pre, .markdown-body pre *,\n .highlight, .highlight *,\n pre code, pre code * {\n filter: none !important;\n }\n \n /* Fix common UI elements */\n .modal, .popup, .dropdown-menu, .tooltip, .popover {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #2d2d44 !important;\n border-color: #444 !important;\n }\n \n /* Scrollbars */\n ::-webkit-scrollbar { background: #1a1a2e !important; }\n ::-webkit-scrollbar-thumb { background: #444 !important; }\n ::-webkit-scrollbar-thumb:hover { background: #555 !important; }\n \n /* Selection */\n ::selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ::-moz-selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ';\n }\n \n function removeDarkMode() {\n var style = document.getElementById('universal-dark-mode-style');\n if (style) style.remove();\n }\n \n // Toggle with Alt+Shift+D\n document.addEventListener('keydown', function(e) {\n if (e.altKey && e.shiftKey && e.key === 'D') {\n e.preventDefault();\n enabled = !enabled;\n if (enabled) {\n applyDarkMode();\n console.log('[Universal Dark Mode] Enabled');\n } else {\n removeDarkMode();\n console.log('[Universal Dark Mode] Disabled');\n }\n }\n });\n \n // Apply on load\n applyDarkMode();\n \n // Re-apply on dynamic content\n var observer = new MutationObserver(function(mutations) {\n if (enabled && !document.getElementById('universal-dark-mode-style')) {\n applyDarkMode();\n }\n });\n observer.observe(document.head, { childList: true });\n \n console.log('[Universal Dark Mode] Loaded - Press Alt+Shift+D to toggle');\n})();", "Universal Dark Mode"); } } catch(__e) { console.warn('[Userscript:Universal Dark Mode]', __e); } })(); })();
Skip to content

Transition to ndjson export format - #10

Merged
hargoniX merged 8 commits into
leanprover:masterfrom
ammkrn:json_output
Jan 23, 2026
Merged

Transition to ndjson export format#10
hargoniX merged 8 commits into
leanprover:masterfrom
ammkrn:json_output

Conversation

@ammkrn

Copy link
Copy Markdown
Contributor

When ready, will close#3 (also see that issue for relevant discussion).

The format is described in format_ndjson and the README has been updated accordingly, along with the recommended command for invocation; the old README recommended invoking lake exe, which will not correctly set up the lake env in the most common use case, something like exporting mathlib.

Comment threadformat_ndjson.md Outdated
Expr.lit (Literal.strVal)
```
{
"lit": {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This does not seem to be what your program currently actually prints?

| .lit (.natVal i) => return .mkObj [("natVal", s!"{i}")]
| .lit (.strVal s) => return .mkObj [("strVal", s)]

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, I've pushed a correction for the format specification.

Comment threadformat_ndjson.md Outdated
"levelParams": Array<integer>,
"type": integer,
"value": integer,
"hints": Array<"opaque" | "abbrev" | integer>

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is also incorrectly documented afaict?

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ditto.

Corrects two errors in the format specification for string literals and
reducibility hints.
Comment threadformat_ndjson.md Outdated
@hargoniXhargoniX mentioned this pull request Jan 7, 2026
Correct a typo of `deBruijnIndex`
Comment threadExport.lean Outdated
Comment threadExport.lean Outdated
@ammkrn

Copy link
Copy Markdown
ContributorAuthor

The open comments should be resolved, and I believe I addressed the remaining points in #10

+ Namespaces backreferences from a generic "i" to "in", "il", and "ie" for name, level, and expr.
+ Abbreviates some json attribute names
@nomeata

nomeata commented Jan 9, 2026

Copy link
Copy Markdown
Contributor

Thanks!

(Seeing how slow lean4export is I’m inclined to rewrite the exporter to use simple string interpolation into hardcoded JSON lines, at least for everything that only contains numbers, but that can wait until other dust has settled.)

Comment threadExport.lean Outdated
Modifies the export procedure and format to export the components of an
inductive declaration (the inductive specification(s), constructors,
and recursors) together.
Consumers are generally going to want to find these related declarations
anyway, and with the introduction of the auxiliary `.rec_<N>` recursors
motivated by nested inductive support, (in my opinion) this is a more
comprehensible format.
@ammkrn

Copy link
Copy Markdown
ContributorAuthor

I've added one commit 50db814 to implement the change mentioned elsewhere, packaging related elements of an inductive declaration.

@nomeata

nomeata commented Jan 13, 2026

Copy link
Copy Markdown
Contributor

Great! Are declarations now following the Lean.Declaration type consistently, so that a lean-using importer can parse to that type directly?

Hmm, it seems we now have a somewhat odd mix of Lean.Declaration and Lean.ConstantInfo. But if its a “best of both worlds” mix it’s maybe fine?

@ammkrn

Copy link
Copy Markdown
ContributorAuthor

Great! Are declarations now following the Lean.Declaration type consistently, so that a lean-using importer can parse to that type directly?

Hmm, it seems we now have a somewhat odd mix of Lean.Declaration and Lean.ConstantInfo. But if its a “best of both worlds” mix it’s maybe fine?

I'm not sure how one would be able to parse anything directly regardless of format since (a) we're using integer pointers and (b) it was already suggested that we make simplifying cuts that would prevent direct parsing, like removing "deBruijnIndex" and changing "binderName".

Lean.Declaration only includes what's strictly necessary for a "full" kernel implementation to do the work of type checking; for example it expects the consumer to both construct and check recursors. The very first exporter implementation followed that pattern.

In the previous round of format changes, it was decided that the exporter should include additional information (like "targets" for the recursors and rec rules, Eq/Quot declarations, etc.). With this additional info it's much easier to bootstrap new checkers, implement certain memory optimizations, and have things like parallel checkers.

Looking at Lean.Declaration, maybe it's worth it to similarly group mutual definitions as well.

@nomeata

Copy link
Copy Markdown
Contributor

I didn’t mean “parse generically”, but I meant “parse export to List Lean.Declaration” (so field name changes or indirections are not a blocker).

But thanks for the historical perspective, agreed on that count.

Comment threadExport.lean
@nomeatanomeata mentioned this pull request Jan 15, 2026
@nomeata

Copy link
Copy Markdown
Contributor

Looking at Lean.Declaration, maybe it's worth it to similarly group mutual definitions as well.

These are always ever unsafe, so not of high importance here? But in principle you are right.

Exports elements of mutual theorem and def/opaque blocks together.
@ammkrn

Copy link
Copy Markdown
ContributorAuthor

Looking at Lean.Declaration, maybe it's worth it to similarly group mutual definitions as well.

These are always ever unsafe, so not of high importance here? But in principle you are right.

Should be done in 7188ec9 modulo style issues.

@nomeata

Copy link
Copy Markdown
Contributor

Am I reading this right that it doesn't distinguish between a normal definition and a singly-recursive definition? It seems that of that's the case, import information is lost. I assume that's why Lean.Declaration is explicit in where the mutual groups are. But maybe lean forgets that information so it's hard to reproduce the declaration here?

@nomeata

Copy link
Copy Markdown
Contributor

Hmm, the Kernel certainly doesn’t seem to remember whether a declaration came from a .mutualDefnDecl or not, so by the time we have a ConstantInfo this information is lost.

Ah! But the kernel treats any unsafe declaration as recursive, even if not in a mutual block. Odd. But not partial declarations.

I find it confusing.

In any case, I think I’m leaning towards an export format that more closely matches Lean.Declaration here, and with a dedicated .mutual tag that is only used when Lean.Declaration.mutualDefnDecl would be used (namely when its unsafe or partial and all is not a singleton). This way parsers for checkers who do not support this can just skip the whole line. But I don't feel strongly about it.

@ammkrn

Copy link
Copy Markdown
ContributorAuthor

In any case, I think I’m leaning towards an export format that more closely matches Lean.Declaration here, and with a dedicated .mutual tag that is only used when Lean.Declaration.mutualDefnDecl would be used (namely when its unsafe or partial and all is not a singleton). This way parsers for checkers who do not support this can just skip the whole line. But I don't feel strongly about it.

I have no issue with that. To the earlier question, I think examination of all (specifically whether there's more than one name in there) would be the context clue.

Adds a separate metadata field for the format version, since it's not
necessarily the same as the exporter version. For example a performance
fix in the exporter might justify an exporter version bump, but not a
format version bump.
Tries to export `Nat` on the appearance of a nat literal, and Char.ofNat
and String.ofList on the appearance of a string literal.
@hargoniX
hargoniX merged commit 4f1e55e into leanprover:masterJan 23, 2026
1 check passed
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[RFC] Switch to JSON

3 participants

@ammkrn@nomeata@hargoniX