Add an auto-generated unicode character category file - #4605

Merged
tnull merged 1 commit into
lightningdevkit:mainfrom
TheBlueMatt:2026-05-unicode-autogen
May 18, 2026
Merged

Add an auto-generated unicode character category file#4605
tnull merged 1 commit into
lightningdevkit:mainfrom
TheBlueMatt:2026-05-unicode-autogen

Conversation

@TheBlueMatt

Copy link
Copy Markdown
Collaborator

1a01b5a added detection of unicode format characters in PrintableString, but used a hard-coded table which may eventually become out of date.

Here we switch to an auto-generated table, include all General_CategoryOther characters, and also ban unallocated code points.

Finally, CI validates that the file is kept up to date.

Written by Claude

@ldk-reviews-bot

ldk-reviews-bot commented May 7, 2026

Copy link
Copy Markdown

👋 Thanks for assigning @tnull as a reviewer!
I'll wait for their review and will help manage the review process.
Once they submit their review, I'll check if a second reviewer would be helpful.

@TheBlueMatt
TheBlueMatt requested a review from tnullMay 7, 2026 18:47
@TheBlueMatt
TheBlueMattforce-pushed the 2026-05-unicode-autogen branch from b6f8c03 to bd75483CompareMay 7, 2026 18:49
Comment on lines +36 to +38
let is_other = is_unicode_general_category_other(c);
let is_unassigned = is_unicode_general_category_unassigned(c);
let c = if c.is_control() || is_other || is_unassigned {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: c.is_control() is now fully redundant — it checks Cc (Control), which is already covered by is_unicode_general_category_other (see 0x0000..=0x001F and 0x007F..=0x009F in unicode.rs). The old code needed it because is_format_char only covered Cf, but the new function covers all of Cc / Cf / Cs / Co.

Not a bug (the || short-circuits harmlessly), but it's potentially confusing because it suggests is_other doesn't handle control characters.

Suggested change
let is_other = is_unicode_general_category_other(c);
let is_unassigned = is_unicode_general_category_unassigned(c);
let c = if c.is_control() || is_other || is_unassigned {
let c = ifis_unicode_general_category_other(c) || is_unicode_general_category_unassigned(c){

@ldk-claude-review-bot

ldk-claude-review-bot commented May 7, 2026

Copy link
Copy Markdown
Collaborator

Review Summary

New issue found

  • .github/workflows/check_unicode.yml:25gh issue create runs without deduplication; will create a new issue every day the file is out of date. build.yml already demonstrates the correct pattern (check for existing open issue, comment on it instead).

Correction to prior review

My three prior comments about the YAML workflow file (missing jobs: key, missing checkout step, fi indentation) were incorrect — the file structure is fine as written. I misread the diff indentation. Apologies for the noise.

Prior comment still applicable

  • lightning-types/src/string.rs:38c.is_control() remains redundant with is_unicode_general_category_other, which already covers all Cc characters.

@codecov

codecovBot commented May 7, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 92.23301% with 40 lines in your changes missing coverage. Please review.
✅ Project coverage is 86.15%. Comparing base (1a01b5a) to head (65e8cc8).
⚠️ Report is 66 commits behind head on main.

Files with missing linesPatch %Lines
lightning-types/src/unicode.rs92.18%40 Missing ⚠️
Additional details and impacted files
@@ Coverage Diff @@## main #4605 +/- ##
==========================================
+ Coverage 86.09% 86.15% +0.06% 
==========================================
Files 157 158 +1 Lines 108828 109323 +495 Branches 108828 109323 +495 ==========================================
+ Hits 93694 94189 +495 + Misses 12519 12518 -1 - Partials 2615 2616 +1 
FlagCoverage Δ
tests86.15% <92.23%> (+0.06%)⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@tnulltnull left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Alright, seems maybe a bit excessive, but fine by me. No need to check this on every CI run though, let's just move it to a weekly cronjob?

Comment thread.github/workflows/build.yml Outdated

@tnulltnull left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good, please squash.

Comment thread.github/workflows/check_unicode.yml
Comment thread.github/workflows/check_unicode.yml
Comment thread.github/workflows/check_unicode.yml Outdated
@tnull

tnull commented May 8, 2026

Copy link
Copy Markdown
Contributor

Claude's reviews seem legit though

1a01b5a added detection of unicode
format characters in `PrintableString`, but used a hard-coded table
which may eventually become out of date.
Here we switch to an auto-generated table, include all
`General_Category` `Other` characters, and also ban unallocated
code points.
Finally, CI validates that the file is kept up to date.
Written by Claude
@TheBlueMatt
TheBlueMattforce-pushed the 2026-05-unicode-autogen branch from 2de5954 to 65e8cc8CompareMay 8, 2026 19:44
@TheBlueMatt

Copy link
Copy Markdown
CollaboratorAuthor

Squashed and fixed claude's comments:

$ git diff-tree -U2 2de5954d9 65e8cc8d5
diff --git a/.github/workflows/check_unicode.yml b/.github/workflows/check_unicode.yml
index c22a4e3b29..a01add3f81 100644
--- a/.github/workflows/check_unicode.yml+++ b/.github/workflows/check_unicode.yml@@ -5,4 +5,5 @@ on:
- cron: '42 3 * * *'
+jobs:
check-unicode:
runs-on: ubuntu-latest
@@ -10,4 +11,6 @@ on:
issues: write
steps:
+ - name: Checkout source code+ uses: actions/checkout@v4
- name: Check unicode file state
env:
@@ -21,3 +24,3 @@ on:
BODY="The unicode character listing is out of date, see $RUN_URL"
gh issue create --title "$TITLE" --body "$BODY"
- fi+ fi

TITLE="Unicode listing out of date: ${{ github.workflow }}"
RUN_URL="https://github.com/${{ github.repository }}/actions/runs/${{ github.run_id }}"
BODY="The unicode character listing is out of date, see $RUN_URL"
gh issue create --title "$TITLE" --body "$BODY"

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bug: No deduplication — this creates a new issue every day the file is out of date. Unicode updates are infrequent (roughly annual), but if nobody regenerates the file promptly, this will spam the issue tracker with identical issues.

build.yml already has the right pattern (lines 380-391): check for an existing open issue with a label, and comment on it instead of creating a duplicate. This workflow should do the same.

Minimal fix sketch:

LABEL="unicode outdated"gh label create "$LABEL" 2>/dev/null || trueEXISTING=$(gh issue list --label "$LABEL" --state open --json number --jq '.[0].number // empty')if [ -n "$EXISTING" ]; thengh issue comment "$EXISTING" --body "$BODY"elsegh issue create --title "$TITLE" --label "$LABEL" --body "$BODY"fi

Copy link
Copy Markdown
CollaboratorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Eh, that takes effort, this is fine we expect to respond quickly.

@tnull
tnull merged commit 71fdc27 into lightningdevkit:mainMay 18, 2026
23 of 24 checks passed
@TheBlueMatt

Copy link
Copy Markdown
CollaboratorAuthor

Backported to 0.1 in #4680.

@TheBlueMatt

Copy link
Copy Markdown
CollaboratorAuthor

Backported to 0.2 in #4683.

TheBlueMatt added a commit that referenced this pull request Jun 19, 2026
v0.1.10 - Jun 18, 2026 - "Loupe de Loupe"
API Updates
===========
* `DefaultMessageRouter` will now always generate blinded message paths that
provide no privacy (where our node is the introduction node) for nodes with
public channels. This works around an issue which will appear for any nodes
with LND peers that enable onion messaging - such peers will refuse to
forward BOLT 12 messages from unknown third parties, which most BOLT 12
payers rely on today (#4647).
* Explicit `amount_msats` of 0 is rejected in BOLT 12 `Offer`s; `OfferBuilder`
now maps 0-amounts to an amount of `None` (#4324).
Bug Fixes
=========
* Async `ChannelMonitorUpdate` persistence operations which complete, but are
not marked as complete in a persisted `ChannelManager` prior to restart,
followed immediately by a block connection and then another restart could
result in some channel operations hanging leading for force-closures (#4377).
* If an MPP payment is claimed but `ChannelMonitorUpdate`s for some parts are
still being completed asynchronously, further channel updates (e.g.
forwarding another payment) are pending and the node restarts, the channel
could have become stuck (#4520).
* The presence of unconfirmed transactions actually no longer causes
`ElectrumSyncClient` to spuriously fail to sync (#4590).
* `FilesystemStore::list_all_keys` will no longer fail if there are stale
intermediate files lying around from a previous unclean shutdown (#4618).
* When forwarding an HTLC while in a blinded path with proportional fees over
200%, LDK will no longer spuriously allow a forward that pays us 1 msat too
little in fees (#4697).
* Fixed a rare case where a channel could get stuck on reconnect when using
both async `ChannelMonitorUpdate` persistence and async signing (#4684).
* `Event::PaymentSent::fee_paid_msat` is no longer `None` in cases where
`ChannelManager::abandon_payment` was called before the payment ultimately
completes anyway (#4651).
* Syncing a `ChainMonitor` using the `Confirm` trait will no longer write some
full `ChannelMonitor`s to disk several times per block (#4544).
* `OMDomainResolver` now correctly accounts for failed queries when rate
limiting, ensuring we continue to respond to queries after failures (#4591).
* Calling `ChannelManager::send_payment_with_route` without a `route_params`
and with an invalid `Route` will no longer panic (#4707).
* `lightning-custom-message`'s handling of `peer_connected` events now ensures
that sub-handlers will see a `peer_disconnected` event if a different
sub-handler refused the connection by `Err`ing `peer_connected` (#4595).
* Incomplete MPP keysend payments will no longer see their HTLCs held until
expiry (#4558).
* `InvoiceRequestBuilder` will no longer accept a `quantity` of `0` for a
BOLT 12 `Offer`, allowing any quantity up to a bound (#4667).
* `lightning-custom-message` handlers that return `Ok(None)` when asked to
deserialize a message in their defined range no longer cause panics (#4709).
* Several spurious debug assertions were fixed (#4537, #4618).
Security
========
0.1.10 fixes a sanitization issue and several denial-of-service vulnerabilities.
* `Bolt11Invoice::recover_payee_pub_key` no longer panics if called on an
invoice which set an explicit public key, rather than relying on public key
recovery. This method is called from `payment_parameters_from_invoice` and
`payment_parameters_from_variable_amount_invoice` (#4717).
* Maliciously-crafted unpayable invoices which have overflowing feerates will
no longer cause an `unwrap` failure panic (#4716).
* `possiblyrandom` did not properly generate random data except when it was
explicitly configured to. By default this means LDK is vulnerable to various
HashDoS attacks (#4719).
* `OMNameResolver` will no longer panic when looking up payment instructions
which include unicode characters at the start of a TXT record (#4718).
* `PrintableString` did not properly sanitize unicode format characters,
allowing an attacker to corrupt the rendering of logs or UI (#4593, #4605).
* RGS data is now limited in how large of a graph it is able to cause a client
to store in memory. Note that RGS data is still considered a DoS vector in
general and you should only use semi-trusted RGS data (#4713).
* Counterparty-provided strings in failure messages are no longer logged in
full, reducing the ability of such a counterparty to spam our logs (#4714).
* Reading a corrupted `ChannelManager` or `ProbabilisticScorer` can no longer
cause us to allocate large amounts of memory (#4712).
Thanks to Project Loupe for reporting most of the issues fixed in this release.
TheBlueMatt added a commit to TheBlueMatt/rust-lightning that referenced this pull request Jun 23, 2026
v0.2.3 - Jun 18, 2026 - "Through the Loupe"
API Updates
===========
* `DefaultMessageRouter` will now always generate blinded message paths that
provide no privacy (where our node is the introduction node) for nodes with
public channels. This works around an issue which will appear for any nodes
with LND peers that enable onion messaging - such peers will refuse to
forward BOLT 12 messages from unknown third parties, which most BOLT 12
payers rely on today (lightningdevkit#4647).
* Explicit `amount_msats` of 0 is rejected in BOLT 12 `Offer`s; `OfferBuilder`
now maps 0-amounts to an amount of `None` (lightningdevkit#4324).
Bug Fixes
=========
* `Features::supports_zero_conf` no longer clears the `ZeroConf` features and
`Features::requires_zero_conf` now correctly reports required, rather than
supported, status (lightningdevkit#4517).
* If an MPP payment is claimed but `ChannelMonitorUpdate`s for some parts are
still being completed asynchronously, further channel updates (e.g.
forwarding another payment) are pending and the node restarts, the channel
could have become stuck (lightningdevkit#4520).
* The presence of unconfirmed transactions actually no longer causes
`ElectrumSyncClient` to spuriously fail to sync (lightningdevkit#4590).
* LSPS1, LSPS2, and LSPS5 persistence will no longer get stuck and refuse to
persist again after a single failure from the KVStore (lightningdevkit#4597, lightningdevkit#4282).
* Dropping the future returned by
`OutputSweeper::regenerate_and_broadcast_spend_if_necessary` no longer
results in future calls to the same method being spuriously ignored (lightningdevkit#4598).
* Used async-receive offers are no longer refreshed on every timer tick once
their refresh time is reached (lightningdevkit#4672).
* `FilesystemStore::list_all_keys` will no longer fail if there are stale
intermediate files lying around from a previous unclean shutdown (lightningdevkit#4618).
* When forwarding an HTLC while in a blinded path with proportional fees over
200%, LDK will no longer spuriously allow a forward that pays us 1 msat too
little in fees (lightningdevkit#4697).
* Fixed a rare case where a channel could get stuck on reconnect when using
both async `ChannelMonitorUpdate` persistence and async signing (lightningdevkit#4684).
* If we had exactly zero balance in a zero-fee-commitment channel, the
counterparty was able to splice all of their balance out, violating the
reserve requirements they'd otherwise be forced to keep (lightningdevkit#4580).
* Providing an `Event::HTLCIntercepted` to the `LSPS2ServiceHandler` twice no
longer results in spuriously opening a channel early (lightningdevkit#4656).
* `Event::PaymentSent::fee_paid_msat` is no longer `None` in cases where
`ChannelManager::abandon_payment` was called before the payment ultimately
completes anyway (lightningdevkit#4651).
* `AnchorDescriptor::previous_utxo` now provides the correct `script_pubkey`
for non-zero-commitment-fee anchor channels (lightningdevkit#4669).
* Syncing a `ChainMonitor` using the `Confirm` trait will no longer write some
full `ChannelMonitor`s to disk several times per block (lightningdevkit#4544).
* `OMDomainResolver` now correctly accounts for failed queries when rate
limiting, ensuring we continue to respond to queries after failures (lightningdevkit#4591).
* Calling `ChannelManager::send_payment_with_route` without a `route_params`
and with an invalid `Route` will no longer panic (lightningdevkit#4707).
* `LSPS2ServiceHandler::channel_open_failed` now correctly fails intercepted
HTLCs rather than allowing them to fail just before expiry (lightningdevkit#4677).
* `StaticInvoice::is_offer_expired` was corrected to check offer, rather than
static invoice, expiry (lightningdevkit#4594).
* `lightning-custom-message`'s handling of `peer_connected` events now ensures
that sub-handlers will see a `peer_disconnected` event if a different
sub-handler refused the connection by `Err`ing `peer_connected` (lightningdevkit#4595).
* Replay protection for LSPS5 signatures now detects replays which are only
different in the encoded signature's case (lightningdevkit#4701).
* When `lightning-liquidity` is configured in the background processor, there
is no longer a stream of `Persisting LiquidityManager...` log spam (lightningdevkit#4246).
* Incomplete MPP keysend payments will no longer see their HTLCs held until
expiry (lightningdevkit#4558).
* `InvoiceRequestBuilder` will no longer accept a `quantity` of `0` for a
BOLT 12 `Offer`, allowing any quantity up to a bound (lightningdevkit#4667).
* `lightning-custom-message` handlers that return `Ok(None)` when asked to
deserialize a message in their defined range no longer cause panics (lightningdevkit#4709).
* Several spurious debug assertions were fixed (lightningdevkit#4537, lightningdevkit#4618, lightningdevkit#4026)
Security
========
0.2.3 fixes several underestimates of the anchor reserves required to ensure we
can reliably close channels, several denial-of-service vulnerabilities and a
sanitization issue.
* `Bolt11Invoice::recover_payee_pub_key` no longer panics if called on an
invoice which set an explicit public key, rather than relying on public key
recovery. Note that this method is called from
`PaymentParameters::from_bolt11_invoice` (lightningdevkit#4717).
* Maliciously-crafted unpayable invoices which have overflowing feerates will
no longer cause an `unwrap` failure panic (lightningdevkit#4716).
* Parsing an `LSPSDateTime` which is before 1970 no longer panics. This is
reachable when parsing messages from counterparties (lightningdevkit#4715).
* `possiblyrandom` did not properly generate random data except when it was
explicitly configured to. By default this means LDK is vulnerable to various
HashDoS attacks (lightningdevkit#4719).
* `OMNameResolver` will no longer panic when looking up payment instructions
which include unicode characters at the start of a TXT record (lightningdevkit#4718).
* When using the `anchor_channel_reserves` module to calculate reserves
required to pay for fees when closing anchor channels, zero-fee-commitment
channels were not considered. This could allow a counterparty to open many
channels, leaving us unable to properly force-close (lightningdevkit#4592).
* The `anchor_channel_reserves` module overestimated the value of `Utxo`s in
the wallet by ignoring the `TxIn` cost to spend them (lightningdevkit#4670).
* `PrintableString` did not properly sanitize unicode format characters,
allowing an attacker to corrupt the rendering of logs or UI (lightningdevkit#4593, lightningdevkit#4605).
* RGS data is now limited in how large of a graph it is able to cause a client
to store in memory. Note that RGS data is still considered a DoS vector in
general and you should only use semi-trusted RGS data (lightningdevkit#4713).
* Counterparty-provided strings in failure messages are no longer logged in
full, reducing the ability of such a counterparty to spam our logs (lightningdevkit#4714).
* Reading a corrupted `ChannelManager` or `ProbabilisticScorer` can no longer
cause us to allocate large amounts of memory (lightningdevkit#4712).
Thanks to Project Loupe for reporting most of the issues fixed in this release.
Conflicts resolved in:
* lightning/src/chain/channelmonitor.rs
* lightning/src/events/mod.rs
* lightning/src/ln/channelmanager.rs
* lightning/src/ln/mod.rs
* lightning/src/ln/offers_tests.rs
* lightning/src/ln/onion_utils.rs
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants

@TheBlueMatt@ldk-reviews-bot@ldk-claude-review-bot@tnull
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Add copy buttons to all
 blocks\n(function() {\n function addCopyButtons() {\n document.querySelectorAll('pre code').forEach(function(codeBlock) {\n if (codeBlock.parentElement.hasAttribute('data-copy-added')) return;\n codeBlock.parentElement.setAttribute('data-copy-added', 'true');\n \n var btn = document.createElement('button');\n btn.textContent = 'Copy';\n btn.style.cssText = 'position:absolute;top:4px;right:4px;padding:2px 8px;font-size:11px;background:#4ecdc4;border:none;border-radius:4px;color:#1a1a2e;cursor:pointer;opacity:0.7;transition:opacity 0.2s;';\n btn.onmouseover = function() { this.style.opacity = '1'; };\n btn.onmouseout = function() { this.style.opacity = '0.7'; };\n btn.onclick = function() {\n navigator.clipboard.writeText(codeBlock.textContent).then(function() {\n btn.textContent = 'Copied!';\n setTimeout(function() { btn.textContent = 'Copy'; }, 1500);\n });\n };\n codeBlock.parentElement.style.position = 'relative';\n codeBlock.parentElement.appendChild(btn);\n });\n }\n \n addCopyButtons();\n \n // Re-run on dynamic content\n var observer = new MutationObserver(addCopyButtons);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Add Copy Buttons to Code Blocks");
}
} catch(__e) { console.warn('[Userscript:Add Copy Buttons to Code Blocks]', __e); }
})();
(function(){
try {
var __m = "github.com";
var __re = new RegExp('^' + "github\\.com" + '
Skip to content

Add an auto-generated unicode character category file - #4605

Merged
tnull merged 1 commit into
lightningdevkit:mainfrom
TheBlueMatt:2026-05-unicode-autogen
May 18, 2026
Merged

Add an auto-generated unicode character category file#4605
tnull merged 1 commit into
lightningdevkit:mainfrom
TheBlueMatt:2026-05-unicode-autogen

Conversation

@TheBlueMatt

Copy link
Copy Markdown
Collaborator

1a01b5a added detection of unicode format characters in PrintableString, but used a hard-coded table which may eventually become out of date.

Here we switch to an auto-generated table, include all General_CategoryOther characters, and also ban unallocated code points.

Finally, CI validates that the file is kept up to date.

Written by Claude

@ldk-reviews-bot

ldk-reviews-bot commented May 7, 2026

Copy link
Copy Markdown

👋 Thanks for assigning @tnull as a reviewer!
I'll wait for their review and will help manage the review process.
Once they submit their review, I'll check if a second reviewer would be helpful.

@TheBlueMatt
TheBlueMatt requested a review from tnullMay 7, 2026 18:47
@TheBlueMatt
TheBlueMattforce-pushed the 2026-05-unicode-autogen branch from b6f8c03 to bd75483CompareMay 7, 2026 18:49
Comment on lines +36 to +38
let is_other = is_unicode_general_category_other(c);
let is_unassigned = is_unicode_general_category_unassigned(c);
let c = if c.is_control() || is_other || is_unassigned {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: c.is_control() is now fully redundant — it checks Cc (Control), which is already covered by is_unicode_general_category_other (see 0x0000..=0x001F and 0x007F..=0x009F in unicode.rs). The old code needed it because is_format_char only covered Cf, but the new function covers all of Cc / Cf / Cs / Co.

Not a bug (the || short-circuits harmlessly), but it's potentially confusing because it suggests is_other doesn't handle control characters.

Suggested change
let is_other = is_unicode_general_category_other(c);
let is_unassigned = is_unicode_general_category_unassigned(c);
let c = if c.is_control() || is_other || is_unassigned {
let c = ifis_unicode_general_category_other(c) || is_unicode_general_category_unassigned(c){

@ldk-claude-review-bot

ldk-claude-review-bot commented May 7, 2026

Copy link
Copy Markdown
Collaborator

Review Summary

New issue found

  • .github/workflows/check_unicode.yml:25gh issue create runs without deduplication; will create a new issue every day the file is out of date. build.yml already demonstrates the correct pattern (check for existing open issue, comment on it instead).

Correction to prior review

My three prior comments about the YAML workflow file (missing jobs: key, missing checkout step, fi indentation) were incorrect — the file structure is fine as written. I misread the diff indentation. Apologies for the noise.

Prior comment still applicable

  • lightning-types/src/string.rs:38c.is_control() remains redundant with is_unicode_general_category_other, which already covers all Cc characters.

@codecov

codecovBot commented May 7, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 92.23301% with 40 lines in your changes missing coverage. Please review.
✅ Project coverage is 86.15%. Comparing base (1a01b5a) to head (65e8cc8).
⚠️ Report is 66 commits behind head on main.

Files with missing linesPatch %Lines
lightning-types/src/unicode.rs92.18%40 Missing ⚠️
Additional details and impacted files
@@ Coverage Diff @@## main #4605 +/- ##
==========================================
+ Coverage 86.09% 86.15% +0.06% 
==========================================
Files 157 158 +1 Lines 108828 109323 +495 Branches 108828 109323 +495 ==========================================
+ Hits 93694 94189 +495 + Misses 12519 12518 -1 - Partials 2615 2616 +1 
FlagCoverage Δ
tests86.15% <92.23%> (+0.06%)⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@tnulltnull left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Alright, seems maybe a bit excessive, but fine by me. No need to check this on every CI run though, let's just move it to a weekly cronjob?

Comment thread.github/workflows/build.yml Outdated

@tnulltnull left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good, please squash.

Comment thread.github/workflows/check_unicode.yml
Comment thread.github/workflows/check_unicode.yml
Comment thread.github/workflows/check_unicode.yml Outdated
@tnull

tnull commented May 8, 2026

Copy link
Copy Markdown
Contributor

Claude's reviews seem legit though

1a01b5a added detection of unicode
format characters in `PrintableString`, but used a hard-coded table
which may eventually become out of date.
Here we switch to an auto-generated table, include all
`General_Category` `Other` characters, and also ban unallocated
code points.
Finally, CI validates that the file is kept up to date.
Written by Claude
@TheBlueMatt
TheBlueMattforce-pushed the 2026-05-unicode-autogen branch from 2de5954 to 65e8cc8CompareMay 8, 2026 19:44
@TheBlueMatt

Copy link
Copy Markdown
CollaboratorAuthor

Squashed and fixed claude's comments:

$ git diff-tree -U2 2de5954d9 65e8cc8d5
diff --git a/.github/workflows/check_unicode.yml b/.github/workflows/check_unicode.yml
index c22a4e3b29..a01add3f81 100644
--- a/.github/workflows/check_unicode.yml+++ b/.github/workflows/check_unicode.yml@@ -5,4 +5,5 @@ on:
- cron: '42 3 * * *'
+jobs:
check-unicode:
runs-on: ubuntu-latest
@@ -10,4 +11,6 @@ on:
issues: write
steps:
+ - name: Checkout source code+ uses: actions/checkout@v4
- name: Check unicode file state
env:
@@ -21,3 +24,3 @@ on:
BODY="The unicode character listing is out of date, see $RUN_URL"
gh issue create --title "$TITLE" --body "$BODY"
- fi+ fi

TITLE="Unicode listing out of date: ${{ github.workflow }}"
RUN_URL="https://github.com/${{ github.repository }}/actions/runs/${{ github.run_id }}"
BODY="The unicode character listing is out of date, see $RUN_URL"
gh issue create --title "$TITLE" --body "$BODY"

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bug: No deduplication — this creates a new issue every day the file is out of date. Unicode updates are infrequent (roughly annual), but if nobody regenerates the file promptly, this will spam the issue tracker with identical issues.

build.yml already has the right pattern (lines 380-391): check for an existing open issue with a label, and comment on it instead of creating a duplicate. This workflow should do the same.

Minimal fix sketch:

LABEL="unicode outdated"gh label create "$LABEL" 2>/dev/null || trueEXISTING=$(gh issue list --label "$LABEL" --state open --json number --jq '.[0].number // empty')if [ -n "$EXISTING" ]; thengh issue comment "$EXISTING" --body "$BODY"elsegh issue create --title "$TITLE" --label "$LABEL" --body "$BODY"fi

Copy link
Copy Markdown
CollaboratorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Eh, that takes effort, this is fine we expect to respond quickly.

@tnull
tnull merged commit 71fdc27 into lightningdevkit:mainMay 18, 2026
23 of 24 checks passed
@TheBlueMatt

Copy link
Copy Markdown
CollaboratorAuthor

Backported to 0.1 in #4680.

@TheBlueMatt

Copy link
Copy Markdown
CollaboratorAuthor

Backported to 0.2 in #4683.

TheBlueMatt added a commit that referenced this pull request Jun 19, 2026
v0.1.10 - Jun 18, 2026 - "Loupe de Loupe"
API Updates
===========
* `DefaultMessageRouter` will now always generate blinded message paths that
provide no privacy (where our node is the introduction node) for nodes with
public channels. This works around an issue which will appear for any nodes
with LND peers that enable onion messaging - such peers will refuse to
forward BOLT 12 messages from unknown third parties, which most BOLT 12
payers rely on today (#4647).
* Explicit `amount_msats` of 0 is rejected in BOLT 12 `Offer`s; `OfferBuilder`
now maps 0-amounts to an amount of `None` (#4324).
Bug Fixes
=========
* Async `ChannelMonitorUpdate` persistence operations which complete, but are
not marked as complete in a persisted `ChannelManager` prior to restart,
followed immediately by a block connection and then another restart could
result in some channel operations hanging leading for force-closures (#4377).
* If an MPP payment is claimed but `ChannelMonitorUpdate`s for some parts are
still being completed asynchronously, further channel updates (e.g.
forwarding another payment) are pending and the node restarts, the channel
could have become stuck (#4520).
* The presence of unconfirmed transactions actually no longer causes
`ElectrumSyncClient` to spuriously fail to sync (#4590).
* `FilesystemStore::list_all_keys` will no longer fail if there are stale
intermediate files lying around from a previous unclean shutdown (#4618).
* When forwarding an HTLC while in a blinded path with proportional fees over
200%, LDK will no longer spuriously allow a forward that pays us 1 msat too
little in fees (#4697).
* Fixed a rare case where a channel could get stuck on reconnect when using
both async `ChannelMonitorUpdate` persistence and async signing (#4684).
* `Event::PaymentSent::fee_paid_msat` is no longer `None` in cases where
`ChannelManager::abandon_payment` was called before the payment ultimately
completes anyway (#4651).
* Syncing a `ChainMonitor` using the `Confirm` trait will no longer write some
full `ChannelMonitor`s to disk several times per block (#4544).
* `OMDomainResolver` now correctly accounts for failed queries when rate
limiting, ensuring we continue to respond to queries after failures (#4591).
* Calling `ChannelManager::send_payment_with_route` without a `route_params`
and with an invalid `Route` will no longer panic (#4707).
* `lightning-custom-message`'s handling of `peer_connected` events now ensures
that sub-handlers will see a `peer_disconnected` event if a different
sub-handler refused the connection by `Err`ing `peer_connected` (#4595).
* Incomplete MPP keysend payments will no longer see their HTLCs held until
expiry (#4558).
* `InvoiceRequestBuilder` will no longer accept a `quantity` of `0` for a
BOLT 12 `Offer`, allowing any quantity up to a bound (#4667).
* `lightning-custom-message` handlers that return `Ok(None)` when asked to
deserialize a message in their defined range no longer cause panics (#4709).
* Several spurious debug assertions were fixed (#4537, #4618).
Security
========
0.1.10 fixes a sanitization issue and several denial-of-service vulnerabilities.
* `Bolt11Invoice::recover_payee_pub_key` no longer panics if called on an
invoice which set an explicit public key, rather than relying on public key
recovery. This method is called from `payment_parameters_from_invoice` and
`payment_parameters_from_variable_amount_invoice` (#4717).
* Maliciously-crafted unpayable invoices which have overflowing feerates will
no longer cause an `unwrap` failure panic (#4716).
* `possiblyrandom` did not properly generate random data except when it was
explicitly configured to. By default this means LDK is vulnerable to various
HashDoS attacks (#4719).
* `OMNameResolver` will no longer panic when looking up payment instructions
which include unicode characters at the start of a TXT record (#4718).
* `PrintableString` did not properly sanitize unicode format characters,
allowing an attacker to corrupt the rendering of logs or UI (#4593, #4605).
* RGS data is now limited in how large of a graph it is able to cause a client
to store in memory. Note that RGS data is still considered a DoS vector in
general and you should only use semi-trusted RGS data (#4713).
* Counterparty-provided strings in failure messages are no longer logged in
full, reducing the ability of such a counterparty to spam our logs (#4714).
* Reading a corrupted `ChannelManager` or `ProbabilisticScorer` can no longer
cause us to allocate large amounts of memory (#4712).
Thanks to Project Loupe for reporting most of the issues fixed in this release.
TheBlueMatt added a commit to TheBlueMatt/rust-lightning that referenced this pull request Jun 23, 2026
v0.2.3 - Jun 18, 2026 - "Through the Loupe"
API Updates
===========
* `DefaultMessageRouter` will now always generate blinded message paths that
provide no privacy (where our node is the introduction node) for nodes with
public channels. This works around an issue which will appear for any nodes
with LND peers that enable onion messaging - such peers will refuse to
forward BOLT 12 messages from unknown third parties, which most BOLT 12
payers rely on today (lightningdevkit#4647).
* Explicit `amount_msats` of 0 is rejected in BOLT 12 `Offer`s; `OfferBuilder`
now maps 0-amounts to an amount of `None` (lightningdevkit#4324).
Bug Fixes
=========
* `Features::supports_zero_conf` no longer clears the `ZeroConf` features and
`Features::requires_zero_conf` now correctly reports required, rather than
supported, status (lightningdevkit#4517).
* If an MPP payment is claimed but `ChannelMonitorUpdate`s for some parts are
still being completed asynchronously, further channel updates (e.g.
forwarding another payment) are pending and the node restarts, the channel
could have become stuck (lightningdevkit#4520).
* The presence of unconfirmed transactions actually no longer causes
`ElectrumSyncClient` to spuriously fail to sync (lightningdevkit#4590).
* LSPS1, LSPS2, and LSPS5 persistence will no longer get stuck and refuse to
persist again after a single failure from the KVStore (lightningdevkit#4597, lightningdevkit#4282).
* Dropping the future returned by
`OutputSweeper::regenerate_and_broadcast_spend_if_necessary` no longer
results in future calls to the same method being spuriously ignored (lightningdevkit#4598).
* Used async-receive offers are no longer refreshed on every timer tick once
their refresh time is reached (lightningdevkit#4672).
* `FilesystemStore::list_all_keys` will no longer fail if there are stale
intermediate files lying around from a previous unclean shutdown (lightningdevkit#4618).
* When forwarding an HTLC while in a blinded path with proportional fees over
200%, LDK will no longer spuriously allow a forward that pays us 1 msat too
little in fees (lightningdevkit#4697).
* Fixed a rare case where a channel could get stuck on reconnect when using
both async `ChannelMonitorUpdate` persistence and async signing (lightningdevkit#4684).
* If we had exactly zero balance in a zero-fee-commitment channel, the
counterparty was able to splice all of their balance out, violating the
reserve requirements they'd otherwise be forced to keep (lightningdevkit#4580).
* Providing an `Event::HTLCIntercepted` to the `LSPS2ServiceHandler` twice no
longer results in spuriously opening a channel early (lightningdevkit#4656).
* `Event::PaymentSent::fee_paid_msat` is no longer `None` in cases where
`ChannelManager::abandon_payment` was called before the payment ultimately
completes anyway (lightningdevkit#4651).
* `AnchorDescriptor::previous_utxo` now provides the correct `script_pubkey`
for non-zero-commitment-fee anchor channels (lightningdevkit#4669).
* Syncing a `ChainMonitor` using the `Confirm` trait will no longer write some
full `ChannelMonitor`s to disk several times per block (lightningdevkit#4544).
* `OMDomainResolver` now correctly accounts for failed queries when rate
limiting, ensuring we continue to respond to queries after failures (lightningdevkit#4591).
* Calling `ChannelManager::send_payment_with_route` without a `route_params`
and with an invalid `Route` will no longer panic (lightningdevkit#4707).
* `LSPS2ServiceHandler::channel_open_failed` now correctly fails intercepted
HTLCs rather than allowing them to fail just before expiry (lightningdevkit#4677).
* `StaticInvoice::is_offer_expired` was corrected to check offer, rather than
static invoice, expiry (lightningdevkit#4594).
* `lightning-custom-message`'s handling of `peer_connected` events now ensures
that sub-handlers will see a `peer_disconnected` event if a different
sub-handler refused the connection by `Err`ing `peer_connected` (lightningdevkit#4595).
* Replay protection for LSPS5 signatures now detects replays which are only
different in the encoded signature's case (lightningdevkit#4701).
* When `lightning-liquidity` is configured in the background processor, there
is no longer a stream of `Persisting LiquidityManager...` log spam (lightningdevkit#4246).
* Incomplete MPP keysend payments will no longer see their HTLCs held until
expiry (lightningdevkit#4558).
* `InvoiceRequestBuilder` will no longer accept a `quantity` of `0` for a
BOLT 12 `Offer`, allowing any quantity up to a bound (lightningdevkit#4667).
* `lightning-custom-message` handlers that return `Ok(None)` when asked to
deserialize a message in their defined range no longer cause panics (lightningdevkit#4709).
* Several spurious debug assertions were fixed (lightningdevkit#4537, lightningdevkit#4618, lightningdevkit#4026)
Security
========
0.2.3 fixes several underestimates of the anchor reserves required to ensure we
can reliably close channels, several denial-of-service vulnerabilities and a
sanitization issue.
* `Bolt11Invoice::recover_payee_pub_key` no longer panics if called on an
invoice which set an explicit public key, rather than relying on public key
recovery. Note that this method is called from
`PaymentParameters::from_bolt11_invoice` (lightningdevkit#4717).
* Maliciously-crafted unpayable invoices which have overflowing feerates will
no longer cause an `unwrap` failure panic (lightningdevkit#4716).
* Parsing an `LSPSDateTime` which is before 1970 no longer panics. This is
reachable when parsing messages from counterparties (lightningdevkit#4715).
* `possiblyrandom` did not properly generate random data except when it was
explicitly configured to. By default this means LDK is vulnerable to various
HashDoS attacks (lightningdevkit#4719).
* `OMNameResolver` will no longer panic when looking up payment instructions
which include unicode characters at the start of a TXT record (lightningdevkit#4718).
* When using the `anchor_channel_reserves` module to calculate reserves
required to pay for fees when closing anchor channels, zero-fee-commitment
channels were not considered. This could allow a counterparty to open many
channels, leaving us unable to properly force-close (lightningdevkit#4592).
* The `anchor_channel_reserves` module overestimated the value of `Utxo`s in
the wallet by ignoring the `TxIn` cost to spend them (lightningdevkit#4670).
* `PrintableString` did not properly sanitize unicode format characters,
allowing an attacker to corrupt the rendering of logs or UI (lightningdevkit#4593, lightningdevkit#4605).
* RGS data is now limited in how large of a graph it is able to cause a client
to store in memory. Note that RGS data is still considered a DoS vector in
general and you should only use semi-trusted RGS data (lightningdevkit#4713).
* Counterparty-provided strings in failure messages are no longer logged in
full, reducing the ability of such a counterparty to spam our logs (lightningdevkit#4714).
* Reading a corrupted `ChannelManager` or `ProbabilisticScorer` can no longer
cause us to allocate large amounts of memory (lightningdevkit#4712).
Thanks to Project Loupe for reporting most of the issues fixed in this release.
Conflicts resolved in:
* lightning/src/chain/channelmonitor.rs
* lightning/src/events/mod.rs
* lightning/src/ln/channelmanager.rs
* lightning/src/ln/mod.rs
* lightning/src/ln/offers_tests.rs
* lightning/src/ln/onion_utils.rs
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants

@TheBlueMatt@ldk-reviews-bot@ldk-claude-review-bot@tnull
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Force GitHub README to respect dark mode\n(function() {\n var style = document.createElement('style');\n style.textContent = '\n .markdown-body {\n color-scheme: dark light;\n }\n .markdown-body pre { background: #161b22 !important; }\n .markdown-body code { background: rgba(110, 118, 129, 0.4) !important; }\n .markdown-body table th, .markdown-body table td { border-color: #30363d !important; }\n .markdown-body img { background: #0d1117; }\n .markdown-body blockquote { border-left-color: #8b949e; }\n .markdown-body hr { border-color: #30363d; }\n ';\n document.head.appendChild(style);\n})();", "GitHub Dark Mode README Fix"); } } catch(__e) { console.warn('[Userscript:GitHub Dark Mode README Fix]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
Skip to content

Add an auto-generated unicode character category file - #4605

Merged
tnull merged 1 commit into
lightningdevkit:mainfrom
TheBlueMatt:2026-05-unicode-autogen
May 18, 2026
Merged

Add an auto-generated unicode character category file#4605
tnull merged 1 commit into
lightningdevkit:mainfrom
TheBlueMatt:2026-05-unicode-autogen

Conversation

@TheBlueMatt

Copy link
Copy Markdown
Collaborator

1a01b5a added detection of unicode format characters in PrintableString, but used a hard-coded table which may eventually become out of date.

Here we switch to an auto-generated table, include all General_CategoryOther characters, and also ban unallocated code points.

Finally, CI validates that the file is kept up to date.

Written by Claude

@ldk-reviews-bot

ldk-reviews-bot commented May 7, 2026

Copy link
Copy Markdown

👋 Thanks for assigning @tnull as a reviewer!
I'll wait for their review and will help manage the review process.
Once they submit their review, I'll check if a second reviewer would be helpful.

@TheBlueMatt
TheBlueMatt requested a review from tnullMay 7, 2026 18:47
@TheBlueMatt
TheBlueMattforce-pushed the 2026-05-unicode-autogen branch from b6f8c03 to bd75483CompareMay 7, 2026 18:49
Comment on lines +36 to +38
let is_other = is_unicode_general_category_other(c);
let is_unassigned = is_unicode_general_category_unassigned(c);
let c = if c.is_control() || is_other || is_unassigned {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: c.is_control() is now fully redundant — it checks Cc (Control), which is already covered by is_unicode_general_category_other (see 0x0000..=0x001F and 0x007F..=0x009F in unicode.rs). The old code needed it because is_format_char only covered Cf, but the new function covers all of Cc / Cf / Cs / Co.

Not a bug (the || short-circuits harmlessly), but it's potentially confusing because it suggests is_other doesn't handle control characters.

Suggested change
let is_other = is_unicode_general_category_other(c);
let is_unassigned = is_unicode_general_category_unassigned(c);
let c = if c.is_control() || is_other || is_unassigned {
let c = ifis_unicode_general_category_other(c) || is_unicode_general_category_unassigned(c){

@ldk-claude-review-bot

ldk-claude-review-bot commented May 7, 2026

Copy link
Copy Markdown
Collaborator

Review Summary

New issue found

  • .github/workflows/check_unicode.yml:25gh issue create runs without deduplication; will create a new issue every day the file is out of date. build.yml already demonstrates the correct pattern (check for existing open issue, comment on it instead).

Correction to prior review

My three prior comments about the YAML workflow file (missing jobs: key, missing checkout step, fi indentation) were incorrect — the file structure is fine as written. I misread the diff indentation. Apologies for the noise.

Prior comment still applicable

  • lightning-types/src/string.rs:38c.is_control() remains redundant with is_unicode_general_category_other, which already covers all Cc characters.

@codecov

codecovBot commented May 7, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 92.23301% with 40 lines in your changes missing coverage. Please review.
✅ Project coverage is 86.15%. Comparing base (1a01b5a) to head (65e8cc8).
⚠️ Report is 66 commits behind head on main.

Files with missing linesPatch %Lines
lightning-types/src/unicode.rs92.18%40 Missing ⚠️
Additional details and impacted files
@@ Coverage Diff @@## main #4605 +/- ##
==========================================
+ Coverage 86.09% 86.15% +0.06% 
==========================================
Files 157 158 +1 Lines 108828 109323 +495 Branches 108828 109323 +495 ==========================================
+ Hits 93694 94189 +495 + Misses 12519 12518 -1 - Partials 2615 2616 +1 
FlagCoverage Δ
tests86.15% <92.23%> (+0.06%)⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@tnulltnull left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Alright, seems maybe a bit excessive, but fine by me. No need to check this on every CI run though, let's just move it to a weekly cronjob?

Comment thread.github/workflows/build.yml Outdated

@tnulltnull left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good, please squash.

Comment thread.github/workflows/check_unicode.yml
Comment thread.github/workflows/check_unicode.yml
Comment thread.github/workflows/check_unicode.yml Outdated
@tnull

tnull commented May 8, 2026

Copy link
Copy Markdown
Contributor

Claude's reviews seem legit though

1a01b5a added detection of unicode
format characters in `PrintableString`, but used a hard-coded table
which may eventually become out of date.
Here we switch to an auto-generated table, include all
`General_Category` `Other` characters, and also ban unallocated
code points.
Finally, CI validates that the file is kept up to date.
Written by Claude
@TheBlueMatt
TheBlueMattforce-pushed the 2026-05-unicode-autogen branch from 2de5954 to 65e8cc8CompareMay 8, 2026 19:44
@TheBlueMatt

Copy link
Copy Markdown
CollaboratorAuthor

Squashed and fixed claude's comments:

$ git diff-tree -U2 2de5954d9 65e8cc8d5
diff --git a/.github/workflows/check_unicode.yml b/.github/workflows/check_unicode.yml
index c22a4e3b29..a01add3f81 100644
--- a/.github/workflows/check_unicode.yml+++ b/.github/workflows/check_unicode.yml@@ -5,4 +5,5 @@ on:
- cron: '42 3 * * *'
+jobs:
check-unicode:
runs-on: ubuntu-latest
@@ -10,4 +11,6 @@ on:
issues: write
steps:
+ - name: Checkout source code+ uses: actions/checkout@v4
- name: Check unicode file state
env:
@@ -21,3 +24,3 @@ on:
BODY="The unicode character listing is out of date, see $RUN_URL"
gh issue create --title "$TITLE" --body "$BODY"
- fi+ fi

TITLE="Unicode listing out of date: ${{ github.workflow }}"
RUN_URL="https://github.com/${{ github.repository }}/actions/runs/${{ github.run_id }}"
BODY="The unicode character listing is out of date, see $RUN_URL"
gh issue create --title "$TITLE" --body "$BODY"

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bug: No deduplication — this creates a new issue every day the file is out of date. Unicode updates are infrequent (roughly annual), but if nobody regenerates the file promptly, this will spam the issue tracker with identical issues.

build.yml already has the right pattern (lines 380-391): check for an existing open issue with a label, and comment on it instead of creating a duplicate. This workflow should do the same.

Minimal fix sketch:

LABEL="unicode outdated"gh label create "$LABEL" 2>/dev/null || trueEXISTING=$(gh issue list --label "$LABEL" --state open --json number --jq '.[0].number // empty')if [ -n "$EXISTING" ]; thengh issue comment "$EXISTING" --body "$BODY"elsegh issue create --title "$TITLE" --label "$LABEL" --body "$BODY"fi

Copy link
Copy Markdown
CollaboratorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Eh, that takes effort, this is fine we expect to respond quickly.

@tnull
tnull merged commit 71fdc27 into lightningdevkit:mainMay 18, 2026
23 of 24 checks passed
@TheBlueMatt

Copy link
Copy Markdown
CollaboratorAuthor

Backported to 0.1 in #4680.

@TheBlueMatt

Copy link
Copy Markdown
CollaboratorAuthor

Backported to 0.2 in #4683.

TheBlueMatt added a commit that referenced this pull request Jun 19, 2026
v0.1.10 - Jun 18, 2026 - "Loupe de Loupe"
API Updates
===========
* `DefaultMessageRouter` will now always generate blinded message paths that
provide no privacy (where our node is the introduction node) for nodes with
public channels. This works around an issue which will appear for any nodes
with LND peers that enable onion messaging - such peers will refuse to
forward BOLT 12 messages from unknown third parties, which most BOLT 12
payers rely on today (#4647).
* Explicit `amount_msats` of 0 is rejected in BOLT 12 `Offer`s; `OfferBuilder`
now maps 0-amounts to an amount of `None` (#4324).
Bug Fixes
=========
* Async `ChannelMonitorUpdate` persistence operations which complete, but are
not marked as complete in a persisted `ChannelManager` prior to restart,
followed immediately by a block connection and then another restart could
result in some channel operations hanging leading for force-closures (#4377).
* If an MPP payment is claimed but `ChannelMonitorUpdate`s for some parts are
still being completed asynchronously, further channel updates (e.g.
forwarding another payment) are pending and the node restarts, the channel
could have become stuck (#4520).
* The presence of unconfirmed transactions actually no longer causes
`ElectrumSyncClient` to spuriously fail to sync (#4590).
* `FilesystemStore::list_all_keys` will no longer fail if there are stale
intermediate files lying around from a previous unclean shutdown (#4618).
* When forwarding an HTLC while in a blinded path with proportional fees over
200%, LDK will no longer spuriously allow a forward that pays us 1 msat too
little in fees (#4697).
* Fixed a rare case where a channel could get stuck on reconnect when using
both async `ChannelMonitorUpdate` persistence and async signing (#4684).
* `Event::PaymentSent::fee_paid_msat` is no longer `None` in cases where
`ChannelManager::abandon_payment` was called before the payment ultimately
completes anyway (#4651).
* Syncing a `ChainMonitor` using the `Confirm` trait will no longer write some
full `ChannelMonitor`s to disk several times per block (#4544).
* `OMDomainResolver` now correctly accounts for failed queries when rate
limiting, ensuring we continue to respond to queries after failures (#4591).
* Calling `ChannelManager::send_payment_with_route` without a `route_params`
and with an invalid `Route` will no longer panic (#4707).
* `lightning-custom-message`'s handling of `peer_connected` events now ensures
that sub-handlers will see a `peer_disconnected` event if a different
sub-handler refused the connection by `Err`ing `peer_connected` (#4595).
* Incomplete MPP keysend payments will no longer see their HTLCs held until
expiry (#4558).
* `InvoiceRequestBuilder` will no longer accept a `quantity` of `0` for a
BOLT 12 `Offer`, allowing any quantity up to a bound (#4667).
* `lightning-custom-message` handlers that return `Ok(None)` when asked to
deserialize a message in their defined range no longer cause panics (#4709).
* Several spurious debug assertions were fixed (#4537, #4618).
Security
========
0.1.10 fixes a sanitization issue and several denial-of-service vulnerabilities.
* `Bolt11Invoice::recover_payee_pub_key` no longer panics if called on an
invoice which set an explicit public key, rather than relying on public key
recovery. This method is called from `payment_parameters_from_invoice` and
`payment_parameters_from_variable_amount_invoice` (#4717).
* Maliciously-crafted unpayable invoices which have overflowing feerates will
no longer cause an `unwrap` failure panic (#4716).
* `possiblyrandom` did not properly generate random data except when it was
explicitly configured to. By default this means LDK is vulnerable to various
HashDoS attacks (#4719).
* `OMNameResolver` will no longer panic when looking up payment instructions
which include unicode characters at the start of a TXT record (#4718).
* `PrintableString` did not properly sanitize unicode format characters,
allowing an attacker to corrupt the rendering of logs or UI (#4593, #4605).
* RGS data is now limited in how large of a graph it is able to cause a client
to store in memory. Note that RGS data is still considered a DoS vector in
general and you should only use semi-trusted RGS data (#4713).
* Counterparty-provided strings in failure messages are no longer logged in
full, reducing the ability of such a counterparty to spam our logs (#4714).
* Reading a corrupted `ChannelManager` or `ProbabilisticScorer` can no longer
cause us to allocate large amounts of memory (#4712).
Thanks to Project Loupe for reporting most of the issues fixed in this release.
TheBlueMatt added a commit to TheBlueMatt/rust-lightning that referenced this pull request Jun 23, 2026
v0.2.3 - Jun 18, 2026 - "Through the Loupe"
API Updates
===========
* `DefaultMessageRouter` will now always generate blinded message paths that
provide no privacy (where our node is the introduction node) for nodes with
public channels. This works around an issue which will appear for any nodes
with LND peers that enable onion messaging - such peers will refuse to
forward BOLT 12 messages from unknown third parties, which most BOLT 12
payers rely on today (lightningdevkit#4647).
* Explicit `amount_msats` of 0 is rejected in BOLT 12 `Offer`s; `OfferBuilder`
now maps 0-amounts to an amount of `None` (lightningdevkit#4324).
Bug Fixes
=========
* `Features::supports_zero_conf` no longer clears the `ZeroConf` features and
`Features::requires_zero_conf` now correctly reports required, rather than
supported, status (lightningdevkit#4517).
* If an MPP payment is claimed but `ChannelMonitorUpdate`s for some parts are
still being completed asynchronously, further channel updates (e.g.
forwarding another payment) are pending and the node restarts, the channel
could have become stuck (lightningdevkit#4520).
* The presence of unconfirmed transactions actually no longer causes
`ElectrumSyncClient` to spuriously fail to sync (lightningdevkit#4590).
* LSPS1, LSPS2, and LSPS5 persistence will no longer get stuck and refuse to
persist again after a single failure from the KVStore (lightningdevkit#4597, lightningdevkit#4282).
* Dropping the future returned by
`OutputSweeper::regenerate_and_broadcast_spend_if_necessary` no longer
results in future calls to the same method being spuriously ignored (lightningdevkit#4598).
* Used async-receive offers are no longer refreshed on every timer tick once
their refresh time is reached (lightningdevkit#4672).
* `FilesystemStore::list_all_keys` will no longer fail if there are stale
intermediate files lying around from a previous unclean shutdown (lightningdevkit#4618).
* When forwarding an HTLC while in a blinded path with proportional fees over
200%, LDK will no longer spuriously allow a forward that pays us 1 msat too
little in fees (lightningdevkit#4697).
* Fixed a rare case where a channel could get stuck on reconnect when using
both async `ChannelMonitorUpdate` persistence and async signing (lightningdevkit#4684).
* If we had exactly zero balance in a zero-fee-commitment channel, the
counterparty was able to splice all of their balance out, violating the
reserve requirements they'd otherwise be forced to keep (lightningdevkit#4580).
* Providing an `Event::HTLCIntercepted` to the `LSPS2ServiceHandler` twice no
longer results in spuriously opening a channel early (lightningdevkit#4656).
* `Event::PaymentSent::fee_paid_msat` is no longer `None` in cases where
`ChannelManager::abandon_payment` was called before the payment ultimately
completes anyway (lightningdevkit#4651).
* `AnchorDescriptor::previous_utxo` now provides the correct `script_pubkey`
for non-zero-commitment-fee anchor channels (lightningdevkit#4669).
* Syncing a `ChainMonitor` using the `Confirm` trait will no longer write some
full `ChannelMonitor`s to disk several times per block (lightningdevkit#4544).
* `OMDomainResolver` now correctly accounts for failed queries when rate
limiting, ensuring we continue to respond to queries after failures (lightningdevkit#4591).
* Calling `ChannelManager::send_payment_with_route` without a `route_params`
and with an invalid `Route` will no longer panic (lightningdevkit#4707).
* `LSPS2ServiceHandler::channel_open_failed` now correctly fails intercepted
HTLCs rather than allowing them to fail just before expiry (lightningdevkit#4677).
* `StaticInvoice::is_offer_expired` was corrected to check offer, rather than
static invoice, expiry (lightningdevkit#4594).
* `lightning-custom-message`'s handling of `peer_connected` events now ensures
that sub-handlers will see a `peer_disconnected` event if a different
sub-handler refused the connection by `Err`ing `peer_connected` (lightningdevkit#4595).
* Replay protection for LSPS5 signatures now detects replays which are only
different in the encoded signature's case (lightningdevkit#4701).
* When `lightning-liquidity` is configured in the background processor, there
is no longer a stream of `Persisting LiquidityManager...` log spam (lightningdevkit#4246).
* Incomplete MPP keysend payments will no longer see their HTLCs held until
expiry (lightningdevkit#4558).
* `InvoiceRequestBuilder` will no longer accept a `quantity` of `0` for a
BOLT 12 `Offer`, allowing any quantity up to a bound (lightningdevkit#4667).
* `lightning-custom-message` handlers that return `Ok(None)` when asked to
deserialize a message in their defined range no longer cause panics (lightningdevkit#4709).
* Several spurious debug assertions were fixed (lightningdevkit#4537, lightningdevkit#4618, lightningdevkit#4026)
Security
========
0.2.3 fixes several underestimates of the anchor reserves required to ensure we
can reliably close channels, several denial-of-service vulnerabilities and a
sanitization issue.
* `Bolt11Invoice::recover_payee_pub_key` no longer panics if called on an
invoice which set an explicit public key, rather than relying on public key
recovery. Note that this method is called from
`PaymentParameters::from_bolt11_invoice` (lightningdevkit#4717).
* Maliciously-crafted unpayable invoices which have overflowing feerates will
no longer cause an `unwrap` failure panic (lightningdevkit#4716).
* Parsing an `LSPSDateTime` which is before 1970 no longer panics. This is
reachable when parsing messages from counterparties (lightningdevkit#4715).
* `possiblyrandom` did not properly generate random data except when it was
explicitly configured to. By default this means LDK is vulnerable to various
HashDoS attacks (lightningdevkit#4719).
* `OMNameResolver` will no longer panic when looking up payment instructions
which include unicode characters at the start of a TXT record (lightningdevkit#4718).
* When using the `anchor_channel_reserves` module to calculate reserves
required to pay for fees when closing anchor channels, zero-fee-commitment
channels were not considered. This could allow a counterparty to open many
channels, leaving us unable to properly force-close (lightningdevkit#4592).
* The `anchor_channel_reserves` module overestimated the value of `Utxo`s in
the wallet by ignoring the `TxIn` cost to spend them (lightningdevkit#4670).
* `PrintableString` did not properly sanitize unicode format characters,
allowing an attacker to corrupt the rendering of logs or UI (lightningdevkit#4593, lightningdevkit#4605).
* RGS data is now limited in how large of a graph it is able to cause a client
to store in memory. Note that RGS data is still considered a DoS vector in
general and you should only use semi-trusted RGS data (lightningdevkit#4713).
* Counterparty-provided strings in failure messages are no longer logged in
full, reducing the ability of such a counterparty to spam our logs (lightningdevkit#4714).
* Reading a corrupted `ChannelManager` or `ProbabilisticScorer` can no longer
cause us to allocate large amounts of memory (lightningdevkit#4712).
Thanks to Project Loupe for reporting most of the issues fixed in this release.
Conflicts resolved in:
* lightning/src/chain/channelmonitor.rs
* lightning/src/events/mod.rs
* lightning/src/ln/channelmanager.rs
* lightning/src/ln/mod.rs
* lightning/src/ln/offers_tests.rs
* lightning/src/ln/onion_utils.rs
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants

@TheBlueMatt@ldk-reviews-bot@ldk-claude-review-bot@tnull
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Highlight search terms from Google/DuckDuckGo/Bing referrer\n(function() {\n var ref = document.referrer;\n var terms = [];\n \n if (ref.includes('google.com') || ref.includes('duckduckgo.com') || ref.includes('bing.com')) {\n var url = new URL(ref);\n var q = url.searchParams.get('q') || url.searchParams.get('p');\n if (q) {\n terms = q.split(/\\s+/).filter(function(t) { return t.length > 2; });\n }\n }\n \n if (terms.length === 0) return;\n \n var style = document.createElement('style');\n style.textContent = '.userscript-highlight { background: #fbbf24; color: #1a1a2e; padding: 1px 3px; border-radius: 2px; }';\n document.head.appendChild(style);\n \n function highlight(node) {\n if (node.nodeType === 3) { // text node\n var text = node.textContent;\n var found = false;\n terms.forEach(function(term) {\n var regex = new RegExp('(' + term.replace(/[.*+?^${}()|[\\]\\\\]/g, '\\\\') + ')', 'gi');\n if (regex.test(text)) {\n found = true;\n var frag = document.createDocumentFragment();\n var parts = text.split(regex);\n parts.forEach(function(part, i) {\n if (i % 2 === 0) {\n frag.appendChild(document.createTextNode(part));\n } else {\n var span = document.createElement('span');\n span.className = 'userscript-highlight';\n span.textContent = part;\n frag.appendChild(span);\n }\n });\n node.parentNode.replaceChild(frag, node);\n }\n });\n } else if (node.nodeType === 1 && node.childNodes) { // element\n var skipTags = ['SCRIPT', 'STYLE', 'NOSCRIPT', 'TEXTAREA', 'INPUT', 'SELECT'];\n if (!skipTags.includes(node.tagName)) {\n Array.from(node.childNodes).forEach(highlight);\n }\n }\n }\n \n highlight(document.body);\n \n // Re-highlight on dynamic content\n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1 || node.nodeType === 3) highlight(node);\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Highlight Search Terms"); } } catch(__e) { console.warn('[Userscript:Highlight Search Terms]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
Skip to content

Add an auto-generated unicode character category file - #4605

Merged
tnull merged 1 commit into
lightningdevkit:mainfrom
TheBlueMatt:2026-05-unicode-autogen
May 18, 2026
Merged

Add an auto-generated unicode character category file#4605
tnull merged 1 commit into
lightningdevkit:mainfrom
TheBlueMatt:2026-05-unicode-autogen

Conversation

@TheBlueMatt

Copy link
Copy Markdown
Collaborator

1a01b5a added detection of unicode format characters in PrintableString, but used a hard-coded table which may eventually become out of date.

Here we switch to an auto-generated table, include all General_CategoryOther characters, and also ban unallocated code points.

Finally, CI validates that the file is kept up to date.

Written by Claude

@ldk-reviews-bot

ldk-reviews-bot commented May 7, 2026

Copy link
Copy Markdown

👋 Thanks for assigning @tnull as a reviewer!
I'll wait for their review and will help manage the review process.
Once they submit their review, I'll check if a second reviewer would be helpful.

@TheBlueMatt
TheBlueMatt requested a review from tnullMay 7, 2026 18:47
@TheBlueMatt
TheBlueMattforce-pushed the 2026-05-unicode-autogen branch from b6f8c03 to bd75483CompareMay 7, 2026 18:49
Comment on lines +36 to +38
let is_other = is_unicode_general_category_other(c);
let is_unassigned = is_unicode_general_category_unassigned(c);
let c = if c.is_control() || is_other || is_unassigned {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: c.is_control() is now fully redundant — it checks Cc (Control), which is already covered by is_unicode_general_category_other (see 0x0000..=0x001F and 0x007F..=0x009F in unicode.rs). The old code needed it because is_format_char only covered Cf, but the new function covers all of Cc / Cf / Cs / Co.

Not a bug (the || short-circuits harmlessly), but it's potentially confusing because it suggests is_other doesn't handle control characters.

Suggested change
let is_other = is_unicode_general_category_other(c);
let is_unassigned = is_unicode_general_category_unassigned(c);
let c = if c.is_control() || is_other || is_unassigned {
let c = ifis_unicode_general_category_other(c) || is_unicode_general_category_unassigned(c){

@ldk-claude-review-bot

ldk-claude-review-bot commented May 7, 2026

Copy link
Copy Markdown
Collaborator

Review Summary

New issue found

  • .github/workflows/check_unicode.yml:25gh issue create runs without deduplication; will create a new issue every day the file is out of date. build.yml already demonstrates the correct pattern (check for existing open issue, comment on it instead).

Correction to prior review

My three prior comments about the YAML workflow file (missing jobs: key, missing checkout step, fi indentation) were incorrect — the file structure is fine as written. I misread the diff indentation. Apologies for the noise.

Prior comment still applicable

  • lightning-types/src/string.rs:38c.is_control() remains redundant with is_unicode_general_category_other, which already covers all Cc characters.

@codecov

codecovBot commented May 7, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 92.23301% with 40 lines in your changes missing coverage. Please review.
✅ Project coverage is 86.15%. Comparing base (1a01b5a) to head (65e8cc8).
⚠️ Report is 66 commits behind head on main.

Files with missing linesPatch %Lines
lightning-types/src/unicode.rs92.18%40 Missing ⚠️
Additional details and impacted files
@@ Coverage Diff @@## main #4605 +/- ##
==========================================
+ Coverage 86.09% 86.15% +0.06% 
==========================================
Files 157 158 +1 Lines 108828 109323 +495 Branches 108828 109323 +495 ==========================================
+ Hits 93694 94189 +495 + Misses 12519 12518 -1 - Partials 2615 2616 +1 
FlagCoverage Δ
tests86.15% <92.23%> (+0.06%)⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@tnulltnull left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Alright, seems maybe a bit excessive, but fine by me. No need to check this on every CI run though, let's just move it to a weekly cronjob?

Comment thread.github/workflows/build.yml Outdated

@tnulltnull left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good, please squash.

Comment thread.github/workflows/check_unicode.yml
Comment thread.github/workflows/check_unicode.yml
Comment thread.github/workflows/check_unicode.yml Outdated
@tnull

tnull commented May 8, 2026

Copy link
Copy Markdown
Contributor

Claude's reviews seem legit though

1a01b5a added detection of unicode
format characters in `PrintableString`, but used a hard-coded table
which may eventually become out of date.
Here we switch to an auto-generated table, include all
`General_Category` `Other` characters, and also ban unallocated
code points.
Finally, CI validates that the file is kept up to date.
Written by Claude
@TheBlueMatt
TheBlueMattforce-pushed the 2026-05-unicode-autogen branch from 2de5954 to 65e8cc8CompareMay 8, 2026 19:44
@TheBlueMatt

Copy link
Copy Markdown
CollaboratorAuthor

Squashed and fixed claude's comments:

$ git diff-tree -U2 2de5954d9 65e8cc8d5
diff --git a/.github/workflows/check_unicode.yml b/.github/workflows/check_unicode.yml
index c22a4e3b29..a01add3f81 100644
--- a/.github/workflows/check_unicode.yml+++ b/.github/workflows/check_unicode.yml@@ -5,4 +5,5 @@ on:
- cron: '42 3 * * *'
+jobs:
check-unicode:
runs-on: ubuntu-latest
@@ -10,4 +11,6 @@ on:
issues: write
steps:
+ - name: Checkout source code+ uses: actions/checkout@v4
- name: Check unicode file state
env:
@@ -21,3 +24,3 @@ on:
BODY="The unicode character listing is out of date, see $RUN_URL"
gh issue create --title "$TITLE" --body "$BODY"
- fi+ fi

TITLE="Unicode listing out of date: ${{ github.workflow }}"
RUN_URL="https://github.com/${{ github.repository }}/actions/runs/${{ github.run_id }}"
BODY="The unicode character listing is out of date, see $RUN_URL"
gh issue create --title "$TITLE" --body "$BODY"

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bug: No deduplication — this creates a new issue every day the file is out of date. Unicode updates are infrequent (roughly annual), but if nobody regenerates the file promptly, this will spam the issue tracker with identical issues.

build.yml already has the right pattern (lines 380-391): check for an existing open issue with a label, and comment on it instead of creating a duplicate. This workflow should do the same.

Minimal fix sketch:

LABEL="unicode outdated"gh label create "$LABEL" 2>/dev/null || trueEXISTING=$(gh issue list --label "$LABEL" --state open --json number --jq '.[0].number // empty')if [ -n "$EXISTING" ]; thengh issue comment "$EXISTING" --body "$BODY"elsegh issue create --title "$TITLE" --label "$LABEL" --body "$BODY"fi

Copy link
Copy Markdown
CollaboratorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Eh, that takes effort, this is fine we expect to respond quickly.

@tnull
tnull merged commit 71fdc27 into lightningdevkit:mainMay 18, 2026
23 of 24 checks passed
@TheBlueMatt

Copy link
Copy Markdown
CollaboratorAuthor

Backported to 0.1 in #4680.

@TheBlueMatt

Copy link
Copy Markdown
CollaboratorAuthor

Backported to 0.2 in #4683.

TheBlueMatt added a commit that referenced this pull request Jun 19, 2026
v0.1.10 - Jun 18, 2026 - "Loupe de Loupe"
API Updates
===========
* `DefaultMessageRouter` will now always generate blinded message paths that
provide no privacy (where our node is the introduction node) for nodes with
public channels. This works around an issue which will appear for any nodes
with LND peers that enable onion messaging - such peers will refuse to
forward BOLT 12 messages from unknown third parties, which most BOLT 12
payers rely on today (#4647).
* Explicit `amount_msats` of 0 is rejected in BOLT 12 `Offer`s; `OfferBuilder`
now maps 0-amounts to an amount of `None` (#4324).
Bug Fixes
=========
* Async `ChannelMonitorUpdate` persistence operations which complete, but are
not marked as complete in a persisted `ChannelManager` prior to restart,
followed immediately by a block connection and then another restart could
result in some channel operations hanging leading for force-closures (#4377).
* If an MPP payment is claimed but `ChannelMonitorUpdate`s for some parts are
still being completed asynchronously, further channel updates (e.g.
forwarding another payment) are pending and the node restarts, the channel
could have become stuck (#4520).
* The presence of unconfirmed transactions actually no longer causes
`ElectrumSyncClient` to spuriously fail to sync (#4590).
* `FilesystemStore::list_all_keys` will no longer fail if there are stale
intermediate files lying around from a previous unclean shutdown (#4618).
* When forwarding an HTLC while in a blinded path with proportional fees over
200%, LDK will no longer spuriously allow a forward that pays us 1 msat too
little in fees (#4697).
* Fixed a rare case where a channel could get stuck on reconnect when using
both async `ChannelMonitorUpdate` persistence and async signing (#4684).
* `Event::PaymentSent::fee_paid_msat` is no longer `None` in cases where
`ChannelManager::abandon_payment` was called before the payment ultimately
completes anyway (#4651).
* Syncing a `ChainMonitor` using the `Confirm` trait will no longer write some
full `ChannelMonitor`s to disk several times per block (#4544).
* `OMDomainResolver` now correctly accounts for failed queries when rate
limiting, ensuring we continue to respond to queries after failures (#4591).
* Calling `ChannelManager::send_payment_with_route` without a `route_params`
and with an invalid `Route` will no longer panic (#4707).
* `lightning-custom-message`'s handling of `peer_connected` events now ensures
that sub-handlers will see a `peer_disconnected` event if a different
sub-handler refused the connection by `Err`ing `peer_connected` (#4595).
* Incomplete MPP keysend payments will no longer see their HTLCs held until
expiry (#4558).
* `InvoiceRequestBuilder` will no longer accept a `quantity` of `0` for a
BOLT 12 `Offer`, allowing any quantity up to a bound (#4667).
* `lightning-custom-message` handlers that return `Ok(None)` when asked to
deserialize a message in their defined range no longer cause panics (#4709).
* Several spurious debug assertions were fixed (#4537, #4618).
Security
========
0.1.10 fixes a sanitization issue and several denial-of-service vulnerabilities.
* `Bolt11Invoice::recover_payee_pub_key` no longer panics if called on an
invoice which set an explicit public key, rather than relying on public key
recovery. This method is called from `payment_parameters_from_invoice` and
`payment_parameters_from_variable_amount_invoice` (#4717).
* Maliciously-crafted unpayable invoices which have overflowing feerates will
no longer cause an `unwrap` failure panic (#4716).
* `possiblyrandom` did not properly generate random data except when it was
explicitly configured to. By default this means LDK is vulnerable to various
HashDoS attacks (#4719).
* `OMNameResolver` will no longer panic when looking up payment instructions
which include unicode characters at the start of a TXT record (#4718).
* `PrintableString` did not properly sanitize unicode format characters,
allowing an attacker to corrupt the rendering of logs or UI (#4593, #4605).
* RGS data is now limited in how large of a graph it is able to cause a client
to store in memory. Note that RGS data is still considered a DoS vector in
general and you should only use semi-trusted RGS data (#4713).
* Counterparty-provided strings in failure messages are no longer logged in
full, reducing the ability of such a counterparty to spam our logs (#4714).
* Reading a corrupted `ChannelManager` or `ProbabilisticScorer` can no longer
cause us to allocate large amounts of memory (#4712).
Thanks to Project Loupe for reporting most of the issues fixed in this release.
TheBlueMatt added a commit to TheBlueMatt/rust-lightning that referenced this pull request Jun 23, 2026
v0.2.3 - Jun 18, 2026 - "Through the Loupe"
API Updates
===========
* `DefaultMessageRouter` will now always generate blinded message paths that
provide no privacy (where our node is the introduction node) for nodes with
public channels. This works around an issue which will appear for any nodes
with LND peers that enable onion messaging - such peers will refuse to
forward BOLT 12 messages from unknown third parties, which most BOLT 12
payers rely on today (lightningdevkit#4647).
* Explicit `amount_msats` of 0 is rejected in BOLT 12 `Offer`s; `OfferBuilder`
now maps 0-amounts to an amount of `None` (lightningdevkit#4324).
Bug Fixes
=========
* `Features::supports_zero_conf` no longer clears the `ZeroConf` features and
`Features::requires_zero_conf` now correctly reports required, rather than
supported, status (lightningdevkit#4517).
* If an MPP payment is claimed but `ChannelMonitorUpdate`s for some parts are
still being completed asynchronously, further channel updates (e.g.
forwarding another payment) are pending and the node restarts, the channel
could have become stuck (lightningdevkit#4520).
* The presence of unconfirmed transactions actually no longer causes
`ElectrumSyncClient` to spuriously fail to sync (lightningdevkit#4590).
* LSPS1, LSPS2, and LSPS5 persistence will no longer get stuck and refuse to
persist again after a single failure from the KVStore (lightningdevkit#4597, lightningdevkit#4282).
* Dropping the future returned by
`OutputSweeper::regenerate_and_broadcast_spend_if_necessary` no longer
results in future calls to the same method being spuriously ignored (lightningdevkit#4598).
* Used async-receive offers are no longer refreshed on every timer tick once
their refresh time is reached (lightningdevkit#4672).
* `FilesystemStore::list_all_keys` will no longer fail if there are stale
intermediate files lying around from a previous unclean shutdown (lightningdevkit#4618).
* When forwarding an HTLC while in a blinded path with proportional fees over
200%, LDK will no longer spuriously allow a forward that pays us 1 msat too
little in fees (lightningdevkit#4697).
* Fixed a rare case where a channel could get stuck on reconnect when using
both async `ChannelMonitorUpdate` persistence and async signing (lightningdevkit#4684).
* If we had exactly zero balance in a zero-fee-commitment channel, the
counterparty was able to splice all of their balance out, violating the
reserve requirements they'd otherwise be forced to keep (lightningdevkit#4580).
* Providing an `Event::HTLCIntercepted` to the `LSPS2ServiceHandler` twice no
longer results in spuriously opening a channel early (lightningdevkit#4656).
* `Event::PaymentSent::fee_paid_msat` is no longer `None` in cases where
`ChannelManager::abandon_payment` was called before the payment ultimately
completes anyway (lightningdevkit#4651).
* `AnchorDescriptor::previous_utxo` now provides the correct `script_pubkey`
for non-zero-commitment-fee anchor channels (lightningdevkit#4669).
* Syncing a `ChainMonitor` using the `Confirm` trait will no longer write some
full `ChannelMonitor`s to disk several times per block (lightningdevkit#4544).
* `OMDomainResolver` now correctly accounts for failed queries when rate
limiting, ensuring we continue to respond to queries after failures (lightningdevkit#4591).
* Calling `ChannelManager::send_payment_with_route` without a `route_params`
and with an invalid `Route` will no longer panic (lightningdevkit#4707).
* `LSPS2ServiceHandler::channel_open_failed` now correctly fails intercepted
HTLCs rather than allowing them to fail just before expiry (lightningdevkit#4677).
* `StaticInvoice::is_offer_expired` was corrected to check offer, rather than
static invoice, expiry (lightningdevkit#4594).
* `lightning-custom-message`'s handling of `peer_connected` events now ensures
that sub-handlers will see a `peer_disconnected` event if a different
sub-handler refused the connection by `Err`ing `peer_connected` (lightningdevkit#4595).
* Replay protection for LSPS5 signatures now detects replays which are only
different in the encoded signature's case (lightningdevkit#4701).
* When `lightning-liquidity` is configured in the background processor, there
is no longer a stream of `Persisting LiquidityManager...` log spam (lightningdevkit#4246).
* Incomplete MPP keysend payments will no longer see their HTLCs held until
expiry (lightningdevkit#4558).
* `InvoiceRequestBuilder` will no longer accept a `quantity` of `0` for a
BOLT 12 `Offer`, allowing any quantity up to a bound (lightningdevkit#4667).
* `lightning-custom-message` handlers that return `Ok(None)` when asked to
deserialize a message in their defined range no longer cause panics (lightningdevkit#4709).
* Several spurious debug assertions were fixed (lightningdevkit#4537, lightningdevkit#4618, lightningdevkit#4026)
Security
========
0.2.3 fixes several underestimates of the anchor reserves required to ensure we
can reliably close channels, several denial-of-service vulnerabilities and a
sanitization issue.
* `Bolt11Invoice::recover_payee_pub_key` no longer panics if called on an
invoice which set an explicit public key, rather than relying on public key
recovery. Note that this method is called from
`PaymentParameters::from_bolt11_invoice` (lightningdevkit#4717).
* Maliciously-crafted unpayable invoices which have overflowing feerates will
no longer cause an `unwrap` failure panic (lightningdevkit#4716).
* Parsing an `LSPSDateTime` which is before 1970 no longer panics. This is
reachable when parsing messages from counterparties (lightningdevkit#4715).
* `possiblyrandom` did not properly generate random data except when it was
explicitly configured to. By default this means LDK is vulnerable to various
HashDoS attacks (lightningdevkit#4719).
* `OMNameResolver` will no longer panic when looking up payment instructions
which include unicode characters at the start of a TXT record (lightningdevkit#4718).
* When using the `anchor_channel_reserves` module to calculate reserves
required to pay for fees when closing anchor channels, zero-fee-commitment
channels were not considered. This could allow a counterparty to open many
channels, leaving us unable to properly force-close (lightningdevkit#4592).
* The `anchor_channel_reserves` module overestimated the value of `Utxo`s in
the wallet by ignoring the `TxIn` cost to spend them (lightningdevkit#4670).
* `PrintableString` did not properly sanitize unicode format characters,
allowing an attacker to corrupt the rendering of logs or UI (lightningdevkit#4593, lightningdevkit#4605).
* RGS data is now limited in how large of a graph it is able to cause a client
to store in memory. Note that RGS data is still considered a DoS vector in
general and you should only use semi-trusted RGS data (lightningdevkit#4713).
* Counterparty-provided strings in failure messages are no longer logged in
full, reducing the ability of such a counterparty to spam our logs (lightningdevkit#4714).
* Reading a corrupted `ChannelManager` or `ProbabilisticScorer` can no longer
cause us to allocate large amounts of memory (lightningdevkit#4712).
Thanks to Project Loupe for reporting most of the issues fixed in this release.
Conflicts resolved in:
* lightning/src/chain/channelmonitor.rs
* lightning/src/events/mod.rs
* lightning/src/ln/channelmanager.rs
* lightning/src/ln/mod.rs
* lightning/src/ln/offers_tests.rs
* lightning/src/ln/onion_utils.rs
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants

@TheBlueMatt@ldk-reviews-bot@ldk-claude-review-bot@tnull
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Strip utm_, fbclid, gclid, etc. from all links on page\n(function() {\n var trackingParams = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content',\n 'fbclid', 'gclid', 'dclid', 'msclkid', 'yclid',\n 'ref', 'ref_src', 'source', 'medium', 'campaign'];\n \n function cleanUrl(url) {\n try {\n var u = new URL(url, window.location.origin);\n var changed = false;\n trackingParams.forEach(function(p) {\n if (u.searchParams.has(p)) {\n u.searchParams.delete(p);\n changed = true;\n }\n });\n return changed ? u.toString() : url;\n } catch (e) {\n return url;\n }\n }\n \n function cleanLinks() {\n document.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n \n cleanLinks();\n \n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1) {\n if (node.tagName === 'A') cleanLinks();\n node.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Remove Tracking Parameters from Links"); } } catch(__e) { console.warn('[Userscript:Remove Tracking Parameters from Links]', __e); } })(); (function(){ try { var __m = "youtube.com"; var __re = new RegExp('^' + "youtube\\.com" + '
Skip to content

Add an auto-generated unicode character category file - #4605

Merged
tnull merged 1 commit into
lightningdevkit:mainfrom
TheBlueMatt:2026-05-unicode-autogen
May 18, 2026
Merged

Add an auto-generated unicode character category file#4605
tnull merged 1 commit into
lightningdevkit:mainfrom
TheBlueMatt:2026-05-unicode-autogen

Conversation

@TheBlueMatt

Copy link
Copy Markdown
Collaborator

1a01b5a added detection of unicode format characters in PrintableString, but used a hard-coded table which may eventually become out of date.

Here we switch to an auto-generated table, include all General_CategoryOther characters, and also ban unallocated code points.

Finally, CI validates that the file is kept up to date.

Written by Claude

@ldk-reviews-bot

ldk-reviews-bot commented May 7, 2026

Copy link
Copy Markdown

👋 Thanks for assigning @tnull as a reviewer!
I'll wait for their review and will help manage the review process.
Once they submit their review, I'll check if a second reviewer would be helpful.

@TheBlueMatt
TheBlueMatt requested a review from tnullMay 7, 2026 18:47
@TheBlueMatt
TheBlueMattforce-pushed the 2026-05-unicode-autogen branch from b6f8c03 to bd75483CompareMay 7, 2026 18:49
Comment on lines +36 to +38
let is_other = is_unicode_general_category_other(c);
let is_unassigned = is_unicode_general_category_unassigned(c);
let c = if c.is_control() || is_other || is_unassigned {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: c.is_control() is now fully redundant — it checks Cc (Control), which is already covered by is_unicode_general_category_other (see 0x0000..=0x001F and 0x007F..=0x009F in unicode.rs). The old code needed it because is_format_char only covered Cf, but the new function covers all of Cc / Cf / Cs / Co.

Not a bug (the || short-circuits harmlessly), but it's potentially confusing because it suggests is_other doesn't handle control characters.

Suggested change
let is_other = is_unicode_general_category_other(c);
let is_unassigned = is_unicode_general_category_unassigned(c);
let c = if c.is_control() || is_other || is_unassigned {
let c = ifis_unicode_general_category_other(c) || is_unicode_general_category_unassigned(c){

@ldk-claude-review-bot

ldk-claude-review-bot commented May 7, 2026

Copy link
Copy Markdown
Collaborator

Review Summary

New issue found

  • .github/workflows/check_unicode.yml:25gh issue create runs without deduplication; will create a new issue every day the file is out of date. build.yml already demonstrates the correct pattern (check for existing open issue, comment on it instead).

Correction to prior review

My three prior comments about the YAML workflow file (missing jobs: key, missing checkout step, fi indentation) were incorrect — the file structure is fine as written. I misread the diff indentation. Apologies for the noise.

Prior comment still applicable

  • lightning-types/src/string.rs:38c.is_control() remains redundant with is_unicode_general_category_other, which already covers all Cc characters.

@codecov

codecovBot commented May 7, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 92.23301% with 40 lines in your changes missing coverage. Please review.
✅ Project coverage is 86.15%. Comparing base (1a01b5a) to head (65e8cc8).
⚠️ Report is 66 commits behind head on main.

Files with missing linesPatch %Lines
lightning-types/src/unicode.rs92.18%40 Missing ⚠️
Additional details and impacted files
@@ Coverage Diff @@## main #4605 +/- ##
==========================================
+ Coverage 86.09% 86.15% +0.06% 
==========================================
Files 157 158 +1 Lines 108828 109323 +495 Branches 108828 109323 +495 ==========================================
+ Hits 93694 94189 +495 + Misses 12519 12518 -1 - Partials 2615 2616 +1 
FlagCoverage Δ
tests86.15% <92.23%> (+0.06%)⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@tnulltnull left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Alright, seems maybe a bit excessive, but fine by me. No need to check this on every CI run though, let's just move it to a weekly cronjob?

Comment thread.github/workflows/build.yml Outdated

@tnulltnull left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good, please squash.

Comment thread.github/workflows/check_unicode.yml
Comment thread.github/workflows/check_unicode.yml
Comment thread.github/workflows/check_unicode.yml Outdated
@tnull

tnull commented May 8, 2026

Copy link
Copy Markdown
Contributor

Claude's reviews seem legit though

1a01b5a added detection of unicode
format characters in `PrintableString`, but used a hard-coded table
which may eventually become out of date.
Here we switch to an auto-generated table, include all
`General_Category` `Other` characters, and also ban unallocated
code points.
Finally, CI validates that the file is kept up to date.
Written by Claude
@TheBlueMatt
TheBlueMattforce-pushed the 2026-05-unicode-autogen branch from 2de5954 to 65e8cc8CompareMay 8, 2026 19:44
@TheBlueMatt

Copy link
Copy Markdown
CollaboratorAuthor

Squashed and fixed claude's comments:

$ git diff-tree -U2 2de5954d9 65e8cc8d5
diff --git a/.github/workflows/check_unicode.yml b/.github/workflows/check_unicode.yml
index c22a4e3b29..a01add3f81 100644
--- a/.github/workflows/check_unicode.yml+++ b/.github/workflows/check_unicode.yml@@ -5,4 +5,5 @@ on:
- cron: '42 3 * * *'
+jobs:
check-unicode:
runs-on: ubuntu-latest
@@ -10,4 +11,6 @@ on:
issues: write
steps:
+ - name: Checkout source code+ uses: actions/checkout@v4
- name: Check unicode file state
env:
@@ -21,3 +24,3 @@ on:
BODY="The unicode character listing is out of date, see $RUN_URL"
gh issue create --title "$TITLE" --body "$BODY"
- fi+ fi

TITLE="Unicode listing out of date: ${{ github.workflow }}"
RUN_URL="https://github.com/${{ github.repository }}/actions/runs/${{ github.run_id }}"
BODY="The unicode character listing is out of date, see $RUN_URL"
gh issue create --title "$TITLE" --body "$BODY"

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bug: No deduplication — this creates a new issue every day the file is out of date. Unicode updates are infrequent (roughly annual), but if nobody regenerates the file promptly, this will spam the issue tracker with identical issues.

build.yml already has the right pattern (lines 380-391): check for an existing open issue with a label, and comment on it instead of creating a duplicate. This workflow should do the same.

Minimal fix sketch:

LABEL="unicode outdated"gh label create "$LABEL" 2>/dev/null || trueEXISTING=$(gh issue list --label "$LABEL" --state open --json number --jq '.[0].number // empty')if [ -n "$EXISTING" ]; thengh issue comment "$EXISTING" --body "$BODY"elsegh issue create --title "$TITLE" --label "$LABEL" --body "$BODY"fi

Copy link
Copy Markdown
CollaboratorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Eh, that takes effort, this is fine we expect to respond quickly.

@tnull
tnull merged commit 71fdc27 into lightningdevkit:mainMay 18, 2026
23 of 24 checks passed
@TheBlueMatt

Copy link
Copy Markdown
CollaboratorAuthor

Backported to 0.1 in #4680.

@TheBlueMatt

Copy link
Copy Markdown
CollaboratorAuthor

Backported to 0.2 in #4683.

TheBlueMatt added a commit that referenced this pull request Jun 19, 2026
v0.1.10 - Jun 18, 2026 - "Loupe de Loupe"
API Updates
===========
* `DefaultMessageRouter` will now always generate blinded message paths that
provide no privacy (where our node is the introduction node) for nodes with
public channels. This works around an issue which will appear for any nodes
with LND peers that enable onion messaging - such peers will refuse to
forward BOLT 12 messages from unknown third parties, which most BOLT 12
payers rely on today (#4647).
* Explicit `amount_msats` of 0 is rejected in BOLT 12 `Offer`s; `OfferBuilder`
now maps 0-amounts to an amount of `None` (#4324).
Bug Fixes
=========
* Async `ChannelMonitorUpdate` persistence operations which complete, but are
not marked as complete in a persisted `ChannelManager` prior to restart,
followed immediately by a block connection and then another restart could
result in some channel operations hanging leading for force-closures (#4377).
* If an MPP payment is claimed but `ChannelMonitorUpdate`s for some parts are
still being completed asynchronously, further channel updates (e.g.
forwarding another payment) are pending and the node restarts, the channel
could have become stuck (#4520).
* The presence of unconfirmed transactions actually no longer causes
`ElectrumSyncClient` to spuriously fail to sync (#4590).
* `FilesystemStore::list_all_keys` will no longer fail if there are stale
intermediate files lying around from a previous unclean shutdown (#4618).
* When forwarding an HTLC while in a blinded path with proportional fees over
200%, LDK will no longer spuriously allow a forward that pays us 1 msat too
little in fees (#4697).
* Fixed a rare case where a channel could get stuck on reconnect when using
both async `ChannelMonitorUpdate` persistence and async signing (#4684).
* `Event::PaymentSent::fee_paid_msat` is no longer `None` in cases where
`ChannelManager::abandon_payment` was called before the payment ultimately
completes anyway (#4651).
* Syncing a `ChainMonitor` using the `Confirm` trait will no longer write some
full `ChannelMonitor`s to disk several times per block (#4544).
* `OMDomainResolver` now correctly accounts for failed queries when rate
limiting, ensuring we continue to respond to queries after failures (#4591).
* Calling `ChannelManager::send_payment_with_route` without a `route_params`
and with an invalid `Route` will no longer panic (#4707).
* `lightning-custom-message`'s handling of `peer_connected` events now ensures
that sub-handlers will see a `peer_disconnected` event if a different
sub-handler refused the connection by `Err`ing `peer_connected` (#4595).
* Incomplete MPP keysend payments will no longer see their HTLCs held until
expiry (#4558).
* `InvoiceRequestBuilder` will no longer accept a `quantity` of `0` for a
BOLT 12 `Offer`, allowing any quantity up to a bound (#4667).
* `lightning-custom-message` handlers that return `Ok(None)` when asked to
deserialize a message in their defined range no longer cause panics (#4709).
* Several spurious debug assertions were fixed (#4537, #4618).
Security
========
0.1.10 fixes a sanitization issue and several denial-of-service vulnerabilities.
* `Bolt11Invoice::recover_payee_pub_key` no longer panics if called on an
invoice which set an explicit public key, rather than relying on public key
recovery. This method is called from `payment_parameters_from_invoice` and
`payment_parameters_from_variable_amount_invoice` (#4717).
* Maliciously-crafted unpayable invoices which have overflowing feerates will
no longer cause an `unwrap` failure panic (#4716).
* `possiblyrandom` did not properly generate random data except when it was
explicitly configured to. By default this means LDK is vulnerable to various
HashDoS attacks (#4719).
* `OMNameResolver` will no longer panic when looking up payment instructions
which include unicode characters at the start of a TXT record (#4718).
* `PrintableString` did not properly sanitize unicode format characters,
allowing an attacker to corrupt the rendering of logs or UI (#4593, #4605).
* RGS data is now limited in how large of a graph it is able to cause a client
to store in memory. Note that RGS data is still considered a DoS vector in
general and you should only use semi-trusted RGS data (#4713).
* Counterparty-provided strings in failure messages are no longer logged in
full, reducing the ability of such a counterparty to spam our logs (#4714).
* Reading a corrupted `ChannelManager` or `ProbabilisticScorer` can no longer
cause us to allocate large amounts of memory (#4712).
Thanks to Project Loupe for reporting most of the issues fixed in this release.
TheBlueMatt added a commit to TheBlueMatt/rust-lightning that referenced this pull request Jun 23, 2026
v0.2.3 - Jun 18, 2026 - "Through the Loupe"
API Updates
===========
* `DefaultMessageRouter` will now always generate blinded message paths that
provide no privacy (where our node is the introduction node) for nodes with
public channels. This works around an issue which will appear for any nodes
with LND peers that enable onion messaging - such peers will refuse to
forward BOLT 12 messages from unknown third parties, which most BOLT 12
payers rely on today (lightningdevkit#4647).
* Explicit `amount_msats` of 0 is rejected in BOLT 12 `Offer`s; `OfferBuilder`
now maps 0-amounts to an amount of `None` (lightningdevkit#4324).
Bug Fixes
=========
* `Features::supports_zero_conf` no longer clears the `ZeroConf` features and
`Features::requires_zero_conf` now correctly reports required, rather than
supported, status (lightningdevkit#4517).
* If an MPP payment is claimed but `ChannelMonitorUpdate`s for some parts are
still being completed asynchronously, further channel updates (e.g.
forwarding another payment) are pending and the node restarts, the channel
could have become stuck (lightningdevkit#4520).
* The presence of unconfirmed transactions actually no longer causes
`ElectrumSyncClient` to spuriously fail to sync (lightningdevkit#4590).
* LSPS1, LSPS2, and LSPS5 persistence will no longer get stuck and refuse to
persist again after a single failure from the KVStore (lightningdevkit#4597, lightningdevkit#4282).
* Dropping the future returned by
`OutputSweeper::regenerate_and_broadcast_spend_if_necessary` no longer
results in future calls to the same method being spuriously ignored (lightningdevkit#4598).
* Used async-receive offers are no longer refreshed on every timer tick once
their refresh time is reached (lightningdevkit#4672).
* `FilesystemStore::list_all_keys` will no longer fail if there are stale
intermediate files lying around from a previous unclean shutdown (lightningdevkit#4618).
* When forwarding an HTLC while in a blinded path with proportional fees over
200%, LDK will no longer spuriously allow a forward that pays us 1 msat too
little in fees (lightningdevkit#4697).
* Fixed a rare case where a channel could get stuck on reconnect when using
both async `ChannelMonitorUpdate` persistence and async signing (lightningdevkit#4684).
* If we had exactly zero balance in a zero-fee-commitment channel, the
counterparty was able to splice all of their balance out, violating the
reserve requirements they'd otherwise be forced to keep (lightningdevkit#4580).
* Providing an `Event::HTLCIntercepted` to the `LSPS2ServiceHandler` twice no
longer results in spuriously opening a channel early (lightningdevkit#4656).
* `Event::PaymentSent::fee_paid_msat` is no longer `None` in cases where
`ChannelManager::abandon_payment` was called before the payment ultimately
completes anyway (lightningdevkit#4651).
* `AnchorDescriptor::previous_utxo` now provides the correct `script_pubkey`
for non-zero-commitment-fee anchor channels (lightningdevkit#4669).
* Syncing a `ChainMonitor` using the `Confirm` trait will no longer write some
full `ChannelMonitor`s to disk several times per block (lightningdevkit#4544).
* `OMDomainResolver` now correctly accounts for failed queries when rate
limiting, ensuring we continue to respond to queries after failures (lightningdevkit#4591).
* Calling `ChannelManager::send_payment_with_route` without a `route_params`
and with an invalid `Route` will no longer panic (lightningdevkit#4707).
* `LSPS2ServiceHandler::channel_open_failed` now correctly fails intercepted
HTLCs rather than allowing them to fail just before expiry (lightningdevkit#4677).
* `StaticInvoice::is_offer_expired` was corrected to check offer, rather than
static invoice, expiry (lightningdevkit#4594).
* `lightning-custom-message`'s handling of `peer_connected` events now ensures
that sub-handlers will see a `peer_disconnected` event if a different
sub-handler refused the connection by `Err`ing `peer_connected` (lightningdevkit#4595).
* Replay protection for LSPS5 signatures now detects replays which are only
different in the encoded signature's case (lightningdevkit#4701).
* When `lightning-liquidity` is configured in the background processor, there
is no longer a stream of `Persisting LiquidityManager...` log spam (lightningdevkit#4246).
* Incomplete MPP keysend payments will no longer see their HTLCs held until
expiry (lightningdevkit#4558).
* `InvoiceRequestBuilder` will no longer accept a `quantity` of `0` for a
BOLT 12 `Offer`, allowing any quantity up to a bound (lightningdevkit#4667).
* `lightning-custom-message` handlers that return `Ok(None)` when asked to
deserialize a message in their defined range no longer cause panics (lightningdevkit#4709).
* Several spurious debug assertions were fixed (lightningdevkit#4537, lightningdevkit#4618, lightningdevkit#4026)
Security
========
0.2.3 fixes several underestimates of the anchor reserves required to ensure we
can reliably close channels, several denial-of-service vulnerabilities and a
sanitization issue.
* `Bolt11Invoice::recover_payee_pub_key` no longer panics if called on an
invoice which set an explicit public key, rather than relying on public key
recovery. Note that this method is called from
`PaymentParameters::from_bolt11_invoice` (lightningdevkit#4717).
* Maliciously-crafted unpayable invoices which have overflowing feerates will
no longer cause an `unwrap` failure panic (lightningdevkit#4716).
* Parsing an `LSPSDateTime` which is before 1970 no longer panics. This is
reachable when parsing messages from counterparties (lightningdevkit#4715).
* `possiblyrandom` did not properly generate random data except when it was
explicitly configured to. By default this means LDK is vulnerable to various
HashDoS attacks (lightningdevkit#4719).
* `OMNameResolver` will no longer panic when looking up payment instructions
which include unicode characters at the start of a TXT record (lightningdevkit#4718).
* When using the `anchor_channel_reserves` module to calculate reserves
required to pay for fees when closing anchor channels, zero-fee-commitment
channels were not considered. This could allow a counterparty to open many
channels, leaving us unable to properly force-close (lightningdevkit#4592).
* The `anchor_channel_reserves` module overestimated the value of `Utxo`s in
the wallet by ignoring the `TxIn` cost to spend them (lightningdevkit#4670).
* `PrintableString` did not properly sanitize unicode format characters,
allowing an attacker to corrupt the rendering of logs or UI (lightningdevkit#4593, lightningdevkit#4605).
* RGS data is now limited in how large of a graph it is able to cause a client
to store in memory. Note that RGS data is still considered a DoS vector in
general and you should only use semi-trusted RGS data (lightningdevkit#4713).
* Counterparty-provided strings in failure messages are no longer logged in
full, reducing the ability of such a counterparty to spam our logs (lightningdevkit#4714).
* Reading a corrupted `ChannelManager` or `ProbabilisticScorer` can no longer
cause us to allocate large amounts of memory (lightningdevkit#4712).
Thanks to Project Loupe for reporting most of the issues fixed in this release.
Conflicts resolved in:
* lightning/src/chain/channelmonitor.rs
* lightning/src/events/mod.rs
* lightning/src/ln/channelmanager.rs
* lightning/src/ln/mod.rs
* lightning/src/ln/offers_tests.rs
* lightning/src/ln/onion_utils.rs
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants

@TheBlueMatt@ldk-reviews-bot@ldk-claude-review-bot@tnull
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Auto-enable theater mode on YouTube\n(function() {\n function tryTheater() {\n var btn = document.querySelector('button[aria-label=\"Theater mode\"], ytd-player #player button[title=\"Theater mode\"]');\n if (btn && !btn.classList.contains('activated')) {\n btn.click();\n }\n }\n \n // Try immediately\n tryTheater();\n \n // Try after navigation (SPA)\n var lastUrl = location.href;\n setInterval(function() {\n if (location.href !== lastUrl) {\n lastUrl = location.href;\n setTimeout(tryTheater, 500);\n }\n }, 1000);\n \n // Also try on player load\n var observer = new MutationObserver(tryTheater);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "YouTube Theater Mode Default"); } } catch(__e) { console.warn('[Userscript:YouTube Theater Mode Default]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
Skip to content

Add an auto-generated unicode character category file - #4605

Merged
tnull merged 1 commit into
lightningdevkit:mainfrom
TheBlueMatt:2026-05-unicode-autogen
May 18, 2026
Merged

Add an auto-generated unicode character category file#4605
tnull merged 1 commit into
lightningdevkit:mainfrom
TheBlueMatt:2026-05-unicode-autogen

Conversation

@TheBlueMatt

Copy link
Copy Markdown
Collaborator

1a01b5a added detection of unicode format characters in PrintableString, but used a hard-coded table which may eventually become out of date.

Here we switch to an auto-generated table, include all General_CategoryOther characters, and also ban unallocated code points.

Finally, CI validates that the file is kept up to date.

Written by Claude

@ldk-reviews-bot

ldk-reviews-bot commented May 7, 2026

Copy link
Copy Markdown

👋 Thanks for assigning @tnull as a reviewer!
I'll wait for their review and will help manage the review process.
Once they submit their review, I'll check if a second reviewer would be helpful.

@TheBlueMatt
TheBlueMatt requested a review from tnullMay 7, 2026 18:47
@TheBlueMatt
TheBlueMattforce-pushed the 2026-05-unicode-autogen branch from b6f8c03 to bd75483CompareMay 7, 2026 18:49
Comment on lines +36 to +38
let is_other = is_unicode_general_category_other(c);
let is_unassigned = is_unicode_general_category_unassigned(c);
let c = if c.is_control() || is_other || is_unassigned {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: c.is_control() is now fully redundant — it checks Cc (Control), which is already covered by is_unicode_general_category_other (see 0x0000..=0x001F and 0x007F..=0x009F in unicode.rs). The old code needed it because is_format_char only covered Cf, but the new function covers all of Cc / Cf / Cs / Co.

Not a bug (the || short-circuits harmlessly), but it's potentially confusing because it suggests is_other doesn't handle control characters.

Suggested change
let is_other = is_unicode_general_category_other(c);
let is_unassigned = is_unicode_general_category_unassigned(c);
let c = if c.is_control() || is_other || is_unassigned {
let c = ifis_unicode_general_category_other(c) || is_unicode_general_category_unassigned(c){

@ldk-claude-review-bot

ldk-claude-review-bot commented May 7, 2026

Copy link
Copy Markdown
Collaborator

Review Summary

New issue found

  • .github/workflows/check_unicode.yml:25gh issue create runs without deduplication; will create a new issue every day the file is out of date. build.yml already demonstrates the correct pattern (check for existing open issue, comment on it instead).

Correction to prior review

My three prior comments about the YAML workflow file (missing jobs: key, missing checkout step, fi indentation) were incorrect — the file structure is fine as written. I misread the diff indentation. Apologies for the noise.

Prior comment still applicable

  • lightning-types/src/string.rs:38c.is_control() remains redundant with is_unicode_general_category_other, which already covers all Cc characters.

@codecov

codecovBot commented May 7, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 92.23301% with 40 lines in your changes missing coverage. Please review.
✅ Project coverage is 86.15%. Comparing base (1a01b5a) to head (65e8cc8).
⚠️ Report is 66 commits behind head on main.

Files with missing linesPatch %Lines
lightning-types/src/unicode.rs92.18%40 Missing ⚠️
Additional details and impacted files
@@ Coverage Diff @@## main #4605 +/- ##
==========================================
+ Coverage 86.09% 86.15% +0.06% 
==========================================
Files 157 158 +1 Lines 108828 109323 +495 Branches 108828 109323 +495 ==========================================
+ Hits 93694 94189 +495 + Misses 12519 12518 -1 - Partials 2615 2616 +1 
FlagCoverage Δ
tests86.15% <92.23%> (+0.06%)⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@tnulltnull left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Alright, seems maybe a bit excessive, but fine by me. No need to check this on every CI run though, let's just move it to a weekly cronjob?

Comment thread.github/workflows/build.yml Outdated

@tnulltnull left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good, please squash.

Comment thread.github/workflows/check_unicode.yml
Comment thread.github/workflows/check_unicode.yml
Comment thread.github/workflows/check_unicode.yml Outdated
@tnull

tnull commented May 8, 2026

Copy link
Copy Markdown
Contributor

Claude's reviews seem legit though

1a01b5a added detection of unicode
format characters in `PrintableString`, but used a hard-coded table
which may eventually become out of date.
Here we switch to an auto-generated table, include all
`General_Category` `Other` characters, and also ban unallocated
code points.
Finally, CI validates that the file is kept up to date.
Written by Claude
@TheBlueMatt
TheBlueMattforce-pushed the 2026-05-unicode-autogen branch from 2de5954 to 65e8cc8CompareMay 8, 2026 19:44
@TheBlueMatt

Copy link
Copy Markdown
CollaboratorAuthor

Squashed and fixed claude's comments:

$ git diff-tree -U2 2de5954d9 65e8cc8d5
diff --git a/.github/workflows/check_unicode.yml b/.github/workflows/check_unicode.yml
index c22a4e3b29..a01add3f81 100644
--- a/.github/workflows/check_unicode.yml+++ b/.github/workflows/check_unicode.yml@@ -5,4 +5,5 @@ on:
- cron: '42 3 * * *'
+jobs:
check-unicode:
runs-on: ubuntu-latest
@@ -10,4 +11,6 @@ on:
issues: write
steps:
+ - name: Checkout source code+ uses: actions/checkout@v4
- name: Check unicode file state
env:
@@ -21,3 +24,3 @@ on:
BODY="The unicode character listing is out of date, see $RUN_URL"
gh issue create --title "$TITLE" --body "$BODY"
- fi+ fi

TITLE="Unicode listing out of date: ${{ github.workflow }}"
RUN_URL="https://github.com/${{ github.repository }}/actions/runs/${{ github.run_id }}"
BODY="The unicode character listing is out of date, see $RUN_URL"
gh issue create --title "$TITLE" --body "$BODY"

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bug: No deduplication — this creates a new issue every day the file is out of date. Unicode updates are infrequent (roughly annual), but if nobody regenerates the file promptly, this will spam the issue tracker with identical issues.

build.yml already has the right pattern (lines 380-391): check for an existing open issue with a label, and comment on it instead of creating a duplicate. This workflow should do the same.

Minimal fix sketch:

LABEL="unicode outdated"gh label create "$LABEL" 2>/dev/null || trueEXISTING=$(gh issue list --label "$LABEL" --state open --json number --jq '.[0].number // empty')if [ -n "$EXISTING" ]; thengh issue comment "$EXISTING" --body "$BODY"elsegh issue create --title "$TITLE" --label "$LABEL" --body "$BODY"fi

Copy link
Copy Markdown
CollaboratorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Eh, that takes effort, this is fine we expect to respond quickly.

@tnull
tnull merged commit 71fdc27 into lightningdevkit:mainMay 18, 2026
23 of 24 checks passed
@TheBlueMatt

Copy link
Copy Markdown
CollaboratorAuthor

Backported to 0.1 in #4680.

@TheBlueMatt

Copy link
Copy Markdown
CollaboratorAuthor

Backported to 0.2 in #4683.

TheBlueMatt added a commit that referenced this pull request Jun 19, 2026
v0.1.10 - Jun 18, 2026 - "Loupe de Loupe"
API Updates
===========
* `DefaultMessageRouter` will now always generate blinded message paths that
provide no privacy (where our node is the introduction node) for nodes with
public channels. This works around an issue which will appear for any nodes
with LND peers that enable onion messaging - such peers will refuse to
forward BOLT 12 messages from unknown third parties, which most BOLT 12
payers rely on today (#4647).
* Explicit `amount_msats` of 0 is rejected in BOLT 12 `Offer`s; `OfferBuilder`
now maps 0-amounts to an amount of `None` (#4324).
Bug Fixes
=========
* Async `ChannelMonitorUpdate` persistence operations which complete, but are
not marked as complete in a persisted `ChannelManager` prior to restart,
followed immediately by a block connection and then another restart could
result in some channel operations hanging leading for force-closures (#4377).
* If an MPP payment is claimed but `ChannelMonitorUpdate`s for some parts are
still being completed asynchronously, further channel updates (e.g.
forwarding another payment) are pending and the node restarts, the channel
could have become stuck (#4520).
* The presence of unconfirmed transactions actually no longer causes
`ElectrumSyncClient` to spuriously fail to sync (#4590).
* `FilesystemStore::list_all_keys` will no longer fail if there are stale
intermediate files lying around from a previous unclean shutdown (#4618).
* When forwarding an HTLC while in a blinded path with proportional fees over
200%, LDK will no longer spuriously allow a forward that pays us 1 msat too
little in fees (#4697).
* Fixed a rare case where a channel could get stuck on reconnect when using
both async `ChannelMonitorUpdate` persistence and async signing (#4684).
* `Event::PaymentSent::fee_paid_msat` is no longer `None` in cases where
`ChannelManager::abandon_payment` was called before the payment ultimately
completes anyway (#4651).
* Syncing a `ChainMonitor` using the `Confirm` trait will no longer write some
full `ChannelMonitor`s to disk several times per block (#4544).
* `OMDomainResolver` now correctly accounts for failed queries when rate
limiting, ensuring we continue to respond to queries after failures (#4591).
* Calling `ChannelManager::send_payment_with_route` without a `route_params`
and with an invalid `Route` will no longer panic (#4707).
* `lightning-custom-message`'s handling of `peer_connected` events now ensures
that sub-handlers will see a `peer_disconnected` event if a different
sub-handler refused the connection by `Err`ing `peer_connected` (#4595).
* Incomplete MPP keysend payments will no longer see their HTLCs held until
expiry (#4558).
* `InvoiceRequestBuilder` will no longer accept a `quantity` of `0` for a
BOLT 12 `Offer`, allowing any quantity up to a bound (#4667).
* `lightning-custom-message` handlers that return `Ok(None)` when asked to
deserialize a message in their defined range no longer cause panics (#4709).
* Several spurious debug assertions were fixed (#4537, #4618).
Security
========
0.1.10 fixes a sanitization issue and several denial-of-service vulnerabilities.
* `Bolt11Invoice::recover_payee_pub_key` no longer panics if called on an
invoice which set an explicit public key, rather than relying on public key
recovery. This method is called from `payment_parameters_from_invoice` and
`payment_parameters_from_variable_amount_invoice` (#4717).
* Maliciously-crafted unpayable invoices which have overflowing feerates will
no longer cause an `unwrap` failure panic (#4716).
* `possiblyrandom` did not properly generate random data except when it was
explicitly configured to. By default this means LDK is vulnerable to various
HashDoS attacks (#4719).
* `OMNameResolver` will no longer panic when looking up payment instructions
which include unicode characters at the start of a TXT record (#4718).
* `PrintableString` did not properly sanitize unicode format characters,
allowing an attacker to corrupt the rendering of logs or UI (#4593, #4605).
* RGS data is now limited in how large of a graph it is able to cause a client
to store in memory. Note that RGS data is still considered a DoS vector in
general and you should only use semi-trusted RGS data (#4713).
* Counterparty-provided strings in failure messages are no longer logged in
full, reducing the ability of such a counterparty to spam our logs (#4714).
* Reading a corrupted `ChannelManager` or `ProbabilisticScorer` can no longer
cause us to allocate large amounts of memory (#4712).
Thanks to Project Loupe for reporting most of the issues fixed in this release.
TheBlueMatt added a commit to TheBlueMatt/rust-lightning that referenced this pull request Jun 23, 2026
v0.2.3 - Jun 18, 2026 - "Through the Loupe"
API Updates
===========
* `DefaultMessageRouter` will now always generate blinded message paths that
provide no privacy (where our node is the introduction node) for nodes with
public channels. This works around an issue which will appear for any nodes
with LND peers that enable onion messaging - such peers will refuse to
forward BOLT 12 messages from unknown third parties, which most BOLT 12
payers rely on today (lightningdevkit#4647).
* Explicit `amount_msats` of 0 is rejected in BOLT 12 `Offer`s; `OfferBuilder`
now maps 0-amounts to an amount of `None` (lightningdevkit#4324).
Bug Fixes
=========
* `Features::supports_zero_conf` no longer clears the `ZeroConf` features and
`Features::requires_zero_conf` now correctly reports required, rather than
supported, status (lightningdevkit#4517).
* If an MPP payment is claimed but `ChannelMonitorUpdate`s for some parts are
still being completed asynchronously, further channel updates (e.g.
forwarding another payment) are pending and the node restarts, the channel
could have become stuck (lightningdevkit#4520).
* The presence of unconfirmed transactions actually no longer causes
`ElectrumSyncClient` to spuriously fail to sync (lightningdevkit#4590).
* LSPS1, LSPS2, and LSPS5 persistence will no longer get stuck and refuse to
persist again after a single failure from the KVStore (lightningdevkit#4597, lightningdevkit#4282).
* Dropping the future returned by
`OutputSweeper::regenerate_and_broadcast_spend_if_necessary` no longer
results in future calls to the same method being spuriously ignored (lightningdevkit#4598).
* Used async-receive offers are no longer refreshed on every timer tick once
their refresh time is reached (lightningdevkit#4672).
* `FilesystemStore::list_all_keys` will no longer fail if there are stale
intermediate files lying around from a previous unclean shutdown (lightningdevkit#4618).
* When forwarding an HTLC while in a blinded path with proportional fees over
200%, LDK will no longer spuriously allow a forward that pays us 1 msat too
little in fees (lightningdevkit#4697).
* Fixed a rare case where a channel could get stuck on reconnect when using
both async `ChannelMonitorUpdate` persistence and async signing (lightningdevkit#4684).
* If we had exactly zero balance in a zero-fee-commitment channel, the
counterparty was able to splice all of their balance out, violating the
reserve requirements they'd otherwise be forced to keep (lightningdevkit#4580).
* Providing an `Event::HTLCIntercepted` to the `LSPS2ServiceHandler` twice no
longer results in spuriously opening a channel early (lightningdevkit#4656).
* `Event::PaymentSent::fee_paid_msat` is no longer `None` in cases where
`ChannelManager::abandon_payment` was called before the payment ultimately
completes anyway (lightningdevkit#4651).
* `AnchorDescriptor::previous_utxo` now provides the correct `script_pubkey`
for non-zero-commitment-fee anchor channels (lightningdevkit#4669).
* Syncing a `ChainMonitor` using the `Confirm` trait will no longer write some
full `ChannelMonitor`s to disk several times per block (lightningdevkit#4544).
* `OMDomainResolver` now correctly accounts for failed queries when rate
limiting, ensuring we continue to respond to queries after failures (lightningdevkit#4591).
* Calling `ChannelManager::send_payment_with_route` without a `route_params`
and with an invalid `Route` will no longer panic (lightningdevkit#4707).
* `LSPS2ServiceHandler::channel_open_failed` now correctly fails intercepted
HTLCs rather than allowing them to fail just before expiry (lightningdevkit#4677).
* `StaticInvoice::is_offer_expired` was corrected to check offer, rather than
static invoice, expiry (lightningdevkit#4594).
* `lightning-custom-message`'s handling of `peer_connected` events now ensures
that sub-handlers will see a `peer_disconnected` event if a different
sub-handler refused the connection by `Err`ing `peer_connected` (lightningdevkit#4595).
* Replay protection for LSPS5 signatures now detects replays which are only
different in the encoded signature's case (lightningdevkit#4701).
* When `lightning-liquidity` is configured in the background processor, there
is no longer a stream of `Persisting LiquidityManager...` log spam (lightningdevkit#4246).
* Incomplete MPP keysend payments will no longer see their HTLCs held until
expiry (lightningdevkit#4558).
* `InvoiceRequestBuilder` will no longer accept a `quantity` of `0` for a
BOLT 12 `Offer`, allowing any quantity up to a bound (lightningdevkit#4667).
* `lightning-custom-message` handlers that return `Ok(None)` when asked to
deserialize a message in their defined range no longer cause panics (lightningdevkit#4709).
* Several spurious debug assertions were fixed (lightningdevkit#4537, lightningdevkit#4618, lightningdevkit#4026)
Security
========
0.2.3 fixes several underestimates of the anchor reserves required to ensure we
can reliably close channels, several denial-of-service vulnerabilities and a
sanitization issue.
* `Bolt11Invoice::recover_payee_pub_key` no longer panics if called on an
invoice which set an explicit public key, rather than relying on public key
recovery. Note that this method is called from
`PaymentParameters::from_bolt11_invoice` (lightningdevkit#4717).
* Maliciously-crafted unpayable invoices which have overflowing feerates will
no longer cause an `unwrap` failure panic (lightningdevkit#4716).
* Parsing an `LSPSDateTime` which is before 1970 no longer panics. This is
reachable when parsing messages from counterparties (lightningdevkit#4715).
* `possiblyrandom` did not properly generate random data except when it was
explicitly configured to. By default this means LDK is vulnerable to various
HashDoS attacks (lightningdevkit#4719).
* `OMNameResolver` will no longer panic when looking up payment instructions
which include unicode characters at the start of a TXT record (lightningdevkit#4718).
* When using the `anchor_channel_reserves` module to calculate reserves
required to pay for fees when closing anchor channels, zero-fee-commitment
channels were not considered. This could allow a counterparty to open many
channels, leaving us unable to properly force-close (lightningdevkit#4592).
* The `anchor_channel_reserves` module overestimated the value of `Utxo`s in
the wallet by ignoring the `TxIn` cost to spend them (lightningdevkit#4670).
* `PrintableString` did not properly sanitize unicode format characters,
allowing an attacker to corrupt the rendering of logs or UI (lightningdevkit#4593, lightningdevkit#4605).
* RGS data is now limited in how large of a graph it is able to cause a client
to store in memory. Note that RGS data is still considered a DoS vector in
general and you should only use semi-trusted RGS data (lightningdevkit#4713).
* Counterparty-provided strings in failure messages are no longer logged in
full, reducing the ability of such a counterparty to spam our logs (lightningdevkit#4714).
* Reading a corrupted `ChannelManager` or `ProbabilisticScorer` can no longer
cause us to allocate large amounts of memory (lightningdevkit#4712).
Thanks to Project Loupe for reporting most of the issues fixed in this release.
Conflicts resolved in:
* lightning/src/chain/channelmonitor.rs
* lightning/src/events/mod.rs
* lightning/src/ln/channelmanager.rs
* lightning/src/ln/mod.rs
* lightning/src/ln/offers_tests.rs
* lightning/src/ln/onion_utils.rs
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants

@TheBlueMatt@ldk-reviews-bot@ldk-claude-review-bot@tnull
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Remove or un-stick sticky/fixed headers that block content\n(function() {\n function unstick() {\n document.querySelectorAll('header, nav, [role=\"banner\"], .header, .navbar, .sticky, .fixed-top, [style*=\"position: fixed\"], [style*=\"position:sticky\"]').forEach(function(el) {\n if (el.style.position === 'fixed' || el.style.position === 'sticky' || \n getComputedStyle(el).position === 'fixed' || getComputedStyle(el).position === 'sticky') {\n el.style.position = 'static';\n el.style.top = 'auto';\n el.style.zIndex = 'auto';\n }\n });\n }\n \n unstick();\n \n var observer = new MutationObserver(unstick);\n observer.observe(document.body, { childList: true, subtree: true, attributes: true, attributeFilter: ['style', 'class'] });\n})();", "Kill Sticky Headers"); } } catch(__e) { console.warn('[Userscript:Kill Sticky Headers]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
Skip to content

Add an auto-generated unicode character category file - #4605

Merged
tnull merged 1 commit into
lightningdevkit:mainfrom
TheBlueMatt:2026-05-unicode-autogen
May 18, 2026
Merged

Add an auto-generated unicode character category file#4605
tnull merged 1 commit into
lightningdevkit:mainfrom
TheBlueMatt:2026-05-unicode-autogen

Conversation

@TheBlueMatt

Copy link
Copy Markdown
Collaborator

1a01b5a added detection of unicode format characters in PrintableString, but used a hard-coded table which may eventually become out of date.

Here we switch to an auto-generated table, include all General_CategoryOther characters, and also ban unallocated code points.

Finally, CI validates that the file is kept up to date.

Written by Claude

@ldk-reviews-bot

ldk-reviews-bot commented May 7, 2026

Copy link
Copy Markdown

👋 Thanks for assigning @tnull as a reviewer!
I'll wait for their review and will help manage the review process.
Once they submit their review, I'll check if a second reviewer would be helpful.

@TheBlueMatt
TheBlueMatt requested a review from tnullMay 7, 2026 18:47
@TheBlueMatt
TheBlueMattforce-pushed the 2026-05-unicode-autogen branch from b6f8c03 to bd75483CompareMay 7, 2026 18:49
Comment on lines +36 to +38
let is_other = is_unicode_general_category_other(c);
let is_unassigned = is_unicode_general_category_unassigned(c);
let c = if c.is_control() || is_other || is_unassigned {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: c.is_control() is now fully redundant — it checks Cc (Control), which is already covered by is_unicode_general_category_other (see 0x0000..=0x001F and 0x007F..=0x009F in unicode.rs). The old code needed it because is_format_char only covered Cf, but the new function covers all of Cc / Cf / Cs / Co.

Not a bug (the || short-circuits harmlessly), but it's potentially confusing because it suggests is_other doesn't handle control characters.

Suggested change
let is_other = is_unicode_general_category_other(c);
let is_unassigned = is_unicode_general_category_unassigned(c);
let c = if c.is_control() || is_other || is_unassigned {
let c = ifis_unicode_general_category_other(c) || is_unicode_general_category_unassigned(c){

@ldk-claude-review-bot

ldk-claude-review-bot commented May 7, 2026

Copy link
Copy Markdown
Collaborator

Review Summary

New issue found

  • .github/workflows/check_unicode.yml:25gh issue create runs without deduplication; will create a new issue every day the file is out of date. build.yml already demonstrates the correct pattern (check for existing open issue, comment on it instead).

Correction to prior review

My three prior comments about the YAML workflow file (missing jobs: key, missing checkout step, fi indentation) were incorrect — the file structure is fine as written. I misread the diff indentation. Apologies for the noise.

Prior comment still applicable

  • lightning-types/src/string.rs:38c.is_control() remains redundant with is_unicode_general_category_other, which already covers all Cc characters.

@codecov

codecovBot commented May 7, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 92.23301% with 40 lines in your changes missing coverage. Please review.
✅ Project coverage is 86.15%. Comparing base (1a01b5a) to head (65e8cc8).
⚠️ Report is 66 commits behind head on main.

Files with missing linesPatch %Lines
lightning-types/src/unicode.rs92.18%40 Missing ⚠️
Additional details and impacted files
@@ Coverage Diff @@## main #4605 +/- ##
==========================================
+ Coverage 86.09% 86.15% +0.06% 
==========================================
Files 157 158 +1 Lines 108828 109323 +495 Branches 108828 109323 +495 ==========================================
+ Hits 93694 94189 +495 + Misses 12519 12518 -1 - Partials 2615 2616 +1 
FlagCoverage Δ
tests86.15% <92.23%> (+0.06%)⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@tnulltnull left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Alright, seems maybe a bit excessive, but fine by me. No need to check this on every CI run though, let's just move it to a weekly cronjob?

Comment thread.github/workflows/build.yml Outdated

@tnulltnull left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good, please squash.

Comment thread.github/workflows/check_unicode.yml
Comment thread.github/workflows/check_unicode.yml
Comment thread.github/workflows/check_unicode.yml Outdated
@tnull

tnull commented May 8, 2026

Copy link
Copy Markdown
Contributor

Claude's reviews seem legit though

1a01b5a added detection of unicode
format characters in `PrintableString`, but used a hard-coded table
which may eventually become out of date.
Here we switch to an auto-generated table, include all
`General_Category` `Other` characters, and also ban unallocated
code points.
Finally, CI validates that the file is kept up to date.
Written by Claude
@TheBlueMatt
TheBlueMattforce-pushed the 2026-05-unicode-autogen branch from 2de5954 to 65e8cc8CompareMay 8, 2026 19:44
@TheBlueMatt

Copy link
Copy Markdown
CollaboratorAuthor

Squashed and fixed claude's comments:

$ git diff-tree -U2 2de5954d9 65e8cc8d5
diff --git a/.github/workflows/check_unicode.yml b/.github/workflows/check_unicode.yml
index c22a4e3b29..a01add3f81 100644
--- a/.github/workflows/check_unicode.yml+++ b/.github/workflows/check_unicode.yml@@ -5,4 +5,5 @@ on:
- cron: '42 3 * * *'
+jobs:
check-unicode:
runs-on: ubuntu-latest
@@ -10,4 +11,6 @@ on:
issues: write
steps:
+ - name: Checkout source code+ uses: actions/checkout@v4
- name: Check unicode file state
env:
@@ -21,3 +24,3 @@ on:
BODY="The unicode character listing is out of date, see $RUN_URL"
gh issue create --title "$TITLE" --body "$BODY"
- fi+ fi

TITLE="Unicode listing out of date: ${{ github.workflow }}"
RUN_URL="https://github.com/${{ github.repository }}/actions/runs/${{ github.run_id }}"
BODY="The unicode character listing is out of date, see $RUN_URL"
gh issue create --title "$TITLE" --body "$BODY"

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bug: No deduplication — this creates a new issue every day the file is out of date. Unicode updates are infrequent (roughly annual), but if nobody regenerates the file promptly, this will spam the issue tracker with identical issues.

build.yml already has the right pattern (lines 380-391): check for an existing open issue with a label, and comment on it instead of creating a duplicate. This workflow should do the same.

Minimal fix sketch:

LABEL="unicode outdated"gh label create "$LABEL" 2>/dev/null || trueEXISTING=$(gh issue list --label "$LABEL" --state open --json number --jq '.[0].number // empty')if [ -n "$EXISTING" ]; thengh issue comment "$EXISTING" --body "$BODY"elsegh issue create --title "$TITLE" --label "$LABEL" --body "$BODY"fi

Copy link
Copy Markdown
CollaboratorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Eh, that takes effort, this is fine we expect to respond quickly.

@tnull
tnull merged commit 71fdc27 into lightningdevkit:mainMay 18, 2026
23 of 24 checks passed
@TheBlueMatt

Copy link
Copy Markdown
CollaboratorAuthor

Backported to 0.1 in #4680.

@TheBlueMatt

Copy link
Copy Markdown
CollaboratorAuthor

Backported to 0.2 in #4683.

TheBlueMatt added a commit that referenced this pull request Jun 19, 2026
v0.1.10 - Jun 18, 2026 - "Loupe de Loupe"
API Updates
===========
* `DefaultMessageRouter` will now always generate blinded message paths that
provide no privacy (where our node is the introduction node) for nodes with
public channels. This works around an issue which will appear for any nodes
with LND peers that enable onion messaging - such peers will refuse to
forward BOLT 12 messages from unknown third parties, which most BOLT 12
payers rely on today (#4647).
* Explicit `amount_msats` of 0 is rejected in BOLT 12 `Offer`s; `OfferBuilder`
now maps 0-amounts to an amount of `None` (#4324).
Bug Fixes
=========
* Async `ChannelMonitorUpdate` persistence operations which complete, but are
not marked as complete in a persisted `ChannelManager` prior to restart,
followed immediately by a block connection and then another restart could
result in some channel operations hanging leading for force-closures (#4377).
* If an MPP payment is claimed but `ChannelMonitorUpdate`s for some parts are
still being completed asynchronously, further channel updates (e.g.
forwarding another payment) are pending and the node restarts, the channel
could have become stuck (#4520).
* The presence of unconfirmed transactions actually no longer causes
`ElectrumSyncClient` to spuriously fail to sync (#4590).
* `FilesystemStore::list_all_keys` will no longer fail if there are stale
intermediate files lying around from a previous unclean shutdown (#4618).
* When forwarding an HTLC while in a blinded path with proportional fees over
200%, LDK will no longer spuriously allow a forward that pays us 1 msat too
little in fees (#4697).
* Fixed a rare case where a channel could get stuck on reconnect when using
both async `ChannelMonitorUpdate` persistence and async signing (#4684).
* `Event::PaymentSent::fee_paid_msat` is no longer `None` in cases where
`ChannelManager::abandon_payment` was called before the payment ultimately
completes anyway (#4651).
* Syncing a `ChainMonitor` using the `Confirm` trait will no longer write some
full `ChannelMonitor`s to disk several times per block (#4544).
* `OMDomainResolver` now correctly accounts for failed queries when rate
limiting, ensuring we continue to respond to queries after failures (#4591).
* Calling `ChannelManager::send_payment_with_route` without a `route_params`
and with an invalid `Route` will no longer panic (#4707).
* `lightning-custom-message`'s handling of `peer_connected` events now ensures
that sub-handlers will see a `peer_disconnected` event if a different
sub-handler refused the connection by `Err`ing `peer_connected` (#4595).
* Incomplete MPP keysend payments will no longer see their HTLCs held until
expiry (#4558).
* `InvoiceRequestBuilder` will no longer accept a `quantity` of `0` for a
BOLT 12 `Offer`, allowing any quantity up to a bound (#4667).
* `lightning-custom-message` handlers that return `Ok(None)` when asked to
deserialize a message in their defined range no longer cause panics (#4709).
* Several spurious debug assertions were fixed (#4537, #4618).
Security
========
0.1.10 fixes a sanitization issue and several denial-of-service vulnerabilities.
* `Bolt11Invoice::recover_payee_pub_key` no longer panics if called on an
invoice which set an explicit public key, rather than relying on public key
recovery. This method is called from `payment_parameters_from_invoice` and
`payment_parameters_from_variable_amount_invoice` (#4717).
* Maliciously-crafted unpayable invoices which have overflowing feerates will
no longer cause an `unwrap` failure panic (#4716).
* `possiblyrandom` did not properly generate random data except when it was
explicitly configured to. By default this means LDK is vulnerable to various
HashDoS attacks (#4719).
* `OMNameResolver` will no longer panic when looking up payment instructions
which include unicode characters at the start of a TXT record (#4718).
* `PrintableString` did not properly sanitize unicode format characters,
allowing an attacker to corrupt the rendering of logs or UI (#4593, #4605).
* RGS data is now limited in how large of a graph it is able to cause a client
to store in memory. Note that RGS data is still considered a DoS vector in
general and you should only use semi-trusted RGS data (#4713).
* Counterparty-provided strings in failure messages are no longer logged in
full, reducing the ability of such a counterparty to spam our logs (#4714).
* Reading a corrupted `ChannelManager` or `ProbabilisticScorer` can no longer
cause us to allocate large amounts of memory (#4712).
Thanks to Project Loupe for reporting most of the issues fixed in this release.
TheBlueMatt added a commit to TheBlueMatt/rust-lightning that referenced this pull request Jun 23, 2026
v0.2.3 - Jun 18, 2026 - "Through the Loupe"
API Updates
===========
* `DefaultMessageRouter` will now always generate blinded message paths that
provide no privacy (where our node is the introduction node) for nodes with
public channels. This works around an issue which will appear for any nodes
with LND peers that enable onion messaging - such peers will refuse to
forward BOLT 12 messages from unknown third parties, which most BOLT 12
payers rely on today (lightningdevkit#4647).
* Explicit `amount_msats` of 0 is rejected in BOLT 12 `Offer`s; `OfferBuilder`
now maps 0-amounts to an amount of `None` (lightningdevkit#4324).
Bug Fixes
=========
* `Features::supports_zero_conf` no longer clears the `ZeroConf` features and
`Features::requires_zero_conf` now correctly reports required, rather than
supported, status (lightningdevkit#4517).
* If an MPP payment is claimed but `ChannelMonitorUpdate`s for some parts are
still being completed asynchronously, further channel updates (e.g.
forwarding another payment) are pending and the node restarts, the channel
could have become stuck (lightningdevkit#4520).
* The presence of unconfirmed transactions actually no longer causes
`ElectrumSyncClient` to spuriously fail to sync (lightningdevkit#4590).
* LSPS1, LSPS2, and LSPS5 persistence will no longer get stuck and refuse to
persist again after a single failure from the KVStore (lightningdevkit#4597, lightningdevkit#4282).
* Dropping the future returned by
`OutputSweeper::regenerate_and_broadcast_spend_if_necessary` no longer
results in future calls to the same method being spuriously ignored (lightningdevkit#4598).
* Used async-receive offers are no longer refreshed on every timer tick once
their refresh time is reached (lightningdevkit#4672).
* `FilesystemStore::list_all_keys` will no longer fail if there are stale
intermediate files lying around from a previous unclean shutdown (lightningdevkit#4618).
* When forwarding an HTLC while in a blinded path with proportional fees over
200%, LDK will no longer spuriously allow a forward that pays us 1 msat too
little in fees (lightningdevkit#4697).
* Fixed a rare case where a channel could get stuck on reconnect when using
both async `ChannelMonitorUpdate` persistence and async signing (lightningdevkit#4684).
* If we had exactly zero balance in a zero-fee-commitment channel, the
counterparty was able to splice all of their balance out, violating the
reserve requirements they'd otherwise be forced to keep (lightningdevkit#4580).
* Providing an `Event::HTLCIntercepted` to the `LSPS2ServiceHandler` twice no
longer results in spuriously opening a channel early (lightningdevkit#4656).
* `Event::PaymentSent::fee_paid_msat` is no longer `None` in cases where
`ChannelManager::abandon_payment` was called before the payment ultimately
completes anyway (lightningdevkit#4651).
* `AnchorDescriptor::previous_utxo` now provides the correct `script_pubkey`
for non-zero-commitment-fee anchor channels (lightningdevkit#4669).
* Syncing a `ChainMonitor` using the `Confirm` trait will no longer write some
full `ChannelMonitor`s to disk several times per block (lightningdevkit#4544).
* `OMDomainResolver` now correctly accounts for failed queries when rate
limiting, ensuring we continue to respond to queries after failures (lightningdevkit#4591).
* Calling `ChannelManager::send_payment_with_route` without a `route_params`
and with an invalid `Route` will no longer panic (lightningdevkit#4707).
* `LSPS2ServiceHandler::channel_open_failed` now correctly fails intercepted
HTLCs rather than allowing them to fail just before expiry (lightningdevkit#4677).
* `StaticInvoice::is_offer_expired` was corrected to check offer, rather than
static invoice, expiry (lightningdevkit#4594).
* `lightning-custom-message`'s handling of `peer_connected` events now ensures
that sub-handlers will see a `peer_disconnected` event if a different
sub-handler refused the connection by `Err`ing `peer_connected` (lightningdevkit#4595).
* Replay protection for LSPS5 signatures now detects replays which are only
different in the encoded signature's case (lightningdevkit#4701).
* When `lightning-liquidity` is configured in the background processor, there
is no longer a stream of `Persisting LiquidityManager...` log spam (lightningdevkit#4246).
* Incomplete MPP keysend payments will no longer see their HTLCs held until
expiry (lightningdevkit#4558).
* `InvoiceRequestBuilder` will no longer accept a `quantity` of `0` for a
BOLT 12 `Offer`, allowing any quantity up to a bound (lightningdevkit#4667).
* `lightning-custom-message` handlers that return `Ok(None)` when asked to
deserialize a message in their defined range no longer cause panics (lightningdevkit#4709).
* Several spurious debug assertions were fixed (lightningdevkit#4537, lightningdevkit#4618, lightningdevkit#4026)
Security
========
0.2.3 fixes several underestimates of the anchor reserves required to ensure we
can reliably close channels, several denial-of-service vulnerabilities and a
sanitization issue.
* `Bolt11Invoice::recover_payee_pub_key` no longer panics if called on an
invoice which set an explicit public key, rather than relying on public key
recovery. Note that this method is called from
`PaymentParameters::from_bolt11_invoice` (lightningdevkit#4717).
* Maliciously-crafted unpayable invoices which have overflowing feerates will
no longer cause an `unwrap` failure panic (lightningdevkit#4716).
* Parsing an `LSPSDateTime` which is before 1970 no longer panics. This is
reachable when parsing messages from counterparties (lightningdevkit#4715).
* `possiblyrandom` did not properly generate random data except when it was
explicitly configured to. By default this means LDK is vulnerable to various
HashDoS attacks (lightningdevkit#4719).
* `OMNameResolver` will no longer panic when looking up payment instructions
which include unicode characters at the start of a TXT record (lightningdevkit#4718).
* When using the `anchor_channel_reserves` module to calculate reserves
required to pay for fees when closing anchor channels, zero-fee-commitment
channels were not considered. This could allow a counterparty to open many
channels, leaving us unable to properly force-close (lightningdevkit#4592).
* The `anchor_channel_reserves` module overestimated the value of `Utxo`s in
the wallet by ignoring the `TxIn` cost to spend them (lightningdevkit#4670).
* `PrintableString` did not properly sanitize unicode format characters,
allowing an attacker to corrupt the rendering of logs or UI (lightningdevkit#4593, lightningdevkit#4605).
* RGS data is now limited in how large of a graph it is able to cause a client
to store in memory. Note that RGS data is still considered a DoS vector in
general and you should only use semi-trusted RGS data (lightningdevkit#4713).
* Counterparty-provided strings in failure messages are no longer logged in
full, reducing the ability of such a counterparty to spam our logs (lightningdevkit#4714).
* Reading a corrupted `ChannelManager` or `ProbabilisticScorer` can no longer
cause us to allocate large amounts of memory (lightningdevkit#4712).
Thanks to Project Loupe for reporting most of the issues fixed in this release.
Conflicts resolved in:
* lightning/src/chain/channelmonitor.rs
* lightning/src/events/mod.rs
* lightning/src/ln/channelmanager.rs
* lightning/src/ln/mod.rs
* lightning/src/ln/offers_tests.rs
* lightning/src/ln/onion_utils.rs
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants

@TheBlueMatt@ldk-reviews-bot@ldk-claude-review-bot@tnull
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Universal Dark Mode - works on any site\n(function() {\n var enabled = true;\n \n function applyDarkMode() {\n if (!enabled) return;\n \n // Create style element if it doesn't exist\n var style = document.getElementById('universal-dark-mode-style');\n if (!style) {\n style = document.createElement('style');\n style.id = 'universal-dark-mode-style';\n document.head.appendChild(style);\n }\n \n // Dark mode CSS - inverts colors but preserves images/video\n style.textContent = '\n /* Invert everything except media */\n html {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #1a1a2e !important;\n }\n \n /* Restore images, videos, iframes, canvas */\n img, video, iframe, canvas, svg, picture, [style*=\"background-image\"] {\n filter: invert(1) hue-rotate(180deg) !important;\n }\n \n /* Preserve specific elements that should not be inverted */\n .no-dark-mode, .no-dark-mode *,\n [data-theme=\"light\"], [data-theme=\"light\"],\n .ace_editor, .ace_editor *,\n .CodeMirror, .CodeMirror *,\n .monaco-editor, .monaco-editor *,\n .markdown-body pre, .markdown-body pre *,\n .highlight, .highlight *,\n pre code, pre code * {\n filter: none !important;\n }\n \n /* Fix common UI elements */\n .modal, .popup, .dropdown-menu, .tooltip, .popover {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #2d2d44 !important;\n border-color: #444 !important;\n }\n \n /* Scrollbars */\n ::-webkit-scrollbar { background: #1a1a2e !important; }\n ::-webkit-scrollbar-thumb { background: #444 !important; }\n ::-webkit-scrollbar-thumb:hover { background: #555 !important; }\n \n /* Selection */\n ::selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ::-moz-selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ';\n }\n \n function removeDarkMode() {\n var style = document.getElementById('universal-dark-mode-style');\n if (style) style.remove();\n }\n \n // Toggle with Alt+Shift+D\n document.addEventListener('keydown', function(e) {\n if (e.altKey && e.shiftKey && e.key === 'D') {\n e.preventDefault();\n enabled = !enabled;\n if (enabled) {\n applyDarkMode();\n console.log('[Universal Dark Mode] Enabled');\n } else {\n removeDarkMode();\n console.log('[Universal Dark Mode] Disabled');\n }\n }\n });\n \n // Apply on load\n applyDarkMode();\n \n // Re-apply on dynamic content\n var observer = new MutationObserver(function(mutations) {\n if (enabled && !document.getElementById('universal-dark-mode-style')) {\n applyDarkMode();\n }\n });\n observer.observe(document.head, { childList: true });\n \n console.log('[Universal Dark Mode] Loaded - Press Alt+Shift+D to toggle');\n})();", "Universal Dark Mode"); } } catch(__e) { console.warn('[Userscript:Universal Dark Mode]', __e); } })(); })();
Skip to content

Add an auto-generated unicode character category file - #4605

Merged
tnull merged 1 commit into
lightningdevkit:mainfrom
TheBlueMatt:2026-05-unicode-autogen
May 18, 2026
Merged

Add an auto-generated unicode character category file#4605
tnull merged 1 commit into
lightningdevkit:mainfrom
TheBlueMatt:2026-05-unicode-autogen

Conversation

@TheBlueMatt

Copy link
Copy Markdown
Collaborator

1a01b5a added detection of unicode format characters in PrintableString, but used a hard-coded table which may eventually become out of date.

Here we switch to an auto-generated table, include all General_CategoryOther characters, and also ban unallocated code points.

Finally, CI validates that the file is kept up to date.

Written by Claude

@ldk-reviews-bot

ldk-reviews-bot commented May 7, 2026

Copy link
Copy Markdown

👋 Thanks for assigning @tnull as a reviewer!
I'll wait for their review and will help manage the review process.
Once they submit their review, I'll check if a second reviewer would be helpful.

@TheBlueMatt
TheBlueMatt requested a review from tnullMay 7, 2026 18:47
@TheBlueMatt
TheBlueMattforce-pushed the 2026-05-unicode-autogen branch from b6f8c03 to bd75483CompareMay 7, 2026 18:49
Comment on lines +36 to +38
let is_other = is_unicode_general_category_other(c);
let is_unassigned = is_unicode_general_category_unassigned(c);
let c = if c.is_control() || is_other || is_unassigned {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: c.is_control() is now fully redundant — it checks Cc (Control), which is already covered by is_unicode_general_category_other (see 0x0000..=0x001F and 0x007F..=0x009F in unicode.rs). The old code needed it because is_format_char only covered Cf, but the new function covers all of Cc / Cf / Cs / Co.

Not a bug (the || short-circuits harmlessly), but it's potentially confusing because it suggests is_other doesn't handle control characters.

Suggested change
let is_other = is_unicode_general_category_other(c);
let is_unassigned = is_unicode_general_category_unassigned(c);
let c = if c.is_control() || is_other || is_unassigned {
let c = ifis_unicode_general_category_other(c) || is_unicode_general_category_unassigned(c){

@ldk-claude-review-bot

ldk-claude-review-bot commented May 7, 2026

Copy link
Copy Markdown
Collaborator

Review Summary

New issue found

  • .github/workflows/check_unicode.yml:25gh issue create runs without deduplication; will create a new issue every day the file is out of date. build.yml already demonstrates the correct pattern (check for existing open issue, comment on it instead).

Correction to prior review

My three prior comments about the YAML workflow file (missing jobs: key, missing checkout step, fi indentation) were incorrect — the file structure is fine as written. I misread the diff indentation. Apologies for the noise.

Prior comment still applicable

  • lightning-types/src/string.rs:38c.is_control() remains redundant with is_unicode_general_category_other, which already covers all Cc characters.

@codecov

codecovBot commented May 7, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 92.23301% with 40 lines in your changes missing coverage. Please review.
✅ Project coverage is 86.15%. Comparing base (1a01b5a) to head (65e8cc8).
⚠️ Report is 66 commits behind head on main.

Files with missing linesPatch %Lines
lightning-types/src/unicode.rs92.18%40 Missing ⚠️
Additional details and impacted files
@@ Coverage Diff @@## main #4605 +/- ##
==========================================
+ Coverage 86.09% 86.15% +0.06% 
==========================================
Files 157 158 +1 Lines 108828 109323 +495 Branches 108828 109323 +495 ==========================================
+ Hits 93694 94189 +495 + Misses 12519 12518 -1 - Partials 2615 2616 +1 
FlagCoverage Δ
tests86.15% <92.23%> (+0.06%)⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@tnulltnull left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Alright, seems maybe a bit excessive, but fine by me. No need to check this on every CI run though, let's just move it to a weekly cronjob?

Comment thread.github/workflows/build.yml Outdated

@tnulltnull left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good, please squash.

Comment thread.github/workflows/check_unicode.yml
Comment thread.github/workflows/check_unicode.yml
Comment thread.github/workflows/check_unicode.yml Outdated
@tnull

tnull commented May 8, 2026

Copy link
Copy Markdown
Contributor

Claude's reviews seem legit though

1a01b5a added detection of unicode
format characters in `PrintableString`, but used a hard-coded table
which may eventually become out of date.
Here we switch to an auto-generated table, include all
`General_Category` `Other` characters, and also ban unallocated
code points.
Finally, CI validates that the file is kept up to date.
Written by Claude
@TheBlueMatt
TheBlueMattforce-pushed the 2026-05-unicode-autogen branch from 2de5954 to 65e8cc8CompareMay 8, 2026 19:44
@TheBlueMatt

Copy link
Copy Markdown
CollaboratorAuthor

Squashed and fixed claude's comments:

$ git diff-tree -U2 2de5954d9 65e8cc8d5
diff --git a/.github/workflows/check_unicode.yml b/.github/workflows/check_unicode.yml
index c22a4e3b29..a01add3f81 100644
--- a/.github/workflows/check_unicode.yml+++ b/.github/workflows/check_unicode.yml@@ -5,4 +5,5 @@ on:
- cron: '42 3 * * *'
+jobs:
check-unicode:
runs-on: ubuntu-latest
@@ -10,4 +11,6 @@ on:
issues: write
steps:
+ - name: Checkout source code+ uses: actions/checkout@v4
- name: Check unicode file state
env:
@@ -21,3 +24,3 @@ on:
BODY="The unicode character listing is out of date, see $RUN_URL"
gh issue create --title "$TITLE" --body "$BODY"
- fi+ fi

TITLE="Unicode listing out of date: ${{ github.workflow }}"
RUN_URL="https://github.com/${{ github.repository }}/actions/runs/${{ github.run_id }}"
BODY="The unicode character listing is out of date, see $RUN_URL"
gh issue create --title "$TITLE" --body "$BODY"

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bug: No deduplication — this creates a new issue every day the file is out of date. Unicode updates are infrequent (roughly annual), but if nobody regenerates the file promptly, this will spam the issue tracker with identical issues.

build.yml already has the right pattern (lines 380-391): check for an existing open issue with a label, and comment on it instead of creating a duplicate. This workflow should do the same.

Minimal fix sketch:

LABEL="unicode outdated"gh label create "$LABEL" 2>/dev/null || trueEXISTING=$(gh issue list --label "$LABEL" --state open --json number --jq '.[0].number // empty')if [ -n "$EXISTING" ]; thengh issue comment "$EXISTING" --body "$BODY"elsegh issue create --title "$TITLE" --label "$LABEL" --body "$BODY"fi

Copy link
Copy Markdown
CollaboratorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Eh, that takes effort, this is fine we expect to respond quickly.

@tnull
tnull merged commit 71fdc27 into lightningdevkit:mainMay 18, 2026
23 of 24 checks passed
@TheBlueMatt

Copy link
Copy Markdown
CollaboratorAuthor

Backported to 0.1 in #4680.

@TheBlueMatt

Copy link
Copy Markdown
CollaboratorAuthor

Backported to 0.2 in #4683.

TheBlueMatt added a commit that referenced this pull request Jun 19, 2026
v0.1.10 - Jun 18, 2026 - "Loupe de Loupe"
API Updates
===========
* `DefaultMessageRouter` will now always generate blinded message paths that
provide no privacy (where our node is the introduction node) for nodes with
public channels. This works around an issue which will appear for any nodes
with LND peers that enable onion messaging - such peers will refuse to
forward BOLT 12 messages from unknown third parties, which most BOLT 12
payers rely on today (#4647).
* Explicit `amount_msats` of 0 is rejected in BOLT 12 `Offer`s; `OfferBuilder`
now maps 0-amounts to an amount of `None` (#4324).
Bug Fixes
=========
* Async `ChannelMonitorUpdate` persistence operations which complete, but are
not marked as complete in a persisted `ChannelManager` prior to restart,
followed immediately by a block connection and then another restart could
result in some channel operations hanging leading for force-closures (#4377).
* If an MPP payment is claimed but `ChannelMonitorUpdate`s for some parts are
still being completed asynchronously, further channel updates (e.g.
forwarding another payment) are pending and the node restarts, the channel
could have become stuck (#4520).
* The presence of unconfirmed transactions actually no longer causes
`ElectrumSyncClient` to spuriously fail to sync (#4590).
* `FilesystemStore::list_all_keys` will no longer fail if there are stale
intermediate files lying around from a previous unclean shutdown (#4618).
* When forwarding an HTLC while in a blinded path with proportional fees over
200%, LDK will no longer spuriously allow a forward that pays us 1 msat too
little in fees (#4697).
* Fixed a rare case where a channel could get stuck on reconnect when using
both async `ChannelMonitorUpdate` persistence and async signing (#4684).
* `Event::PaymentSent::fee_paid_msat` is no longer `None` in cases where
`ChannelManager::abandon_payment` was called before the payment ultimately
completes anyway (#4651).
* Syncing a `ChainMonitor` using the `Confirm` trait will no longer write some
full `ChannelMonitor`s to disk several times per block (#4544).
* `OMDomainResolver` now correctly accounts for failed queries when rate
limiting, ensuring we continue to respond to queries after failures (#4591).
* Calling `ChannelManager::send_payment_with_route` without a `route_params`
and with an invalid `Route` will no longer panic (#4707).
* `lightning-custom-message`'s handling of `peer_connected` events now ensures
that sub-handlers will see a `peer_disconnected` event if a different
sub-handler refused the connection by `Err`ing `peer_connected` (#4595).
* Incomplete MPP keysend payments will no longer see their HTLCs held until
expiry (#4558).
* `InvoiceRequestBuilder` will no longer accept a `quantity` of `0` for a
BOLT 12 `Offer`, allowing any quantity up to a bound (#4667).
* `lightning-custom-message` handlers that return `Ok(None)` when asked to
deserialize a message in their defined range no longer cause panics (#4709).
* Several spurious debug assertions were fixed (#4537, #4618).
Security
========
0.1.10 fixes a sanitization issue and several denial-of-service vulnerabilities.
* `Bolt11Invoice::recover_payee_pub_key` no longer panics if called on an
invoice which set an explicit public key, rather than relying on public key
recovery. This method is called from `payment_parameters_from_invoice` and
`payment_parameters_from_variable_amount_invoice` (#4717).
* Maliciously-crafted unpayable invoices which have overflowing feerates will
no longer cause an `unwrap` failure panic (#4716).
* `possiblyrandom` did not properly generate random data except when it was
explicitly configured to. By default this means LDK is vulnerable to various
HashDoS attacks (#4719).
* `OMNameResolver` will no longer panic when looking up payment instructions
which include unicode characters at the start of a TXT record (#4718).
* `PrintableString` did not properly sanitize unicode format characters,
allowing an attacker to corrupt the rendering of logs or UI (#4593, #4605).
* RGS data is now limited in how large of a graph it is able to cause a client
to store in memory. Note that RGS data is still considered a DoS vector in
general and you should only use semi-trusted RGS data (#4713).
* Counterparty-provided strings in failure messages are no longer logged in
full, reducing the ability of such a counterparty to spam our logs (#4714).
* Reading a corrupted `ChannelManager` or `ProbabilisticScorer` can no longer
cause us to allocate large amounts of memory (#4712).
Thanks to Project Loupe for reporting most of the issues fixed in this release.
TheBlueMatt added a commit to TheBlueMatt/rust-lightning that referenced this pull request Jun 23, 2026
v0.2.3 - Jun 18, 2026 - "Through the Loupe"
API Updates
===========
* `DefaultMessageRouter` will now always generate blinded message paths that
provide no privacy (where our node is the introduction node) for nodes with
public channels. This works around an issue which will appear for any nodes
with LND peers that enable onion messaging - such peers will refuse to
forward BOLT 12 messages from unknown third parties, which most BOLT 12
payers rely on today (lightningdevkit#4647).
* Explicit `amount_msats` of 0 is rejected in BOLT 12 `Offer`s; `OfferBuilder`
now maps 0-amounts to an amount of `None` (lightningdevkit#4324).
Bug Fixes
=========
* `Features::supports_zero_conf` no longer clears the `ZeroConf` features and
`Features::requires_zero_conf` now correctly reports required, rather than
supported, status (lightningdevkit#4517).
* If an MPP payment is claimed but `ChannelMonitorUpdate`s for some parts are
still being completed asynchronously, further channel updates (e.g.
forwarding another payment) are pending and the node restarts, the channel
could have become stuck (lightningdevkit#4520).
* The presence of unconfirmed transactions actually no longer causes
`ElectrumSyncClient` to spuriously fail to sync (lightningdevkit#4590).
* LSPS1, LSPS2, and LSPS5 persistence will no longer get stuck and refuse to
persist again after a single failure from the KVStore (lightningdevkit#4597, lightningdevkit#4282).
* Dropping the future returned by
`OutputSweeper::regenerate_and_broadcast_spend_if_necessary` no longer
results in future calls to the same method being spuriously ignored (lightningdevkit#4598).
* Used async-receive offers are no longer refreshed on every timer tick once
their refresh time is reached (lightningdevkit#4672).
* `FilesystemStore::list_all_keys` will no longer fail if there are stale
intermediate files lying around from a previous unclean shutdown (lightningdevkit#4618).
* When forwarding an HTLC while in a blinded path with proportional fees over
200%, LDK will no longer spuriously allow a forward that pays us 1 msat too
little in fees (lightningdevkit#4697).
* Fixed a rare case where a channel could get stuck on reconnect when using
both async `ChannelMonitorUpdate` persistence and async signing (lightningdevkit#4684).
* If we had exactly zero balance in a zero-fee-commitment channel, the
counterparty was able to splice all of their balance out, violating the
reserve requirements they'd otherwise be forced to keep (lightningdevkit#4580).
* Providing an `Event::HTLCIntercepted` to the `LSPS2ServiceHandler` twice no
longer results in spuriously opening a channel early (lightningdevkit#4656).
* `Event::PaymentSent::fee_paid_msat` is no longer `None` in cases where
`ChannelManager::abandon_payment` was called before the payment ultimately
completes anyway (lightningdevkit#4651).
* `AnchorDescriptor::previous_utxo` now provides the correct `script_pubkey`
for non-zero-commitment-fee anchor channels (lightningdevkit#4669).
* Syncing a `ChainMonitor` using the `Confirm` trait will no longer write some
full `ChannelMonitor`s to disk several times per block (lightningdevkit#4544).
* `OMDomainResolver` now correctly accounts for failed queries when rate
limiting, ensuring we continue to respond to queries after failures (lightningdevkit#4591).
* Calling `ChannelManager::send_payment_with_route` without a `route_params`
and with an invalid `Route` will no longer panic (lightningdevkit#4707).
* `LSPS2ServiceHandler::channel_open_failed` now correctly fails intercepted
HTLCs rather than allowing them to fail just before expiry (lightningdevkit#4677).
* `StaticInvoice::is_offer_expired` was corrected to check offer, rather than
static invoice, expiry (lightningdevkit#4594).
* `lightning-custom-message`'s handling of `peer_connected` events now ensures
that sub-handlers will see a `peer_disconnected` event if a different
sub-handler refused the connection by `Err`ing `peer_connected` (lightningdevkit#4595).
* Replay protection for LSPS5 signatures now detects replays which are only
different in the encoded signature's case (lightningdevkit#4701).
* When `lightning-liquidity` is configured in the background processor, there
is no longer a stream of `Persisting LiquidityManager...` log spam (lightningdevkit#4246).
* Incomplete MPP keysend payments will no longer see their HTLCs held until
expiry (lightningdevkit#4558).
* `InvoiceRequestBuilder` will no longer accept a `quantity` of `0` for a
BOLT 12 `Offer`, allowing any quantity up to a bound (lightningdevkit#4667).
* `lightning-custom-message` handlers that return `Ok(None)` when asked to
deserialize a message in their defined range no longer cause panics (lightningdevkit#4709).
* Several spurious debug assertions were fixed (lightningdevkit#4537, lightningdevkit#4618, lightningdevkit#4026)
Security
========
0.2.3 fixes several underestimates of the anchor reserves required to ensure we
can reliably close channels, several denial-of-service vulnerabilities and a
sanitization issue.
* `Bolt11Invoice::recover_payee_pub_key` no longer panics if called on an
invoice which set an explicit public key, rather than relying on public key
recovery. Note that this method is called from
`PaymentParameters::from_bolt11_invoice` (lightningdevkit#4717).
* Maliciously-crafted unpayable invoices which have overflowing feerates will
no longer cause an `unwrap` failure panic (lightningdevkit#4716).
* Parsing an `LSPSDateTime` which is before 1970 no longer panics. This is
reachable when parsing messages from counterparties (lightningdevkit#4715).
* `possiblyrandom` did not properly generate random data except when it was
explicitly configured to. By default this means LDK is vulnerable to various
HashDoS attacks (lightningdevkit#4719).
* `OMNameResolver` will no longer panic when looking up payment instructions
which include unicode characters at the start of a TXT record (lightningdevkit#4718).
* When using the `anchor_channel_reserves` module to calculate reserves
required to pay for fees when closing anchor channels, zero-fee-commitment
channels were not considered. This could allow a counterparty to open many
channels, leaving us unable to properly force-close (lightningdevkit#4592).
* The `anchor_channel_reserves` module overestimated the value of `Utxo`s in
the wallet by ignoring the `TxIn` cost to spend them (lightningdevkit#4670).
* `PrintableString` did not properly sanitize unicode format characters,
allowing an attacker to corrupt the rendering of logs or UI (lightningdevkit#4593, lightningdevkit#4605).
* RGS data is now limited in how large of a graph it is able to cause a client
to store in memory. Note that RGS data is still considered a DoS vector in
general and you should only use semi-trusted RGS data (lightningdevkit#4713).
* Counterparty-provided strings in failure messages are no longer logged in
full, reducing the ability of such a counterparty to spam our logs (lightningdevkit#4714).
* Reading a corrupted `ChannelManager` or `ProbabilisticScorer` can no longer
cause us to allocate large amounts of memory (lightningdevkit#4712).
Thanks to Project Loupe for reporting most of the issues fixed in this release.
Conflicts resolved in:
* lightning/src/chain/channelmonitor.rs
* lightning/src/events/mod.rs
* lightning/src/ln/channelmanager.rs
* lightning/src/ln/mod.rs
* lightning/src/ln/offers_tests.rs
* lightning/src/ln/onion_utils.rs
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants

@TheBlueMatt@ldk-reviews-bot@ldk-claude-review-bot@tnull