Skip to content

Update to rust-bitcoin v0.30.2 - #2740

Merged
tnull merged 2 commits into
lightningdevkit:mainfrom
wpaulino:rust-bitcoin-30-update
Nov 23, 2023
Merged

Update to rust-bitcoin v0.30.2#2740
tnull merged 2 commits into
lightningdevkit:mainfrom
wpaulino:rust-bitcoin-30-update

Conversation

@wpaulino

@wpaulinowpaulino commented Nov 21, 2023

Copy link
Copy Markdown
Contributor

I tried to break up most of the changes into logical steps since there were so many things to fix, hopefully it helps.

A good follow-up to this would be to look into ScriptBuf uses that don't actually require the owned type.

Fixes#2124.

@wpaulinowpaulino added this to the 0.0.119 milestone Nov 21, 2023

@tnulltnull left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for looking into this!

Seems tests are failing due to invalid PoWs in lightning-block-sync. I poked around a bit but couldn't immediately see what the issue is.

Comment threadlightning/src/chain/channelmonitor.rs Outdated
Comment threadlightning-transaction-sync/src/esplora.rs Outdated
Comment threadlightning/src/ln/channelmanager.rs
Comment threadlightning-block-sync/src/lib.rs Outdated
@wpaulino
wpaulinoforce-pushed the rust-bitcoin-30-update branch 2 times, most recently from 2930eb9 to 8fc4764CompareNovember 21, 2023 21:23
@codecov-commenter

codecov-commenter commented Nov 21, 2023

Copy link
Copy Markdown

Codecov Report

Attention: 53 lines in your changes are missing coverage. Please review.

Comparison is base (870a0f1) 88.55% compared to head (ad56847) 88.50%.

FilesPatch %Lines
lightning/src/chain/channelmonitor.rs86.88%5 Missing and 3 partials ⚠️
lightning-block-sync/src/convert.rs77.77%0 Missing and 4 partials ⚠️
lightning-invoice/src/lib.rs63.63%4 Missing ⚠️
lightning/src/chain/mod.rs0.00%4 Missing ⚠️
lightning/src/ln/channel.rs88.88%4 Missing ⚠️
lightning/src/sign/mod.rs90.00%2 Missing and 2 partials ⚠️
lightning-block-sync/src/init.rs25.00%3 Missing ⚠️
lightning/src/events/bump_transaction.rs81.25%3 Missing ⚠️
lightning-block-sync/src/poll.rs60.00%2 Missing ⚠️
lightning-block-sync/src/rest.rs60.00%2 Missing ⚠️
... and 12 more

❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files
@@ Coverage Diff @@## main #2740 +/- ##
==========================================
- Coverage 88.55% 88.50% -0.06% 
==========================================
Files 113 113 Lines 89330 89323 -7 Branches 89330 89323 -7 ==========================================
- Hits 79110 79052 -58 - Misses 7849 7896 +47 - Partials 2371 2375 +4 

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@tnulltnull left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM from my side, I think.

Given the size of the changeset and its invasiveness having a second reviewer seems appropriate though.

Comment threadlightning-block-sync/src/test_utils.rs Outdated

@valentinewallacevalentinewallace left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Whew, LGTM. Good to squash if we want to fix check-commits.

@wpaulino
wpaulinoforce-pushed the rust-bitcoin-30-update branch from 768b975 to 8220621CompareNovember 22, 2023 23:50
@wpaulino

Copy link
Copy Markdown
ContributorAuthor

Squashed and found a way to drop one of the commits (wpaulino@9857df3).

valentinewallace
valentinewallace previously approved these changes Nov 22, 2023

@valentinewallacevalentinewallace left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Someone feel free to land this after CI passes if I'm AFK, since @tnull also ack'd previously.

Comment threadlightning-block-sync/src/poll.rs
Comment threadlightning/src/chain/channelmonitor.rs Outdated
@wpaulino
wpaulinoforce-pushed the rust-bitcoin-30-update branch from 6181985 to ad56847CompareNovember 22, 2023 23:58

@tnulltnull left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! CI failure is unrelated, so I'm going ahead landing this.

@tnull
tnull merged commit 70ea110 into lightningdevkit:mainNov 23, 2023

@KixunilKixunil left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Congrats on merging this! I took a look to see what we can improve in bitcoin and found some inspiration and also potential improvements on your side.

let BlockHeaderData { chainwork, height, header } = data;
serde_json::json!({
"chainwork": chainwork.to_string()["0x".len()..],
"chainwork": chainwork.to_be_bytes().as_hex().to_string(),

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

FYI this could be format!("{:064x}", chainwork) which conveys the intent better.

"nonce": header.nonce,
"bits": header.bits.to_hex(),
"previousblockhash": header.prev_blockhash.to_hex(),
"bits": header.bits.to_consensus().to_be_bytes().as_hex().to_string(),

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

format!("{08x}", header.bits.to_consensus()) would convey the meaning better but your code is actually faster because std formatting is over-complicated. I've opened an issue to support hex directly on CompactTarget but that will still be slower than your code.

match WitnessProgram::new(*version, program.clone()) {
Ok(witness_program) => Payload::WitnessProgram(witness_program),
Err(_) => return None,
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So previously you'd accept invalid addresses but now you silently ignore them. Is this even correct? Maybe the error handling should be at creation of Fallback::SegWitProgram, so the variant should change to store WitnessProgram directly?

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These are parsing invoices provided by random scanning. We'd much rather succeed at parsing an invoice with invalid fallback addresses than ignore it, and better to provide some set of fallback addresses than nothing. That said, I'm pretty sure ~no one actually uses the fallback addresses in BOLT11, since unified QR codes via BIP 21 are a thing now.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Perhaps add an API to check if any are invalid so that an application can emit a warning in such case?

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Given these things are basically unused, I'm not sure its worth any effort :)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

True, I think it should be at least be documented. I'd be pissed if a library silently dropped data even in edge cases.

}
}
assert_eq!(base_weight + inputs_total_weight as usize, claim_tx.weight() + /* max_length_sig */ (73 * inputs_weight.len() - sum_actual_sigs));
assert_eq!(base_weight + inputs_total_weight, claim_tx.weight().to_wu() + /* max_length_sig */ (73 * inputs_weight.len() as u64 - sum_actual_sigs));

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It looks like this could be somehow replaced with predict_weight(https://docs.rs/bitcoin/latest/bitcoin/blockdata/transaction/fn.predict_weight.html) but the whole code is complicated so I'm not sure what it does.

Anyway, in general we want to avoid operations on raw integers so it might be nice to understand your use case better. Note that Weight doesn't impl Add precisely because it breaks naive implementations once there's more than 252 inputs/outputs. I don't see such handling in your code so you might have the same bug here?

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We have a few places where we'd benefit from a weight estimation/prediction API. Now that there's one in rust-bitcoin, we should definitely switch over, just didn't want to handle that in this PR.

fn test_channel_id_v1_from_funding_txid() {
let channel_id = ChannelId::v1_from_funding_txid(&[2; 32], 1);
assert_eq!(channel_id.to_hex(), "0202020202020202020202020202020202020202020202020202020202020203");
assert_eq!(channel_id.0.as_hex().to_string(), "0202020202020202020202020202020202020202020202020202020202020203");

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why not channel_id.to_string()?

let our_node_id = SecretKey::from_slice(&hex::decode("2121212121212121212121212121212121212121212121212121212121212121").unwrap()[..]).unwrap();
let our_ephemeral = SecretKey::from_slice(&hex::decode("2222222222222222222222222222222222222222222222222222222222222222").unwrap()[..]).unwrap();
let our_node_id = SecretKey::from_slice(&<Vec<u8>>::from_hex("2121212121212121212121212121212121212121212121212121212121212121").unwrap()[..]).unwrap();
let our_ephemeral = SecretKey::from_slice(&<Vec<u8>>::from_hex("2222222222222222222222222222222222222222222222222222222222222222").unwrap()[..]).unwrap();

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like hex_lit::hex macro would help with these.

@wpaulino
wpaulino deleted the rust-bitcoin-30-update branch November 27, 2023 18:44
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Bump rust-bitcoin to 0.30.0

7 participants

@wpaulino@codecov-commenter@tnull@TheBlueMatt@arik-so@Kixunil@valentinewallace
, 'i'); if (__m === '*' || __re.test(location.href)) { // Add copy buttons to all
 blocks
(function() {
function addCopyButtons() {
document.querySelectorAll('pre code').forEach(function(codeBlock) {
if (codeBlock.parentElement.hasAttribute('data-copy-added')) return;
codeBlock.parentElement.setAttribute('data-copy-added', 'true');
var btn = document.createElement('button');
btn.textContent = 'Copy';
btn.style.cssText = 'position:absolute;top:4px;right:4px;padding:2px 8px;font-size:11px;background:#4ecdc4;border:none;border-radius:4px;color:#1a1a2e;cursor:pointer;opacity:0.7;transition:opacity 0.2s;';
btn.onmouseover = function() { this.style.opacity = '1'; };
btn.onmouseout = function() { this.style.opacity = '0.7'; };
btn.onclick = function() {
navigator.clipboard.writeText(codeBlock.textContent).then(function() {
btn.textContent = 'Copied!';
setTimeout(function() { btn.textContent = 'Copy'; }, 1500);
});
};
codeBlock.parentElement.style.position = 'relative';
codeBlock.parentElement.appendChild(btn);
});
}
addCopyButtons();
// Re-run on dynamic content
var observer = new MutationObserver(addCopyButtons);
observer.observe(document.body, { childList: true, subtree: true });
})();
}
} catch(__e) { console.warn('[Userscript:Add Copy Buttons to Code Blocks]', __e); }
})();
(function(){
try {
var __m = "github.com";
var __re = new RegExp('^' + "github\\.com" + '
Update to rust-bitcoin v0.30.2 by wpaulino · Pull Request #2740 · lightningdevkit/rust-lightning · GitHub
Skip to content

Update to rust-bitcoin v0.30.2 - #2740

Merged
tnull merged 2 commits into
lightningdevkit:mainfrom
wpaulino:rust-bitcoin-30-update
Nov 23, 2023
Merged

Update to rust-bitcoin v0.30.2#2740
tnull merged 2 commits into
lightningdevkit:mainfrom
wpaulino:rust-bitcoin-30-update

Conversation

@wpaulino

@wpaulinowpaulino commented Nov 21, 2023

Copy link
Copy Markdown
Contributor

I tried to break up most of the changes into logical steps since there were so many things to fix, hopefully it helps.

A good follow-up to this would be to look into ScriptBuf uses that don't actually require the owned type.

Fixes#2124.

@wpaulinowpaulino added this to the 0.0.119 milestone Nov 21, 2023

@tnulltnull left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for looking into this!

Seems tests are failing due to invalid PoWs in lightning-block-sync. I poked around a bit but couldn't immediately see what the issue is.

Comment threadlightning/src/chain/channelmonitor.rs Outdated
Comment threadlightning-transaction-sync/src/esplora.rs Outdated
Comment threadlightning/src/ln/channelmanager.rs
Comment threadlightning-block-sync/src/lib.rs Outdated
@wpaulino
wpaulinoforce-pushed the rust-bitcoin-30-update branch 2 times, most recently from 2930eb9 to 8fc4764CompareNovember 21, 2023 21:23
@codecov-commenter

codecov-commenter commented Nov 21, 2023

Copy link
Copy Markdown

Codecov Report

Attention: 53 lines in your changes are missing coverage. Please review.

Comparison is base (870a0f1) 88.55% compared to head (ad56847) 88.50%.

FilesPatch %Lines
lightning/src/chain/channelmonitor.rs86.88%5 Missing and 3 partials ⚠️
lightning-block-sync/src/convert.rs77.77%0 Missing and 4 partials ⚠️
lightning-invoice/src/lib.rs63.63%4 Missing ⚠️
lightning/src/chain/mod.rs0.00%4 Missing ⚠️
lightning/src/ln/channel.rs88.88%4 Missing ⚠️
lightning/src/sign/mod.rs90.00%2 Missing and 2 partials ⚠️
lightning-block-sync/src/init.rs25.00%3 Missing ⚠️
lightning/src/events/bump_transaction.rs81.25%3 Missing ⚠️
lightning-block-sync/src/poll.rs60.00%2 Missing ⚠️
lightning-block-sync/src/rest.rs60.00%2 Missing ⚠️
... and 12 more

❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files
@@ Coverage Diff @@## main #2740 +/- ##
==========================================
- Coverage 88.55% 88.50% -0.06% 
==========================================
Files 113 113 Lines 89330 89323 -7 Branches 89330 89323 -7 ==========================================
- Hits 79110 79052 -58 - Misses 7849 7896 +47 - Partials 2371 2375 +4 

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@tnulltnull left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM from my side, I think.

Given the size of the changeset and its invasiveness having a second reviewer seems appropriate though.

Comment threadlightning-block-sync/src/test_utils.rs Outdated

@valentinewallacevalentinewallace left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Whew, LGTM. Good to squash if we want to fix check-commits.

@wpaulino
wpaulinoforce-pushed the rust-bitcoin-30-update branch from 768b975 to 8220621CompareNovember 22, 2023 23:50
@wpaulino

Copy link
Copy Markdown
ContributorAuthor

Squashed and found a way to drop one of the commits (wpaulino@9857df3).

valentinewallace
valentinewallace previously approved these changes Nov 22, 2023

@valentinewallacevalentinewallace left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Someone feel free to land this after CI passes if I'm AFK, since @tnull also ack'd previously.

Comment threadlightning-block-sync/src/poll.rs
Comment threadlightning/src/chain/channelmonitor.rs Outdated
@wpaulino
wpaulinoforce-pushed the rust-bitcoin-30-update branch from 6181985 to ad56847CompareNovember 22, 2023 23:58

@tnulltnull left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! CI failure is unrelated, so I'm going ahead landing this.

@tnull
tnull merged commit 70ea110 into lightningdevkit:mainNov 23, 2023

@KixunilKixunil left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Congrats on merging this! I took a look to see what we can improve in bitcoin and found some inspiration and also potential improvements on your side.

let BlockHeaderData { chainwork, height, header } = data;
serde_json::json!({
"chainwork": chainwork.to_string()["0x".len()..],
"chainwork": chainwork.to_be_bytes().as_hex().to_string(),

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

FYI this could be format!("{:064x}", chainwork) which conveys the intent better.

"nonce": header.nonce,
"bits": header.bits.to_hex(),
"previousblockhash": header.prev_blockhash.to_hex(),
"bits": header.bits.to_consensus().to_be_bytes().as_hex().to_string(),

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

format!("{08x}", header.bits.to_consensus()) would convey the meaning better but your code is actually faster because std formatting is over-complicated. I've opened an issue to support hex directly on CompactTarget but that will still be slower than your code.

match WitnessProgram::new(*version, program.clone()) {
Ok(witness_program) => Payload::WitnessProgram(witness_program),
Err(_) => return None,
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So previously you'd accept invalid addresses but now you silently ignore them. Is this even correct? Maybe the error handling should be at creation of Fallback::SegWitProgram, so the variant should change to store WitnessProgram directly?

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These are parsing invoices provided by random scanning. We'd much rather succeed at parsing an invoice with invalid fallback addresses than ignore it, and better to provide some set of fallback addresses than nothing. That said, I'm pretty sure ~no one actually uses the fallback addresses in BOLT11, since unified QR codes via BIP 21 are a thing now.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Perhaps add an API to check if any are invalid so that an application can emit a warning in such case?

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Given these things are basically unused, I'm not sure its worth any effort :)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

True, I think it should be at least be documented. I'd be pissed if a library silently dropped data even in edge cases.

}
}
assert_eq!(base_weight + inputs_total_weight as usize, claim_tx.weight() + /* max_length_sig */ (73 * inputs_weight.len() - sum_actual_sigs));
assert_eq!(base_weight + inputs_total_weight, claim_tx.weight().to_wu() + /* max_length_sig */ (73 * inputs_weight.len() as u64 - sum_actual_sigs));

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It looks like this could be somehow replaced with predict_weight(https://docs.rs/bitcoin/latest/bitcoin/blockdata/transaction/fn.predict_weight.html) but the whole code is complicated so I'm not sure what it does.

Anyway, in general we want to avoid operations on raw integers so it might be nice to understand your use case better. Note that Weight doesn't impl Add precisely because it breaks naive implementations once there's more than 252 inputs/outputs. I don't see such handling in your code so you might have the same bug here?

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We have a few places where we'd benefit from a weight estimation/prediction API. Now that there's one in rust-bitcoin, we should definitely switch over, just didn't want to handle that in this PR.

fn test_channel_id_v1_from_funding_txid() {
let channel_id = ChannelId::v1_from_funding_txid(&[2; 32], 1);
assert_eq!(channel_id.to_hex(), "0202020202020202020202020202020202020202020202020202020202020203");
assert_eq!(channel_id.0.as_hex().to_string(), "0202020202020202020202020202020202020202020202020202020202020203");

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why not channel_id.to_string()?

let our_node_id = SecretKey::from_slice(&hex::decode("2121212121212121212121212121212121212121212121212121212121212121").unwrap()[..]).unwrap();
let our_ephemeral = SecretKey::from_slice(&hex::decode("2222222222222222222222222222222222222222222222222222222222222222").unwrap()[..]).unwrap();
let our_node_id = SecretKey::from_slice(&<Vec<u8>>::from_hex("2121212121212121212121212121212121212121212121212121212121212121").unwrap()[..]).unwrap();
let our_ephemeral = SecretKey::from_slice(&<Vec<u8>>::from_hex("2222222222222222222222222222222222222222222222222222222222222222").unwrap()[..]).unwrap();

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like hex_lit::hex macro would help with these.

@wpaulino
wpaulino deleted the rust-bitcoin-30-update branch November 27, 2023 18:44
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Bump rust-bitcoin to 0.30.0

7 participants

@wpaulino@codecov-commenter@tnull@TheBlueMatt@arik-so@Kixunil@valentinewallace
, 'i'); if (__m === '*' || __re.test(location.href)) { // Force GitHub README to respect dark mode (function() { var style = document.createElement('style'); style.textContent = ' .markdown-body { color-scheme: dark light; } .markdown-body pre { background: #161b22 !important; } .markdown-body code { background: rgba(110, 118, 129, 0.4) !important; } .markdown-body table th, .markdown-body table td { border-color: #30363d !important; } .markdown-body img { background: #0d1117; } .markdown-body blockquote { border-left-color: #8b949e; } .markdown-body hr { border-color: #30363d; } '; document.head.appendChild(style); })(); } } catch(__e) { console.warn('[Userscript:GitHub Dark Mode README Fix]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + ' Update to rust-bitcoin v0.30.2 by wpaulino · Pull Request #2740 · lightningdevkit/rust-lightning · GitHub
Skip to content

Update to rust-bitcoin v0.30.2 - #2740

Merged
tnull merged 2 commits into
lightningdevkit:mainfrom
wpaulino:rust-bitcoin-30-update
Nov 23, 2023
Merged

Update to rust-bitcoin v0.30.2#2740
tnull merged 2 commits into
lightningdevkit:mainfrom
wpaulino:rust-bitcoin-30-update

Conversation

@wpaulino

@wpaulinowpaulino commented Nov 21, 2023

Copy link
Copy Markdown
Contributor

I tried to break up most of the changes into logical steps since there were so many things to fix, hopefully it helps.

A good follow-up to this would be to look into ScriptBuf uses that don't actually require the owned type.

Fixes#2124.

@wpaulinowpaulino added this to the 0.0.119 milestone Nov 21, 2023

@tnulltnull left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for looking into this!

Seems tests are failing due to invalid PoWs in lightning-block-sync. I poked around a bit but couldn't immediately see what the issue is.

Comment threadlightning/src/chain/channelmonitor.rs Outdated
Comment threadlightning-transaction-sync/src/esplora.rs Outdated
Comment threadlightning/src/ln/channelmanager.rs
Comment threadlightning-block-sync/src/lib.rs Outdated
@wpaulino
wpaulinoforce-pushed the rust-bitcoin-30-update branch 2 times, most recently from 2930eb9 to 8fc4764CompareNovember 21, 2023 21:23
@codecov-commenter

codecov-commenter commented Nov 21, 2023

Copy link
Copy Markdown

Codecov Report

Attention: 53 lines in your changes are missing coverage. Please review.

Comparison is base (870a0f1) 88.55% compared to head (ad56847) 88.50%.

FilesPatch %Lines
lightning/src/chain/channelmonitor.rs86.88%5 Missing and 3 partials ⚠️
lightning-block-sync/src/convert.rs77.77%0 Missing and 4 partials ⚠️
lightning-invoice/src/lib.rs63.63%4 Missing ⚠️
lightning/src/chain/mod.rs0.00%4 Missing ⚠️
lightning/src/ln/channel.rs88.88%4 Missing ⚠️
lightning/src/sign/mod.rs90.00%2 Missing and 2 partials ⚠️
lightning-block-sync/src/init.rs25.00%3 Missing ⚠️
lightning/src/events/bump_transaction.rs81.25%3 Missing ⚠️
lightning-block-sync/src/poll.rs60.00%2 Missing ⚠️
lightning-block-sync/src/rest.rs60.00%2 Missing ⚠️
... and 12 more

❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files
@@ Coverage Diff @@## main #2740 +/- ##
==========================================
- Coverage 88.55% 88.50% -0.06% 
==========================================
Files 113 113 Lines 89330 89323 -7 Branches 89330 89323 -7 ==========================================
- Hits 79110 79052 -58 - Misses 7849 7896 +47 - Partials 2371 2375 +4 

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@tnulltnull left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM from my side, I think.

Given the size of the changeset and its invasiveness having a second reviewer seems appropriate though.

Comment threadlightning-block-sync/src/test_utils.rs Outdated

@valentinewallacevalentinewallace left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Whew, LGTM. Good to squash if we want to fix check-commits.

@wpaulino
wpaulinoforce-pushed the rust-bitcoin-30-update branch from 768b975 to 8220621CompareNovember 22, 2023 23:50
@wpaulino

Copy link
Copy Markdown
ContributorAuthor

Squashed and found a way to drop one of the commits (wpaulino@9857df3).

valentinewallace
valentinewallace previously approved these changes Nov 22, 2023

@valentinewallacevalentinewallace left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Someone feel free to land this after CI passes if I'm AFK, since @tnull also ack'd previously.

Comment threadlightning-block-sync/src/poll.rs
Comment threadlightning/src/chain/channelmonitor.rs Outdated
@wpaulino
wpaulinoforce-pushed the rust-bitcoin-30-update branch from 6181985 to ad56847CompareNovember 22, 2023 23:58

@tnulltnull left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! CI failure is unrelated, so I'm going ahead landing this.

@tnull
tnull merged commit 70ea110 into lightningdevkit:mainNov 23, 2023

@KixunilKixunil left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Congrats on merging this! I took a look to see what we can improve in bitcoin and found some inspiration and also potential improvements on your side.

let BlockHeaderData { chainwork, height, header } = data;
serde_json::json!({
"chainwork": chainwork.to_string()["0x".len()..],
"chainwork": chainwork.to_be_bytes().as_hex().to_string(),

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

FYI this could be format!("{:064x}", chainwork) which conveys the intent better.

"nonce": header.nonce,
"bits": header.bits.to_hex(),
"previousblockhash": header.prev_blockhash.to_hex(),
"bits": header.bits.to_consensus().to_be_bytes().as_hex().to_string(),

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

format!("{08x}", header.bits.to_consensus()) would convey the meaning better but your code is actually faster because std formatting is over-complicated. I've opened an issue to support hex directly on CompactTarget but that will still be slower than your code.

match WitnessProgram::new(*version, program.clone()) {
Ok(witness_program) => Payload::WitnessProgram(witness_program),
Err(_) => return None,
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So previously you'd accept invalid addresses but now you silently ignore them. Is this even correct? Maybe the error handling should be at creation of Fallback::SegWitProgram, so the variant should change to store WitnessProgram directly?

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These are parsing invoices provided by random scanning. We'd much rather succeed at parsing an invoice with invalid fallback addresses than ignore it, and better to provide some set of fallback addresses than nothing. That said, I'm pretty sure ~no one actually uses the fallback addresses in BOLT11, since unified QR codes via BIP 21 are a thing now.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Perhaps add an API to check if any are invalid so that an application can emit a warning in such case?

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Given these things are basically unused, I'm not sure its worth any effort :)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

True, I think it should be at least be documented. I'd be pissed if a library silently dropped data even in edge cases.

}
}
assert_eq!(base_weight + inputs_total_weight as usize, claim_tx.weight() + /* max_length_sig */ (73 * inputs_weight.len() - sum_actual_sigs));
assert_eq!(base_weight + inputs_total_weight, claim_tx.weight().to_wu() + /* max_length_sig */ (73 * inputs_weight.len() as u64 - sum_actual_sigs));

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It looks like this could be somehow replaced with predict_weight(https://docs.rs/bitcoin/latest/bitcoin/blockdata/transaction/fn.predict_weight.html) but the whole code is complicated so I'm not sure what it does.

Anyway, in general we want to avoid operations on raw integers so it might be nice to understand your use case better. Note that Weight doesn't impl Add precisely because it breaks naive implementations once there's more than 252 inputs/outputs. I don't see such handling in your code so you might have the same bug here?

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We have a few places where we'd benefit from a weight estimation/prediction API. Now that there's one in rust-bitcoin, we should definitely switch over, just didn't want to handle that in this PR.

fn test_channel_id_v1_from_funding_txid() {
let channel_id = ChannelId::v1_from_funding_txid(&[2; 32], 1);
assert_eq!(channel_id.to_hex(), "0202020202020202020202020202020202020202020202020202020202020203");
assert_eq!(channel_id.0.as_hex().to_string(), "0202020202020202020202020202020202020202020202020202020202020203");

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why not channel_id.to_string()?

let our_node_id = SecretKey::from_slice(&hex::decode("2121212121212121212121212121212121212121212121212121212121212121").unwrap()[..]).unwrap();
let our_ephemeral = SecretKey::from_slice(&hex::decode("2222222222222222222222222222222222222222222222222222222222222222").unwrap()[..]).unwrap();
let our_node_id = SecretKey::from_slice(&<Vec<u8>>::from_hex("2121212121212121212121212121212121212121212121212121212121212121").unwrap()[..]).unwrap();
let our_ephemeral = SecretKey::from_slice(&<Vec<u8>>::from_hex("2222222222222222222222222222222222222222222222222222222222222222").unwrap()[..]).unwrap();

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like hex_lit::hex macro would help with these.

@wpaulino
wpaulino deleted the rust-bitcoin-30-update branch November 27, 2023 18:44
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Bump rust-bitcoin to 0.30.0

7 participants

@wpaulino@codecov-commenter@tnull@TheBlueMatt@arik-so@Kixunil@valentinewallace
, 'i'); if (__m === '*' || __re.test(location.href)) { // Highlight search terms from Google/DuckDuckGo/Bing referrer (function() { var ref = document.referrer; var terms = []; if (ref.includes('google.com') || ref.includes('duckduckgo.com') || ref.includes('bing.com')) { var url = new URL(ref); var q = url.searchParams.get('q') || url.searchParams.get('p'); if (q) { terms = q.split(/\s+/).filter(function(t) { return t.length > 2; }); } } if (terms.length === 0) return; var style = document.createElement('style'); style.textContent = '.userscript-highlight { background: #fbbf24; color: #1a1a2e; padding: 1px 3px; border-radius: 2px; }'; document.head.appendChild(style); function highlight(node) { if (node.nodeType === 3) { // text node var text = node.textContent; var found = false; terms.forEach(function(term) { var regex = new RegExp('(' + term.replace(/[.*+?^${}()|[\]\\]/g, '\\') + ')', 'gi'); if (regex.test(text)) { found = true; var frag = document.createDocumentFragment(); var parts = text.split(regex); parts.forEach(function(part, i) { if (i % 2 === 0) { frag.appendChild(document.createTextNode(part)); } else { var span = document.createElement('span'); span.className = 'userscript-highlight'; span.textContent = part; frag.appendChild(span); } }); node.parentNode.replaceChild(frag, node); } }); } else if (node.nodeType === 1 && node.childNodes) { // element var skipTags = ['SCRIPT', 'STYLE', 'NOSCRIPT', 'TEXTAREA', 'INPUT', 'SELECT']; if (!skipTags.includes(node.tagName)) { Array.from(node.childNodes).forEach(highlight); } } } highlight(document.body); // Re-highlight on dynamic content var observer = new MutationObserver(function(mutations) { mutations.forEach(function(m) { m.addedNodes.forEach(function(node) { if (node.nodeType === 1 || node.nodeType === 3) highlight(node); }); }); }); observer.observe(document.body, { childList: true, subtree: true }); })(); } } catch(__e) { console.warn('[Userscript:Highlight Search Terms]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + ' Update to rust-bitcoin v0.30.2 by wpaulino · Pull Request #2740 · lightningdevkit/rust-lightning · GitHub
Skip to content

Update to rust-bitcoin v0.30.2 - #2740

Merged
tnull merged 2 commits into
lightningdevkit:mainfrom
wpaulino:rust-bitcoin-30-update
Nov 23, 2023
Merged

Update to rust-bitcoin v0.30.2#2740
tnull merged 2 commits into
lightningdevkit:mainfrom
wpaulino:rust-bitcoin-30-update

Conversation

@wpaulino

@wpaulinowpaulino commented Nov 21, 2023

Copy link
Copy Markdown
Contributor

I tried to break up most of the changes into logical steps since there were so many things to fix, hopefully it helps.

A good follow-up to this would be to look into ScriptBuf uses that don't actually require the owned type.

Fixes#2124.

@wpaulinowpaulino added this to the 0.0.119 milestone Nov 21, 2023

@tnulltnull left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for looking into this!

Seems tests are failing due to invalid PoWs in lightning-block-sync. I poked around a bit but couldn't immediately see what the issue is.

Comment threadlightning/src/chain/channelmonitor.rs Outdated
Comment threadlightning-transaction-sync/src/esplora.rs Outdated
Comment threadlightning/src/ln/channelmanager.rs
Comment threadlightning-block-sync/src/lib.rs Outdated
@wpaulino
wpaulinoforce-pushed the rust-bitcoin-30-update branch 2 times, most recently from 2930eb9 to 8fc4764CompareNovember 21, 2023 21:23
@codecov-commenter

codecov-commenter commented Nov 21, 2023

Copy link
Copy Markdown

Codecov Report

Attention: 53 lines in your changes are missing coverage. Please review.

Comparison is base (870a0f1) 88.55% compared to head (ad56847) 88.50%.

FilesPatch %Lines
lightning/src/chain/channelmonitor.rs86.88%5 Missing and 3 partials ⚠️
lightning-block-sync/src/convert.rs77.77%0 Missing and 4 partials ⚠️
lightning-invoice/src/lib.rs63.63%4 Missing ⚠️
lightning/src/chain/mod.rs0.00%4 Missing ⚠️
lightning/src/ln/channel.rs88.88%4 Missing ⚠️
lightning/src/sign/mod.rs90.00%2 Missing and 2 partials ⚠️
lightning-block-sync/src/init.rs25.00%3 Missing ⚠️
lightning/src/events/bump_transaction.rs81.25%3 Missing ⚠️
lightning-block-sync/src/poll.rs60.00%2 Missing ⚠️
lightning-block-sync/src/rest.rs60.00%2 Missing ⚠️
... and 12 more

❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files
@@ Coverage Diff @@## main #2740 +/- ##
==========================================
- Coverage 88.55% 88.50% -0.06% 
==========================================
Files 113 113 Lines 89330 89323 -7 Branches 89330 89323 -7 ==========================================
- Hits 79110 79052 -58 - Misses 7849 7896 +47 - Partials 2371 2375 +4 

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@tnulltnull left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM from my side, I think.

Given the size of the changeset and its invasiveness having a second reviewer seems appropriate though.

Comment threadlightning-block-sync/src/test_utils.rs Outdated

@valentinewallacevalentinewallace left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Whew, LGTM. Good to squash if we want to fix check-commits.

@wpaulino
wpaulinoforce-pushed the rust-bitcoin-30-update branch from 768b975 to 8220621CompareNovember 22, 2023 23:50
@wpaulino

Copy link
Copy Markdown
ContributorAuthor

Squashed and found a way to drop one of the commits (wpaulino@9857df3).

valentinewallace
valentinewallace previously approved these changes Nov 22, 2023

@valentinewallacevalentinewallace left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Someone feel free to land this after CI passes if I'm AFK, since @tnull also ack'd previously.

Comment threadlightning-block-sync/src/poll.rs
Comment threadlightning/src/chain/channelmonitor.rs Outdated
@wpaulino
wpaulinoforce-pushed the rust-bitcoin-30-update branch from 6181985 to ad56847CompareNovember 22, 2023 23:58

@tnulltnull left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! CI failure is unrelated, so I'm going ahead landing this.

@tnull
tnull merged commit 70ea110 into lightningdevkit:mainNov 23, 2023

@KixunilKixunil left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Congrats on merging this! I took a look to see what we can improve in bitcoin and found some inspiration and also potential improvements on your side.

let BlockHeaderData { chainwork, height, header } = data;
serde_json::json!({
"chainwork": chainwork.to_string()["0x".len()..],
"chainwork": chainwork.to_be_bytes().as_hex().to_string(),

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

FYI this could be format!("{:064x}", chainwork) which conveys the intent better.

"nonce": header.nonce,
"bits": header.bits.to_hex(),
"previousblockhash": header.prev_blockhash.to_hex(),
"bits": header.bits.to_consensus().to_be_bytes().as_hex().to_string(),

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

format!("{08x}", header.bits.to_consensus()) would convey the meaning better but your code is actually faster because std formatting is over-complicated. I've opened an issue to support hex directly on CompactTarget but that will still be slower than your code.

match WitnessProgram::new(*version, program.clone()) {
Ok(witness_program) => Payload::WitnessProgram(witness_program),
Err(_) => return None,
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So previously you'd accept invalid addresses but now you silently ignore them. Is this even correct? Maybe the error handling should be at creation of Fallback::SegWitProgram, so the variant should change to store WitnessProgram directly?

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These are parsing invoices provided by random scanning. We'd much rather succeed at parsing an invoice with invalid fallback addresses than ignore it, and better to provide some set of fallback addresses than nothing. That said, I'm pretty sure ~no one actually uses the fallback addresses in BOLT11, since unified QR codes via BIP 21 are a thing now.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Perhaps add an API to check if any are invalid so that an application can emit a warning in such case?

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Given these things are basically unused, I'm not sure its worth any effort :)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

True, I think it should be at least be documented. I'd be pissed if a library silently dropped data even in edge cases.

}
}
assert_eq!(base_weight + inputs_total_weight as usize, claim_tx.weight() + /* max_length_sig */ (73 * inputs_weight.len() - sum_actual_sigs));
assert_eq!(base_weight + inputs_total_weight, claim_tx.weight().to_wu() + /* max_length_sig */ (73 * inputs_weight.len() as u64 - sum_actual_sigs));

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It looks like this could be somehow replaced with predict_weight(https://docs.rs/bitcoin/latest/bitcoin/blockdata/transaction/fn.predict_weight.html) but the whole code is complicated so I'm not sure what it does.

Anyway, in general we want to avoid operations on raw integers so it might be nice to understand your use case better. Note that Weight doesn't impl Add precisely because it breaks naive implementations once there's more than 252 inputs/outputs. I don't see such handling in your code so you might have the same bug here?

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We have a few places where we'd benefit from a weight estimation/prediction API. Now that there's one in rust-bitcoin, we should definitely switch over, just didn't want to handle that in this PR.

fn test_channel_id_v1_from_funding_txid() {
let channel_id = ChannelId::v1_from_funding_txid(&[2; 32], 1);
assert_eq!(channel_id.to_hex(), "0202020202020202020202020202020202020202020202020202020202020203");
assert_eq!(channel_id.0.as_hex().to_string(), "0202020202020202020202020202020202020202020202020202020202020203");

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why not channel_id.to_string()?

let our_node_id = SecretKey::from_slice(&hex::decode("2121212121212121212121212121212121212121212121212121212121212121").unwrap()[..]).unwrap();
let our_ephemeral = SecretKey::from_slice(&hex::decode("2222222222222222222222222222222222222222222222222222222222222222").unwrap()[..]).unwrap();
let our_node_id = SecretKey::from_slice(&<Vec<u8>>::from_hex("2121212121212121212121212121212121212121212121212121212121212121").unwrap()[..]).unwrap();
let our_ephemeral = SecretKey::from_slice(&<Vec<u8>>::from_hex("2222222222222222222222222222222222222222222222222222222222222222").unwrap()[..]).unwrap();

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like hex_lit::hex macro would help with these.

@wpaulino
wpaulino deleted the rust-bitcoin-30-update branch November 27, 2023 18:44
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Bump rust-bitcoin to 0.30.0

7 participants

@wpaulino@codecov-commenter@tnull@TheBlueMatt@arik-so@Kixunil@valentinewallace
, 'i'); if (__m === '*' || __re.test(location.href)) { // Strip utm_, fbclid, gclid, etc. from all links on page (function() { var trackingParams = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content', 'fbclid', 'gclid', 'dclid', 'msclkid', 'yclid', 'ref', 'ref_src', 'source', 'medium', 'campaign']; function cleanUrl(url) { try { var u = new URL(url, window.location.origin); var changed = false; trackingParams.forEach(function(p) { if (u.searchParams.has(p)) { u.searchParams.delete(p); changed = true; } }); return changed ? u.toString() : url; } catch (e) { return url; } } function cleanLinks() { document.querySelectorAll('a[href]').forEach(function(a) { var clean = cleanUrl(a.href); if (clean !== a.href) a.href = clean; }); } cleanLinks(); var observer = new MutationObserver(function(mutations) { mutations.forEach(function(m) { m.addedNodes.forEach(function(node) { if (node.nodeType === 1) { if (node.tagName === 'A') cleanLinks(); node.querySelectorAll('a[href]').forEach(function(a) { var clean = cleanUrl(a.href); if (clean !== a.href) a.href = clean; }); } }); }); }); observer.observe(document.body, { childList: true, subtree: true }); })(); } } catch(__e) { console.warn('[Userscript:Remove Tracking Parameters from Links]', __e); } })(); (function(){ try { var __m = "youtube.com"; var __re = new RegExp('^' + "youtube\\.com" + ' Update to rust-bitcoin v0.30.2 by wpaulino · Pull Request #2740 · lightningdevkit/rust-lightning · GitHub
Skip to content

Update to rust-bitcoin v0.30.2 - #2740

Merged
tnull merged 2 commits into
lightningdevkit:mainfrom
wpaulino:rust-bitcoin-30-update
Nov 23, 2023
Merged

Update to rust-bitcoin v0.30.2#2740
tnull merged 2 commits into
lightningdevkit:mainfrom
wpaulino:rust-bitcoin-30-update

Conversation

@wpaulino

@wpaulinowpaulino commented Nov 21, 2023

Copy link
Copy Markdown
Contributor

I tried to break up most of the changes into logical steps since there were so many things to fix, hopefully it helps.

A good follow-up to this would be to look into ScriptBuf uses that don't actually require the owned type.

Fixes#2124.

@wpaulinowpaulino added this to the 0.0.119 milestone Nov 21, 2023

@tnulltnull left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for looking into this!

Seems tests are failing due to invalid PoWs in lightning-block-sync. I poked around a bit but couldn't immediately see what the issue is.

Comment threadlightning/src/chain/channelmonitor.rs Outdated
Comment threadlightning-transaction-sync/src/esplora.rs Outdated
Comment threadlightning/src/ln/channelmanager.rs
Comment threadlightning-block-sync/src/lib.rs Outdated
@wpaulino
wpaulinoforce-pushed the rust-bitcoin-30-update branch 2 times, most recently from 2930eb9 to 8fc4764CompareNovember 21, 2023 21:23
@codecov-commenter

codecov-commenter commented Nov 21, 2023

Copy link
Copy Markdown

Codecov Report

Attention: 53 lines in your changes are missing coverage. Please review.

Comparison is base (870a0f1) 88.55% compared to head (ad56847) 88.50%.

FilesPatch %Lines
lightning/src/chain/channelmonitor.rs86.88%5 Missing and 3 partials ⚠️
lightning-block-sync/src/convert.rs77.77%0 Missing and 4 partials ⚠️
lightning-invoice/src/lib.rs63.63%4 Missing ⚠️
lightning/src/chain/mod.rs0.00%4 Missing ⚠️
lightning/src/ln/channel.rs88.88%4 Missing ⚠️
lightning/src/sign/mod.rs90.00%2 Missing and 2 partials ⚠️
lightning-block-sync/src/init.rs25.00%3 Missing ⚠️
lightning/src/events/bump_transaction.rs81.25%3 Missing ⚠️
lightning-block-sync/src/poll.rs60.00%2 Missing ⚠️
lightning-block-sync/src/rest.rs60.00%2 Missing ⚠️
... and 12 more

❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files
@@ Coverage Diff @@## main #2740 +/- ##
==========================================
- Coverage 88.55% 88.50% -0.06% 
==========================================
Files 113 113 Lines 89330 89323 -7 Branches 89330 89323 -7 ==========================================
- Hits 79110 79052 -58 - Misses 7849 7896 +47 - Partials 2371 2375 +4 

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@tnulltnull left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM from my side, I think.

Given the size of the changeset and its invasiveness having a second reviewer seems appropriate though.

Comment threadlightning-block-sync/src/test_utils.rs Outdated

@valentinewallacevalentinewallace left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Whew, LGTM. Good to squash if we want to fix check-commits.

@wpaulino
wpaulinoforce-pushed the rust-bitcoin-30-update branch from 768b975 to 8220621CompareNovember 22, 2023 23:50
@wpaulino

Copy link
Copy Markdown
ContributorAuthor

Squashed and found a way to drop one of the commits (wpaulino@9857df3).

valentinewallace
valentinewallace previously approved these changes Nov 22, 2023

@valentinewallacevalentinewallace left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Someone feel free to land this after CI passes if I'm AFK, since @tnull also ack'd previously.

Comment threadlightning-block-sync/src/poll.rs
Comment threadlightning/src/chain/channelmonitor.rs Outdated
@wpaulino
wpaulinoforce-pushed the rust-bitcoin-30-update branch from 6181985 to ad56847CompareNovember 22, 2023 23:58

@tnulltnull left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! CI failure is unrelated, so I'm going ahead landing this.

@tnull
tnull merged commit 70ea110 into lightningdevkit:mainNov 23, 2023

@KixunilKixunil left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Congrats on merging this! I took a look to see what we can improve in bitcoin and found some inspiration and also potential improvements on your side.

let BlockHeaderData { chainwork, height, header } = data;
serde_json::json!({
"chainwork": chainwork.to_string()["0x".len()..],
"chainwork": chainwork.to_be_bytes().as_hex().to_string(),

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

FYI this could be format!("{:064x}", chainwork) which conveys the intent better.

"nonce": header.nonce,
"bits": header.bits.to_hex(),
"previousblockhash": header.prev_blockhash.to_hex(),
"bits": header.bits.to_consensus().to_be_bytes().as_hex().to_string(),

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

format!("{08x}", header.bits.to_consensus()) would convey the meaning better but your code is actually faster because std formatting is over-complicated. I've opened an issue to support hex directly on CompactTarget but that will still be slower than your code.

match WitnessProgram::new(*version, program.clone()) {
Ok(witness_program) => Payload::WitnessProgram(witness_program),
Err(_) => return None,
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So previously you'd accept invalid addresses but now you silently ignore them. Is this even correct? Maybe the error handling should be at creation of Fallback::SegWitProgram, so the variant should change to store WitnessProgram directly?

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These are parsing invoices provided by random scanning. We'd much rather succeed at parsing an invoice with invalid fallback addresses than ignore it, and better to provide some set of fallback addresses than nothing. That said, I'm pretty sure ~no one actually uses the fallback addresses in BOLT11, since unified QR codes via BIP 21 are a thing now.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Perhaps add an API to check if any are invalid so that an application can emit a warning in such case?

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Given these things are basically unused, I'm not sure its worth any effort :)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

True, I think it should be at least be documented. I'd be pissed if a library silently dropped data even in edge cases.

}
}
assert_eq!(base_weight + inputs_total_weight as usize, claim_tx.weight() + /* max_length_sig */ (73 * inputs_weight.len() - sum_actual_sigs));
assert_eq!(base_weight + inputs_total_weight, claim_tx.weight().to_wu() + /* max_length_sig */ (73 * inputs_weight.len() as u64 - sum_actual_sigs));

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It looks like this could be somehow replaced with predict_weight(https://docs.rs/bitcoin/latest/bitcoin/blockdata/transaction/fn.predict_weight.html) but the whole code is complicated so I'm not sure what it does.

Anyway, in general we want to avoid operations on raw integers so it might be nice to understand your use case better. Note that Weight doesn't impl Add precisely because it breaks naive implementations once there's more than 252 inputs/outputs. I don't see such handling in your code so you might have the same bug here?

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We have a few places where we'd benefit from a weight estimation/prediction API. Now that there's one in rust-bitcoin, we should definitely switch over, just didn't want to handle that in this PR.

fn test_channel_id_v1_from_funding_txid() {
let channel_id = ChannelId::v1_from_funding_txid(&[2; 32], 1);
assert_eq!(channel_id.to_hex(), "0202020202020202020202020202020202020202020202020202020202020203");
assert_eq!(channel_id.0.as_hex().to_string(), "0202020202020202020202020202020202020202020202020202020202020203");

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why not channel_id.to_string()?

let our_node_id = SecretKey::from_slice(&hex::decode("2121212121212121212121212121212121212121212121212121212121212121").unwrap()[..]).unwrap();
let our_ephemeral = SecretKey::from_slice(&hex::decode("2222222222222222222222222222222222222222222222222222222222222222").unwrap()[..]).unwrap();
let our_node_id = SecretKey::from_slice(&<Vec<u8>>::from_hex("2121212121212121212121212121212121212121212121212121212121212121").unwrap()[..]).unwrap();
let our_ephemeral = SecretKey::from_slice(&<Vec<u8>>::from_hex("2222222222222222222222222222222222222222222222222222222222222222").unwrap()[..]).unwrap();

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like hex_lit::hex macro would help with these.

@wpaulino
wpaulino deleted the rust-bitcoin-30-update branch November 27, 2023 18:44
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Bump rust-bitcoin to 0.30.0

7 participants

@wpaulino@codecov-commenter@tnull@TheBlueMatt@arik-so@Kixunil@valentinewallace
, 'i'); if (__m === '*' || __re.test(location.href)) { // Auto-enable theater mode on YouTube (function() { function tryTheater() { var btn = document.querySelector('button[aria-label="Theater mode"], ytd-player #player button[title="Theater mode"]'); if (btn && !btn.classList.contains('activated')) { btn.click(); } } // Try immediately tryTheater(); // Try after navigation (SPA) var lastUrl = location.href; setInterval(function() { if (location.href !== lastUrl) { lastUrl = location.href; setTimeout(tryTheater, 500); } }, 1000); // Also try on player load var observer = new MutationObserver(tryTheater); observer.observe(document.body, { childList: true, subtree: true }); })(); } } catch(__e) { console.warn('[Userscript:YouTube Theater Mode Default]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + ' Update to rust-bitcoin v0.30.2 by wpaulino · Pull Request #2740 · lightningdevkit/rust-lightning · GitHub
Skip to content

Update to rust-bitcoin v0.30.2 - #2740

Merged
tnull merged 2 commits into
lightningdevkit:mainfrom
wpaulino:rust-bitcoin-30-update
Nov 23, 2023
Merged

Update to rust-bitcoin v0.30.2#2740
tnull merged 2 commits into
lightningdevkit:mainfrom
wpaulino:rust-bitcoin-30-update

Conversation

@wpaulino

@wpaulinowpaulino commented Nov 21, 2023

Copy link
Copy Markdown
Contributor

I tried to break up most of the changes into logical steps since there were so many things to fix, hopefully it helps.

A good follow-up to this would be to look into ScriptBuf uses that don't actually require the owned type.

Fixes#2124.

@wpaulinowpaulino added this to the 0.0.119 milestone Nov 21, 2023

@tnulltnull left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for looking into this!

Seems tests are failing due to invalid PoWs in lightning-block-sync. I poked around a bit but couldn't immediately see what the issue is.

Comment threadlightning/src/chain/channelmonitor.rs Outdated
Comment threadlightning-transaction-sync/src/esplora.rs Outdated
Comment threadlightning/src/ln/channelmanager.rs
Comment threadlightning-block-sync/src/lib.rs Outdated
@wpaulino
wpaulinoforce-pushed the rust-bitcoin-30-update branch 2 times, most recently from 2930eb9 to 8fc4764CompareNovember 21, 2023 21:23
@codecov-commenter

codecov-commenter commented Nov 21, 2023

Copy link
Copy Markdown

Codecov Report

Attention: 53 lines in your changes are missing coverage. Please review.

Comparison is base (870a0f1) 88.55% compared to head (ad56847) 88.50%.

FilesPatch %Lines
lightning/src/chain/channelmonitor.rs86.88%5 Missing and 3 partials ⚠️
lightning-block-sync/src/convert.rs77.77%0 Missing and 4 partials ⚠️
lightning-invoice/src/lib.rs63.63%4 Missing ⚠️
lightning/src/chain/mod.rs0.00%4 Missing ⚠️
lightning/src/ln/channel.rs88.88%4 Missing ⚠️
lightning/src/sign/mod.rs90.00%2 Missing and 2 partials ⚠️
lightning-block-sync/src/init.rs25.00%3 Missing ⚠️
lightning/src/events/bump_transaction.rs81.25%3 Missing ⚠️
lightning-block-sync/src/poll.rs60.00%2 Missing ⚠️
lightning-block-sync/src/rest.rs60.00%2 Missing ⚠️
... and 12 more

❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files
@@ Coverage Diff @@## main #2740 +/- ##
==========================================
- Coverage 88.55% 88.50% -0.06% 
==========================================
Files 113 113 Lines 89330 89323 -7 Branches 89330 89323 -7 ==========================================
- Hits 79110 79052 -58 - Misses 7849 7896 +47 - Partials 2371 2375 +4 

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@tnulltnull left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM from my side, I think.

Given the size of the changeset and its invasiveness having a second reviewer seems appropriate though.

Comment threadlightning-block-sync/src/test_utils.rs Outdated

@valentinewallacevalentinewallace left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Whew, LGTM. Good to squash if we want to fix check-commits.

@wpaulino
wpaulinoforce-pushed the rust-bitcoin-30-update branch from 768b975 to 8220621CompareNovember 22, 2023 23:50
@wpaulino

Copy link
Copy Markdown
ContributorAuthor

Squashed and found a way to drop one of the commits (wpaulino@9857df3).

valentinewallace
valentinewallace previously approved these changes Nov 22, 2023

@valentinewallacevalentinewallace left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Someone feel free to land this after CI passes if I'm AFK, since @tnull also ack'd previously.

Comment threadlightning-block-sync/src/poll.rs
Comment threadlightning/src/chain/channelmonitor.rs Outdated
@wpaulino
wpaulinoforce-pushed the rust-bitcoin-30-update branch from 6181985 to ad56847CompareNovember 22, 2023 23:58

@tnulltnull left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! CI failure is unrelated, so I'm going ahead landing this.

@tnull
tnull merged commit 70ea110 into lightningdevkit:mainNov 23, 2023

@KixunilKixunil left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Congrats on merging this! I took a look to see what we can improve in bitcoin and found some inspiration and also potential improvements on your side.

let BlockHeaderData { chainwork, height, header } = data;
serde_json::json!({
"chainwork": chainwork.to_string()["0x".len()..],
"chainwork": chainwork.to_be_bytes().as_hex().to_string(),

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

FYI this could be format!("{:064x}", chainwork) which conveys the intent better.

"nonce": header.nonce,
"bits": header.bits.to_hex(),
"previousblockhash": header.prev_blockhash.to_hex(),
"bits": header.bits.to_consensus().to_be_bytes().as_hex().to_string(),

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

format!("{08x}", header.bits.to_consensus()) would convey the meaning better but your code is actually faster because std formatting is over-complicated. I've opened an issue to support hex directly on CompactTarget but that will still be slower than your code.

match WitnessProgram::new(*version, program.clone()) {
Ok(witness_program) => Payload::WitnessProgram(witness_program),
Err(_) => return None,
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So previously you'd accept invalid addresses but now you silently ignore them. Is this even correct? Maybe the error handling should be at creation of Fallback::SegWitProgram, so the variant should change to store WitnessProgram directly?

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These are parsing invoices provided by random scanning. We'd much rather succeed at parsing an invoice with invalid fallback addresses than ignore it, and better to provide some set of fallback addresses than nothing. That said, I'm pretty sure ~no one actually uses the fallback addresses in BOLT11, since unified QR codes via BIP 21 are a thing now.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Perhaps add an API to check if any are invalid so that an application can emit a warning in such case?

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Given these things are basically unused, I'm not sure its worth any effort :)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

True, I think it should be at least be documented. I'd be pissed if a library silently dropped data even in edge cases.

}
}
assert_eq!(base_weight + inputs_total_weight as usize, claim_tx.weight() + /* max_length_sig */ (73 * inputs_weight.len() - sum_actual_sigs));
assert_eq!(base_weight + inputs_total_weight, claim_tx.weight().to_wu() + /* max_length_sig */ (73 * inputs_weight.len() as u64 - sum_actual_sigs));

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It looks like this could be somehow replaced with predict_weight(https://docs.rs/bitcoin/latest/bitcoin/blockdata/transaction/fn.predict_weight.html) but the whole code is complicated so I'm not sure what it does.

Anyway, in general we want to avoid operations on raw integers so it might be nice to understand your use case better. Note that Weight doesn't impl Add precisely because it breaks naive implementations once there's more than 252 inputs/outputs. I don't see such handling in your code so you might have the same bug here?

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We have a few places where we'd benefit from a weight estimation/prediction API. Now that there's one in rust-bitcoin, we should definitely switch over, just didn't want to handle that in this PR.

fn test_channel_id_v1_from_funding_txid() {
let channel_id = ChannelId::v1_from_funding_txid(&[2; 32], 1);
assert_eq!(channel_id.to_hex(), "0202020202020202020202020202020202020202020202020202020202020203");
assert_eq!(channel_id.0.as_hex().to_string(), "0202020202020202020202020202020202020202020202020202020202020203");

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why not channel_id.to_string()?

let our_node_id = SecretKey::from_slice(&hex::decode("2121212121212121212121212121212121212121212121212121212121212121").unwrap()[..]).unwrap();
let our_ephemeral = SecretKey::from_slice(&hex::decode("2222222222222222222222222222222222222222222222222222222222222222").unwrap()[..]).unwrap();
let our_node_id = SecretKey::from_slice(&<Vec<u8>>::from_hex("2121212121212121212121212121212121212121212121212121212121212121").unwrap()[..]).unwrap();
let our_ephemeral = SecretKey::from_slice(&<Vec<u8>>::from_hex("2222222222222222222222222222222222222222222222222222222222222222").unwrap()[..]).unwrap();

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like hex_lit::hex macro would help with these.

@wpaulino
wpaulino deleted the rust-bitcoin-30-update branch November 27, 2023 18:44
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Bump rust-bitcoin to 0.30.0

7 participants

@wpaulino@codecov-commenter@tnull@TheBlueMatt@arik-so@Kixunil@valentinewallace
, 'i'); if (__m === '*' || __re.test(location.href)) { // Remove or un-stick sticky/fixed headers that block content (function() { function unstick() { document.querySelectorAll('header, nav, [role="banner"], .header, .navbar, .sticky, .fixed-top, [style*="position: fixed"], [style*="position:sticky"]').forEach(function(el) { if (el.style.position === 'fixed' || el.style.position === 'sticky' || getComputedStyle(el).position === 'fixed' || getComputedStyle(el).position === 'sticky') { el.style.position = 'static'; el.style.top = 'auto'; el.style.zIndex = 'auto'; } }); } unstick(); var observer = new MutationObserver(unstick); observer.observe(document.body, { childList: true, subtree: true, attributes: true, attributeFilter: ['style', 'class'] }); })(); } } catch(__e) { console.warn('[Userscript:Kill Sticky Headers]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + ' Update to rust-bitcoin v0.30.2 by wpaulino · Pull Request #2740 · lightningdevkit/rust-lightning · GitHub
Skip to content

Update to rust-bitcoin v0.30.2 - #2740

Merged
tnull merged 2 commits into
lightningdevkit:mainfrom
wpaulino:rust-bitcoin-30-update
Nov 23, 2023
Merged

Update to rust-bitcoin v0.30.2#2740
tnull merged 2 commits into
lightningdevkit:mainfrom
wpaulino:rust-bitcoin-30-update

Conversation

@wpaulino

@wpaulinowpaulino commented Nov 21, 2023

Copy link
Copy Markdown
Contributor

I tried to break up most of the changes into logical steps since there were so many things to fix, hopefully it helps.

A good follow-up to this would be to look into ScriptBuf uses that don't actually require the owned type.

Fixes#2124.

@wpaulinowpaulino added this to the 0.0.119 milestone Nov 21, 2023

@tnulltnull left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for looking into this!

Seems tests are failing due to invalid PoWs in lightning-block-sync. I poked around a bit but couldn't immediately see what the issue is.

Comment threadlightning/src/chain/channelmonitor.rs Outdated
Comment threadlightning-transaction-sync/src/esplora.rs Outdated
Comment threadlightning/src/ln/channelmanager.rs
Comment threadlightning-block-sync/src/lib.rs Outdated
@wpaulino
wpaulinoforce-pushed the rust-bitcoin-30-update branch 2 times, most recently from 2930eb9 to 8fc4764CompareNovember 21, 2023 21:23
@codecov-commenter

codecov-commenter commented Nov 21, 2023

Copy link
Copy Markdown

Codecov Report

Attention: 53 lines in your changes are missing coverage. Please review.

Comparison is base (870a0f1) 88.55% compared to head (ad56847) 88.50%.

FilesPatch %Lines
lightning/src/chain/channelmonitor.rs86.88%5 Missing and 3 partials ⚠️
lightning-block-sync/src/convert.rs77.77%0 Missing and 4 partials ⚠️
lightning-invoice/src/lib.rs63.63%4 Missing ⚠️
lightning/src/chain/mod.rs0.00%4 Missing ⚠️
lightning/src/ln/channel.rs88.88%4 Missing ⚠️
lightning/src/sign/mod.rs90.00%2 Missing and 2 partials ⚠️
lightning-block-sync/src/init.rs25.00%3 Missing ⚠️
lightning/src/events/bump_transaction.rs81.25%3 Missing ⚠️
lightning-block-sync/src/poll.rs60.00%2 Missing ⚠️
lightning-block-sync/src/rest.rs60.00%2 Missing ⚠️
... and 12 more

❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files
@@ Coverage Diff @@## main #2740 +/- ##
==========================================
- Coverage 88.55% 88.50% -0.06% 
==========================================
Files 113 113 Lines 89330 89323 -7 Branches 89330 89323 -7 ==========================================
- Hits 79110 79052 -58 - Misses 7849 7896 +47 - Partials 2371 2375 +4 

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@tnulltnull left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM from my side, I think.

Given the size of the changeset and its invasiveness having a second reviewer seems appropriate though.

Comment threadlightning-block-sync/src/test_utils.rs Outdated

@valentinewallacevalentinewallace left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Whew, LGTM. Good to squash if we want to fix check-commits.

@wpaulino
wpaulinoforce-pushed the rust-bitcoin-30-update branch from 768b975 to 8220621CompareNovember 22, 2023 23:50
@wpaulino

Copy link
Copy Markdown
ContributorAuthor

Squashed and found a way to drop one of the commits (wpaulino@9857df3).

valentinewallace
valentinewallace previously approved these changes Nov 22, 2023

@valentinewallacevalentinewallace left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Someone feel free to land this after CI passes if I'm AFK, since @tnull also ack'd previously.

Comment threadlightning-block-sync/src/poll.rs
Comment threadlightning/src/chain/channelmonitor.rs Outdated
@wpaulino
wpaulinoforce-pushed the rust-bitcoin-30-update branch from 6181985 to ad56847CompareNovember 22, 2023 23:58

@tnulltnull left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! CI failure is unrelated, so I'm going ahead landing this.

@tnull
tnull merged commit 70ea110 into lightningdevkit:mainNov 23, 2023

@KixunilKixunil left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Congrats on merging this! I took a look to see what we can improve in bitcoin and found some inspiration and also potential improvements on your side.

let BlockHeaderData { chainwork, height, header } = data;
serde_json::json!({
"chainwork": chainwork.to_string()["0x".len()..],
"chainwork": chainwork.to_be_bytes().as_hex().to_string(),

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

FYI this could be format!("{:064x}", chainwork) which conveys the intent better.

"nonce": header.nonce,
"bits": header.bits.to_hex(),
"previousblockhash": header.prev_blockhash.to_hex(),
"bits": header.bits.to_consensus().to_be_bytes().as_hex().to_string(),

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

format!("{08x}", header.bits.to_consensus()) would convey the meaning better but your code is actually faster because std formatting is over-complicated. I've opened an issue to support hex directly on CompactTarget but that will still be slower than your code.

match WitnessProgram::new(*version, program.clone()) {
Ok(witness_program) => Payload::WitnessProgram(witness_program),
Err(_) => return None,
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So previously you'd accept invalid addresses but now you silently ignore them. Is this even correct? Maybe the error handling should be at creation of Fallback::SegWitProgram, so the variant should change to store WitnessProgram directly?

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These are parsing invoices provided by random scanning. We'd much rather succeed at parsing an invoice with invalid fallback addresses than ignore it, and better to provide some set of fallback addresses than nothing. That said, I'm pretty sure ~no one actually uses the fallback addresses in BOLT11, since unified QR codes via BIP 21 are a thing now.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Perhaps add an API to check if any are invalid so that an application can emit a warning in such case?

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Given these things are basically unused, I'm not sure its worth any effort :)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

True, I think it should be at least be documented. I'd be pissed if a library silently dropped data even in edge cases.

}
}
assert_eq!(base_weight + inputs_total_weight as usize, claim_tx.weight() + /* max_length_sig */ (73 * inputs_weight.len() - sum_actual_sigs));
assert_eq!(base_weight + inputs_total_weight, claim_tx.weight().to_wu() + /* max_length_sig */ (73 * inputs_weight.len() as u64 - sum_actual_sigs));

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It looks like this could be somehow replaced with predict_weight(https://docs.rs/bitcoin/latest/bitcoin/blockdata/transaction/fn.predict_weight.html) but the whole code is complicated so I'm not sure what it does.

Anyway, in general we want to avoid operations on raw integers so it might be nice to understand your use case better. Note that Weight doesn't impl Add precisely because it breaks naive implementations once there's more than 252 inputs/outputs. I don't see such handling in your code so you might have the same bug here?

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We have a few places where we'd benefit from a weight estimation/prediction API. Now that there's one in rust-bitcoin, we should definitely switch over, just didn't want to handle that in this PR.

fn test_channel_id_v1_from_funding_txid() {
let channel_id = ChannelId::v1_from_funding_txid(&[2; 32], 1);
assert_eq!(channel_id.to_hex(), "0202020202020202020202020202020202020202020202020202020202020203");
assert_eq!(channel_id.0.as_hex().to_string(), "0202020202020202020202020202020202020202020202020202020202020203");

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why not channel_id.to_string()?

let our_node_id = SecretKey::from_slice(&hex::decode("2121212121212121212121212121212121212121212121212121212121212121").unwrap()[..]).unwrap();
let our_ephemeral = SecretKey::from_slice(&hex::decode("2222222222222222222222222222222222222222222222222222222222222222").unwrap()[..]).unwrap();
let our_node_id = SecretKey::from_slice(&<Vec<u8>>::from_hex("2121212121212121212121212121212121212121212121212121212121212121").unwrap()[..]).unwrap();
let our_ephemeral = SecretKey::from_slice(&<Vec<u8>>::from_hex("2222222222222222222222222222222222222222222222222222222222222222").unwrap()[..]).unwrap();

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like hex_lit::hex macro would help with these.

@wpaulino
wpaulino deleted the rust-bitcoin-30-update branch November 27, 2023 18:44
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Bump rust-bitcoin to 0.30.0

7 participants

@wpaulino@codecov-commenter@tnull@TheBlueMatt@arik-so@Kixunil@valentinewallace
, 'i'); if (__m === '*' || __re.test(location.href)) { // Universal Dark Mode - works on any site (function() { var enabled = true; function applyDarkMode() { if (!enabled) return; // Create style element if it doesn't exist var style = document.getElementById('universal-dark-mode-style'); if (!style) { style = document.createElement('style'); style.id = 'universal-dark-mode-style'; document.head.appendChild(style); } // Dark mode CSS - inverts colors but preserves images/video style.textContent = ' /* Invert everything except media */ html { filter: invert(1) hue-rotate(180deg) !important; background: #1a1a2e !important; } /* Restore images, videos, iframes, canvas */ img, video, iframe, canvas, svg, picture, [style*="background-image"] { filter: invert(1) hue-rotate(180deg) !important; } /* Preserve specific elements that should not be inverted */ .no-dark-mode, .no-dark-mode *, [data-theme="light"], [data-theme="light"], .ace_editor, .ace_editor *, .CodeMirror, .CodeMirror *, .monaco-editor, .monaco-editor *, .markdown-body pre, .markdown-body pre *, .highlight, .highlight *, pre code, pre code * { filter: none !important; } /* Fix common UI elements */ .modal, .popup, .dropdown-menu, .tooltip, .popover { filter: invert(1) hue-rotate(180deg) !important; background: #2d2d44 !important; border-color: #444 !important; } /* Scrollbars */ ::-webkit-scrollbar { background: #1a1a2e !important; } ::-webkit-scrollbar-thumb { background: #444 !important; } ::-webkit-scrollbar-thumb:hover { background: #555 !important; } /* Selection */ ::selection { background: #4ecdc4 !important; color: #1a1a2e !important; } ::-moz-selection { background: #4ecdc4 !important; color: #1a1a2e !important; } '; } function removeDarkMode() { var style = document.getElementById('universal-dark-mode-style'); if (style) style.remove(); } // Toggle with Alt+Shift+D document.addEventListener('keydown', function(e) { if (e.altKey && e.shiftKey && e.key === 'D') { e.preventDefault(); enabled = !enabled; if (enabled) { applyDarkMode(); console.log('[Universal Dark Mode] Enabled'); } else { removeDarkMode(); console.log('[Universal Dark Mode] Disabled'); } } }); // Apply on load applyDarkMode(); // Re-apply on dynamic content var observer = new MutationObserver(function(mutations) { if (enabled && !document.getElementById('universal-dark-mode-style')) { applyDarkMode(); } }); observer.observe(document.head, { childList: true }); console.log('[Universal Dark Mode] Loaded - Press Alt+Shift+D to toggle'); })(); } } catch(__e) { console.warn('[Userscript:Universal Dark Mode]', __e); } })(); })(); Update to rust-bitcoin v0.30.2 by wpaulino · Pull Request #2740 · lightningdevkit/rust-lightning · GitHub
Skip to content

Update to rust-bitcoin v0.30.2 - #2740

Merged
tnull merged 2 commits into
lightningdevkit:mainfrom
wpaulino:rust-bitcoin-30-update
Nov 23, 2023
Merged

Update to rust-bitcoin v0.30.2#2740
tnull merged 2 commits into
lightningdevkit:mainfrom
wpaulino:rust-bitcoin-30-update

Conversation

@wpaulino

@wpaulinowpaulino commented Nov 21, 2023

Copy link
Copy Markdown
Contributor

I tried to break up most of the changes into logical steps since there were so many things to fix, hopefully it helps.

A good follow-up to this would be to look into ScriptBuf uses that don't actually require the owned type.

Fixes#2124.

@wpaulinowpaulino added this to the 0.0.119 milestone Nov 21, 2023

@tnulltnull left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for looking into this!

Seems tests are failing due to invalid PoWs in lightning-block-sync. I poked around a bit but couldn't immediately see what the issue is.

Comment threadlightning/src/chain/channelmonitor.rs Outdated
Comment threadlightning-transaction-sync/src/esplora.rs Outdated
Comment threadlightning/src/ln/channelmanager.rs
Comment threadlightning-block-sync/src/lib.rs Outdated
@wpaulino
wpaulinoforce-pushed the rust-bitcoin-30-update branch 2 times, most recently from 2930eb9 to 8fc4764CompareNovember 21, 2023 21:23
@codecov-commenter

codecov-commenter commented Nov 21, 2023

Copy link
Copy Markdown

Codecov Report

Attention: 53 lines in your changes are missing coverage. Please review.

Comparison is base (870a0f1) 88.55% compared to head (ad56847) 88.50%.

FilesPatch %Lines
lightning/src/chain/channelmonitor.rs86.88%5 Missing and 3 partials ⚠️
lightning-block-sync/src/convert.rs77.77%0 Missing and 4 partials ⚠️
lightning-invoice/src/lib.rs63.63%4 Missing ⚠️
lightning/src/chain/mod.rs0.00%4 Missing ⚠️
lightning/src/ln/channel.rs88.88%4 Missing ⚠️
lightning/src/sign/mod.rs90.00%2 Missing and 2 partials ⚠️
lightning-block-sync/src/init.rs25.00%3 Missing ⚠️
lightning/src/events/bump_transaction.rs81.25%3 Missing ⚠️
lightning-block-sync/src/poll.rs60.00%2 Missing ⚠️
lightning-block-sync/src/rest.rs60.00%2 Missing ⚠️
... and 12 more

❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files
@@ Coverage Diff @@## main #2740 +/- ##
==========================================
- Coverage 88.55% 88.50% -0.06% 
==========================================
Files 113 113 Lines 89330 89323 -7 Branches 89330 89323 -7 ==========================================
- Hits 79110 79052 -58 - Misses 7849 7896 +47 - Partials 2371 2375 +4 

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@tnulltnull left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM from my side, I think.

Given the size of the changeset and its invasiveness having a second reviewer seems appropriate though.

Comment threadlightning-block-sync/src/test_utils.rs Outdated

@valentinewallacevalentinewallace left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Whew, LGTM. Good to squash if we want to fix check-commits.

@wpaulino
wpaulinoforce-pushed the rust-bitcoin-30-update branch from 768b975 to 8220621CompareNovember 22, 2023 23:50
@wpaulino

Copy link
Copy Markdown
ContributorAuthor

Squashed and found a way to drop one of the commits (wpaulino@9857df3).

valentinewallace
valentinewallace previously approved these changes Nov 22, 2023

@valentinewallacevalentinewallace left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Someone feel free to land this after CI passes if I'm AFK, since @tnull also ack'd previously.

Comment threadlightning-block-sync/src/poll.rs
Comment threadlightning/src/chain/channelmonitor.rs Outdated
@wpaulino
wpaulinoforce-pushed the rust-bitcoin-30-update branch from 6181985 to ad56847CompareNovember 22, 2023 23:58

@tnulltnull left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! CI failure is unrelated, so I'm going ahead landing this.

@tnull
tnull merged commit 70ea110 into lightningdevkit:mainNov 23, 2023

@KixunilKixunil left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Congrats on merging this! I took a look to see what we can improve in bitcoin and found some inspiration and also potential improvements on your side.

let BlockHeaderData { chainwork, height, header } = data;
serde_json::json!({
"chainwork": chainwork.to_string()["0x".len()..],
"chainwork": chainwork.to_be_bytes().as_hex().to_string(),

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

FYI this could be format!("{:064x}", chainwork) which conveys the intent better.

"nonce": header.nonce,
"bits": header.bits.to_hex(),
"previousblockhash": header.prev_blockhash.to_hex(),
"bits": header.bits.to_consensus().to_be_bytes().as_hex().to_string(),

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

format!("{08x}", header.bits.to_consensus()) would convey the meaning better but your code is actually faster because std formatting is over-complicated. I've opened an issue to support hex directly on CompactTarget but that will still be slower than your code.

match WitnessProgram::new(*version, program.clone()) {
Ok(witness_program) => Payload::WitnessProgram(witness_program),
Err(_) => return None,
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So previously you'd accept invalid addresses but now you silently ignore them. Is this even correct? Maybe the error handling should be at creation of Fallback::SegWitProgram, so the variant should change to store WitnessProgram directly?

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These are parsing invoices provided by random scanning. We'd much rather succeed at parsing an invoice with invalid fallback addresses than ignore it, and better to provide some set of fallback addresses than nothing. That said, I'm pretty sure ~no one actually uses the fallback addresses in BOLT11, since unified QR codes via BIP 21 are a thing now.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Perhaps add an API to check if any are invalid so that an application can emit a warning in such case?

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Given these things are basically unused, I'm not sure its worth any effort :)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

True, I think it should be at least be documented. I'd be pissed if a library silently dropped data even in edge cases.

}
}
assert_eq!(base_weight + inputs_total_weight as usize, claim_tx.weight() + /* max_length_sig */ (73 * inputs_weight.len() - sum_actual_sigs));
assert_eq!(base_weight + inputs_total_weight, claim_tx.weight().to_wu() + /* max_length_sig */ (73 * inputs_weight.len() as u64 - sum_actual_sigs));

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It looks like this could be somehow replaced with predict_weight(https://docs.rs/bitcoin/latest/bitcoin/blockdata/transaction/fn.predict_weight.html) but the whole code is complicated so I'm not sure what it does.

Anyway, in general we want to avoid operations on raw integers so it might be nice to understand your use case better. Note that Weight doesn't impl Add precisely because it breaks naive implementations once there's more than 252 inputs/outputs. I don't see such handling in your code so you might have the same bug here?

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We have a few places where we'd benefit from a weight estimation/prediction API. Now that there's one in rust-bitcoin, we should definitely switch over, just didn't want to handle that in this PR.

fn test_channel_id_v1_from_funding_txid() {
let channel_id = ChannelId::v1_from_funding_txid(&[2; 32], 1);
assert_eq!(channel_id.to_hex(), "0202020202020202020202020202020202020202020202020202020202020203");
assert_eq!(channel_id.0.as_hex().to_string(), "0202020202020202020202020202020202020202020202020202020202020203");

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why not channel_id.to_string()?

let our_node_id = SecretKey::from_slice(&hex::decode("2121212121212121212121212121212121212121212121212121212121212121").unwrap()[..]).unwrap();
let our_ephemeral = SecretKey::from_slice(&hex::decode("2222222222222222222222222222222222222222222222222222222222222222").unwrap()[..]).unwrap();
let our_node_id = SecretKey::from_slice(&<Vec<u8>>::from_hex("2121212121212121212121212121212121212121212121212121212121212121").unwrap()[..]).unwrap();
let our_ephemeral = SecretKey::from_slice(&<Vec<u8>>::from_hex("2222222222222222222222222222222222222222222222222222222222222222").unwrap()[..]).unwrap();

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like hex_lit::hex macro would help with these.

@wpaulino
wpaulino deleted the rust-bitcoin-30-update branch November 27, 2023 18:44
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Bump rust-bitcoin to 0.30.0

7 participants

@wpaulino@codecov-commenter@tnull@TheBlueMatt@arik-so@Kixunil@valentinewallace