Fix esplora integration test runner

[tcharding]

Description

Recently we added sub-modules to the esplora module for different HTTP
backends. We (Tobin) failed to update the integration test runner.

Modify the test running macro so that the backend is explicitly
selected:

cargo test --no-default-features --features=use-esplora-ureq,test-esplora

Notes to the reviewers

I can't actually get the integration tests to run to completion. They all (test-rpc, test-electrum, test-esplora) hang

$ echo $BITCOIND_EXE ✘ 130 /opt/bitcoin/bin/bitcoind
$ echo $ELECTRS_EXE
/home/tobin/.cargo/bin/electrs
$ cargo test --no-default-features --features=use-esplora-ureq,test-esplora
test blockchain::esplora::bdk_blockchain_tests::test_sync_address_reuse has been running for over 60 seconds
test blockchain::esplora::bdk_blockchain_tests::test_sync_after_send has been running for over 60 seconds
test blockchain::esplora::bdk_blockchain_tests::test_sync_before_and_after_receive has been running for over 60 seconds
test blockchain::esplora::bdk_blockchain_tests::test_sync_bump_fee_add_input_no_change has been running for over 60 seconds

However without this patch the integration tests do not run at all for esplora so I believe this is still correct. And it seems like a good way to ask if I am doing something wrong with how I'm trying to run the integration tests? Any suggestions please?

Checklists

All Submissions:

• I've signed all my commits
• I followed the contribution guidelines
• I ran cargo fmt and cargo clippy before committing

New Features:

• I've added tests for the new feature
• I've added docs for the new feature
• I've updated CHANGELOG.md

Bugfixes:

• This pull request breaks the existing API
• I've added tests to reproduce the issue which are now passing
• I'm linking the issue being fixed by this PR

[RCasatta]

I think the hanging is more likely due to the esplora version pointed by your env var. For esplora it's needed a specific electrs fork (branch new-index in https://github.com/blockstream/electrs).

bitcoind and electrsd can run with env var set or by setting a specific feature that during the build process download the needed executables. At the moment the feature is used in our tests, see:

test-esplora = ["electrsd/legacy", "electrsd/esplora_a33e97e1", "test-blockchains"]

a33e97e1 stands for the specific commit of electrs, so you can either use the feature, or build electrs at that commit. Also electrsd/legacy feature is needed because there has been changes in electrs parameters...

[tcharding]

Thanks for the tips @RCasatta, I tried as you suggested (with no env vars set) but still tests hang. Can see from the log output that its pulling down bitcoind/electrs to run the tests. Any other ideas please?

$ cargo test --features test-electrum
...
[2021-08-02T22:27:17Z DEBUG bdk::testutils::blockchain_tests] launching /home/tobin/.cargo/bitcoin/bitcoin-0.21.1/bin/bitcoind and /home/tobin/.cargo/electrs/electrs_linux_v0.8.10/electrs
...
[2021-08-02T22:27:18Z DEBUG electrsd] args: ["-vvv", "--db-dir", "/tmp/.tmphAsIio", "--network", "regtest", "--cookie-file", "/tmp/.tmpDleo7B/regtest/.cookie", "--daemon-rpc-addr", "127.0.0.1:42461", "--jsonrpc-import", "--electrum-rpc-addr", "0.0.0.0:46495", "--monitoring-addr", "0.0.0.0:44203"]
test blockchain::electrum::bdk_blockchain_tests::test_sync_address_reuse has been running for over 60 seconds
test blockchain::electrum::bdk_blockchain_tests::test_sync_after_send has been running for over 60 seconds
test blockchain::electrum::bdk_blockchain_tests::test_sync_before_and_after_receive has been running for over 60 seconds
test blockchain::electrum::bdk_blockchain_tests::test_sync_bump_fee_add_input_no_change has been running for over 60 seconds

[RCasatta]

I can't reproduce on my system and neither on the CI (run in this PR looks failing for unrelated HTTP issue, this one https://github.com/RCasatta/bdk/actions/runs/1093394555 looks fine)

Maybe some more details on your system could help to reproduce.

Also, maybe running this:

RUST_LOG=bdk=debug cargo test --features test-electrum -- test_sync_address_reuse

also with this patch:

diff --git a/src/testutils/blockchain_tests.rs b/src/testutils/blockchain_tests.rs
index 2df72d3..e374cb0 100644
--- a/src/testutils/blockchain_tests.rs
+++ b/src/testutils/blockchain_tests.rs
@@ -24,12 +24,17 @@ pub struct TestClient {
impl TestClient {
pub fn new(bitcoind_exe: String, electrs_exe: String) -> Self {
debug!("launching {} and {}", &bitcoind_exe, &electrs_exe);
- let bitcoind = BitcoinD::new(bitcoind_exe).unwrap();
+ let conf = bitcoind::Conf {
+ view_stdout: true,
+ ..Default::default()
+ };
+ let bitcoind = BitcoinD::with_conf(bitcoind_exe).unwrap();
let http_enabled = cfg!(feature = "test-esplora");
let conf = Conf {
http_enabled,
+ view_stderr: true,
..Default::default()
};
let electrsd = ElectrsD::with_conf(electrs_exe, &bitcoind, &conf).unwrap();

May give insight on what's happening under the hood

UPDATE: the patch seems useful to have in case you have a Debug log level, so I opened a PR #417

[notmandatory]

Works for me locally (osx) and in CI so ACK 1f6dd33

I just noticed we also need to update the CI test-blockchains job to add the use-esplora-ureq or use-esplora-request features. I'll add a commit for this before merging.

Need to update CI and fix hanging

[tcharding]

Thanks @notmandatory, picking up the pieces for me :)

[notmandatory]

I think @rajarshimaitra's #430 will go in first then we can add this change. 🙂

[tcharding]

#433 solves this issue in a better way.

[tcharding]

Description

Recently we added sub-modules to the esplora module for different HTTP
backends. We (Tobin) failed to update the integration test runner.

Modify the test running macro so that the backend is explicitly
selected:

cargo test --no-default-features --features=use-esplora-ureq,test-esplora

Notes to the reviewers

I can't actually get the integration tests to run to completion. They all (test-rpc, test-electrum, test-esplora) hang

$ echo $BITCOIND_EXE ✘ 130 /opt/bitcoin/bin/bitcoind
$ echo $ELECTRS_EXE
/home/tobin/.cargo/bin/electrs
$ cargo test --no-default-features --features=use-esplora-ureq,test-esplora
test blockchain::esplora::bdk_blockchain_tests::test_sync_address_reuse has been running for over 60 seconds
test blockchain::esplora::bdk_blockchain_tests::test_sync_after_send has been running for over 60 seconds
test blockchain::esplora::bdk_blockchain_tests::test_sync_before_and_after_receive has been running for over 60 seconds
test blockchain::esplora::bdk_blockchain_tests::test_sync_bump_fee_add_input_no_change has been running for over 60 seconds

However without this patch the integration tests do not run at all for esplora so I believe this is still correct. And it seems like a good way to ask if I am doing something wrong with how I'm trying to run the integration tests? Any suggestions please?

Checklists

All Submissions:

• I've signed all my commits
• I followed the contribution guidelines
• I ran cargo fmt and cargo clippy before committing

New Features:

• I've added tests for the new feature
• I've added docs for the new feature
• I've updated CHANGELOG.md

Bugfixes:

• This pull request breaks the existing API
• I've added tests to reproduce the issue which are now passing
• I'm linking the issue being fixed by this PR

[RCasatta]

I think the hanging is more likely due to the esplora version pointed by your env var. For esplora it's needed a specific electrs fork (branch new-index in https://github.com/blockstream/electrs).

bitcoind and electrsd can run with env var set or by setting a specific feature that during the build process download the needed executables. At the moment the feature is used in our tests, see:

test-esplora = ["electrsd/legacy", "electrsd/esplora_a33e97e1", "test-blockchains"]

a33e97e1 stands for the specific commit of electrs, so you can either use the feature, or build electrs at that commit. Also electrsd/legacy feature is needed because there has been changes in electrs parameters...

[tcharding]

Thanks for the tips @RCasatta, I tried as you suggested (with no env vars set) but still tests hang. Can see from the log output that its pulling down bitcoind/electrs to run the tests. Any other ideas please?

$ cargo test --features test-electrum
...
[2021-08-02T22:27:17Z DEBUG bdk::testutils::blockchain_tests] launching /home/tobin/.cargo/bitcoin/bitcoin-0.21.1/bin/bitcoind and /home/tobin/.cargo/electrs/electrs_linux_v0.8.10/electrs
...
[2021-08-02T22:27:18Z DEBUG electrsd] args: ["-vvv", "--db-dir", "/tmp/.tmphAsIio", "--network", "regtest", "--cookie-file", "/tmp/.tmpDleo7B/regtest/.cookie", "--daemon-rpc-addr", "127.0.0.1:42461", "--jsonrpc-import", "--electrum-rpc-addr", "0.0.0.0:46495", "--monitoring-addr", "0.0.0.0:44203"]
test blockchain::electrum::bdk_blockchain_tests::test_sync_address_reuse has been running for over 60 seconds
test blockchain::electrum::bdk_blockchain_tests::test_sync_after_send has been running for over 60 seconds
test blockchain::electrum::bdk_blockchain_tests::test_sync_before_and_after_receive has been running for over 60 seconds
test blockchain::electrum::bdk_blockchain_tests::test_sync_bump_fee_add_input_no_change has been running for over 60 seconds

[RCasatta]

I can't reproduce on my system and neither on the CI (run in this PR looks failing for unrelated HTTP issue, this one https://github.com/RCasatta/bdk/actions/runs/1093394555 looks fine)

Maybe some more details on your system could help to reproduce.

Also, maybe running this:

RUST_LOG=bdk=debug cargo test --features test-electrum -- test_sync_address_reuse

also with this patch:

diff --git a/src/testutils/blockchain_tests.rs b/src/testutils/blockchain_tests.rs
index 2df72d3..e374cb0 100644
--- a/src/testutils/blockchain_tests.rs
+++ b/src/testutils/blockchain_tests.rs
@@ -24,12 +24,17 @@ pub struct TestClient {
impl TestClient {
pub fn new(bitcoind_exe: String, electrs_exe: String) -> Self {
debug!("launching {} and {}", &bitcoind_exe, &electrs_exe);
- let bitcoind = BitcoinD::new(bitcoind_exe).unwrap();
+ let conf = bitcoind::Conf {
+ view_stdout: true,
+ ..Default::default()
+ };
+ let bitcoind = BitcoinD::with_conf(bitcoind_exe).unwrap();
let http_enabled = cfg!(feature = "test-esplora");
let conf = Conf {
http_enabled,
+ view_stderr: true,
..Default::default()
};
let electrsd = ElectrsD::with_conf(electrs_exe, &bitcoind, &conf).unwrap();

May give insight on what's happening under the hood

UPDATE: the patch seems useful to have in case you have a Debug log level, so I opened a PR #417

[notmandatory]

Works for me locally (osx) and in CI so ACK 1f6dd33

I just noticed we also need to update the CI test-blockchains job to add the use-esplora-ureq or use-esplora-request features. I'll add a commit for this before merging.

Need to update CI and fix hanging

[tcharding]

Thanks @notmandatory, picking up the pieces for me :)

[notmandatory]

I think @rajarshimaitra's #430 will go in first then we can add this change. 🙂

[tcharding]

#433 solves this issue in a better way.

[tcharding]

Description

Recently we added sub-modules to the esplora module for different HTTP
backends. We (Tobin) failed to update the integration test runner.

Modify the test running macro so that the backend is explicitly
selected:

cargo test --no-default-features --features=use-esplora-ureq,test-esplora

Notes to the reviewers

I can't actually get the integration tests to run to completion. They all (test-rpc, test-electrum, test-esplora) hang

$ echo $BITCOIND_EXE ✘ 130 /opt/bitcoin/bin/bitcoind
$ echo $ELECTRS_EXE
/home/tobin/.cargo/bin/electrs
$ cargo test --no-default-features --features=use-esplora-ureq,test-esplora
test blockchain::esplora::bdk_blockchain_tests::test_sync_address_reuse has been running for over 60 seconds
test blockchain::esplora::bdk_blockchain_tests::test_sync_after_send has been running for over 60 seconds
test blockchain::esplora::bdk_blockchain_tests::test_sync_before_and_after_receive has been running for over 60 seconds
test blockchain::esplora::bdk_blockchain_tests::test_sync_bump_fee_add_input_no_change has been running for over 60 seconds

However without this patch the integration tests do not run at all for esplora so I believe this is still correct. And it seems like a good way to ask if I am doing something wrong with how I'm trying to run the integration tests? Any suggestions please?

Checklists

All Submissions:

• I've signed all my commits
• I followed the contribution guidelines
• I ran cargo fmt and cargo clippy before committing

New Features:

• I've added tests for the new feature
• I've added docs for the new feature
• I've updated CHANGELOG.md

Bugfixes:

• This pull request breaks the existing API
• I've added tests to reproduce the issue which are now passing
• I'm linking the issue being fixed by this PR

[RCasatta]

I think the hanging is more likely due to the esplora version pointed by your env var. For esplora it's needed a specific electrs fork (branch new-index in https://github.com/blockstream/electrs).

bitcoind and electrsd can run with env var set or by setting a specific feature that during the build process download the needed executables. At the moment the feature is used in our tests, see:

test-esplora = ["electrsd/legacy", "electrsd/esplora_a33e97e1", "test-blockchains"]

a33e97e1 stands for the specific commit of electrs, so you can either use the feature, or build electrs at that commit. Also electrsd/legacy feature is needed because there has been changes in electrs parameters...

[tcharding]

Thanks for the tips @RCasatta, I tried as you suggested (with no env vars set) but still tests hang. Can see from the log output that its pulling down bitcoind/electrs to run the tests. Any other ideas please?

$ cargo test --features test-electrum
...
[2021-08-02T22:27:17Z DEBUG bdk::testutils::blockchain_tests] launching /home/tobin/.cargo/bitcoin/bitcoin-0.21.1/bin/bitcoind and /home/tobin/.cargo/electrs/electrs_linux_v0.8.10/electrs
...
[2021-08-02T22:27:18Z DEBUG electrsd] args: ["-vvv", "--db-dir", "/tmp/.tmphAsIio", "--network", "regtest", "--cookie-file", "/tmp/.tmpDleo7B/regtest/.cookie", "--daemon-rpc-addr", "127.0.0.1:42461", "--jsonrpc-import", "--electrum-rpc-addr", "0.0.0.0:46495", "--monitoring-addr", "0.0.0.0:44203"]
test blockchain::electrum::bdk_blockchain_tests::test_sync_address_reuse has been running for over 60 seconds
test blockchain::electrum::bdk_blockchain_tests::test_sync_after_send has been running for over 60 seconds
test blockchain::electrum::bdk_blockchain_tests::test_sync_before_and_after_receive has been running for over 60 seconds
test blockchain::electrum::bdk_blockchain_tests::test_sync_bump_fee_add_input_no_change has been running for over 60 seconds

[RCasatta]

I can't reproduce on my system and neither on the CI (run in this PR looks failing for unrelated HTTP issue, this one https://github.com/RCasatta/bdk/actions/runs/1093394555 looks fine)

Maybe some more details on your system could help to reproduce.

Also, maybe running this:

RUST_LOG=bdk=debug cargo test --features test-electrum -- test_sync_address_reuse

also with this patch:

diff --git a/src/testutils/blockchain_tests.rs b/src/testutils/blockchain_tests.rs
index 2df72d3..e374cb0 100644
--- a/src/testutils/blockchain_tests.rs
+++ b/src/testutils/blockchain_tests.rs
@@ -24,12 +24,17 @@ pub struct TestClient {
impl TestClient {
pub fn new(bitcoind_exe: String, electrs_exe: String) -> Self {
debug!("launching {} and {}", &bitcoind_exe, &electrs_exe);
- let bitcoind = BitcoinD::new(bitcoind_exe).unwrap();
+ let conf = bitcoind::Conf {
+ view_stdout: true,
+ ..Default::default()
+ };
+ let bitcoind = BitcoinD::with_conf(bitcoind_exe).unwrap();
let http_enabled = cfg!(feature = "test-esplora");
let conf = Conf {
http_enabled,
+ view_stderr: true,
..Default::default()
};
let electrsd = ElectrsD::with_conf(electrs_exe, &bitcoind, &conf).unwrap();

May give insight on what's happening under the hood

UPDATE: the patch seems useful to have in case you have a Debug log level, so I opened a PR #417

[notmandatory]

Works for me locally (osx) and in CI so ACK 1f6dd33

I just noticed we also need to update the CI test-blockchains job to add the use-esplora-ureq or use-esplora-request features. I'll add a commit for this before merging.

Need to update CI and fix hanging

[tcharding]

Thanks @notmandatory, picking up the pieces for me :)

[notmandatory]

I think @rajarshimaitra's #430 will go in first then we can add this change. 🙂

[tcharding]

#433 solves this issue in a better way.

[tcharding]

Description

Recently we added sub-modules to the esplora module for different HTTP
backends. We (Tobin) failed to update the integration test runner.

Modify the test running macro so that the backend is explicitly
selected:

cargo test --no-default-features --features=use-esplora-ureq,test-esplora

Notes to the reviewers

I can't actually get the integration tests to run to completion. They all (test-rpc, test-electrum, test-esplora) hang

$ echo $BITCOIND_EXE ✘ 130 /opt/bitcoin/bin/bitcoind
$ echo $ELECTRS_EXE
/home/tobin/.cargo/bin/electrs
$ cargo test --no-default-features --features=use-esplora-ureq,test-esplora
test blockchain::esplora::bdk_blockchain_tests::test_sync_address_reuse has been running for over 60 seconds
test blockchain::esplora::bdk_blockchain_tests::test_sync_after_send has been running for over 60 seconds
test blockchain::esplora::bdk_blockchain_tests::test_sync_before_and_after_receive has been running for over 60 seconds
test blockchain::esplora::bdk_blockchain_tests::test_sync_bump_fee_add_input_no_change has been running for over 60 seconds

However without this patch the integration tests do not run at all for esplora so I believe this is still correct. And it seems like a good way to ask if I am doing something wrong with how I'm trying to run the integration tests? Any suggestions please?

Checklists

All Submissions:

• I've signed all my commits
• I followed the contribution guidelines
• I ran cargo fmt and cargo clippy before committing

New Features:

• I've added tests for the new feature
• I've added docs for the new feature
• I've updated CHANGELOG.md

Bugfixes:

• This pull request breaks the existing API
• I've added tests to reproduce the issue which are now passing
• I'm linking the issue being fixed by this PR

[RCasatta]

I think the hanging is more likely due to the esplora version pointed by your env var. For esplora it's needed a specific electrs fork (branch new-index in https://github.com/blockstream/electrs).

bitcoind and electrsd can run with env var set or by setting a specific feature that during the build process download the needed executables. At the moment the feature is used in our tests, see:

test-esplora = ["electrsd/legacy", "electrsd/esplora_a33e97e1", "test-blockchains"]

a33e97e1 stands for the specific commit of electrs, so you can either use the feature, or build electrs at that commit. Also electrsd/legacy feature is needed because there has been changes in electrs parameters...

[tcharding]

Thanks for the tips @RCasatta, I tried as you suggested (with no env vars set) but still tests hang. Can see from the log output that its pulling down bitcoind/electrs to run the tests. Any other ideas please?

$ cargo test --features test-electrum
...
[2021-08-02T22:27:17Z DEBUG bdk::testutils::blockchain_tests] launching /home/tobin/.cargo/bitcoin/bitcoin-0.21.1/bin/bitcoind and /home/tobin/.cargo/electrs/electrs_linux_v0.8.10/electrs
...
[2021-08-02T22:27:18Z DEBUG electrsd] args: ["-vvv", "--db-dir", "/tmp/.tmphAsIio", "--network", "regtest", "--cookie-file", "/tmp/.tmpDleo7B/regtest/.cookie", "--daemon-rpc-addr", "127.0.0.1:42461", "--jsonrpc-import", "--electrum-rpc-addr", "0.0.0.0:46495", "--monitoring-addr", "0.0.0.0:44203"]
test blockchain::electrum::bdk_blockchain_tests::test_sync_address_reuse has been running for over 60 seconds
test blockchain::electrum::bdk_blockchain_tests::test_sync_after_send has been running for over 60 seconds
test blockchain::electrum::bdk_blockchain_tests::test_sync_before_and_after_receive has been running for over 60 seconds
test blockchain::electrum::bdk_blockchain_tests::test_sync_bump_fee_add_input_no_change has been running for over 60 seconds

[RCasatta]

I can't reproduce on my system and neither on the CI (run in this PR looks failing for unrelated HTTP issue, this one https://github.com/RCasatta/bdk/actions/runs/1093394555 looks fine)

Maybe some more details on your system could help to reproduce.

Also, maybe running this:

RUST_LOG=bdk=debug cargo test --features test-electrum -- test_sync_address_reuse

also with this patch:

diff --git a/src/testutils/blockchain_tests.rs b/src/testutils/blockchain_tests.rs
index 2df72d3..e374cb0 100644
--- a/src/testutils/blockchain_tests.rs
+++ b/src/testutils/blockchain_tests.rs
@@ -24,12 +24,17 @@ pub struct TestClient {
impl TestClient {
pub fn new(bitcoind_exe: String, electrs_exe: String) -> Self {
debug!("launching {} and {}", &bitcoind_exe, &electrs_exe);
- let bitcoind = BitcoinD::new(bitcoind_exe).unwrap();
+ let conf = bitcoind::Conf {
+ view_stdout: true,
+ ..Default::default()
+ };
+ let bitcoind = BitcoinD::with_conf(bitcoind_exe).unwrap();
let http_enabled = cfg!(feature = "test-esplora");
let conf = Conf {
http_enabled,
+ view_stderr: true,
..Default::default()
};
let electrsd = ElectrsD::with_conf(electrs_exe, &bitcoind, &conf).unwrap();

May give insight on what's happening under the hood

UPDATE: the patch seems useful to have in case you have a Debug log level, so I opened a PR #417

[notmandatory]

Works for me locally (osx) and in CI so ACK 1f6dd33

I just noticed we also need to update the CI test-blockchains job to add the use-esplora-ureq or use-esplora-request features. I'll add a commit for this before merging.

Need to update CI and fix hanging

[tcharding]

Thanks @notmandatory, picking up the pieces for me :)

[notmandatory]

I think @rajarshimaitra's #430 will go in first then we can add this change. 🙂

[tcharding]

#433 solves this issue in a better way.

[tcharding]

Description

Recently we added sub-modules to the esplora module for different HTTP
backends. We (Tobin) failed to update the integration test runner.

Modify the test running macro so that the backend is explicitly
selected:

cargo test --no-default-features --features=use-esplora-ureq,test-esplora

Notes to the reviewers

I can't actually get the integration tests to run to completion. They all (test-rpc, test-electrum, test-esplora) hang

$ echo $BITCOIND_EXE ✘ 130 /opt/bitcoin/bin/bitcoind
$ echo $ELECTRS_EXE
/home/tobin/.cargo/bin/electrs
$ cargo test --no-default-features --features=use-esplora-ureq,test-esplora
test blockchain::esplora::bdk_blockchain_tests::test_sync_address_reuse has been running for over 60 seconds
test blockchain::esplora::bdk_blockchain_tests::test_sync_after_send has been running for over 60 seconds
test blockchain::esplora::bdk_blockchain_tests::test_sync_before_and_after_receive has been running for over 60 seconds
test blockchain::esplora::bdk_blockchain_tests::test_sync_bump_fee_add_input_no_change has been running for over 60 seconds

However without this patch the integration tests do not run at all for esplora so I believe this is still correct. And it seems like a good way to ask if I am doing something wrong with how I'm trying to run the integration tests? Any suggestions please?

Checklists

All Submissions:

• I've signed all my commits
• I followed the contribution guidelines
• I ran cargo fmt and cargo clippy before committing

New Features:

• I've added tests for the new feature
• I've added docs for the new feature
• I've updated CHANGELOG.md

Bugfixes:

• This pull request breaks the existing API
• I've added tests to reproduce the issue which are now passing
• I'm linking the issue being fixed by this PR

[RCasatta]

I think the hanging is more likely due to the esplora version pointed by your env var. For esplora it's needed a specific electrs fork (branch new-index in https://github.com/blockstream/electrs).

bitcoind and electrsd can run with env var set or by setting a specific feature that during the build process download the needed executables. At the moment the feature is used in our tests, see:

test-esplora = ["electrsd/legacy", "electrsd/esplora_a33e97e1", "test-blockchains"]

a33e97e1 stands for the specific commit of electrs, so you can either use the feature, or build electrs at that commit. Also electrsd/legacy feature is needed because there has been changes in electrs parameters...

[tcharding]

Thanks for the tips @RCasatta, I tried as you suggested (with no env vars set) but still tests hang. Can see from the log output that its pulling down bitcoind/electrs to run the tests. Any other ideas please?

$ cargo test --features test-electrum
...
[2021-08-02T22:27:17Z DEBUG bdk::testutils::blockchain_tests] launching /home/tobin/.cargo/bitcoin/bitcoin-0.21.1/bin/bitcoind and /home/tobin/.cargo/electrs/electrs_linux_v0.8.10/electrs
...
[2021-08-02T22:27:18Z DEBUG electrsd] args: ["-vvv", "--db-dir", "/tmp/.tmphAsIio", "--network", "regtest", "--cookie-file", "/tmp/.tmpDleo7B/regtest/.cookie", "--daemon-rpc-addr", "127.0.0.1:42461", "--jsonrpc-import", "--electrum-rpc-addr", "0.0.0.0:46495", "--monitoring-addr", "0.0.0.0:44203"]
test blockchain::electrum::bdk_blockchain_tests::test_sync_address_reuse has been running for over 60 seconds
test blockchain::electrum::bdk_blockchain_tests::test_sync_after_send has been running for over 60 seconds
test blockchain::electrum::bdk_blockchain_tests::test_sync_before_and_after_receive has been running for over 60 seconds
test blockchain::electrum::bdk_blockchain_tests::test_sync_bump_fee_add_input_no_change has been running for over 60 seconds

[RCasatta]

I can't reproduce on my system and neither on the CI (run in this PR looks failing for unrelated HTTP issue, this one https://github.com/RCasatta/bdk/actions/runs/1093394555 looks fine)

Maybe some more details on your system could help to reproduce.

Also, maybe running this:

RUST_LOG=bdk=debug cargo test --features test-electrum -- test_sync_address_reuse

also with this patch:

diff --git a/src/testutils/blockchain_tests.rs b/src/testutils/blockchain_tests.rs
index 2df72d3..e374cb0 100644
--- a/src/testutils/blockchain_tests.rs
+++ b/src/testutils/blockchain_tests.rs
@@ -24,12 +24,17 @@ pub struct TestClient {
impl TestClient {
pub fn new(bitcoind_exe: String, electrs_exe: String) -> Self {
debug!("launching {} and {}", &bitcoind_exe, &electrs_exe);
- let bitcoind = BitcoinD::new(bitcoind_exe).unwrap();
+ let conf = bitcoind::Conf {
+ view_stdout: true,
+ ..Default::default()
+ };
+ let bitcoind = BitcoinD::with_conf(bitcoind_exe).unwrap();
let http_enabled = cfg!(feature = "test-esplora");
let conf = Conf {
http_enabled,
+ view_stderr: true,
..Default::default()
};
let electrsd = ElectrsD::with_conf(electrs_exe, &bitcoind, &conf).unwrap();

May give insight on what's happening under the hood

UPDATE: the patch seems useful to have in case you have a Debug log level, so I opened a PR #417

[notmandatory]

Works for me locally (osx) and in CI so ACK 1f6dd33

I just noticed we also need to update the CI test-blockchains job to add the use-esplora-ureq or use-esplora-request features. I'll add a commit for this before merging.

Need to update CI and fix hanging

[tcharding]

Thanks @notmandatory, picking up the pieces for me :)

[notmandatory]

I think @rajarshimaitra's #430 will go in first then we can add this change. 🙂

[tcharding]

#433 solves this issue in a better way.

[tcharding]

Description

Recently we added sub-modules to the esplora module for different HTTP
backends. We (Tobin) failed to update the integration test runner.

Modify the test running macro so that the backend is explicitly
selected:

cargo test --no-default-features --features=use-esplora-ureq,test-esplora

Notes to the reviewers

I can't actually get the integration tests to run to completion. They all (test-rpc, test-electrum, test-esplora) hang

$ echo $BITCOIND_EXE ✘ 130 /opt/bitcoin/bin/bitcoind
$ echo $ELECTRS_EXE
/home/tobin/.cargo/bin/electrs
$ cargo test --no-default-features --features=use-esplora-ureq,test-esplora
test blockchain::esplora::bdk_blockchain_tests::test_sync_address_reuse has been running for over 60 seconds
test blockchain::esplora::bdk_blockchain_tests::test_sync_after_send has been running for over 60 seconds
test blockchain::esplora::bdk_blockchain_tests::test_sync_before_and_after_receive has been running for over 60 seconds
test blockchain::esplora::bdk_blockchain_tests::test_sync_bump_fee_add_input_no_change has been running for over 60 seconds

However without this patch the integration tests do not run at all for esplora so I believe this is still correct. And it seems like a good way to ask if I am doing something wrong with how I'm trying to run the integration tests? Any suggestions please?

Checklists

All Submissions:

• I've signed all my commits
• I followed the contribution guidelines
• I ran cargo fmt and cargo clippy before committing

New Features:

• I've added tests for the new feature
• I've added docs for the new feature
• I've updated CHANGELOG.md

Bugfixes:

• This pull request breaks the existing API
• I've added tests to reproduce the issue which are now passing
• I'm linking the issue being fixed by this PR

[RCasatta]

I think the hanging is more likely due to the esplora version pointed by your env var. For esplora it's needed a specific electrs fork (branch new-index in https://github.com/blockstream/electrs).

bitcoind and electrsd can run with env var set or by setting a specific feature that during the build process download the needed executables. At the moment the feature is used in our tests, see:

test-esplora = ["electrsd/legacy", "electrsd/esplora_a33e97e1", "test-blockchains"]

a33e97e1 stands for the specific commit of electrs, so you can either use the feature, or build electrs at that commit. Also electrsd/legacy feature is needed because there has been changes in electrs parameters...

[tcharding]

Thanks for the tips @RCasatta, I tried as you suggested (with no env vars set) but still tests hang. Can see from the log output that its pulling down bitcoind/electrs to run the tests. Any other ideas please?

$ cargo test --features test-electrum
...
[2021-08-02T22:27:17Z DEBUG bdk::testutils::blockchain_tests] launching /home/tobin/.cargo/bitcoin/bitcoin-0.21.1/bin/bitcoind and /home/tobin/.cargo/electrs/electrs_linux_v0.8.10/electrs
...
[2021-08-02T22:27:18Z DEBUG electrsd] args: ["-vvv", "--db-dir", "/tmp/.tmphAsIio", "--network", "regtest", "--cookie-file", "/tmp/.tmpDleo7B/regtest/.cookie", "--daemon-rpc-addr", "127.0.0.1:42461", "--jsonrpc-import", "--electrum-rpc-addr", "0.0.0.0:46495", "--monitoring-addr", "0.0.0.0:44203"]
test blockchain::electrum::bdk_blockchain_tests::test_sync_address_reuse has been running for over 60 seconds
test blockchain::electrum::bdk_blockchain_tests::test_sync_after_send has been running for over 60 seconds
test blockchain::electrum::bdk_blockchain_tests::test_sync_before_and_after_receive has been running for over 60 seconds
test blockchain::electrum::bdk_blockchain_tests::test_sync_bump_fee_add_input_no_change has been running for over 60 seconds

[RCasatta]

I can't reproduce on my system and neither on the CI (run in this PR looks failing for unrelated HTTP issue, this one https://github.com/RCasatta/bdk/actions/runs/1093394555 looks fine)

Maybe some more details on your system could help to reproduce.

Also, maybe running this:

RUST_LOG=bdk=debug cargo test --features test-electrum -- test_sync_address_reuse

also with this patch:

diff --git a/src/testutils/blockchain_tests.rs b/src/testutils/blockchain_tests.rs
index 2df72d3..e374cb0 100644
--- a/src/testutils/blockchain_tests.rs
+++ b/src/testutils/blockchain_tests.rs
@@ -24,12 +24,17 @@ pub struct TestClient {
impl TestClient {
pub fn new(bitcoind_exe: String, electrs_exe: String) -> Self {
debug!("launching {} and {}", &bitcoind_exe, &electrs_exe);
- let bitcoind = BitcoinD::new(bitcoind_exe).unwrap();
+ let conf = bitcoind::Conf {
+ view_stdout: true,
+ ..Default::default()
+ };
+ let bitcoind = BitcoinD::with_conf(bitcoind_exe).unwrap();
let http_enabled = cfg!(feature = "test-esplora");
let conf = Conf {
http_enabled,
+ view_stderr: true,
..Default::default()
};
let electrsd = ElectrsD::with_conf(electrs_exe, &bitcoind, &conf).unwrap();

May give insight on what's happening under the hood

UPDATE: the patch seems useful to have in case you have a Debug log level, so I opened a PR #417

[notmandatory]

Works for me locally (osx) and in CI so ACK 1f6dd33

I just noticed we also need to update the CI test-blockchains job to add the use-esplora-ureq or use-esplora-request features. I'll add a commit for this before merging.

Need to update CI and fix hanging

[tcharding]

Thanks @notmandatory, picking up the pieces for me :)

[notmandatory]

I think @rajarshimaitra's #430 will go in first then we can add this change. 🙂

[tcharding]

#433 solves this issue in a better way.

[tcharding]

Description

Recently we added sub-modules to the esplora module for different HTTP
backends. We (Tobin) failed to update the integration test runner.

Modify the test running macro so that the backend is explicitly
selected:

cargo test --no-default-features --features=use-esplora-ureq,test-esplora

Notes to the reviewers

I can't actually get the integration tests to run to completion. They all (test-rpc, test-electrum, test-esplora) hang

$ echo $BITCOIND_EXE ✘ 130 /opt/bitcoin/bin/bitcoind
$ echo $ELECTRS_EXE
/home/tobin/.cargo/bin/electrs
$ cargo test --no-default-features --features=use-esplora-ureq,test-esplora
test blockchain::esplora::bdk_blockchain_tests::test_sync_address_reuse has been running for over 60 seconds
test blockchain::esplora::bdk_blockchain_tests::test_sync_after_send has been running for over 60 seconds
test blockchain::esplora::bdk_blockchain_tests::test_sync_before_and_after_receive has been running for over 60 seconds
test blockchain::esplora::bdk_blockchain_tests::test_sync_bump_fee_add_input_no_change has been running for over 60 seconds

However without this patch the integration tests do not run at all for esplora so I believe this is still correct. And it seems like a good way to ask if I am doing something wrong with how I'm trying to run the integration tests? Any suggestions please?

Checklists

All Submissions:

• I've signed all my commits
• I followed the contribution guidelines
• I ran cargo fmt and cargo clippy before committing

New Features:

• I've added tests for the new feature
• I've added docs for the new feature
• I've updated CHANGELOG.md

Bugfixes:

• This pull request breaks the existing API
• I've added tests to reproduce the issue which are now passing
• I'm linking the issue being fixed by this PR

[RCasatta]

I think the hanging is more likely due to the esplora version pointed by your env var. For esplora it's needed a specific electrs fork (branch new-index in https://github.com/blockstream/electrs).

bitcoind and electrsd can run with env var set or by setting a specific feature that during the build process download the needed executables. At the moment the feature is used in our tests, see:

test-esplora = ["electrsd/legacy", "electrsd/esplora_a33e97e1", "test-blockchains"]

a33e97e1 stands for the specific commit of electrs, so you can either use the feature, or build electrs at that commit. Also electrsd/legacy feature is needed because there has been changes in electrs parameters...

[tcharding]

Thanks for the tips @RCasatta, I tried as you suggested (with no env vars set) but still tests hang. Can see from the log output that its pulling down bitcoind/electrs to run the tests. Any other ideas please?

$ cargo test --features test-electrum
...
[2021-08-02T22:27:17Z DEBUG bdk::testutils::blockchain_tests] launching /home/tobin/.cargo/bitcoin/bitcoin-0.21.1/bin/bitcoind and /home/tobin/.cargo/electrs/electrs_linux_v0.8.10/electrs
...
[2021-08-02T22:27:18Z DEBUG electrsd] args: ["-vvv", "--db-dir", "/tmp/.tmphAsIio", "--network", "regtest", "--cookie-file", "/tmp/.tmpDleo7B/regtest/.cookie", "--daemon-rpc-addr", "127.0.0.1:42461", "--jsonrpc-import", "--electrum-rpc-addr", "0.0.0.0:46495", "--monitoring-addr", "0.0.0.0:44203"]
test blockchain::electrum::bdk_blockchain_tests::test_sync_address_reuse has been running for over 60 seconds
test blockchain::electrum::bdk_blockchain_tests::test_sync_after_send has been running for over 60 seconds
test blockchain::electrum::bdk_blockchain_tests::test_sync_before_and_after_receive has been running for over 60 seconds
test blockchain::electrum::bdk_blockchain_tests::test_sync_bump_fee_add_input_no_change has been running for over 60 seconds

[RCasatta]

I can't reproduce on my system and neither on the CI (run in this PR looks failing for unrelated HTTP issue, this one https://github.com/RCasatta/bdk/actions/runs/1093394555 looks fine)

Maybe some more details on your system could help to reproduce.

Also, maybe running this:

RUST_LOG=bdk=debug cargo test --features test-electrum -- test_sync_address_reuse

also with this patch:

diff --git a/src/testutils/blockchain_tests.rs b/src/testutils/blockchain_tests.rs
index 2df72d3..e374cb0 100644
--- a/src/testutils/blockchain_tests.rs
+++ b/src/testutils/blockchain_tests.rs
@@ -24,12 +24,17 @@ pub struct TestClient {
impl TestClient {
pub fn new(bitcoind_exe: String, electrs_exe: String) -> Self {
debug!("launching {} and {}", &bitcoind_exe, &electrs_exe);
- let bitcoind = BitcoinD::new(bitcoind_exe).unwrap();
+ let conf = bitcoind::Conf {
+ view_stdout: true,
+ ..Default::default()
+ };
+ let bitcoind = BitcoinD::with_conf(bitcoind_exe).unwrap();
let http_enabled = cfg!(feature = "test-esplora");
let conf = Conf {
http_enabled,
+ view_stderr: true,
..Default::default()
};
let electrsd = ElectrsD::with_conf(electrs_exe, &bitcoind, &conf).unwrap();

May give insight on what's happening under the hood

UPDATE: the patch seems useful to have in case you have a Debug log level, so I opened a PR #417

[notmandatory]

Works for me locally (osx) and in CI so ACK 1f6dd33

I just noticed we also need to update the CI test-blockchains job to add the use-esplora-ureq or use-esplora-request features. I'll add a commit for this before merging.

Need to update CI and fix hanging

[tcharding]

Thanks @notmandatory, picking up the pieces for me :)

[notmandatory]

I think @rajarshimaitra's #430 will go in first then we can add this change. 🙂

[tcharding]

#433 solves this issue in a better way.

[tcharding]

Description

Recently we added sub-modules to the esplora module for different HTTP
backends. We (Tobin) failed to update the integration test runner.

Modify the test running macro so that the backend is explicitly
selected:

cargo test --no-default-features --features=use-esplora-ureq,test-esplora

Notes to the reviewers

I can't actually get the integration tests to run to completion. They all (test-rpc, test-electrum, test-esplora) hang

$ echo $BITCOIND_EXE ✘ 130 /opt/bitcoin/bin/bitcoind
$ echo $ELECTRS_EXE
/home/tobin/.cargo/bin/electrs
$ cargo test --no-default-features --features=use-esplora-ureq,test-esplora
test blockchain::esplora::bdk_blockchain_tests::test_sync_address_reuse has been running for over 60 seconds
test blockchain::esplora::bdk_blockchain_tests::test_sync_after_send has been running for over 60 seconds
test blockchain::esplora::bdk_blockchain_tests::test_sync_before_and_after_receive has been running for over 60 seconds
test blockchain::esplora::bdk_blockchain_tests::test_sync_bump_fee_add_input_no_change has been running for over 60 seconds

However without this patch the integration tests do not run at all for esplora so I believe this is still correct. And it seems like a good way to ask if I am doing something wrong with how I'm trying to run the integration tests? Any suggestions please?

Checklists

All Submissions:

• I've signed all my commits
• I followed the contribution guidelines
• I ran cargo fmt and cargo clippy before committing

New Features:

• I've added tests for the new feature
• I've added docs for the new feature
• I've updated CHANGELOG.md

Bugfixes:

• This pull request breaks the existing API
• I've added tests to reproduce the issue which are now passing
• I'm linking the issue being fixed by this PR

[RCasatta]

I think the hanging is more likely due to the esplora version pointed by your env var. For esplora it's needed a specific electrs fork (branch new-index in https://github.com/blockstream/electrs).

bitcoind and electrsd can run with env var set or by setting a specific feature that during the build process download the needed executables. At the moment the feature is used in our tests, see:

test-esplora = ["electrsd/legacy", "electrsd/esplora_a33e97e1", "test-blockchains"]

a33e97e1 stands for the specific commit of electrs, so you can either use the feature, or build electrs at that commit. Also electrsd/legacy feature is needed because there has been changes in electrs parameters...

[tcharding]

Thanks for the tips @RCasatta, I tried as you suggested (with no env vars set) but still tests hang. Can see from the log output that its pulling down bitcoind/electrs to run the tests. Any other ideas please?

$ cargo test --features test-electrum
...
[2021-08-02T22:27:17Z DEBUG bdk::testutils::blockchain_tests] launching /home/tobin/.cargo/bitcoin/bitcoin-0.21.1/bin/bitcoind and /home/tobin/.cargo/electrs/electrs_linux_v0.8.10/electrs
...
[2021-08-02T22:27:18Z DEBUG electrsd] args: ["-vvv", "--db-dir", "/tmp/.tmphAsIio", "--network", "regtest", "--cookie-file", "/tmp/.tmpDleo7B/regtest/.cookie", "--daemon-rpc-addr", "127.0.0.1:42461", "--jsonrpc-import", "--electrum-rpc-addr", "0.0.0.0:46495", "--monitoring-addr", "0.0.0.0:44203"]
test blockchain::electrum::bdk_blockchain_tests::test_sync_address_reuse has been running for over 60 seconds
test blockchain::electrum::bdk_blockchain_tests::test_sync_after_send has been running for over 60 seconds
test blockchain::electrum::bdk_blockchain_tests::test_sync_before_and_after_receive has been running for over 60 seconds
test blockchain::electrum::bdk_blockchain_tests::test_sync_bump_fee_add_input_no_change has been running for over 60 seconds

[RCasatta]

I can't reproduce on my system and neither on the CI (run in this PR looks failing for unrelated HTTP issue, this one https://github.com/RCasatta/bdk/actions/runs/1093394555 looks fine)

Maybe some more details on your system could help to reproduce.

Also, maybe running this:

RUST_LOG=bdk=debug cargo test --features test-electrum -- test_sync_address_reuse

also with this patch:

diff --git a/src/testutils/blockchain_tests.rs b/src/testutils/blockchain_tests.rs
index 2df72d3..e374cb0 100644
--- a/src/testutils/blockchain_tests.rs
+++ b/src/testutils/blockchain_tests.rs
@@ -24,12 +24,17 @@ pub struct TestClient {
impl TestClient {
pub fn new(bitcoind_exe: String, electrs_exe: String) -> Self {
debug!("launching {} and {}", &bitcoind_exe, &electrs_exe);
- let bitcoind = BitcoinD::new(bitcoind_exe).unwrap();
+ let conf = bitcoind::Conf {
+ view_stdout: true,
+ ..Default::default()
+ };
+ let bitcoind = BitcoinD::with_conf(bitcoind_exe).unwrap();
let http_enabled = cfg!(feature = "test-esplora");
let conf = Conf {
http_enabled,
+ view_stderr: true,
..Default::default()
};
let electrsd = ElectrsD::with_conf(electrs_exe, &bitcoind, &conf).unwrap();

May give insight on what's happening under the hood

UPDATE: the patch seems useful to have in case you have a Debug log level, so I opened a PR #417

[notmandatory]

Works for me locally (osx) and in CI so ACK 1f6dd33

I just noticed we also need to update the CI test-blockchains job to add the use-esplora-ureq or use-esplora-request features. I'll add a commit for this before merging.

Need to update CI and fix hanging

[tcharding]

Thanks @notmandatory, picking up the pieces for me :)

[notmandatory]

I think @rajarshimaitra's #430 will go in first then we can add this change. 🙂

[tcharding]

#433 solves this issue in a better way.