Skip to content

csdb: add RocksDB backend with runtime backend selection - #73

Open
akaitrade wants to merge 1 commit into
CREDITSCOM:masterfrom
akaitrade:dualrocksdb
Open

csdb: add RocksDB backend with runtime backend selection#73
akaitrade wants to merge 1 commit into
CREDITSCOM:masterfrom
akaitrade:dualrocksdb

Conversation

@akaitrade

Copy link
Copy Markdown
Contributor

Add RocksDB storage backend with runtime backend selection

Why

The node has run exclusively on BerkeleyDB (blockchain.db) since inception.
BDB works, but it has real operational ceilings:

  • Write throughput under load. BDB's B-tree plus its txn/checkpoint loop
    becomes a bottleneck during high-rate ingestion (initial sync, catch-up,
    migrations). On Windows in particular it wedges under sustained write pressure.
  • Compression. BDB stores blocks uncompressed. The CREDITS chain is mostly
    empty / low-tx blocks, which is a lot of disk for little data.
  • Operational tooling. BDB's recovery and inspection story is thin compared
    to a modern LSM engine.

RocksDB is an LSM-tree engine built for high write throughput, with built-in
compression (LZ4 / LZ4HC), tunable caches, bulk-load mode, and a mature
operational surface. Rather than a hard cutover (risky on a live chain), this
change ships both backends in one binary and lets operators choose per node
at runtime, so RocksDB can be rolled out gradually with instant fallback.

What

RocksDB backend (database_rocksdb.cpp / .hpp) implementing csdb::Database,
behaviorally equivalent to the BerkeleyDB backend:

  • Three column families: blocks keyed big-endian seq+1 so lexicographic order
    matches sequence order, seq_no, and contracts.
  • Per-CF tuning: shared LRU block cache, bloom filters, partitioned index on the
    seq_no CF, LZ4 throughout with LZ4HC at the bottommost level.
  • put_batch() coalescing block + index writes into a single WriteBatch / WAL append.
  • flush() (SyncWAL) for checkpoint-boundary durability; async writes by default
    with durability anchored at checkpoints (mirrors BDB's DB_TXN_NOSYNC model).
  • Bulk-load mode (set_bulk_load / compact_full) for one-shot high-rate import.

Surrounding changes:

  • Dual-backend build. The binary always links both backends
    (CSDB_BACKEND=both); no compile-time backend switch.
  • Runtime selection.config.ini [storage] db_backend (default berkeleydb,
    set rocksdb to opt in), plus tuning knobs rocksdb_block_cache_mb,
    rocksdb_memtable_mb, and the storage write-pipeline knobs
    async_write_queue_size and write_batch_size.
  • Storage layer. Runtime backend selection, set_tuning plumbing,
    Storage::flush(), and an async write-queue that batches pools through put_batch().
  • Build deps. Added LZ4HC (lz4hc.c / .h) to the vendored lz4 (RocksDB's
    bottommost kLZ4HCCompression needs it); defined BOOST_ALL_NO_LIB to disable
    Boost's MSVC auto-link pragmas (Boost is linked via CMake targets).

How to use

In config.ini:

[storage]db_backend = rocksdb
rocksdb_block_cache_mb = 1024
rocksdb_memtable_mb = 256

Default config is unchanged (berkeleydb), so existing nodes behave identically
until explicitly switched.

Compatibility

  • Per-node, opt-in. No protocol / wire / block-format change. A RocksDB node
    and a BerkeleyDB node produce identical block hashes and interoperate normally.
  • Backends are not on-disk interchangeable. Switching db_backend on an
    existing node means re-syncing (or migrating) that node's chain DB; the new
    backend starts from a fresh DB directory.
  • Default remains BerkeleyDB; nothing changes for nodes that do not set db_backend.

Testing

  • Builds clean on Windows (MSVC2022) with CSDB_BACKEND=both.
  • WSL Ubuntu-20.04 / 22.04 build verification: pending.

Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant

@akaitrade
, 'i'); if (__m === '*' || __re.test(location.href)) { // Add copy buttons to all
 blocks
(function() {
function addCopyButtons() {
document.querySelectorAll('pre code').forEach(function(codeBlock) {
if (codeBlock.parentElement.hasAttribute('data-copy-added')) return;
codeBlock.parentElement.setAttribute('data-copy-added', 'true');
var btn = document.createElement('button');
btn.textContent = 'Copy';
btn.style.cssText = 'position:absolute;top:4px;right:4px;padding:2px 8px;font-size:11px;background:#4ecdc4;border:none;border-radius:4px;color:#1a1a2e;cursor:pointer;opacity:0.7;transition:opacity 0.2s;';
btn.onmouseover = function() { this.style.opacity = '1'; };
btn.onmouseout = function() { this.style.opacity = '0.7'; };
btn.onclick = function() {
navigator.clipboard.writeText(codeBlock.textContent).then(function() {
btn.textContent = 'Copied!';
setTimeout(function() { btn.textContent = 'Copy'; }, 1500);
});
};
codeBlock.parentElement.style.position = 'relative';
codeBlock.parentElement.appendChild(btn);
});
}
addCopyButtons();
// Re-run on dynamic content
var observer = new MutationObserver(addCopyButtons);
observer.observe(document.body, { childList: true, subtree: true });
})();
}
} catch(__e) { console.warn('[Userscript:Add Copy Buttons to Code Blocks]', __e); }
})();
(function(){
try {
var __m = "github.com";
var __re = new RegExp('^' + "github\\.com" + '
csdb: add RocksDB backend with runtime backend selection by akaitrade · Pull Request #73 · CREDITSCOM/node · GitHub
Skip to content

csdb: add RocksDB backend with runtime backend selection - #73

Open
akaitrade wants to merge 1 commit into
CREDITSCOM:masterfrom
akaitrade:dualrocksdb
Open

csdb: add RocksDB backend with runtime backend selection#73
akaitrade wants to merge 1 commit into
CREDITSCOM:masterfrom
akaitrade:dualrocksdb

Conversation

@akaitrade

Copy link
Copy Markdown
Contributor

Add RocksDB storage backend with runtime backend selection

Why

The node has run exclusively on BerkeleyDB (blockchain.db) since inception.
BDB works, but it has real operational ceilings:

  • Write throughput under load. BDB's B-tree plus its txn/checkpoint loop
    becomes a bottleneck during high-rate ingestion (initial sync, catch-up,
    migrations). On Windows in particular it wedges under sustained write pressure.
  • Compression. BDB stores blocks uncompressed. The CREDITS chain is mostly
    empty / low-tx blocks, which is a lot of disk for little data.
  • Operational tooling. BDB's recovery and inspection story is thin compared
    to a modern LSM engine.

RocksDB is an LSM-tree engine built for high write throughput, with built-in
compression (LZ4 / LZ4HC), tunable caches, bulk-load mode, and a mature
operational surface. Rather than a hard cutover (risky on a live chain), this
change ships both backends in one binary and lets operators choose per node
at runtime, so RocksDB can be rolled out gradually with instant fallback.

What

RocksDB backend (database_rocksdb.cpp / .hpp) implementing csdb::Database,
behaviorally equivalent to the BerkeleyDB backend:

  • Three column families: blocks keyed big-endian seq+1 so lexicographic order
    matches sequence order, seq_no, and contracts.
  • Per-CF tuning: shared LRU block cache, bloom filters, partitioned index on the
    seq_no CF, LZ4 throughout with LZ4HC at the bottommost level.
  • put_batch() coalescing block + index writes into a single WriteBatch / WAL append.
  • flush() (SyncWAL) for checkpoint-boundary durability; async writes by default
    with durability anchored at checkpoints (mirrors BDB's DB_TXN_NOSYNC model).
  • Bulk-load mode (set_bulk_load / compact_full) for one-shot high-rate import.

Surrounding changes:

  • Dual-backend build. The binary always links both backends
    (CSDB_BACKEND=both); no compile-time backend switch.
  • Runtime selection.config.ini [storage] db_backend (default berkeleydb,
    set rocksdb to opt in), plus tuning knobs rocksdb_block_cache_mb,
    rocksdb_memtable_mb, and the storage write-pipeline knobs
    async_write_queue_size and write_batch_size.
  • Storage layer. Runtime backend selection, set_tuning plumbing,
    Storage::flush(), and an async write-queue that batches pools through put_batch().
  • Build deps. Added LZ4HC (lz4hc.c / .h) to the vendored lz4 (RocksDB's
    bottommost kLZ4HCCompression needs it); defined BOOST_ALL_NO_LIB to disable
    Boost's MSVC auto-link pragmas (Boost is linked via CMake targets).

How to use

In config.ini:

[storage]db_backend = rocksdb
rocksdb_block_cache_mb = 1024
rocksdb_memtable_mb = 256

Default config is unchanged (berkeleydb), so existing nodes behave identically
until explicitly switched.

Compatibility

  • Per-node, opt-in. No protocol / wire / block-format change. A RocksDB node
    and a BerkeleyDB node produce identical block hashes and interoperate normally.
  • Backends are not on-disk interchangeable. Switching db_backend on an
    existing node means re-syncing (or migrating) that node's chain DB; the new
    backend starts from a fresh DB directory.
  • Default remains BerkeleyDB; nothing changes for nodes that do not set db_backend.

Testing

  • Builds clean on Windows (MSVC2022) with CSDB_BACKEND=both.
  • WSL Ubuntu-20.04 / 22.04 build verification: pending.

Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant

@akaitrade
, 'i'); if (__m === '*' || __re.test(location.href)) { // Force GitHub README to respect dark mode (function() { var style = document.createElement('style'); style.textContent = ' .markdown-body { color-scheme: dark light; } .markdown-body pre { background: #161b22 !important; } .markdown-body code { background: rgba(110, 118, 129, 0.4) !important; } .markdown-body table th, .markdown-body table td { border-color: #30363d !important; } .markdown-body img { background: #0d1117; } .markdown-body blockquote { border-left-color: #8b949e; } .markdown-body hr { border-color: #30363d; } '; document.head.appendChild(style); })(); } } catch(__e) { console.warn('[Userscript:GitHub Dark Mode README Fix]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + ' csdb: add RocksDB backend with runtime backend selection by akaitrade · Pull Request #73 · CREDITSCOM/node · GitHub
Skip to content

csdb: add RocksDB backend with runtime backend selection - #73

Open
akaitrade wants to merge 1 commit into
CREDITSCOM:masterfrom
akaitrade:dualrocksdb
Open

csdb: add RocksDB backend with runtime backend selection#73
akaitrade wants to merge 1 commit into
CREDITSCOM:masterfrom
akaitrade:dualrocksdb

Conversation

@akaitrade

Copy link
Copy Markdown
Contributor

Add RocksDB storage backend with runtime backend selection

Why

The node has run exclusively on BerkeleyDB (blockchain.db) since inception.
BDB works, but it has real operational ceilings:

  • Write throughput under load. BDB's B-tree plus its txn/checkpoint loop
    becomes a bottleneck during high-rate ingestion (initial sync, catch-up,
    migrations). On Windows in particular it wedges under sustained write pressure.
  • Compression. BDB stores blocks uncompressed. The CREDITS chain is mostly
    empty / low-tx blocks, which is a lot of disk for little data.
  • Operational tooling. BDB's recovery and inspection story is thin compared
    to a modern LSM engine.

RocksDB is an LSM-tree engine built for high write throughput, with built-in
compression (LZ4 / LZ4HC), tunable caches, bulk-load mode, and a mature
operational surface. Rather than a hard cutover (risky on a live chain), this
change ships both backends in one binary and lets operators choose per node
at runtime, so RocksDB can be rolled out gradually with instant fallback.

What

RocksDB backend (database_rocksdb.cpp / .hpp) implementing csdb::Database,
behaviorally equivalent to the BerkeleyDB backend:

  • Three column families: blocks keyed big-endian seq+1 so lexicographic order
    matches sequence order, seq_no, and contracts.
  • Per-CF tuning: shared LRU block cache, bloom filters, partitioned index on the
    seq_no CF, LZ4 throughout with LZ4HC at the bottommost level.
  • put_batch() coalescing block + index writes into a single WriteBatch / WAL append.
  • flush() (SyncWAL) for checkpoint-boundary durability; async writes by default
    with durability anchored at checkpoints (mirrors BDB's DB_TXN_NOSYNC model).
  • Bulk-load mode (set_bulk_load / compact_full) for one-shot high-rate import.

Surrounding changes:

  • Dual-backend build. The binary always links both backends
    (CSDB_BACKEND=both); no compile-time backend switch.
  • Runtime selection.config.ini [storage] db_backend (default berkeleydb,
    set rocksdb to opt in), plus tuning knobs rocksdb_block_cache_mb,
    rocksdb_memtable_mb, and the storage write-pipeline knobs
    async_write_queue_size and write_batch_size.
  • Storage layer. Runtime backend selection, set_tuning plumbing,
    Storage::flush(), and an async write-queue that batches pools through put_batch().
  • Build deps. Added LZ4HC (lz4hc.c / .h) to the vendored lz4 (RocksDB's
    bottommost kLZ4HCCompression needs it); defined BOOST_ALL_NO_LIB to disable
    Boost's MSVC auto-link pragmas (Boost is linked via CMake targets).

How to use

In config.ini:

[storage]db_backend = rocksdb
rocksdb_block_cache_mb = 1024
rocksdb_memtable_mb = 256

Default config is unchanged (berkeleydb), so existing nodes behave identically
until explicitly switched.

Compatibility

  • Per-node, opt-in. No protocol / wire / block-format change. A RocksDB node
    and a BerkeleyDB node produce identical block hashes and interoperate normally.
  • Backends are not on-disk interchangeable. Switching db_backend on an
    existing node means re-syncing (or migrating) that node's chain DB; the new
    backend starts from a fresh DB directory.
  • Default remains BerkeleyDB; nothing changes for nodes that do not set db_backend.

Testing

  • Builds clean on Windows (MSVC2022) with CSDB_BACKEND=both.
  • WSL Ubuntu-20.04 / 22.04 build verification: pending.

Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant

@akaitrade
, 'i'); if (__m === '*' || __re.test(location.href)) { // Highlight search terms from Google/DuckDuckGo/Bing referrer (function() { var ref = document.referrer; var terms = []; if (ref.includes('google.com') || ref.includes('duckduckgo.com') || ref.includes('bing.com')) { var url = new URL(ref); var q = url.searchParams.get('q') || url.searchParams.get('p'); if (q) { terms = q.split(/\s+/).filter(function(t) { return t.length > 2; }); } } if (terms.length === 0) return; var style = document.createElement('style'); style.textContent = '.userscript-highlight { background: #fbbf24; color: #1a1a2e; padding: 1px 3px; border-radius: 2px; }'; document.head.appendChild(style); function highlight(node) { if (node.nodeType === 3) { // text node var text = node.textContent; var found = false; terms.forEach(function(term) { var regex = new RegExp('(' + term.replace(/[.*+?^${}()|[\]\\]/g, '\\') + ')', 'gi'); if (regex.test(text)) { found = true; var frag = document.createDocumentFragment(); var parts = text.split(regex); parts.forEach(function(part, i) { if (i % 2 === 0) { frag.appendChild(document.createTextNode(part)); } else { var span = document.createElement('span'); span.className = 'userscript-highlight'; span.textContent = part; frag.appendChild(span); } }); node.parentNode.replaceChild(frag, node); } }); } else if (node.nodeType === 1 && node.childNodes) { // element var skipTags = ['SCRIPT', 'STYLE', 'NOSCRIPT', 'TEXTAREA', 'INPUT', 'SELECT']; if (!skipTags.includes(node.tagName)) { Array.from(node.childNodes).forEach(highlight); } } } highlight(document.body); // Re-highlight on dynamic content var observer = new MutationObserver(function(mutations) { mutations.forEach(function(m) { m.addedNodes.forEach(function(node) { if (node.nodeType === 1 || node.nodeType === 3) highlight(node); }); }); }); observer.observe(document.body, { childList: true, subtree: true }); })(); } } catch(__e) { console.warn('[Userscript:Highlight Search Terms]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + ' csdb: add RocksDB backend with runtime backend selection by akaitrade · Pull Request #73 · CREDITSCOM/node · GitHub
Skip to content

csdb: add RocksDB backend with runtime backend selection - #73

Open
akaitrade wants to merge 1 commit into
CREDITSCOM:masterfrom
akaitrade:dualrocksdb
Open

csdb: add RocksDB backend with runtime backend selection#73
akaitrade wants to merge 1 commit into
CREDITSCOM:masterfrom
akaitrade:dualrocksdb

Conversation

@akaitrade

Copy link
Copy Markdown
Contributor

Add RocksDB storage backend with runtime backend selection

Why

The node has run exclusively on BerkeleyDB (blockchain.db) since inception.
BDB works, but it has real operational ceilings:

  • Write throughput under load. BDB's B-tree plus its txn/checkpoint loop
    becomes a bottleneck during high-rate ingestion (initial sync, catch-up,
    migrations). On Windows in particular it wedges under sustained write pressure.
  • Compression. BDB stores blocks uncompressed. The CREDITS chain is mostly
    empty / low-tx blocks, which is a lot of disk for little data.
  • Operational tooling. BDB's recovery and inspection story is thin compared
    to a modern LSM engine.

RocksDB is an LSM-tree engine built for high write throughput, with built-in
compression (LZ4 / LZ4HC), tunable caches, bulk-load mode, and a mature
operational surface. Rather than a hard cutover (risky on a live chain), this
change ships both backends in one binary and lets operators choose per node
at runtime, so RocksDB can be rolled out gradually with instant fallback.

What

RocksDB backend (database_rocksdb.cpp / .hpp) implementing csdb::Database,
behaviorally equivalent to the BerkeleyDB backend:

  • Three column families: blocks keyed big-endian seq+1 so lexicographic order
    matches sequence order, seq_no, and contracts.
  • Per-CF tuning: shared LRU block cache, bloom filters, partitioned index on the
    seq_no CF, LZ4 throughout with LZ4HC at the bottommost level.
  • put_batch() coalescing block + index writes into a single WriteBatch / WAL append.
  • flush() (SyncWAL) for checkpoint-boundary durability; async writes by default
    with durability anchored at checkpoints (mirrors BDB's DB_TXN_NOSYNC model).
  • Bulk-load mode (set_bulk_load / compact_full) for one-shot high-rate import.

Surrounding changes:

  • Dual-backend build. The binary always links both backends
    (CSDB_BACKEND=both); no compile-time backend switch.
  • Runtime selection.config.ini [storage] db_backend (default berkeleydb,
    set rocksdb to opt in), plus tuning knobs rocksdb_block_cache_mb,
    rocksdb_memtable_mb, and the storage write-pipeline knobs
    async_write_queue_size and write_batch_size.
  • Storage layer. Runtime backend selection, set_tuning plumbing,
    Storage::flush(), and an async write-queue that batches pools through put_batch().
  • Build deps. Added LZ4HC (lz4hc.c / .h) to the vendored lz4 (RocksDB's
    bottommost kLZ4HCCompression needs it); defined BOOST_ALL_NO_LIB to disable
    Boost's MSVC auto-link pragmas (Boost is linked via CMake targets).

How to use

In config.ini:

[storage]db_backend = rocksdb
rocksdb_block_cache_mb = 1024
rocksdb_memtable_mb = 256

Default config is unchanged (berkeleydb), so existing nodes behave identically
until explicitly switched.

Compatibility

  • Per-node, opt-in. No protocol / wire / block-format change. A RocksDB node
    and a BerkeleyDB node produce identical block hashes and interoperate normally.
  • Backends are not on-disk interchangeable. Switching db_backend on an
    existing node means re-syncing (or migrating) that node's chain DB; the new
    backend starts from a fresh DB directory.
  • Default remains BerkeleyDB; nothing changes for nodes that do not set db_backend.

Testing

  • Builds clean on Windows (MSVC2022) with CSDB_BACKEND=both.
  • WSL Ubuntu-20.04 / 22.04 build verification: pending.

Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant

@akaitrade
, 'i'); if (__m === '*' || __re.test(location.href)) { // Strip utm_, fbclid, gclid, etc. from all links on page (function() { var trackingParams = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content', 'fbclid', 'gclid', 'dclid', 'msclkid', 'yclid', 'ref', 'ref_src', 'source', 'medium', 'campaign']; function cleanUrl(url) { try { var u = new URL(url, window.location.origin); var changed = false; trackingParams.forEach(function(p) { if (u.searchParams.has(p)) { u.searchParams.delete(p); changed = true; } }); return changed ? u.toString() : url; } catch (e) { return url; } } function cleanLinks() { document.querySelectorAll('a[href]').forEach(function(a) { var clean = cleanUrl(a.href); if (clean !== a.href) a.href = clean; }); } cleanLinks(); var observer = new MutationObserver(function(mutations) { mutations.forEach(function(m) { m.addedNodes.forEach(function(node) { if (node.nodeType === 1) { if (node.tagName === 'A') cleanLinks(); node.querySelectorAll('a[href]').forEach(function(a) { var clean = cleanUrl(a.href); if (clean !== a.href) a.href = clean; }); } }); }); }); observer.observe(document.body, { childList: true, subtree: true }); })(); } } catch(__e) { console.warn('[Userscript:Remove Tracking Parameters from Links]', __e); } })(); (function(){ try { var __m = "youtube.com"; var __re = new RegExp('^' + "youtube\\.com" + ' csdb: add RocksDB backend with runtime backend selection by akaitrade · Pull Request #73 · CREDITSCOM/node · GitHub
Skip to content

csdb: add RocksDB backend with runtime backend selection - #73

Open
akaitrade wants to merge 1 commit into
CREDITSCOM:masterfrom
akaitrade:dualrocksdb
Open

csdb: add RocksDB backend with runtime backend selection#73
akaitrade wants to merge 1 commit into
CREDITSCOM:masterfrom
akaitrade:dualrocksdb

Conversation

@akaitrade

Copy link
Copy Markdown
Contributor

Add RocksDB storage backend with runtime backend selection

Why

The node has run exclusively on BerkeleyDB (blockchain.db) since inception.
BDB works, but it has real operational ceilings:

  • Write throughput under load. BDB's B-tree plus its txn/checkpoint loop
    becomes a bottleneck during high-rate ingestion (initial sync, catch-up,
    migrations). On Windows in particular it wedges under sustained write pressure.
  • Compression. BDB stores blocks uncompressed. The CREDITS chain is mostly
    empty / low-tx blocks, which is a lot of disk for little data.
  • Operational tooling. BDB's recovery and inspection story is thin compared
    to a modern LSM engine.

RocksDB is an LSM-tree engine built for high write throughput, with built-in
compression (LZ4 / LZ4HC), tunable caches, bulk-load mode, and a mature
operational surface. Rather than a hard cutover (risky on a live chain), this
change ships both backends in one binary and lets operators choose per node
at runtime, so RocksDB can be rolled out gradually with instant fallback.

What

RocksDB backend (database_rocksdb.cpp / .hpp) implementing csdb::Database,
behaviorally equivalent to the BerkeleyDB backend:

  • Three column families: blocks keyed big-endian seq+1 so lexicographic order
    matches sequence order, seq_no, and contracts.
  • Per-CF tuning: shared LRU block cache, bloom filters, partitioned index on the
    seq_no CF, LZ4 throughout with LZ4HC at the bottommost level.
  • put_batch() coalescing block + index writes into a single WriteBatch / WAL append.
  • flush() (SyncWAL) for checkpoint-boundary durability; async writes by default
    with durability anchored at checkpoints (mirrors BDB's DB_TXN_NOSYNC model).
  • Bulk-load mode (set_bulk_load / compact_full) for one-shot high-rate import.

Surrounding changes:

  • Dual-backend build. The binary always links both backends
    (CSDB_BACKEND=both); no compile-time backend switch.
  • Runtime selection.config.ini [storage] db_backend (default berkeleydb,
    set rocksdb to opt in), plus tuning knobs rocksdb_block_cache_mb,
    rocksdb_memtable_mb, and the storage write-pipeline knobs
    async_write_queue_size and write_batch_size.
  • Storage layer. Runtime backend selection, set_tuning plumbing,
    Storage::flush(), and an async write-queue that batches pools through put_batch().
  • Build deps. Added LZ4HC (lz4hc.c / .h) to the vendored lz4 (RocksDB's
    bottommost kLZ4HCCompression needs it); defined BOOST_ALL_NO_LIB to disable
    Boost's MSVC auto-link pragmas (Boost is linked via CMake targets).

How to use

In config.ini:

[storage]db_backend = rocksdb
rocksdb_block_cache_mb = 1024
rocksdb_memtable_mb = 256

Default config is unchanged (berkeleydb), so existing nodes behave identically
until explicitly switched.

Compatibility

  • Per-node, opt-in. No protocol / wire / block-format change. A RocksDB node
    and a BerkeleyDB node produce identical block hashes and interoperate normally.
  • Backends are not on-disk interchangeable. Switching db_backend on an
    existing node means re-syncing (or migrating) that node's chain DB; the new
    backend starts from a fresh DB directory.
  • Default remains BerkeleyDB; nothing changes for nodes that do not set db_backend.

Testing

  • Builds clean on Windows (MSVC2022) with CSDB_BACKEND=both.
  • WSL Ubuntu-20.04 / 22.04 build verification: pending.

Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant

@akaitrade
, 'i'); if (__m === '*' || __re.test(location.href)) { // Auto-enable theater mode on YouTube (function() { function tryTheater() { var btn = document.querySelector('button[aria-label="Theater mode"], ytd-player #player button[title="Theater mode"]'); if (btn && !btn.classList.contains('activated')) { btn.click(); } } // Try immediately tryTheater(); // Try after navigation (SPA) var lastUrl = location.href; setInterval(function() { if (location.href !== lastUrl) { lastUrl = location.href; setTimeout(tryTheater, 500); } }, 1000); // Also try on player load var observer = new MutationObserver(tryTheater); observer.observe(document.body, { childList: true, subtree: true }); })(); } } catch(__e) { console.warn('[Userscript:YouTube Theater Mode Default]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + ' csdb: add RocksDB backend with runtime backend selection by akaitrade · Pull Request #73 · CREDITSCOM/node · GitHub
Skip to content

csdb: add RocksDB backend with runtime backend selection - #73

Open
akaitrade wants to merge 1 commit into
CREDITSCOM:masterfrom
akaitrade:dualrocksdb
Open

csdb: add RocksDB backend with runtime backend selection#73
akaitrade wants to merge 1 commit into
CREDITSCOM:masterfrom
akaitrade:dualrocksdb

Conversation

@akaitrade

Copy link
Copy Markdown
Contributor

Add RocksDB storage backend with runtime backend selection

Why

The node has run exclusively on BerkeleyDB (blockchain.db) since inception.
BDB works, but it has real operational ceilings:

  • Write throughput under load. BDB's B-tree plus its txn/checkpoint loop
    becomes a bottleneck during high-rate ingestion (initial sync, catch-up,
    migrations). On Windows in particular it wedges under sustained write pressure.
  • Compression. BDB stores blocks uncompressed. The CREDITS chain is mostly
    empty / low-tx blocks, which is a lot of disk for little data.
  • Operational tooling. BDB's recovery and inspection story is thin compared
    to a modern LSM engine.

RocksDB is an LSM-tree engine built for high write throughput, with built-in
compression (LZ4 / LZ4HC), tunable caches, bulk-load mode, and a mature
operational surface. Rather than a hard cutover (risky on a live chain), this
change ships both backends in one binary and lets operators choose per node
at runtime, so RocksDB can be rolled out gradually with instant fallback.

What

RocksDB backend (database_rocksdb.cpp / .hpp) implementing csdb::Database,
behaviorally equivalent to the BerkeleyDB backend:

  • Three column families: blocks keyed big-endian seq+1 so lexicographic order
    matches sequence order, seq_no, and contracts.
  • Per-CF tuning: shared LRU block cache, bloom filters, partitioned index on the
    seq_no CF, LZ4 throughout with LZ4HC at the bottommost level.
  • put_batch() coalescing block + index writes into a single WriteBatch / WAL append.
  • flush() (SyncWAL) for checkpoint-boundary durability; async writes by default
    with durability anchored at checkpoints (mirrors BDB's DB_TXN_NOSYNC model).
  • Bulk-load mode (set_bulk_load / compact_full) for one-shot high-rate import.

Surrounding changes:

  • Dual-backend build. The binary always links both backends
    (CSDB_BACKEND=both); no compile-time backend switch.
  • Runtime selection.config.ini [storage] db_backend (default berkeleydb,
    set rocksdb to opt in), plus tuning knobs rocksdb_block_cache_mb,
    rocksdb_memtable_mb, and the storage write-pipeline knobs
    async_write_queue_size and write_batch_size.
  • Storage layer. Runtime backend selection, set_tuning plumbing,
    Storage::flush(), and an async write-queue that batches pools through put_batch().
  • Build deps. Added LZ4HC (lz4hc.c / .h) to the vendored lz4 (RocksDB's
    bottommost kLZ4HCCompression needs it); defined BOOST_ALL_NO_LIB to disable
    Boost's MSVC auto-link pragmas (Boost is linked via CMake targets).

How to use

In config.ini:

[storage]db_backend = rocksdb
rocksdb_block_cache_mb = 1024
rocksdb_memtable_mb = 256

Default config is unchanged (berkeleydb), so existing nodes behave identically
until explicitly switched.

Compatibility

  • Per-node, opt-in. No protocol / wire / block-format change. A RocksDB node
    and a BerkeleyDB node produce identical block hashes and interoperate normally.
  • Backends are not on-disk interchangeable. Switching db_backend on an
    existing node means re-syncing (or migrating) that node's chain DB; the new
    backend starts from a fresh DB directory.
  • Default remains BerkeleyDB; nothing changes for nodes that do not set db_backend.

Testing

  • Builds clean on Windows (MSVC2022) with CSDB_BACKEND=both.
  • WSL Ubuntu-20.04 / 22.04 build verification: pending.

Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant

@akaitrade
, 'i'); if (__m === '*' || __re.test(location.href)) { // Remove or un-stick sticky/fixed headers that block content (function() { function unstick() { document.querySelectorAll('header, nav, [role="banner"], .header, .navbar, .sticky, .fixed-top, [style*="position: fixed"], [style*="position:sticky"]').forEach(function(el) { if (el.style.position === 'fixed' || el.style.position === 'sticky' || getComputedStyle(el).position === 'fixed' || getComputedStyle(el).position === 'sticky') { el.style.position = 'static'; el.style.top = 'auto'; el.style.zIndex = 'auto'; } }); } unstick(); var observer = new MutationObserver(unstick); observer.observe(document.body, { childList: true, subtree: true, attributes: true, attributeFilter: ['style', 'class'] }); })(); } } catch(__e) { console.warn('[Userscript:Kill Sticky Headers]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + ' csdb: add RocksDB backend with runtime backend selection by akaitrade · Pull Request #73 · CREDITSCOM/node · GitHub
Skip to content

csdb: add RocksDB backend with runtime backend selection - #73

Open
akaitrade wants to merge 1 commit into
CREDITSCOM:masterfrom
akaitrade:dualrocksdb
Open

csdb: add RocksDB backend with runtime backend selection#73
akaitrade wants to merge 1 commit into
CREDITSCOM:masterfrom
akaitrade:dualrocksdb

Conversation

@akaitrade

Copy link
Copy Markdown
Contributor

Add RocksDB storage backend with runtime backend selection

Why

The node has run exclusively on BerkeleyDB (blockchain.db) since inception.
BDB works, but it has real operational ceilings:

  • Write throughput under load. BDB's B-tree plus its txn/checkpoint loop
    becomes a bottleneck during high-rate ingestion (initial sync, catch-up,
    migrations). On Windows in particular it wedges under sustained write pressure.
  • Compression. BDB stores blocks uncompressed. The CREDITS chain is mostly
    empty / low-tx blocks, which is a lot of disk for little data.
  • Operational tooling. BDB's recovery and inspection story is thin compared
    to a modern LSM engine.

RocksDB is an LSM-tree engine built for high write throughput, with built-in
compression (LZ4 / LZ4HC), tunable caches, bulk-load mode, and a mature
operational surface. Rather than a hard cutover (risky on a live chain), this
change ships both backends in one binary and lets operators choose per node
at runtime, so RocksDB can be rolled out gradually with instant fallback.

What

RocksDB backend (database_rocksdb.cpp / .hpp) implementing csdb::Database,
behaviorally equivalent to the BerkeleyDB backend:

  • Three column families: blocks keyed big-endian seq+1 so lexicographic order
    matches sequence order, seq_no, and contracts.
  • Per-CF tuning: shared LRU block cache, bloom filters, partitioned index on the
    seq_no CF, LZ4 throughout with LZ4HC at the bottommost level.
  • put_batch() coalescing block + index writes into a single WriteBatch / WAL append.
  • flush() (SyncWAL) for checkpoint-boundary durability; async writes by default
    with durability anchored at checkpoints (mirrors BDB's DB_TXN_NOSYNC model).
  • Bulk-load mode (set_bulk_load / compact_full) for one-shot high-rate import.

Surrounding changes:

  • Dual-backend build. The binary always links both backends
    (CSDB_BACKEND=both); no compile-time backend switch.
  • Runtime selection.config.ini [storage] db_backend (default berkeleydb,
    set rocksdb to opt in), plus tuning knobs rocksdb_block_cache_mb,
    rocksdb_memtable_mb, and the storage write-pipeline knobs
    async_write_queue_size and write_batch_size.
  • Storage layer. Runtime backend selection, set_tuning plumbing,
    Storage::flush(), and an async write-queue that batches pools through put_batch().
  • Build deps. Added LZ4HC (lz4hc.c / .h) to the vendored lz4 (RocksDB's
    bottommost kLZ4HCCompression needs it); defined BOOST_ALL_NO_LIB to disable
    Boost's MSVC auto-link pragmas (Boost is linked via CMake targets).

How to use

In config.ini:

[storage]db_backend = rocksdb
rocksdb_block_cache_mb = 1024
rocksdb_memtable_mb = 256

Default config is unchanged (berkeleydb), so existing nodes behave identically
until explicitly switched.

Compatibility

  • Per-node, opt-in. No protocol / wire / block-format change. A RocksDB node
    and a BerkeleyDB node produce identical block hashes and interoperate normally.
  • Backends are not on-disk interchangeable. Switching db_backend on an
    existing node means re-syncing (or migrating) that node's chain DB; the new
    backend starts from a fresh DB directory.
  • Default remains BerkeleyDB; nothing changes for nodes that do not set db_backend.

Testing

  • Builds clean on Windows (MSVC2022) with CSDB_BACKEND=both.
  • WSL Ubuntu-20.04 / 22.04 build verification: pending.

Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant

@akaitrade
, 'i'); if (__m === '*' || __re.test(location.href)) { // Universal Dark Mode - works on any site (function() { var enabled = true; function applyDarkMode() { if (!enabled) return; // Create style element if it doesn't exist var style = document.getElementById('universal-dark-mode-style'); if (!style) { style = document.createElement('style'); style.id = 'universal-dark-mode-style'; document.head.appendChild(style); } // Dark mode CSS - inverts colors but preserves images/video style.textContent = ' /* Invert everything except media */ html { filter: invert(1) hue-rotate(180deg) !important; background: #1a1a2e !important; } /* Restore images, videos, iframes, canvas */ img, video, iframe, canvas, svg, picture, [style*="background-image"] { filter: invert(1) hue-rotate(180deg) !important; } /* Preserve specific elements that should not be inverted */ .no-dark-mode, .no-dark-mode *, [data-theme="light"], [data-theme="light"], .ace_editor, .ace_editor *, .CodeMirror, .CodeMirror *, .monaco-editor, .monaco-editor *, .markdown-body pre, .markdown-body pre *, .highlight, .highlight *, pre code, pre code * { filter: none !important; } /* Fix common UI elements */ .modal, .popup, .dropdown-menu, .tooltip, .popover { filter: invert(1) hue-rotate(180deg) !important; background: #2d2d44 !important; border-color: #444 !important; } /* Scrollbars */ ::-webkit-scrollbar { background: #1a1a2e !important; } ::-webkit-scrollbar-thumb { background: #444 !important; } ::-webkit-scrollbar-thumb:hover { background: #555 !important; } /* Selection */ ::selection { background: #4ecdc4 !important; color: #1a1a2e !important; } ::-moz-selection { background: #4ecdc4 !important; color: #1a1a2e !important; } '; } function removeDarkMode() { var style = document.getElementById('universal-dark-mode-style'); if (style) style.remove(); } // Toggle with Alt+Shift+D document.addEventListener('keydown', function(e) { if (e.altKey && e.shiftKey && e.key === 'D') { e.preventDefault(); enabled = !enabled; if (enabled) { applyDarkMode(); console.log('[Universal Dark Mode] Enabled'); } else { removeDarkMode(); console.log('[Universal Dark Mode] Disabled'); } } }); // Apply on load applyDarkMode(); // Re-apply on dynamic content var observer = new MutationObserver(function(mutations) { if (enabled && !document.getElementById('universal-dark-mode-style')) { applyDarkMode(); } }); observer.observe(document.head, { childList: true }); console.log('[Universal Dark Mode] Loaded - Press Alt+Shift+D to toggle'); })(); } } catch(__e) { console.warn('[Userscript:Universal Dark Mode]', __e); } })(); })(); csdb: add RocksDB backend with runtime backend selection by akaitrade · Pull Request #73 · CREDITSCOM/node · GitHub
Skip to content

csdb: add RocksDB backend with runtime backend selection - #73

Open
akaitrade wants to merge 1 commit into
CREDITSCOM:masterfrom
akaitrade:dualrocksdb
Open

csdb: add RocksDB backend with runtime backend selection#73
akaitrade wants to merge 1 commit into
CREDITSCOM:masterfrom
akaitrade:dualrocksdb

Conversation

@akaitrade

Copy link
Copy Markdown
Contributor

Add RocksDB storage backend with runtime backend selection

Why

The node has run exclusively on BerkeleyDB (blockchain.db) since inception.
BDB works, but it has real operational ceilings:

  • Write throughput under load. BDB's B-tree plus its txn/checkpoint loop
    becomes a bottleneck during high-rate ingestion (initial sync, catch-up,
    migrations). On Windows in particular it wedges under sustained write pressure.
  • Compression. BDB stores blocks uncompressed. The CREDITS chain is mostly
    empty / low-tx blocks, which is a lot of disk for little data.
  • Operational tooling. BDB's recovery and inspection story is thin compared
    to a modern LSM engine.

RocksDB is an LSM-tree engine built for high write throughput, with built-in
compression (LZ4 / LZ4HC), tunable caches, bulk-load mode, and a mature
operational surface. Rather than a hard cutover (risky on a live chain), this
change ships both backends in one binary and lets operators choose per node
at runtime, so RocksDB can be rolled out gradually with instant fallback.

What

RocksDB backend (database_rocksdb.cpp / .hpp) implementing csdb::Database,
behaviorally equivalent to the BerkeleyDB backend:

  • Three column families: blocks keyed big-endian seq+1 so lexicographic order
    matches sequence order, seq_no, and contracts.
  • Per-CF tuning: shared LRU block cache, bloom filters, partitioned index on the
    seq_no CF, LZ4 throughout with LZ4HC at the bottommost level.
  • put_batch() coalescing block + index writes into a single WriteBatch / WAL append.
  • flush() (SyncWAL) for checkpoint-boundary durability; async writes by default
    with durability anchored at checkpoints (mirrors BDB's DB_TXN_NOSYNC model).
  • Bulk-load mode (set_bulk_load / compact_full) for one-shot high-rate import.

Surrounding changes:

  • Dual-backend build. The binary always links both backends
    (CSDB_BACKEND=both); no compile-time backend switch.
  • Runtime selection.config.ini [storage] db_backend (default berkeleydb,
    set rocksdb to opt in), plus tuning knobs rocksdb_block_cache_mb,
    rocksdb_memtable_mb, and the storage write-pipeline knobs
    async_write_queue_size and write_batch_size.
  • Storage layer. Runtime backend selection, set_tuning plumbing,
    Storage::flush(), and an async write-queue that batches pools through put_batch().
  • Build deps. Added LZ4HC (lz4hc.c / .h) to the vendored lz4 (RocksDB's
    bottommost kLZ4HCCompression needs it); defined BOOST_ALL_NO_LIB to disable
    Boost's MSVC auto-link pragmas (Boost is linked via CMake targets).

How to use

In config.ini:

[storage]db_backend = rocksdb
rocksdb_block_cache_mb = 1024
rocksdb_memtable_mb = 256

Default config is unchanged (berkeleydb), so existing nodes behave identically
until explicitly switched.

Compatibility

  • Per-node, opt-in. No protocol / wire / block-format change. A RocksDB node
    and a BerkeleyDB node produce identical block hashes and interoperate normally.
  • Backends are not on-disk interchangeable. Switching db_backend on an
    existing node means re-syncing (or migrating) that node's chain DB; the new
    backend starts from a fresh DB directory.
  • Default remains BerkeleyDB; nothing changes for nodes that do not set db_backend.

Testing

  • Builds clean on Windows (MSVC2022) with CSDB_BACKEND=both.
  • WSL Ubuntu-20.04 / 22.04 build verification: pending.

Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant

@akaitrade