Skip to content

refactor: update cliento to blob api and remove core da - #2887

Closed
randygrok wants to merge 39 commits into
mainfrom
celestia-blob-api-client
Closed

refactor: update cliento to blob api and remove core da#2887
randygrok wants to merge 39 commits into
mainfrom
celestia-blob-api-client

Conversation

@randygrok

Copy link
Copy Markdown
Contributor

Overview

Add native Celestia blob API client using celestia-node's blob API.
- Client with JSON-RPC connection to celestia-node
- Native types (Namespace, Blob, Commitment, Proof)
- Stub methods (Submit, Get, GetAll, GetProof, Included)
- Unit tests for types and client
Implement Submit method for Celestia blob API client. Validation
is delegated to celestia-node rather than duplicating checks in
the client layer.
Implement Get, GetAll, GetProof, and Included methods for the
Celestia blob API client. All methods delegate to celestia-node
via JSON-RPC with appropriate logging.
Add temporary adapter that implements da.DA interface using the native
Celestia blob API client. This allows ev-node to use the new Celestia
client while maintaining compatibility with existing code.
The adapter bridges the gap between:
- Celestia's height-based blob API
- ev-node's ID-based DA abstraction
IDs are encoded as [height (8 bytes)][commitment] for compatibility.
Replace jsonrpc.NewClient with the new Celestia blob API adapter
in testapp initialization. This uses the native Celestia blob API
instead of the deprecated DA API.
Replace jsonrpc.NewClient with the new Celestia blob API adapter
in evm and grpc app initialization. All apps now use the native
Celestia blob API instead of the deprecated DA API.
Remove the generic JSON-RPC DA abstraction layer now that all
applications use the Celestia blob API adapter directly.
Changes:
- Remove da/jsonrpc package (client, server, tests)
- Remove tools/da-debug debugging tool
- Embed JSON-RPC server directly in local-da for testing
- Update documentation to reflect Celestia as the DA implementation
- Remove da-debug from tools.mk build configuration
Move all DA-related code from core/da module into a single da/da.go file.
The DA interface is now co-located with its implementation (Celestia adapter)
rather than being a separate core module.
Changes:
- Consolidate core/da/{da.go,errors.go,namespace.go} into da/da.go
- Move DummyDA test utility to da/testing.go
- Update all imports from github.com/evstack/ev-node/core/da to github.com/evstack/ev-node/da
- Remove core/da directory entirely
- Fix package shadowing issues in block/internal/{submitting,syncing}
The DA interface is kept for testability with mocks, but no longer
needs to be a separate module since Celestia is the only implementation.
- Move all DA interface methods (Submit, Get, GetIDs, GetProofs, Commit, Validate) directly into Client
- Rename internal Celestia blob API methods to lowercase (submit, get, getAll, getProof, included)
- Update all applications to use celestiada.NewClient instead of celestiada.NewAdapter
- Remove adapter.go and adapter_test.go (397 lines)
- Update client tests to call lowercase internal methods
- Move all DA interface methods (Submit, Get, GetIDs, GetProofs, Commit, Validate) directly into Client
- Rename internal Celestia blob API methods to lowercase (submit, get, getAll, getProof, included)
- Update all applications to use celestiada.NewClient instead of celestiada.NewAdapter
- Remove adapter.go and adapter_test.go (397 lines)
- Update client tests to call lowercase internal methods
…v-node into celestia-blob-api-client
# Conflicts:
#	da/celestia/client.go
- bring back command-line tool `da-debug` for inspecting data availability layers in blockchain.
- Added functionality to query and search for blobs at specific DA heights and namespaces.
- Introduced detailed logging and error handling for better user experience.
- Created unit tests for key functions including namespace parsing, hex decoding, and blob data handling.
@randygrokrandygrok changed the title Celestia blob api clientrefactor: update cliento to blob api and remove core daNov 26, 2025
@claude

claudeBot commented Nov 26, 2025

Copy link
Copy Markdown
Contributor

Claude finished @randygrok's task —— View job


Comprehensive Code Review Complete

I've completed a thorough code review of this DA layer refactoring. This PR represents a significant architectural improvement, migrating from the previous core/da interface to a new da package with direct Celestia blob API integration. Here are my findings:


1. Code Quality & Architecture ⭐⭐⭐⭐½

Strengths

  • Clean interface design: The new da.DA interface (da/da.go:15-39) is well-structured with clear method signatures and comprehensive status codes
  • Proper separation of concerns: Moving DA logic out of core/ into its own da/ package improves modularity
  • Type safety improvements: Using typed Blob, ID, Commitment, and Proof as byte slices provides flexibility while maintaining clarity
  • Structured results: ResultSubmit and ResultRetrieve provide rich feedback with status codes, timestamps, and metadata

Areas for Improvement

1. Namespace handling could be more type-safe (da/da.go:180-264)

// Current approach mixes []byte and Namespace structtypeNamespacestruct {
Versionuint8ID [NamespaceIDSize]byte
}
// But many methods accept []byte for namespacefunc (c*Client) Submit(ctx context.Context, blobs []da.Blob, gasPricefloat64, namespace []byte)

Recommendation: Consider consistently using the Namespace type throughout the API instead of []byte to prevent invalid namespace errors at compile time.

2. Duplicate makeID function (da/da.go:101, da/celestia/client.go:347)

// Both da/da.go and da/celestia/client.go define makeIDfuncmakeID(heightuint64, commitment []byte) []byte

Recommendation: Export the function from da/da.go and remove the duplicate to follow DRY principles.

3. Magic numbers in client (da/celestia/client.go:20, da/celestia/commitment.go:13)

constdefaultRetrieveTimeout=10*time.SecondconstsubtreeRootThreshold=64

Recommendation: Add comments explaining why these specific values were chosen, particularly subtreeRootThreshold which must match celestia-app's configuration.


2. Error Handling & Resilience ⭐⭐⭐⭐

Strengths

  • Comprehensive error mapping: The handleSubmitError and handleRetrieveError methods (da/celestia/client.go:500-617) properly map JSON-RPC errors to status codes
  • Context-aware error handling: Proper handling of context.Canceled and context.DeadlineExceeded
  • Retry logic with backoff: The submitter implements exponential backoff with clamping (block/internal/submitting/da_submitter.go:62-93)
  • Structured error types: Well-defined error variables (da/da.go:118-127) using sentinel errors

Areas for Improvement

1. Error wrapping inconsistency (da/celestia/client.go:238, 290)

// Some errors are wrapped, others are notreturnnil, fmt.Errorf("invalid ID at index %d: %w", i, err) // Good ✓returnnil, fmt.Errorf("failed to get blobs at height %d: %w", height, err) // Good ✓

Most error handling is good, but ensure ALL errors use %w for proper unwrapping.

2. Silent error in namespace creation (da/da.go:245)

funcNamespaceFromString(sstring) *Namespace {
hash:=sha256.Sum256([]byte(s))
ns, _:=NewNamespaceV0(hash[:NamespaceVersionZeroDataSize])
returnns// Ignoring error - could return nil!
}

Critical Issue: This function can return nil without the caller knowing. This could cause nil pointer panics.
Recommendation: Change signature to func NamespaceFromString(s string) (*Namespace, error) or document that it never fails for valid input.

3. Missing timeout on commitment calculation (da/celestia/client.go:305-314)

func (c*Client) Commit(ctx context.Context, blobs []da.Blob, namespace []byte) ([]da.Commitment, error) {
// ctx is accepted but never usedfori, blob:=rangeblobs {
commitment, err:=CreateCommitment(blob, namespace)
// No timeout or cancellation check
}
}

Recommendation: Either use the context for cancellation checks in loops, or remove it from the signature if not needed.

4. Batch size halving could oscillate (block/internal/submitting/da_submitter.go:444-466)

case dapkg.StatusTooBig:
iflen(items) ==1 {
returnfmt.Errorf("unrecoverable error...")
}
half:=len(items) /2ifhalf==0 {
half=1
}
items=items[:half]

Observation: This halving strategy works but doesn't account for per-blob vs total-batch size limits. The pre-check at line 392-400 helps, but consider a more sophisticated size calculation.


3. Security ⭐⭐⭐⭐⭐

Strengths

  • No hardcoded secrets: Authentication tokens are properly parameterized (da/celestia/client.go:39-62)
  • Namespace validation: Strict validation prevents invalid namespace submissions (da/celestia/types.go:58-73)
  • Signature verification: Proper Ed25519 signature validation in syncer (block/internal/syncing/da_retriever.go:315-340)
  • Input validation: Comprehensive validation of blob sizes, namespaces, and commitment formats
  • Proposer address verification: Ensures only expected proposers can submit data (block/internal/syncing/da_retriever.go:306-313)

Areas for Improvement

1. Potential information leak in error messages (da/celestia/client.go:374, 409)

Message: fmt.Sprintf("invalid namespace: %s", err.Error()),
Message: fmt.Sprintf("failed to create commitment for blob %d: %s", i, err.Error()),

Low Priority: These expose internal error details to callers. Generally acceptable for a blockchain DA layer, but consider if any sensitive info could leak.

2. No rate limiting on client (da/celestia/client.go)
The client doesn't implement any rate limiting for requests to celestia-node. While this may be handled at the celestia-node level, consider adding client-side rate limiting for defense in depth.


4. Performance & Resource Efficiency ⭐⭐⭐⭐

Strengths

  • Concurrent marshaling: Smart use of goroutines with semaphore (32 workers) for marshaling items (block/internal/submitting/da_submitter.go:524-571)
  • Connection reuse: Single JSON-RPC connection with proper cleanup (da/celestia/client.go:64-92)
  • Efficient ID grouping: Groups IDs by height to minimize DA queries (da/celestia/client.go:227-259)
  • Namespace byte caching: Pre-computes namespace bytes once (block/internal/submitting/da_submitter.go:105-107, 151-152)

Areas for Improvement

1. Unbounded memory growth in Get (da/celestia/client.go:222-262)

func (c*Client) Get(ctx context.Context, ids []da.ID, namespace []byte) ([]da.Blob, error) {
heightGroups:=make(map[uint64][]Commitment)
idToIndex:=make(map[blobKey]int)
fori, id:=rangeids {
heightGroups[height] =append(heightGroups[height], commitment)
idToIndex[blobKey{height, string(commitment)}] =i
}
// Maps grow unbounded with number of IDs
}

Recommendation: For very large ID sets, consider batching or streaming to limit memory usage.

2. Missing context cancellation checks in loops (da/celestia/client.go:244-259, 281-300)
Long-running loops don't check ctx.Done(), which could delay cancellation.
Recommendation:

fori, id:=rangeids {
select {
case<-ctx.Done():
returnnil, ctx.Err()
default:
}
// process ID
}

3. Goroutine leak potential in marshaling (block/internal/submitting/da_submitter.go:544-562)

fori, item:=rangeitems {
gofunc(idxint, itmT) {
sem<-struct{}{}
deferfunc() { <-sem }()
// If context is canceled early, some goroutines may still be waiting on sem
}(i, item)
}

Minor Issue: If parent context is canceled, goroutines waiting on the semaphore might not exit immediately.
Recommendation: Add context check before acquiring semaphore.

4. Inefficient double namespace filtering (block/internal/syncing/da_retriever.go:80-133)
The code fetches from both header and data namespaces, then filters by namespace in processBlobs. This could be optimized.


5. Testing & Reliability ⭐⭐⭐

Strengths

  • Good unit test coverage for commitment logic: da/celestia/commitment_test.go has comprehensive determinism and edge case tests
  • DummyDA for testing: Well-designed mock implementation (da/testing.go:13-314)
  • Table-driven tests: Good use of table-driven test pattern (da/celestia/client_test.go:14-87)

Critical Issues

1. Very low test coverage on main client (Codecov report)

  • da/celestia/client.go: 29.53% coverage (210 lines missing)
  • da/testing.go: 59.03% coverage (34 lines missing)
  • da/da.go: 59.21% coverage (31 lines missing)

The main client implementation has minimal test coverage. Most tests just verify error paths with an unreachable server (da/celestia/client_test.go:148-225).

Recommendation: Add integration tests with a mock celestia-node server or expand unit tests to cover:

  • Successful submission and retrieval flows
  • Error recovery and retry logic
  • Namespace filtering
  • ID splitting and commitment verification
  • The critical Get, GetIDs, GetProofs, and Validate methods

2. No tests for error mapping logic (da/celestia/client.go:500-617)
The sophisticated error mapping in handleSubmitError and handleRetrieveError has no tests.

Recommendation: Add unit tests for error classification:

funcTestHandleSubmitError(t*testing.T) {
tests:= []struct{
namestringerrerrorwantStatus da.StatusCode
}{
{"timeout error", context.DeadlineExceeded, da.StatusContextDeadline},
{"mempool error", errors.New("already in mempool"), da.StatusAlreadyInMempool},
// etc.
}
}

3. DummyDA height ticker goroutine not tested (da/testing.go:53-74)
The StartHeightTicker/StopHeightTicker mechanism has no tests verifying proper start/stop behavior or goroutine cleanup.

4. No concurrency tests
No tests verify thread-safety of concurrent operations, despite the code having concurrent marshaling and map access in DummyDA.


6. Documentation & Developer Experience ⭐⭐⭐½

Strengths

  • Clear interface documentation: DA interface methods have good godoc comments (da/da.go:15-39)
  • Well-documented constants: Namespace constants are clearly explained (da/da.go:159-178)
  • Updated da-debug tool: The debugging tool (tools/da-debug/main.go) is well-structured with good help text
  • Error variables documented: All error types have clear names and are exported (da/da.go:118-127)

Areas for Improvement

1. Missing package-level documentation

// da/da.gopackage da // No package comment explaining the DA abstraction

Recommendation: Add package-level documentation:

// Package da provides an abstraction layer for Data Availability layers.// It defines the DA interface and common types for interacting with DA layers// such as Celestia. The package handles blob submission, retrieval, and verification// with proper error handling and status codes.package da

2. Undocumented public types (da/celestia/types.go:19-41)

typeBlobstruct {
NamespaceNamespace`json:"namespace"`Data []byte`json:"data"`ShareVeruint8`json:"share_version"`// What is ShareVer?CommitmentCommitment`json:"commitment"`Signer []byte`json:"signer,omitempty"`// What format?Indexint`json:"index"`// Index of what?
}

Recommendation: Add field-level comments explaining format and purpose.

3. No migration guide
With such a significant refactor (removing core/da, changing interface), a migration guide would help.
Recommendation: Add docs/adr/adr-NNN-celestia-blob-api-migration.md explaining:

  • Why the change was made
  • How to update custom DA implementations
  • Breaking changes and compatibility notes

4. SubmitOptions not fully documented (da/celestia/types.go:32-41)

typeSubmitOptionsstruct {
GasPricefloat64`json:"gas_price,omitempty"`IsGasPriceSetbool`json:"is_gas_price_set,omitempty"`// Why is this separate?// Other fields...
}

The relationship between fields and their effects on submission isn't clear.

5. Missing examples
No example code showing how to:

  • Create and use a Celestia client
  • Handle different status codes
  • Implement a custom DA layer

7. Additional Observations

Positive Changes

Removed complex JSON-RPC proxy: Deleted da/jsonrpc/ (241 lines) in favor of direct blob API calls - simpler and more maintainable
Better error granularity: New status codes like StatusIncorrectAccountSequence provide better error handling
Commitment calculation: Local commitment calculation (da/celestia/commitment.go) matches celestia-node's algorithm - good for offline verification
Multi-account support: Round-robin address selector (block/internal/submitting/da_submitter.go:134-142) is a smart addition for scaling

Breaking Changes (Expected for Refactor)

  • core/da interface completely removed
  • Method signatures changed (e.g., Submit now returns ResultSubmit not BaseResult)
  • Namespace handling changed from core/da.Namespace to da.Namespace

Code Smells

1. Mixed responsibilities in submitter (block/internal/submitting/da_submitter.go:307-343)
The mergeSubmitOptions function does JSON manipulation in a struct that's primarily about DA submission. Consider moving to a separate options package.

2. Global variable for DA visualization (pkg/rpc/server/server.go likely)
The pattern server.GetDAVisualizationServer() suggests a global singleton, which makes testing harder.

3. Large function (block/internal/submitting/da_submitter.go:346-497)
The submitToDA function is 151 lines with complex retry logic. Consider extracting retry logic into a separate function.


Summary & Recommendations

Priority Issues to Address

PriorityIssueFileImpact
🔴 CriticalNamespaceFromString can return nilda/da.go:243Potential nil pointer panics
🔴 CriticalVery low test coverage (29-59%)da/celestia/client.go, da/testing.go, da/da.goReliability risk
🟡 HighDuplicate makeID functionda/da.go:101, da/celestia/client.go:347Maintainability
🟡 HighMissing tests for error mappingda/celestia/client.go:500-617Error handling reliability
🟢 MediumUnused context parameterda/celestia/client.go:305API consistency
🟢 MediumMissing context cancellation in loopsda/celestia/client.go:244-300Cancellation responsiveness
🟢 MediumMissing package documentationda/da.goDeveloper experience

Overall Assessment

This is a solid refactoring that improves the architecture and maintainability of the DA layer. The code quality is generally high, with good error handling patterns and performance considerations. However, the test coverage is concerningly low for such a critical component.

Scores:

  • Code Quality & Architecture: ⭐⭐⭐⭐½ (4.5/5)
  • Error Handling & Resilience: ⭐⭐⭐⭐ (4/5)
  • Security: ⭐⭐⭐⭐⭐ (5/5)
  • Performance: ⭐⭐⭐⭐ (4/5)
  • Testing: ⭐⭐⭐ (3/5) - Needs improvement
  • Documentation: ⭐⭐⭐½ (3.5/5)

Overall: ⭐⭐⭐⭐ (4/5) - Approve with recommendations for test coverage improvements.

Recommended Actions Before Merge

  1. Fix critical nil return issue in NamespaceFromString
  2. Add integration tests for the Celestia client with a mock server
  3. Add unit tests for error mapping logic
  4. Remove duplicatemakeID function
  5. Add package-level documentation

The architectural improvements are excellent, but the low test coverage (48.69% patch coverage, 333 missing lines) is a concern for a core component. I recommend adding comprehensive tests before merging to production.


@github-actions

github-actionsBot commented Nov 26, 2025

Copy link
Copy Markdown
Contributor

The latest Buf updates on your PR. Results from workflow CI / buf-check (pull_request).

BuildFormatLintBreakingUpdated (UTC)
✅ passed⏩ skipped✅ passed✅ passedNov 27, 2025, 8:46 PM

@github-actions

github-actionsBot commented Nov 26, 2025

Copy link
Copy Markdown
Contributor
PR Preview Action v1.6.3
Preview removed because the pull request was closed.
2025-11-28 09:45 UTC

@codecov

codecovBot commented Nov 26, 2025

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 42.93478% with 420 lines in your changes missing coverage. Please review.
✅ Project coverage is 61.50%. Comparing base (f8a2cdb) to head (a6639a6).

Files with missing linesPatch %Lines
da/celestia/client.go22.85%297 Missing ⚠️
da/testing.go59.03%29 Missing and 5 partials ⚠️
da/da.go59.21%30 Missing and 1 partial ⚠️
tools/da-debug/main.go3.44%27 Missing and 1 partial ⚠️
pkg/rpc/server/server.go85.48%6 Missing and 3 partials ⚠️
pkg/rpc/server/da_visualization.go53.33%7 Missing ⚠️
da/celestia/types.go64.70%3 Missing and 3 partials ⚠️
da/celestia/commitment.go78.94%2 Missing and 2 partials ⚠️
block/internal/syncing/da_retriever.go93.10%0 Missing and 2 partials ⚠️
block/internal/submitting/da_submitter.go92.30%1 Missing ⚠️
... and 1 more
Additional details and impacted files
@@ Coverage Diff @@## main #2887 +/- ##
==========================================
- Coverage 64.72% 61.50% -3.22% 
==========================================
Files 81 79 -2 Lines 7347 7530 +183 ==========================================
- Hits 4755 4631 -124 - Misses 2050 2364 +314 + Partials 542 535 -7 
FlagCoverage Δ
combined61.50% <42.93%> (-3.22%)⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

randygrokand others added 10 commits November 26, 2025 10:48
- Fix relative paths in apps/grpc/README.md (../../../ -> ../../)
- Add missing .md extension in docs/guides/full-node.md link
- Wrap localhost URL in backticks in tools/da-debug/README.md
Medium returns 403 for automated requests, causing false positives
in the CI link checker.
@randygrok

Copy link
Copy Markdown
ContributorAuthor

close in favor of #2893

Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant

@randygrok