fix: make signature verification backwards compatible

[tac0turtle]

Overview

This pr creates a custom encoding and decoding function to decode and encode legacy header to allow current networks to sync from genesis even with the new slimmed down header type

[claude]

Claude finished @tac0turtle's task —— View job

---

Comprehensive Code Review

I've performed a thorough review of the backwards compatible signature verification implementation. This is a well-architected solution for handling legacy header formats during network upgrades. Below are my findings across multiple dimensions:

---

1. Code Quality & Architecture ⭐⭐⭐⭐

Strengths:

• Clean separation of concerns: The legacy compatibility layer is well-isolated in LegacyHeaderFields struct with clear documentation (types/header.go:142-189)
• Multi-format fallback mechanism: The signature verification strategy (custom → slim → legacy) is elegant and future-proof (types/signed_header.go:124-185)
• Idiomatic Go: Proper use of error wrapping, nil checking, and byte slice handling throughout

Areas for Improvement:

1.1 Memory Efficiency in Signature Verification

Location: types/signed_header.go:124-145 and 203-224

The tryPayload function is duplicated between ValidateBasic() and ValidateBasicWithData(). This violates DRY principles and increases maintenance burden.

Recommendation: Extract the common payload verification logic:

func (sh*SignedHeader) verifyPayload(payload []byte, tried [][]byte) (bool, [][]byte, error) {
iflen(payload) ==0 {
returnfalse, tried, nil
}
for_, p:=rangetried {
ifbytes.Equal(p, payload) {
returnfalse, tried, nil
}
}
tried=append(tried, payload)
verified, err:=sh.Signer.PubKey.Verify(payload, sh.Signature)
iferr!=nil {
returnfalse, tried, err
}
returnverified, tried, nil
}

1.2 Clone Method Should Be a Pointer Method

Location: types/header.go:235-245

The Clone() method uses a value receiver but mutates fields. This can be confusing.

Recommendation: Change to pointer receiver for clarity:

func (h*Header) Clone() *Header {
ifh==nil {
returnnil
}
clone:=*hclone.LastHeaderHash=cloneBytes(h.LastHeaderHash)
// ... rest of cloningreturn&clone
}

---

2. Error Handling & Resilience ⭐⭐⭐⭐⭐

Strengths:

• Comprehensive error wrapping: All errors include context (types/serialization.go:445, 476)
• DoS protection: Smart size limits for unknown fields (types/serialization.go:431, 435) prevent memory exhaustion attacks
• Graceful degradation: The fallback verification strategy ensures compatibility without hard failures

Critical Security Issue:

2.1 Protowire Parsing Error Handling

Location: types/serialization.go:450-496

The protowire parsing code correctly checks for negative return values indicating errors, but the error messages could be more descriptive for debugging.

Recommendation: Add more context to protowire errors:

ifn<0 {
returnnil, fmt.Errorf("failed to parse protowire tag at offset %d: %w", len(pHeader.ProtoReflect().GetUnknown())-len(unknown), protowire.ParseError(n))
}

---

3. Security 🔒 ⭐⭐⭐⭐

Strengths:

• DoS Protection: Excellent size limits on unknown fields (maxUnknownFieldSize=1MB, maxLegacyHashSize=1KB)
• No secret leakage: Addresses and hashes are handled safely
• Signature verification: Multi-format approach ensures signatures are always validated

Security Concerns:

3.1 Potential Timing Attack in Payload Verification

Location: types/signed_header.go:134

The sequential comparison using bytes.Equal in the duplicate detection could theoretically leak timing information about which payloads were tried.

Severity: Low (requires local access and precise timing measurements)

Recommendation: This is likely acceptable for this use case, but document the decision if timing attacks were considered.

3.2 Legacy Field Size Validation

Location: types/serialization.go:474-478

Good validation of field sizes, but consider if 1KB is appropriate for all hash algorithms. Modern hash functions like SHA-256 use 32 bytes, but some systems might use longer identifiers.

Recommendation: Document why 1KB was chosen and which hash algorithms are supported.

---

4. Performance & Resource Efficiency ⭐⭐⭐⭐

Strengths:

• Minimal allocations: Good use of append([]byte(nil), ...) pattern for defensive copying
• Efficient duplicate detection: Slice-based tracking instead of map allocation (types/signed_header.go:126)
• Lazy legacy field initialization: Only allocates LegacyHeaderFields when needed

Performance Issues:

4.1 Triple Marshal Operation in Verification

Location: types/signed_header.go:161-177

The signature verification performs up to 3 marshaling operations:

1. Custom provider
2. Slim format (MarshalBinary)
3. Legacy format (MarshalBinaryLegacy)

Impact: For headers that require legacy verification, this means two failed marshal attempts before success.

Recommendation:

• Add metrics/logging to track which format is used most often
• Consider caching the marshaled payloads if signature verification is called multiple times
• Document the performance trade-off in the migration guide

4.2 Inefficient Legacy Encoding

Location: types/serialization.go:513-591

The marshalLegacyHeader function manually constructs protobuf wire format using multiple append operations. Each appendBytesField could cause reallocation.

Recommendation: Pre-calculate buffer size:

funcmarshalLegacyHeader(h*Header) ([]byte, error) {
// ... existing validation ...// Pre-calculate size to avoid reallocationssize:=estimateLegacyHeaderSize(h)
payload:=make([]byte, 0, size)
// ... rest of encoding ...
}
funcestimateLegacyHeaderSize(h*Header) int {
// Rough estimate: fixed fields + variable length fieldssize:=100// overhead for tags and varintssize+=len(h.LastHeaderHash)
size+=len(h.DataHash)
// ... etcreturnsize
}

---

5. Testing & Reliability ⭐⭐⭐

Strengths:

• Good test coverage: Tests verify both legacy and slim formats (types/signed_header_test.go:251-338)
• Table-driven tests: Systematic verification of error cases
• Integration testing: Tests verify round-trip encoding/decoding

Critical Coverage Gaps:

5.1 Missing Edge Case Tests

Recommendation: Add tests for:

// Test 1: Malformed protowire datafuncTestDecodeLegacyHeaderFields_MalformedData(t*testing.T) {
// Test with truncated protowire data// Test with negative field numbers// Test with invalid wire types
}
// Test 2: Maximum size edge casesfuncTestDecodeLegacyHeaderFields_MaxSize(t*testing.T) {
// Test with exactly maxUnknownFieldSize bytes// Test with exactly maxLegacyHashSize bytes// Test with maxSize + 1 (should fail)
}
// Test 3: Multiple legacy fields in unknown fieldsfuncTestDecodeLegacyHeaderFields_AllLegacyFields(t*testing.T) {
// Test with all three legacy fields populated// Test with duplicate field numbers// Test with fields in different orders
}
// Test 4: Concurrent signature verificationfuncTestSignedHeader_ConcurrentValidation(t*testing.T) {
// Verify thread-safety of ValidateBasic// Ensure no race conditions in signature verification
}
// Test 5: Legacy encoding produces identical bytesfuncTestMarshalLegacyHeader_Deterministic(t*testing.T) {
// Verify that legacy encoding is deterministic// Encode same header multiple times, verify bytes match
}

5.2 Insufficient Benchmark Coverage

Location: No benchmark tests found

Recommendation: Add benchmarks to track performance:

funcBenchmarkSignedHeader_ValidateBasic_SlimFormat(b*testing.B)
funcBenchmarkSignedHeader_ValidateBasic_LegacyFormat(b*testing.B)
funcBenchmarkMarshalBinaryLegacy(b*testing.B)
funcBenchmarkDecodeLegacyHeaderFields(b*testing.B)

5.3 Test Coverage Numbers

The codecov report shows:

• types/signed_header.go: 22.72% coverage (45 missing lines, 6 partial) ⚠️
• types/serialization.go: 62.50% coverage (25 missing lines, 11 partial) ⚠️

Critical: The signature verification fallback logic (lines 124-185 in signed_header.go) appears to have very low coverage. This is the core security mechanism and MUST be thoroughly tested.

---

6. Documentation & Developer Experience ⭐⭐⭐⭐⭐

Strengths:

• Excellent migration guide: The LegacyHeaderFields documentation (types/header.go:142-189) is comprehensive and user-friendly
• Clear format detection strategy: Well-documented fallback mechanism
• Inline comments: Good explanations of non-obvious logic (e.g., protowire parsing)

Minor Improvements:

6.1 Add Examples to Migration Guide

Location: types/header.go:142-189

Recommendation: Add code examples:

// ## Example: Creating a Legacy Header for Testing//// header := &Header{// BaseHeader: BaseHeader{Height: 1, Time: now, ChainID: "test"},// // ... other fields ...// Legacy: &LegacyHeaderFields{// LastCommitHash: make(Hash, 32),// ConsensusHash: make(Hash, 32),// LastResultsHash: make(Hash, 32),// },// }// header.ApplyLegacyDefaults() // Ensures ConsensusHash is initialized

6.2 Document Performance Characteristics

Location: types/signed_header.go:109

Recommendation: Add complexity analysis:

// ValidateBasic performs signature verification using a multi-format fallback:// 1. Custom provider (if set)// 2. Slim format (new format)// 3. Legacy format (old format with deprecated fields)//// Performance: O(3) marshal operations in worst case (legacy headers).// Memory: O(1) additional allocation for duplicate tracking slice.// Network: For historical sync, ~99% of headers should verify on first or second attempt.

---

7. Additional Findings

7.1 Inconsistent Error Variables

Location: types/signed_header.go:93-107

Some errors are package-level variables (good), but the list is incomplete. ErrSignatureVerificationFailed is at line 100 but could benefit from more specific error types.

Recommendation: Consider adding:

var (
ErrCustomProviderFailed=errors.New("custom signature provider failed")
ErrSlimFormatVerificationFailed=errors.New("slim format signature verification failed")
ErrLegacyFormatVerificationFailed=errors.New("legacy format signature verification failed")
)

This would help with debugging by indicating which format failed.

7.2 Missing Validation in ApplyLegacyDefaults

Location: types/header.go:226-231

ApplyLegacyDefaults() silently initializes fields without validating if the header is in a consistent state.

Recommendation: Add validation or document assumptions:

// ApplyLegacyDefaults ensures the Header has a Legacy block initialised with// the expected defaults. This should only be called on headers that require// legacy encoding (e.g., when MarshalBinaryLegacy is used).//// Note: This mutates the header in place. For immutability, use Clone() first.

7.3 Potential Nil Pointer Dereference

Location: types/serialization.go:559, 569

clone.Legacy.ConsensusHash and similar accesses assume Legacy is non-nil after ApplyLegacyDefaults().

Current code is safe because ApplyLegacyDefaults() always initializes Legacy, but this is a subtle dependency.

Recommendation: Add defensive check or comment:

clone.ApplyLegacyDefaults()
// Legacy is guaranteed non-nil after ApplyLegacyDefaultsiflen(clone.Legacy.ConsensusHash) >0 {
payload=appendBytesField(payload, legacyConsensusHashField, clone.Legacy.ConsensusHash)
}

---

Summary & Priority Recommendations

Must Fix (High Priority):

1. Increase test coverage for signature verification fallback logic (types/signed_header.go:124-185) - currently at 22.72% ⚠️
2. Add edge case tests for malformed protowire data and size limits
3. Document performance characteristics of multi-format verification

Should Fix (Medium Priority):

4. Extract duplicate tryPayload logic to reduce code duplication
5. Add benchmarks to track performance regression
6. Pre-calculate buffer size in marshalLegacyHeader to avoid reallocations

Nice to Have (Low Priority):

7. Add more specific error types for debugging
8. Add code examples to migration guide
9. Change Clone() to pointer receiver for consistency

---

Overall Assessment

This is a solid, production-ready implementation of backwards compatible signature verification. The architecture is well thought out, with good separation of concerns and comprehensive documentation. The main concern is the low test coverage (especially for the signature verification fallback logic), which should be addressed before merging to ensure reliability during network upgrades.

Recommendation: Address the test coverage issues, then this PR is ready to merge.

---

[github-actions]

The latest Buf updates on your PR. Results from workflow CI and Release / buf-check (pull_request).

Build	Format	Lint	Breaking	Updated (UTC)
✅ passed	⏩ skipped	✅ passed	✅ passed	Oct 27, 2025, 9:15 AM

[codecov]

Codecov Report

❌ Patch coverage is 54.59184% with 89 lines in your changes missing coverage. Please review.
✅ Project coverage is 62.71%. Comparing base (67877e1) to head (db77e79).
⚠️ Report is 1 commits behind head on main.

Files with missing lines	Patch %	Lines
types/signed_header.go	22.72%	45 Missing and 6 partials ⚠️
types/serialization.go	62.50%	25 Missing and 11 partials ⚠️
types/header.go	94.11%	1 Missing and 1 partial ⚠️

Additional details and impacted files

@@ Coverage Diff @@## main #2782 +/- ##
==========================================
- Coverage 62.86% 62.71% -0.15% 
==========================================
Files 82 82 Lines 6912 7081 +169 ==========================================
+ Hits 4345 4441 +96 - Misses 2063 2118 +55 - Partials 504 522 +18 

Flag	Coverage Δ
combined	62.71% <54.59%> (-0.15%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[julienrbrt]

utACK