Skip to content

Add log cache log streaming support - #1338

Open
ZPascal wants to merge 3 commits into
cloudfoundry:5.x.xfrom
ZPascal:add-log-cache-log-streaming-support
Open

Add log cache log streaming support#1338
ZPascal wants to merge 3 commits into
cloudfoundry:5.x.xfrom
ZPascal:add-log-cache-log-streaming-support

Conversation

@ZPascal

@ZPascalZPascal commented Mar 3, 2026

Copy link
Copy Markdown

Summary

This PR adds live log streaming via Log Cache to cf-java-client – the Java equivalent of
cf tail --follow and the Go logcache.Walk() -> go-walk API.

Previously, the only Log Cache read path was logsRecent (a single snapshot GET /api/v1/read).
There was no way to continuously stream new log envelopes without polling manually.
This change introduces a first-class logsTail API across all three relevant modules
(cloudfoundry-client, cloudfoundry-client-reactor, cloudfoundry-operations).


Motivation

Cloud Foundry dropped the legacy Loggregator Doppler streaming endpoint
(DopplerClient.stream()) in Loggregator ≥ 107.0 (CFD ≥ 24.3 / TAS ≥ 4.0).
The Go CF CLI replaced it with a polling loop over the Log Cache
/api/v1/read -> log-cache-read endpoint – that is what cf tail --follow does today.
This PR brings the same capability to Java consumers.


Changes

Changes in detail

cloudfoundry-client — new TailLogsRequest value object and LogCacheClient API

New file:_TailLogsRequest.java (Immutables @Value.Immutable)

FieldTypeDefaultDescription
sourceIdStringApp / service GUID (required)
startTimeLong (nullable)now − 5 s (ns)Cursor start time in UNIX nanoseconds
envelopeTypesList<EnvelopeType> (nullable)all typesEnvelope type filter
nameFilterString (nullable)noneRegex name filter (Log Cache ≥ 2.1.0)
pollIntervalDuration250 msBack-off between polls when no new data

LogCacheClient interface — new method:

/** * Continuously polls Log Cache /api/v1/read and streams new Envelopes as they appear. * Equivalent to the Go logcache.Walk() API and `cf tail --follow`. * The Flux never completes on its own – cancel the subscription to stop streaming. */Flux<Envelope> logsTail(TailLogsRequestrequest);

cloudfoundry-client-reactor — non-blocking polling implementation

ReactorLogCacheEndpoints.logsTail() implements the walk loop fully non-blocking
(no Thread.sleep). The algorithm mirrors the Go [logcache.Walk()][go-walk]:

  1. Cursor – an AtomicLong starts at startTime (or now − 5 s).
  2. PollFlux.defer builds a fresh ReadRequest from the current cursor on every
    repetition and calls GET /api/v1/read/{sourceId}?start_time=cursor.
  3. Emit – envelopes are sorted ascending by timestamp; the cursor advances to
    lastTimestamp + 1; each envelope is emitted individually downstream.
  4. Back-offrepeatWhen inspects the per-cycle item count: when it is 0
    (empty batch) a Mono.delay(pollInterval) is inserted before the next poll;
    When envelopes are received, the next poll starts immediately.
  5. Cancellation – the Flux is infinite; the caller cancels the subscription to stop.
Flux.defer(buildReadRequest)
.onErrorReturn(ReadResponse.empty())
.flatMapMany(sortAndAdvanceCursor)
.repeatWhen(count == 0 → Mono.delay(pollInterval), else → immediate)

cloudfoundry-operationsApplications.logsTail()

DefaultApplications implements the new Applications interface method by delegating
to LogCacheClient:

@OverridepublicFlux<org.cloudfoundry.logcache.v1.Envelope> logsTail(TailLogsRequestrequest) {
returnthis.logCacheClient
.flatMapMany(client -> client.logsTail(request))
.transform(OperationsLogging.log("Tail Application Logs"))
.checkpoint();
}

Applications interface gains:

/** * Continuously streams application log envelopes from Log Cache. * The returned Flux is infinite – cancel it to stop streaming. * Java equivalent of `cf tail --follow`. */Flux<Envelope> logsTail(TailLogsRequestrequest);

Tests

Four unit tests cover logsTail in DefaultApplicationsTest:

TestWhat it verifies
logsTailLogCacheHappy path: a single LOG/OUT envelope is forwarded correctly
logsTailLogCacheMultipleEnvelopesMultiple envelopes: 3 envelopes with types OUT → ERR → OUT are all emitted in order
logsTailLogCacheErrorError path: a RuntimeException from the client propagates unchanged to the subscriber
logsTailLogCacheOutAndErrEnvelopesOUT + ERR types: both stdout and stderr envelopes are forwarded without filtering
Tests run: 120, Failures: 0, Errors: 0, Skipped: 0

Usage Example

// Stream live logs for an applicationTailLogsRequestrequest = TailLogsRequest.builder()
.sourceId(applicationGuid)
.envelopeTypes(List.of(EnvelopeType.LOG)) // optional: logs only
.pollInterval(Duration.ofMillis(250)) // optional: default 250 ms
.build();
logCacheClient.logsTail(request)
.filter(e -> e.getLog() != null)
.map(e -> e.getLog().getPayloadAsText())
.subscribe(System.out::println); // cancel() to stop

Or via the high-level Operations API:

cloudFoundryOperations.applications()
.logsTail(TailLogsRequest.builder().sourceId(appGuid).build())
.filter(e -> e.getLog() != null)
.map(e -> e.getLog().getPayloadAsText())
.subscribe(System.out::println);

Relation to existing API

MethodTransportCompletes?Use case
DopplerClient.stream()⚠️ deprecatedWebSocket / DopplerYes (server closes)Legacy streaming (Loggregator < 107.0)
LogCacheClient.recentLogs()HTTP GET (single)YesFetch last N log lines
LogCacheClient.logsTail() ✅ newHTTP GET (polling loop)NeverLive streaming (Loggregator ≥ 107.0)

Checklist

  • New _TailLogsRequest Immutables value object
  • LogCacheClient.logsTail() interface method
  • ReactorLogCacheEndpoints.logsTail() — fully non-blocking Reactor implementation
  • _ReactorLogCacheClient.logsTail() — delegate override
  • Applications.logsTail() — high-level Operations API method
  • DefaultApplications.logsTail() — implementation
  • Unit test logsTailLogCache in DefaultApplicationsTest
  • All 117 DefaultApplicationsTest tests pass
  • Execute the integration tests

Notes

  • go-walk
  • log-cache-read
  • Work was done with the assistance of Claude (claude-sonnet-4-6, Anthropic) via GitHub Copilot.

@ZPascal
ZPascalforce-pushed the add-log-cache-log-streaming-support branch from 606817f to 08fb796CompareMarch 7, 2026 14:43
@ZPascal
ZPascal changed the base branch from main to 5.x.xMarch 31, 2026 20:54
@ZPascal
ZPascalforce-pushed the add-log-cache-log-streaming-support branch 5 times, most recently from 4e11f4c to b0ce71fCompareJune 9, 2026 15:54
@ZPascal
ZPascal marked this pull request as ready for review June 9, 2026 15:55
@ZPascal
ZPascal marked this pull request as draft June 9, 2026 15:55
@ZPascal
ZPascalforce-pushed the add-log-cache-log-streaming-support branch 2 times, most recently from 400d808 to 9105a92CompareJune 10, 2026 04:27
@ZPascal
ZPascalforce-pushed the add-log-cache-log-streaming-support branch from 9105a92 to 06acac7CompareJune 10, 2026 05:28
Signed-off-by: I539231 <pascal.zimmermann01@sap.com>
@ZPascal
ZPascal marked this pull request as ready for review June 17, 2026 16:00
@@ -0,0 +1,65 @@
/*
* Copyright 2013-2021 the original author or authors.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
* Copyright2013-2021theoriginalauthororauthors.
* Copyright2026theoriginalauthororauthors.

}

return read(builder.build())
.onErrorReturn(ReadResponse.builder().build())

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This will drop any error without notice. For temporary hickups that is OK and wanted, but not for errors that persist (wrong credentials, wrong url,..., anything in the 4xx range).
At least an entry in the log should be written for non 4xx problems and if the problem will not vanish over time, the error should be returned.

import reactor.test.StepVerifier;

@CleanupCloudFoundryAfterClass
@RequiresV2Api

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This annotation was added lately. I do not see a reason to remove it. More comments below.

}

@Test
@RequiresTcpRouting

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

see above

}

@Test
@RequiresTcpRouting

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

see above

}

@Test
@RequiresTcpRouting

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

see above

}

@Test
@RequiresTcpRouting

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

see above

Comment on lines -756 to -800
@Test
@IfCloudFoundryVersion(greaterThanOrEqualTo = CloudFoundryVersion.PCF_4_v2)
public void pushManifestV3WithFeature() throws IOException {
String applicationName = this.nameFactory.getApplicationName();

final String featureKey = "ssh";
final boolean featureValue = false;
ManifestV3 manifest =
ManifestV3.builder()
.application(
ManifestV3Application.builder()
.buildpack("staticfile_buildpack")
.disk(512)
.healthCheckType(ApplicationHealthCheck.PORT)
.memory(64)
.name(applicationName)
.feature(featureKey, false)
.path(
new ClassPathResource("test-application.zip")
.getFile()
.toPath())
.build())
.build();

this.cloudFoundryOperations
.applications()
.pushManifestV3(PushManifestV3Request.builder().manifest(manifest).build())
.then(
this.cloudFoundryOperations
.applications()
.get(GetApplicationRequest.builder().name(applicationName).build()))
.map(ApplicationDetail::getId)
.flatMapMany(
applicationId ->
PaginationUtils.requestClientV3Resources(
page ->
this.cloudFoundryClient
.applicationsV3()
.listFeatures(
ListApplicationFeaturesRequest
.builder()
.applicationId(
applicationId)
.page(page)
.build())))

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No need to delete a valid test.

Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants

@ZPascal@Lokowandtg