cluster: add control-connection query fallback

[dkropachev]

Summary

Add an opt-in control-connection fallback for application queries when the driver cannot populate normal node pools, which happens in deployments that expose the cluster through a non-broadcast IP address such as a TCP proxy or a node public IP. In that mode the driver can still execute queries over the single control connection, but throughput is poor and connection churn increases the chance of request errors. This option is intentionally disabled by default and should not be used in production.

Also propagate keyspace updates on the fallback path so USE keeps the control connection in sync.

Fixes: #720

Tests

• uv run python - <<\"PY\" ... pytest tests/unit/test_cluster.py -q -k set_keyspace_for_all_pools_reports_all_errors ... PY
• uv run python - <<\"PY\" ... pytest tests/unit/test_response_future.py -q -k control_connection_fallback_updates_connection_keyspace ... PY

[dkropachev]

cqlsh manual test report

I tested cqlsh with the local driver from /extra/scylladb/python-driver-4, installed into the cqlsh venv as editable scylla-driver 3.29.9.

Test setup:

Scylla container: cqlsh-nodepool-test
Docker network: bridge
Reachable CQL endpoint: 127.0.0.1:19042
Advertised node endpoint: 10.255.255.1:19042

This setup makes the initial control connection reachable, but makes discovered node-pool connections unusable.

Results:

Mode	Session start	Node-pool behavior	Query result
Disabled	Fails	no usable pool	NoHostAvailable
Fallback	Succeeds	tries pools, tolerates none usable	query succeeds via control connection
NoNodePoolFallback	Succeeds	skips pool creation	query succeeds via control connection

Default Disabled behavior failed as expected:

Connection error: ('Unable to connect to any servers', ['10.255.255.1'])

With ControlConnectionQueryFallback.Fallback, cqlsh succeeded:

No usable node pools; falling back to control connection for host 127.0.0.1:19042
system.now()
--------------------------------------
11708f30-4a04-11f1-9b89-4372a228dacb
(1 rows)

I also verified directly through the driver that the session starts with no node pools and still executes the query:

pools_at_session_start {}
query_result OrderedDict([('system.now()', UUID(...))])
pools_after_query {}

With ControlConnectionQueryFallback.NoNodePoolFallback, cqlsh also succeeded, and the driver skipped pool creation entirely:

pools_at_session_start {}
initial_connect_futures set()
query_result OrderedDict([('system.now()', UUID(...))])
pools_after_query {}

Conclusion: Fallback now handles the intended case: cqlsh can start even when no node pools are reachable, and queries run through the control connection. NoNodePoolFallback also works and explicitly skips node-pool creation.

[Lorak-mmk]

I have to admit I don't understand the logic and flow around _control_connection_query_attempted. I'll have to just assume it works.

I have 2 more questions regarding SkipPoolCreation:

• WDYT about changing ProfileManager to return IGNORED distance for all hosts in this mode? It would be additional hardening about unexpected code paths doing something weird. One possible problem is that it may affect control connection, in which case we should not do it.
• I think Cluster.on_up and Cluster.on_down should also be guarded in SkipPoolCreation mode. We don't want reconnectors to start etc.

[dkropachev]

I have to admit I don't understand the logic and flow around _control_connection_query_attempted. I'll have to just assume it works.

I have 2 more questions regarding SkipPoolCreation:

• WDYT about changing ProfileManager to return IGNORED distance for all hosts in this mode? It would be additional hardening about unexpected code paths doing something weird. One possible problem is that it may affect control connection, in which case we should not do it.
• I think Cluster.on_up and Cluster.on_down should also be guarded in SkipPoolCreation mode. We don't want reconnectors to start etc.

Great idea, easily can see how it can stop driver spin on pool creation, added.

[sylwiaszunejko]

Looks good now