[Java] infinite loop during Auth when Python `adbc_driver_flightsql.connect` to Java Flight Server

[xinyiZzz]

Describe the usage question you have. Please include as many useful details as possible.

I implemented an Arrow Flight SQL Server using JAVA and tried to connect to it using Python, using Apache Arrow 13.0.0. I'm having a Infinite loop problem and looking for help. Infinite loop at the code position below:


It seems to because AuthObserver::onNext is not being called correctly, Is this related to the Protobuf version?

Arrow Flight SQL Server using JAVA, complete code is here: apache/doris#24314

<arrow.version>13.0.0</arrow.version>
<grpc.version>1.56.0</grpc.version>
<protobuf.version>3.23.1</protobuf.version>
<protoc.artifact.version>3.24.3</protoc.artifact.version>

FlightSqlService.java

public FlightSqlService(int port) {
 BufferAllocator allocator = new RootAllocator();
 Location location = Location.forGrpcInsecure("0.0.0.0", port);
 FlightSqlServiceImpl producer = new FlightSqlServiceImpl(location);
 flightServer = FlightServer.builder(allocator, location, producer)
 .authHandler(new BasicServerAuthHandler(new FlightServerBasicAuthValidator())).build();
 }

flightServer.start();

FlightSqlServiceImpl.java

public class FlightSqlServiceImpl implements FlightSqlProducer, AutoCloseable {
 public FlightSqlServiceImpl(final Location location) {
 this.location = location;
 sqlInfoBuilder = new SqlInfoBuilder();
 sqlInfoBuilder.withFlightSqlServerName("DorisFE")
 .withFlightSqlServerVersion("1.0")
 .withFlightSqlServerArrowVersion("13.0.0")
 .withFlightSqlServerReadOnly(false)
 .withSqlIdentifierQuoteChar("`")
 .withSqlDdlCatalog(true)
 .withSqlDdlSchema(false)
 .withSqlDdlTable(false)
 .withSqlIdentifierCase(SqlSupportedCaseSensitivity.SQL_CASE_SENSITIVITY_CASE_INSENSITIVE)
 .withSqlQuotedIdentifierCase(SqlSupportedCaseSensitivity.SQL_CASE_SENSITIVITY_CASE_INSENSITIVE);
 }
}

FlightServerBasicAuthValidator.java

public class FlightServerBasicAuthValidator implements BasicServerAuthHandler.BasicAuthValidator {
 private static final String myToken = "DORIS_READ_WRITE_TOKEN";

 @Override
 public byte[] getToken(String username, String password) {
 return myToken.getBytes(Charsets.UTF_8);
 }

 @Override
 public Optional<String> isValid(byte[] bytes) {
 return Optional.of(myToken);
 }
}

Connect using Python ADBC

tp version:
adbc_driver_flightsql-0.6.0
adbc_driver_manager-0.6.0
pyarrow-13.0.0

import adbc_driver_flightsql.dbapi as flight_sql
import adbc_driver_flightsql
import adbc_driver_manager

db = adbc_driver_flightsql.connect(uri="grpc://xxx:10478", db_kwargs={
 adbc_driver_manager.DatabaseOptions.USERNAME.value: "root",
 adbc_driver_manager.DatabaseOptions.PASSWORD.value: "",
 })
conn = adbc_driver_manager.AdbcConnection(db)
stmt = adbc_driver_manager.AdbcStatement(conn)

if not add DatabaseOptions.USERNAME and DatabaseOptions.PASSWORD, can connect it normally

db = adbc_driver_flightsql.connect(uri="grpc://xxx:10478")

Component(s)

FlightRPC

[lidavidm]

Thanks for the report, I'll try to find some time to replicate and debug

[xinyiZzz]

thanks,
It seems that Authenticate(ServerAuthSender *outgoing, ServerAuthReader *incoming) deprecated in 13.0.0. Implement the Authentication() with ServerCallContext version instead.
https://arrow.apache.org/docs/cpp/api/flight.html#_CPPv4N5arrow6flight17ServerAuthHandler12AuthenticateEP16ServerAuthSenderP16ServerAuthReader

Is it more recommended to use auth2? use ServerCallContext?
Authenticate(const ServerCallContext &context, ServerAuthSender *outgoing, ServerAuthReader *incoming)

[xinyiZzz]

Fortunately! I successfully implemented authentication using auth2.

[lidavidm]

Ah - thanks for the update, unfortunately I haven't had time to look at this yet

[xinyiZzz]

I reproduced the same problem on cpp flight server. stuck in incoming->Read(&token)

auth.h
class FlightSqlServerAuthHandler : public arrow::flight::ServerAuthHandler {
 arrow::Status Authenticate(const arrow::flight::ServerCallContext& context, arrow::flight::ServerAuthSender* outgoing,
 arrow::flight::ServerAuthReader* incoming) override;
};


auth.cpp
arrow::Status FlightSqlServerAuthHandler::Authenticate(const arrow::flight::ServerCallContext& context, arrow::flight::ServerAuthSender* outgoing,
 arrow::flight::ServerAuthReader* incoming) {
 std::string token;
 RETURN_NOT_OK(incoming->Read(&token)); // stuck here
}

server.cpp
flight_options.auth_handler = std::make_unique<FlightSqlServerAuthHandler>();
RETURN_DORIS_STATUS_IF_ERROR(Init(flight_options));

[lidavidm]

sorry - I'm going to block off some time tomorrow so I can actually look at this, thanks for your patience

[xinyiZzz]

Thanks for your reply, I will use ServerMiddleware instead of ServerAuthHandler. As you said #37824, ServerAuthHandler has been deprecated in Arrow 13.0.0

If Arrow 13.0.0 no longer supports old authentication, it seems that can be marked in the Documentation and Example~

[lidavidm]

Ok, I looked at the Java side and yes - this is also the 'old' auth implementation (that tries to use a stateful "login" flow). So it's getting stuck waiting for a message that never comes (since the Flight SQL client uses a more modern authentication flow). I'll mark it deprecated (without intent to remove) and explain the issues. Thanks for reporting and thanks for bearing with us.

[xinyiZzz]

Many thanks 🙏