Skip to content

FEAT: Comprehensive Logging Framework with Python-C++ Bridge & Instrumentation - #312

Merged
Gaurav Sharma (bewithgaurav) merged 96 commits into
mainfrom
bewithgaurav/logging_framework
Nov 13, 2025
Merged

FEAT: Comprehensive Logging Framework with Python-C++ Bridge & Instrumentation#312
Gaurav Sharma (bewithgaurav) merged 96 commits into
mainfrom
bewithgaurav/logging_framework

Conversation

@bewithgaurav

@bewithgauravGaurav Sharma (bewithgaurav) commented Oct 31, 2025

Copy link
Copy Markdown
Collaborator

Work Item / Issue Reference

AB#39180
AB#40126


Summary

Logging Framework Implementation

Overview

Implements a comprehensive logging system for mssql-python with both Python and C++ components, replacing the previous fragmented approach with a unified, performance-optimized solution.

Key Changes

🎯 Core Logging System

  • Unified Python Logger (mssql_python/logging.py): Single DEBUG-level logger with file/stdout/both output modes
  • C++ Logger Bridge (mssql_python/pybind/logger_bridge.cpp): High-performance bridge from C++ to Python logging with zero overhead when disabled
  • Simplified API: setup_logging(output='file'|'stdout'|'both') - all-or-nothing DEBUG logging for troubleshooting

🔧 Technical Improvements

  • Atomic level checks: Lock-free reads in C++ for minimal performance impact
  • Cached Python logger: Import once, reuse everywhere
  • Lazy message formatting: Only formats strings when logging is enabled
  • GIL-aware: Proper Python GIL management in C++ bridge
  • Thread-safe: Concurrent access supported via mutex protection

📝 Logging Coverage

  • Replaced all warnings.warn() calls with proper logging
  • Added strategic LOG statements at architectural boundaries (connections, transactions, driver loading)
  • ~150 diagnostic LOG statements across C++ codebase for production debugging

🧪 Testing & Quality

  • Updated all tests to use new logging system
  • Fixed test bug: rownumber assertion (was expecting None, should be -1)
  • DevSkim security: Added ignore directives for required vsnprintf usage
  • Type safety: Replaced snprintf with std::ostringstream where appropriate

📚 Documentation

  • Comprehensive learnings documentation in learnings/ directory
  • Migration guide for coverage logging (deferred to future PR due to performance)
  • Philosophy: Logging is for troubleshooting, not production monitoring

Performance Impact

  • When disabled (default): Zero overhead via early returns
  • When enabled: Massive overhead for comprehensive DEBUG logging (Not recommended to use in prod)
  • Designed for temporary enablement during issue investigation

Breaking Changes

  • ⚠️ Removed old logging_config.py and get_logger() API
  • ✅ New API: setup_logging(output='file') for DEBUG logging
  • ✅ All functionality preserved, just different interface

Testing

  • ✅ All existing tests pass with new logging system
  • ✅ Coverage maintained at 77%+ (LOG statements excluded, see comment below)
  • ✅ Validated on Windows, macOS, Linux, RHEL, Alpine (x86_64 + ARM64)

Related Work

  • Addresses feedback from security review about logging practices
  • Foundation for future observability improvements
  • Documented roadmap for coverage improvements in separate PR

@github-actionsgithub-actionsBot added the pr-size: large Substantial code update label Oct 31, 2025
Comment threadmssql_python/pybind/logger_bridge.cpp Fixed
Comment threadmssql_python/pybind/logger_bridge.cpp Fixed
Comment threadmssql_python/pybind/logger_bridge.cpp Fixed
Comment threadmssql_python/pybind/logger_bridge.cpp Fixed
Comment threadtests/test_007_logging.py Fixed
@github-actions

github-actionsBot commented Oct 31, 2025

Copy link
Copy Markdown

📊 Code Coverage Report

🔥 Diff Coverage

63%


🎯 Overall Coverage

77%


📈 Total Lines Covered:5045 out of 6513
📁 Project:mssql-python


Diff Coverage

Diff: main...HEAD, staged and unstaged changes

  • mssql_python/init.py (100%)
  • mssql_python/auth.py (83.3%): Missing lines 170
  • mssql_python/connection.py (100%)
  • mssql_python/connection_string_parser.py (100%)
  • mssql_python/cursor.py (100%)
  • mssql_python/exceptions.py (100%)
  • mssql_python/helpers.py (77.8%): Missing lines 43,46
  • mssql_python/logging.py (85.7%): Missing lines 42,44,132-133,141-142,158-159,230,254-255,261-262,290,302,326,328-332,383-384,474-475,479-480,493,508,510,530-531,535,538-539
  • mssql_python/pooling.py (100%)
  • mssql_python/pybind/connection/connection.cpp (57.1%): Missing lines 218,231,247,253,268,270,274,278,326
  • mssql_python/pybind/connection/connection_pool.cpp (0.0%): Missing lines 75,106
  • mssql_python/pybind/ddbc_bindings.cpp (43.1%): Missing lines 297,399,610-611,659,664,671,678,684,764,787-788,932-933,1315,1323,1354,1418-1420,1422,1447,1460,1545,1558,1598,1663-1664,1680-1681,1699-1700,1709-1710,1744-1745,1771,1784,1806-1807,1809,1842-1843,1868-1869,1894-1895,1897,1908,1916-1917,1921,1928,1940,1985,2000,2006,2021,2027,2046,2101,2112,2116-2117,2129,2155-2156,2166,2187,2206,2211,2240-2242,2258-2260,2264-2266,2273,2276,2281-2285,2288-2289,2293-2298,2301-2304,2309-2310,2312-2314,2318-2319,2321-2323,2325,2328,2330-2331,2333-2336,2338,2348,2362,2370,2454,2456,2491,2585,2606,2638,2647,2650,2654,2701,2704,2708,2730,2741,2784,2789,2802,2813,2850,2874,2923,2956,2960,2972,2983,3012,3022,3033,3195,3204,3374,3485,3583,3615,3654,3664,3697,3774,3830,3839,3841,3847,3857,3864,4045-4046,4054
  • mssql_python/pybind/logger_bridge.cpp (59.3%): Missing lines 30-31,59-64,73-75,77-79,96-97,106-107,111-112,115-117,119,125-126,135-138,141-142,148-149,153-154,176-180,215-216,218-219,222
  • mssql_python/pybind/logger_bridge.hpp (100%)
  • mssql_python/row.py (100%)

Summary

  • Total: 712 lines
  • Missing: 263 lines
  • Coverage: 63%

mssql_python/auth.py

Lines 166-174

166try:
167token=AADAuth.get_token(auth_type)
168logger.info('get_auth_token: Token acquired successfully - auth_type=%s', auth_type)
169returntoken
! 170except (ValueError, RuntimeError) ase:
171logger.warning('get_auth_token: Token acquisition failed - auth_type=%s, error=%s', auth_type, str(e))
172returnNone173

mssql_python/helpers.py

Lines 39-50

39connection_attributes=connection_str.split(";")
40final_connection_attributes= []
4142# Iterate through the attributes and exclude any existing driver attribute
! 43driver_found=False44forattributeinconnection_attributes:
45ifattribute.lower().split("=")[0] =="driver":
! 46driver_found=True47logger.debug('add_driver_to_connection_str: Existing driver attribute found, removing')
48continue49final_connection_attributes.append(attribute)

mssql_python/logging.py

Lines 38-48

38"""Add thread_id (OS native) attribute to log record."""39# Use OS native thread ID for debugging compatibility40try:
41thread_id=threading.get_native_id()
! 42exceptAttributeError:
43# Fallback for Python < 3.8
! 44thread_id=threading.current_thread().ident45record.thread_id=thread_id46returnTrue47

Lines 128-137

128# Flush and close each handler while holding its lock129forhandlerinold_handlers:
130try:
131handler.flush() # Flush BEFORE close
! 132except:
! 133pass# Ignore flush errors134handler.close()
135self._logger.removeHandler(handler)
136finally:
137# Release locks on old handlers

Lines 137-146

137# Release locks on old handlers138forhandlerinold_handlers:
139try:
140handler.release()
! 141except:
! 142pass# Handler might already be closed143144self._file_handler=None145self._stdout_handler=None146

Lines 154-163

154end_bracket=msg.index(']')
155source=msg[1:end_bracket]
156message=msg[end_bracket+2:].strip() # Skip '] '157else:
! 158source='Unknown'
! 159message=msg160161# Format timestamp with milliseconds using period separator162timestamp=self.formatTime(record, '%Y-%m-%d %H:%M:%S')
163timestamp_with_ms=f"{timestamp}.{int(record.msecs):03d}"

Lines 226-234

226""" 227 Reconfigure handlers when output mode changes. 228 Closes existing handlers and creates new ones based on current output mode. 229 """
! 230self._setup_handlers()
231232def_cleanup_handlers(self):
233 """
234Cleanupallhandlersonprocessexit.

Lines 250-259

250forhandlerinhandlers:
251try:
252handler.flush()
253handler.close()
! 254except:
! 255pass# Ignore errors during cleanup256self._logger.removeHandler(handler)
257finally:
258forhandlerinhandlers:
259try:

Lines 257-266

257finally:
258forhandlerinhandlers:
259try:
260handler.release()
! 261except:
! 262pass263264def_validate_log_file_extension(self, file_path: str) ->None:
265 """
266Validatethatthelogfilehasanallowedextension.

Lines 286-294

286WriteCSVheaderandmetadatatothelogfile.
287Calledoncewhenlogfileiscreated.
288 """
289ifnotself._log_fileornotself._file_handler:
! 290return291292try:
293# Get script name from sys.argv or __main__294script_name=os.path.basename(sys.argv[0]) ifsys.argvelse'<interactive>'

Lines 298-306

298299# Get driver version (try to import from package)300try:
301frommssql_pythonimport__version__
! 302driver_version=__version__303except:
304driver_version='unknown'305306# Get current time

Lines 322-336

322withopen(self._log_file, 'a') asf:
323f.write(header_line)
324f.write(csv_header)
325 ! 326exceptExceptionase:
327# Notify on stderr so user knows why header is missing
! 328try:
! 329sys.stderr.write(f"[MSSQL-Python] Warning: Failed to write log header to {self._log_file}: {type(e).__name__}\n")
! 330sys.stderr.flush()
! 331except:
! 332pass# Even stderr notification failed333# Don't crash - logging continues without header334335def_log(self, level: int, msg: str, add_prefix: bool=True, *args, **kwargs):
336 """

Lines 379-388

379importsys380level_name=logging.getLevelName(level)
381sys.stderr.write(f"[MSSQL-Python Logging Failed - {level_name}] {msgif'msg'inlocals() else'Unable to format message'}\n")
382sys.stderr.flush()
! 383except:
! 384pass# Even stderr failed - give up silently385386# Convenience methods for logging387388defdebug(self, msg: str, *args, **kwargs):

Lines 470-484

470# Handler management471472defaddHandler(self, handler: logging.Handler):
473"""Add a handler to the logger (thread-safe)"""
! 474withself._handler_lock:
! 475self._logger.addHandler(handler)
476477defremoveHandler(self, handler: logging.Handler):
478"""Remove a handler from the logger (thread-safe)"""
! 479withself._handler_lock:
! 480self._logger.removeHandler(handler)
481482 @property483defhandlers(self) ->list:
484"""Get list of handlers attached to the logger (thread-safe)"""

Lines 489-497

489""" 490 Reset/recreate handlers. 491 Useful when log file has been deleted or needs to be recreated. 492 """
! 493self._setup_handlers()
494495def_notify_cpp_level_change(self, level: int):
496 """
497NotifyC++bridgethatloglevelhaschanged.

Lines 504-514

504# Import here to avoid circular dependency505from . importddbc_bindings506ifhasattr(ddbc_bindings, 'update_log_level'):
507ddbc_bindings.update_log_level(level)
! 508except (ImportError, AttributeError):
509# C++ bindings not available or not yet initialized
! 510pass511512# Properties513514 @property

Lines 526-543

526527Raises:
528ValueError: IfmodeisnotavalidOutputModevalue529"""! 530 if mode not in (FILE, STDOUT, BOTH):! 531 raise ValueError( 532 f"Invalid output mode: {mode}. " 533 f"Must be one of: {FILE}, {STDOUT}, {BOTH}" 534 )! 535 self._output_mode = mode 536  537 # Only reconfigure if handlers were already initialized! 538 if self._handlers_initialized:! 539 self._reconfigure_handlers() 540  541 @property 542 def log_file(self) -> Optional[str]: 543 """Getthecurrentlogfilepath (Noneiffileoutputisdisabled)"""

mssql_python/pybind/connection/connection.cpp

Lines 214-222

214215// Convert to wide string216 std::wstring wstr = Utf8ToWString(utf8_str);
217if (wstr.empty() && !utf8_str.empty()) {
! 218LOG("Failed to convert string value to wide string for attribute=%d", attribute);
219returnSQL_ERROR;
220 }
221this->wstrStringBuffer.clear();
222this->wstrStringBuffer = std::move(wstr);

Lines 227-235

227 #ifdefined(__APPLE__) || defined(__linux__)
228 // For macOS/Linux, convert wstring to SQLWCHAR buffer
229 std::vector<SQLWCHAR> sqlwcharBuffer = WStringToSQLWCHAR(this->wstrStringBuffer);
230if (sqlwcharBuffer.empty() && !this->wstrStringBuffer.empty()) {
! 231LOG("Failed to convert wide string to SQLWCHAR buffer for attribute=%d", attribute);
232returnSQL_ERROR;
233 }
234235 ptr = sqlwcharBuffer.data();

Lines 243-251

243244SQLRETURN ret = SQLSetConnectAttr_ptr(_dbcHandle->get(),
245 attribute, ptr, length);
246if (!SQL_SUCCEEDED(ret)) {
! 247LOG("Failed to set string attribute=%d, ret=%d", attribute, ret);
248 } else {
249LOG("Set string attribute=%d successfully", attribute);
250 }
251return ret;

Lines 249-257

249LOG("Set string attribute=%d successfully", attribute);
250 }
251return ret;
252 } catch (const std::exception& e) {
! 253LOG("Exception during string attribute=%d setting: %s", attribute, e.what());
254returnSQL_ERROR;
255 }
256 } elseif (py::isinstance<py::bytes>(value) ||
257 py::isinstance<py::bytearray>(value)) {

Lines 264-282

264265SQLRETURN ret = SQLSetConnectAttr_ptr(_dbcHandle->get(),
266 attribute, ptr, length);
267if (!SQL_SUCCEEDED(ret)) {
! 268LOG("Failed to set binary attribute=%d, ret=%d", attribute, ret);
269 } else {
! 270LOG("Set binary attribute=%d successfully (length=%d)", attribute, length);
271 }
272return ret;
273 } catch (const std::exception& e) {
! 274LOG("Exception during binary attribute=%d setting: %s", attribute, e.what());
275returnSQL_ERROR;
276 }
277 } else {
! 278LOG("Unsupported attribute value type for attribute=%d", attribute);
279returnSQL_ERROR;
280 }
281 }

Lines 322-330

322SQL_ATTR_RESET_CONNECTION,
323 (SQLPOINTER)SQL_RESET_CONNECTION_YES,
324SQL_IS_INTEGER);
325if (!SQL_SUCCEEDED(ret)) {
! 326LOG("Failed to reset connection (ret=%d). Marking as dead.", ret);
327disconnect();
328returnfalse;
329 }
330updateLastUsed();

mssql_python/pybind/connection/connection_pool.cpp

Lines 71-79

71for (auto& conn : to_disconnect) {
72try {
73 conn->disconnect();
74 } catch (const std::exception& ex) {
! 75LOG("Disconnect bad/expired connections failed: %s", ex.what());
76 }
77 }
78return valid_conn;
79 }

Lines 102-110

102for (auto& conn : to_close) {
103try {
104 conn->disconnect();
105 } catch (const std::exception& ex) {
! 106LOG("ConnectionPool::close: disconnect failed: %s", ex.what());
107 }
108 }
109 }

mssql_python/pybind/ddbc_bindings.cpp

Lines 293-301

293 !py::isinstance<py::bytes>(param)) {
294ThrowStdException(MakeParamMismatchErrorStr(paramInfo.paramCType, paramIndex));
295 }
296if (paramInfo.isDAE) {
! 297LOG("BindParameters: param[%d] SQL_C_CHAR - Using DAE (Data-At-Execution) for large string streaming", paramIndex);
298 dataPtr = const_cast<void*>(reinterpret_cast<constvoid*>(&paramInfos[paramIndex]));
299 strLenOrIndPtr = AllocateParamBuffer<SQLLEN>(paramBuffers);
300 *strLenOrIndPtr = SQL_LEN_DATA_AT_EXEC(0);
301 bufferLength = 0;

Lines 395-403

395 &describedDigits,
396 &nullable
397 );
398if (!SQL_SUCCEEDED(rc)) {
! 399LOG("BindParameters: SQLDescribeParam failed for param[%d] (NULL parameter) - SQLRETURN=%d", paramIndex, rc);
400return rc;
401 }
402 sqlType = describedType;
403 columnSize = describedSize;

Lines 606-615

606 }
607 py::bytes uuid_bytes = param.cast<py::bytes>();
608constunsignedchar* uuid_data = reinterpret_cast<constunsignedchar*>(PyBytes_AS_STRING(uuid_bytes.ptr()));
609if (PyBytes_GET_SIZE(uuid_bytes.ptr()) != 16) {
! 610LOG("BindParameters: param[%d] SQL_C_GUID - Invalid UUID length: expected 16 bytes, got %ld bytes", ! 611 paramIndex, PyBytes_GET_SIZE(uuid_bytes.ptr()));
612ThrowStdException("UUID binary data must be exactly 16 bytes long.");
613 }
614SQLGUID* guid_data_ptr = AllocateParamBuffer<SQLGUID>(paramBuffers);
615 guid_data_ptr->Data1 =

Lines 655-668

655if (paramInfo.paramCType == SQL_C_NUMERIC) {
656SQLHDESC hDesc = nullptr;
657 rc = SQLGetStmtAttr_ptr(hStmt, SQL_ATTR_APP_PARAM_DESC, &hDesc, 0, NULL);
658if(!SQL_SUCCEEDED(rc)) {
! 659LOG("BindParameters: SQLGetStmtAttr(SQL_ATTR_APP_PARAM_DESC) failed for param[%d] - SQLRETURN=%d", paramIndex, rc);
660return rc;
661 }
662 rc = SQLSetDescField_ptr(hDesc, 1, SQL_DESC_TYPE, (SQLPOINTER) SQL_C_NUMERIC, 0);
663if(!SQL_SUCCEEDED(rc)) {
! 664LOG("BindParameters: SQLSetDescField(SQL_DESC_TYPE) failed for param[%d] - SQLRETURN=%d", paramIndex, rc);
665return rc;
666 }
667SQL_NUMERIC_STRUCT* numericPtr = reinterpret_cast<SQL_NUMERIC_STRUCT*>(dataPtr);
668 rc = SQLSetDescField_ptr(hDesc, 1, SQL_DESC_PRECISION,

Lines 667-675

667SQL_NUMERIC_STRUCT* numericPtr = reinterpret_cast<SQL_NUMERIC_STRUCT*>(dataPtr);
668 rc = SQLSetDescField_ptr(hDesc, 1, SQL_DESC_PRECISION,
669 (SQLPOINTER) numericPtr->precision, 0);
670if(!SQL_SUCCEEDED(rc)) {
! 671LOG("BindParameters: SQLSetDescField(SQL_DESC_PRECISION) failed for param[%d] - SQLRETURN=%d", paramIndex, rc);
672return rc;
673 }
674675 rc = SQLSetDescField_ptr(hDesc, 1, SQL_DESC_SCALE,

Lines 674-682

674675 rc = SQLSetDescField_ptr(hDesc, 1, SQL_DESC_SCALE,
676 (SQLPOINTER) numericPtr->scale, 0);
677if(!SQL_SUCCEEDED(rc)) {
! 678LOG("BindParameters: SQLSetDescField(SQL_DESC_SCALE) failed for param[%d] - SQLRETURN=%d", paramIndex, rc);
679return rc;
680 }
681682 rc = SQLSetDescField_ptr(hDesc, 1, SQL_DESC_DATA_PTR, (SQLPOINTER) numericPtr, 0);

Lines 680-688

680 }
681682 rc = SQLSetDescField_ptr(hDesc, 1, SQL_DESC_DATA_PTR, (SQLPOINTER) numericPtr, 0);
683if(!SQL_SUCCEEDED(rc)) {
! 684LOG("BindParameters: SQLSetDescField(SQL_DESC_DATA_PTR) failed for param[%d] - SQLRETURN=%d", paramIndex, rc);
685return rc;
686 }
687 }
688 }

Lines 760-768

760if (pos != std::string::npos) {
761 std::string dir = module_file.substr(0, pos);
762return dir;
763 }
! 764LOG("GetModuleDirectory: Could not extract directory from module path - path='%s'", module_file.c_str());
765return module_file;
766 #endif
767 }

Lines 783-792

783 #else784// macOS/Unix: Use dlopen785void* handle = dlopen(driverPath.c_str(), RTLD_LAZY);
786if (!handle) {
! 787LOG("LoadDriverLibrary: dlopen failed for path='%s' - %s", ! 788 driverPath.c_str(), dlerror() ? dlerror() : "unknown error");
789 }
790return handle;
791 #endif
792 }

Lines 928-937

928 }
929930 DriverHandle handle = LoadDriverLibrary(driverPath.string());
931if (!handle) {
! 932LOG("LoadDriverOrThrowException: Failed to load ODBC driver - path='%s', error='%s'", ! 933 driverPath.string().c_str(), GetLastErrorMessage().c_str());
934ThrowStdException("Failed to load the driver. Please read the documentation (https://github.com/microsoft/mssql-python#installation) to install the required dependencies.");
935 }
936LOG("LoadDriverOrThrowException: ODBC driver library loaded successfully from '%s'", driverPath.string().c_str());

Lines 1311-1319

1311 ErrorInfo SQLCheckError_Wrap(SQLSMALLINT handleType, SqlHandlePtr handle, SQLRETURN retcode) {
1312LOG("SQLCheckError: Checking ODBC errors - handleType=%d, retcode=%d", handleType, retcode);
1313 ErrorInfo errorInfo;
1314if (retcode == SQL_INVALID_HANDLE) {
! 1315LOG("SQLCheckError: SQL_INVALID_HANDLE detected - handle is invalid");
1316 errorInfo.ddbcErrorMsg = std::wstring( L"Invalid handle!");
1317return errorInfo;
1318 }
1319assert(handle != 0);

Lines 1319-1327

1319assert(handle != 0);
1320SQLHANDLE rawHandle = handle->get();
1321if (!SQL_SUCCEEDED(retcode)) {
1322if (!SQLGetDiagRec_ptr) {
! 1323LOG("SQLCheckError: SQLGetDiagRec function pointer not initialized, loading driver");
1324DriverLoader::getInstance().loadDriver(); // Load the driver1325 }
13261327SQLWCHAR sqlState[6], message[SQL_MAX_MESSAGE_LENGTH];

Lines 1350-1358

1350 py::list SQLGetAllDiagRecords(SqlHandlePtr handle) {
1351LOG("SQLGetAllDiagRecords: Retrieving all diagnostic records for handle %p, handleType=%d", 1352 (void*)handle->get(), handle->type());
1353if (!SQLGetDiagRec_ptr) {
! 1354LOG("SQLGetAllDiagRecords: SQLGetDiagRec function pointer not initialized, loading driver");
1355DriverLoader::getInstance().loadDriver();
1356 }
13571358 py::list records;

Lines 1414-1426

1414 }
14151416// Wrap SQLExecDirect1417SQLRETURNSQLExecDirect_wrap(SqlHandlePtr StatementHandle, const std::wstring& Query) {
! 1418 std::string queryUtf8 = WideToUTF8(Query);
! 1419LOG("SQLExecDirect: Executing query directly - statement_handle=%p, query_length=%zu chars", ! 1420 (void*)StatementHandle->get(), Query.length());
1421if (!SQLExecDirect_ptr) {
! 1422LOG("SQLExecDirect: Function pointer not initialized, loading driver");
1423DriverLoader::getInstance().loadDriver(); // Load the driver1424 }
14251426// Configure forward-only cursor

Lines 1443-1451

1443 queryPtr = const_cast<SQLWCHAR*>(Query.c_str());
1444 #endif
1445SQLRETURN ret = SQLExecDirect_ptr(StatementHandle->get(), queryPtr, SQL_NTS);
1446if (!SQL_SUCCEEDED(ret)) {
! 1447LOG("SQLExecDirect: Query execution failed - SQLRETURN=%d", ret);
1448 }
1449return ret;
1450 }

Lines 1456-1464

1456const std::wstring& table,
1457const std::wstring& tableType) {
14581459if (!SQLTables_ptr) {
! 1460LOG("SQLTables: Function pointer not initialized, loading driver");
1461DriverLoader::getInstance().loadDriver();
1462 }
14631464SQLWCHAR* catalogPtr = nullptr;

Lines 1541-1549

1541 py::list& isStmtPrepared, constbool usePrepare = true) {
1542LOG("SQLExecute: Executing %s query - statement_handle=%p, param_count=%zu, query_length=%zu chars", 1543 (params.size() > 0 ? "parameterized" : "direct"), (void*)statementHandle->get(), params.size(), query.length());
1544if (!SQLPrepare_ptr) {
! 1545LOG("SQLExecute: Function pointer not initialized, loading driver");
1546DriverLoader::getInstance().loadDriver(); // Load the driver1547 }
1548assert(SQLPrepare_ptr && SQLBindParameter_ptr && SQLExecute_ptr && SQLExecDirect_ptr);

Lines 1554-1562

15541555RETCODE rc;
1556SQLHANDLE hStmt = statementHandle->get();
1557if (!statementHandle || !statementHandle->get()) {
! 1558LOG("SQLExecute: Statement handle is null or invalid");
1559 }
15601561// Configure forward-only cursor1562if (SQLSetStmtAttr_ptr && hStmt) {

Lines 1594-1602

1594assert(isStmtPrepared.size() == 1);
1595if (usePrepare) {
1596 rc = SQLPrepare_ptr(hStmt, queryPtr, SQL_NTS);
1597if (!SQL_SUCCEEDED(rc)) {
! 1598LOG("SQLExecute: SQLPrepare failed - SQLRETURN=%d, statement_handle=%p", rc, (void*)hStmt);
1599return rc;
1600 }
1601 isStmtPrepared[0] = py::cast(true);
1602 } else {

Lines 1659-1668

1659ThrowStdException("Chunk size exceeds maximum allowed by SQLLEN");
1660 }
1661 rc = SQLPutData_ptr(hStmt, (SQLPOINTER)(dataPtr + offset), static_cast<SQLLEN>(lenBytes));
1662if (!SQL_SUCCEEDED(rc)) {
! 1663LOG("SQLExecute: SQLPutData failed for SQL_C_WCHAR chunk - offset=%zu", ! 1664 offset, totalChars, lenBytes, rc);
1665return rc;
1666 }
1667 offset += len;
1668 }

Lines 1676-1685

1676size_t len = std::min(chunkBytes, totalBytes - offset);
16771678 rc = SQLPutData_ptr(hStmt, (SQLPOINTER)(dataPtr + offset), static_cast<SQLLEN>(len));
1679if (!SQL_SUCCEEDED(rc)) {
! 1680LOG("SQLExecute: SQLPutData failed for SQL_C_CHAR chunk - offset=%zu", ! 1681 offset, totalBytes, len, rc);
1682return rc;
1683 }
1684 offset += len;
1685 }

Lines 1695-1704

1695for (size_t offset = 0; offset < totalBytes; offset += chunkSize) {
1696size_t len = std::min(chunkSize, totalBytes - offset);
1697 rc = SQLPutData_ptr(hStmt, (SQLPOINTER)(dataPtr + offset), static_cast<SQLLEN>(len));
1698if (!SQL_SUCCEEDED(rc)) {
! 1699LOG("SQLExecute: SQLPutData failed for binary/bytes chunk - offset=%zu", ! 1700 offset, totalBytes, len, rc);
1701return rc;
1702 }
1703 }
1704 } else {

Lines 1705-1714

1705ThrowStdException("DAE only supported for str or bytes");
1706 }
1707 }
1708if (!SQL_SUCCEEDED(rc)) {
! 1709LOG("SQLExecute: SQLParamData final call %s - SQLRETURN=%d", ! 1710 (rc == SQL_NO_DATA ? "completed with no data" : "failed"), rc);
1711return rc;
1712 }
1713LOG("SQLExecute: DAE streaming completed successfully, SQLExecute resumed");
1714 }

Lines 1740-1749

1740const ParamInfo& info = paramInfos[paramIndex];
1741LOG("BindParameterArray: Processing param_index=%d, C_type=%d, SQL_type=%d, column_size=%zu, decimal_digits=%d", 1742 paramIndex, info.paramCType, info.paramSQLType, info.columnSize, info.decimalDigits);
1743if (columnValues.size() != paramSetSize) {
! 1744LOG("BindParameterArray: Size mismatch - param_index=%d, expected=%zu, actual=%zu", ! 1745 paramIndex, paramSetSize, columnValues.size());
1746ThrowStdException("Column " + std::to_string(paramIndex) + " has mismatched size.");
1747 }
1748void* dataPtr = nullptr;
1749SQLLEN* strLenOrIndArray = nullptr;

Lines 1767-1775

1767 dataPtr = dataArray;
1768break;
1769 }
1770caseSQL_C_DOUBLE: {
! 1771LOG("BindParameterArray: Binding SQL_C_DOUBLE array - param_index=%d, count=%zu", paramIndex, paramSetSize);
1772double* dataArray = AllocateParamBufferArray<double>(tempBuffers, paramSetSize);
1773for (size_t i = 0; i < paramSetSize; ++i) {
1774if (columnValues[i].is_none()) {
1775if (!strLenOrIndArray)

Lines 1780-1788

1780 dataArray[i] = columnValues[i].cast<double>();
1781if (strLenOrIndArray) strLenOrIndArray[i] = 0;
1782 }
1783 }
! 1784LOG("BindParameterArray: SQL_C_DOUBLE bound - param_index=%d", paramIndex);
1785 dataPtr = dataArray;
1786break;
1787 }
1788caseSQL_C_WCHAR: {

Lines 1802-1813

1802size_t utf16_len = utf16Buf.size() > 0 ? utf16Buf.size() - 1 : 0;
1803// Check UTF-16 length (excluding null terminator) against column size1804if (utf16Buf.size() > 0 && utf16_len > info.columnSize) {
1805 std::string offending = WideToUTF8(wstr);
! 1806LOG("BindParameterArray: SQL_C_WCHAR string too long - param_index=%d, row=%zu, utf16_length=%zu, max=%zu",
! 1807 paramIndex, i, utf16_len, info.columnSize);
1808ThrowStdException("Input string UTF-16 length exceeds allowed column size at parameter index " + std::to_string(paramIndex) + ! 1809". UTF-16 length: " + std::to_string(utf16_len) + ", Column size: " + std::to_string(info.columnSize));
1810 }
1811// If we reach here, the UTF-16 string fits - copy it completely1812std::memcpy(wcharArray + i * (info.columnSize + 1), utf16Buf.data(), utf16Buf.size() * sizeof(SQLWCHAR));
1813 #else

Lines 1838-1847

1838 strLenOrIndArray[i] = SQL_NULL_DATA;
1839 } else {
1840int intVal = columnValues[i].cast<int>();
1841if (intVal < 0 || intVal > 255) {
! 1842LOG("BindParameterArray: TINYINT value out of range - param_index=%d, row=%zu, value=%d",
! 1843 paramIndex, i, intVal);
1844ThrowStdException("UTINYINT value out of range at rowIndex " + std::to_string(i));
1845 }
1846 dataArray[i] = static_cast<unsignedchar>(intVal);
1847if (strLenOrIndArray) strLenOrIndArray[i] = 0;

Lines 1864-1873

1864 } else {
1865int intVal = columnValues[i].cast<int>();
1866if (intVal < std::numeric_limits<short>::min() ||
1867 intVal > std::numeric_limits<short>::max()) {
! 1868LOG("BindParameterArray: SHORT value out of range - param_index=%d, row=%zu, value=%d",
! 1869 paramIndex, i, intVal);
1870ThrowStdException("SHORT value out of range at rowIndex " + std::to_string(i));
1871 }
1872 dataArray[i] = static_cast<short>(intVal);
1873if (strLenOrIndArray) strLenOrIndArray[i] = 0;

Lines 1890-1901

1890std::memset(charArray + i * (info.columnSize + 1), 0, info.columnSize + 1);
1891 } else {
1892 std::string str = columnValues[i].cast<std::string>();
1893if (str.size() > info.columnSize) {
! 1894LOG("BindParameterArray: String/binary too long - param_index=%d, row=%zu, size=%zu, max=%zu",
! 1895 paramIndex, i, str.size(), info.columnSize);
1896ThrowStdException("Input exceeds column size at index " + std::to_string(i));
! 1897 }
1898std::memcpy(charArray + i * (info.columnSize + 1), str.c_str(), str.size());
1899 strLenOrIndArray[i] = static_cast<SQLLEN>(str.size());
1900 }
1901 }

Lines 1904-1912

1904 bufferLength = info.columnSize + 1;
1905break;
1906 }
1907caseSQL_C_BIT: {
! 1908LOG("BindParameterArray: Binding SQL_C_BIT array - param_index=%d, count=%zu", paramIndex, paramSetSize);
1909char* boolArray = AllocateParamBufferArray<char>(tempBuffers, paramSetSize);
1910 strLenOrIndArray = AllocateParamBufferArray<SQLLEN>(tempBuffers, paramSetSize);
1911for (size_t i = 0; i < paramSetSize; ++i) {
1912if (columnValues[i].is_none()) {

Lines 1912-1925

1912if (columnValues[i].is_none()) {
1913 boolArray[i] = 0;
1914 strLenOrIndArray[i] = SQL_NULL_DATA;
1915 } else {
! 1916bool val = columnValues[i].cast<bool>();
! 1917 boolArray[i] = val ? 1 : 0;
1918 strLenOrIndArray[i] = 0;
1919 }
1920 }
! 1921LOG("BindParameterArray: SQL_C_BIT bound - param_index=%d", paramIndex);
1922 dataPtr = boolArray;
1923 bufferLength = sizeof(char);
1924break;
1925 }

Lines 1924-1932

1924break;
1925 }
1926caseSQL_C_STINYINT:
1927caseSQL_C_USHORT: {
! 1928LOG("BindParameterArray: Binding SQL_C_USHORT/STINYINT array - param_index=%d, count=%zu", paramIndex, paramSetSize);
1929unsignedshort* dataArray = AllocateParamBufferArray<unsignedshort>(tempBuffers, paramSetSize);
1930 strLenOrIndArray = AllocateParamBufferArray<SQLLEN>(tempBuffers, paramSetSize);
1931for (size_t i = 0; i < paramSetSize; ++i) {
1932if (columnValues[i].is_none()) {

Lines 1936-1944

1936 dataArray[i] = columnValues[i].cast<unsignedshort>();
1937 strLenOrIndArray[i] = 0;
1938 }
1939 }
! 1940LOG("BindParameterArray: SQL_C_USHORT bound - param_index=%d", paramIndex);
1941 dataPtr = dataArray;
1942 bufferLength = sizeof(unsignedshort);
1943break;
1944 }

Lines 1981-1989

1981 bufferLength = sizeof(float);
1982break;
1983 }
1984caseSQL_C_TYPE_DATE: {
! 1985LOG("BindParameterArray: Binding SQL_C_TYPE_DATE array - param_index=%d, count=%zu", paramIndex, paramSetSize);
1986SQL_DATE_STRUCT* dateArray = AllocateParamBufferArray<SQL_DATE_STRUCT>(tempBuffers, paramSetSize);
1987 strLenOrIndArray = AllocateParamBufferArray<SQLLEN>(tempBuffers, paramSetSize);
1988for (size_t i = 0; i < paramSetSize; ++i) {
1989if (columnValues[i].is_none()) {

Lines 1996-2004

1996 dateArray[i].day = dateObj.attr("day").cast<SQLUSMALLINT>();
1997 strLenOrIndArray[i] = 0;
1998 }
1999 }
! 2000LOG("BindParameterArray: SQL_C_TYPE_DATE bound - param_index=%d", paramIndex);
2001 dataPtr = dateArray;
2002 bufferLength = sizeof(SQL_DATE_STRUCT);
2003break;
2004 }

Lines 2002-2010

2002 bufferLength = sizeof(SQL_DATE_STRUCT);
2003break;
2004 }
2005caseSQL_C_TYPE_TIME: {
! 2006LOG("BindParameterArray: Binding SQL_C_TYPE_TIME array - param_index=%d, count=%zu", paramIndex, paramSetSize);
2007SQL_TIME_STRUCT* timeArray = AllocateParamBufferArray<SQL_TIME_STRUCT>(tempBuffers, paramSetSize);
2008 strLenOrIndArray = AllocateParamBufferArray<SQLLEN>(tempBuffers, paramSetSize);
2009for (size_t i = 0; i < paramSetSize; ++i) {
2010if (columnValues[i].is_none()) {

Lines 2017-2025

2017 timeArray[i].second = timeObj.attr("second").cast<SQLUSMALLINT>();
2018 strLenOrIndArray[i] = 0;
2019 }
2020 }
! 2021LOG("BindParameterArray: SQL_C_TYPE_TIME bound - param_index=%d", paramIndex);
2022 dataPtr = timeArray;
2023 bufferLength = sizeof(SQL_TIME_STRUCT);
2024break;
2025 }

Lines 2023-2031

2023 bufferLength = sizeof(SQL_TIME_STRUCT);
2024break;
2025 }
2026caseSQL_C_TYPE_TIMESTAMP: {
! 2027LOG("BindParameterArray: Binding SQL_C_TYPE_TIMESTAMP array - param_index=%d, count=%zu", paramIndex, paramSetSize);
2028SQL_TIMESTAMP_STRUCT* tsArray = AllocateParamBufferArray<SQL_TIMESTAMP_STRUCT>(tempBuffers, paramSetSize);
2029 strLenOrIndArray = AllocateParamBufferArray<SQLLEN>(tempBuffers, paramSetSize);
2030for (size_t i = 0; i < paramSetSize; ++i) {
2031if (columnValues[i].is_none()) {

Lines 2042-2050

2042 tsArray[i].fraction = static_cast<SQLUINTEGER>(dtObj.attr("microsecond").cast<int>() * 1000); // µs to ns2043 strLenOrIndArray[i] = 0;
2044 }
2045 }
! 2046LOG("BindParameterArray: SQL_C_TYPE_TIMESTAMP bound - param_index=%d", paramIndex);
2047 dataPtr = tsArray;
2048 bufferLength = sizeof(SQL_TIMESTAMP_STRUCT);
2049break;
2050 }

Lines 2097-2105

2097 bufferLength = sizeof(DateTimeOffset);
2098break;
2099 }
2100caseSQL_C_NUMERIC: {
! 2101LOG("BindParameterArray: Binding SQL_C_NUMERIC array - param_index=%d, count=%zu", paramIndex, paramSetSize);
2102SQL_NUMERIC_STRUCT* numericArray = AllocateParamBufferArray<SQL_NUMERIC_STRUCT>(tempBuffers, paramSetSize);
2103 strLenOrIndArray = AllocateParamBufferArray<SQLLEN>(tempBuffers, paramSetSize);
2104for (size_t i = 0; i < paramSetSize; ++i) {
2105const py::handle& element = columnValues[i];

Lines 2108-2121

2108std::memset(&numericArray[i], 0, sizeof(SQL_NUMERIC_STRUCT));
2109continue;
2110 }
2111if (!py::isinstance<NumericData>(element)) {
! 2112LOG("BindParameterArray: NUMERIC type mismatch - param_index=%d, row=%zu", paramIndex, i);
2113throwstd::runtime_error(MakeParamMismatchErrorStr(info.paramCType, paramIndex));
2114 }
2115 NumericData decimalParam = element.cast<NumericData>();
! 2116LOG("BindParameterArray: NUMERIC value - param_index=%d, row=%zu, precision=%d, scale=%d, sign=%d",
! 2117 paramIndex, i, decimalParam.precision, decimalParam.scale, decimalParam.sign);
2118SQL_NUMERIC_STRUCT& target = numericArray[i];
2119std::memset(&target, 0, sizeof(SQL_NUMERIC_STRUCT));
2120 target.precision = decimalParam.precision;
2121 target.scale = decimalParam.scale;

Lines 2125-2133

2125std::memcpy(target.val, decimalParam.val.data(), copyLen);
2126 }
2127 strLenOrIndArray[i] = sizeof(SQL_NUMERIC_STRUCT);
2128 }
! 2129LOG("BindParameterArray: SQL_C_NUMERIC bound - param_index=%d", paramIndex);
2130 dataPtr = numericArray;
2131 bufferLength = sizeof(SQL_NUMERIC_STRUCT);
2132break;
2133 }

Lines 2151-2160

2151 }
2152elseif (py::isinstance<py::bytes>(element)) {
2153 py::bytes b = element.cast<py::bytes>();
2154if (PyBytes_GET_SIZE(b.ptr()) != 16) {
! 2155LOG("BindParameterArray: GUID bytes wrong length - param_index=%d, row=%zu, length=%d",
! 2156 paramIndex, i, PyBytes_GET_SIZE(b.ptr()));
2157ThrowStdException("UUID binary data must be exactly 16 bytes long.");
2158 }
2159std::memcpy(uuid_bytes.data(), PyBytes_AS_STRING(b.ptr()), 16);
2160 }

Lines 2162-2170

2162 py::bytes b = element.attr("bytes_le").cast<py::bytes>();
2163std::memcpy(uuid_bytes.data(), PyBytes_AS_STRING(b.ptr()), 16);
2164 }
2165else {
! 2166LOG("BindParameterArray: GUID type mismatch - param_index=%d, row=%zu", paramIndex, i);
2167ThrowStdException(MakeParamMismatchErrorStr(info.paramCType, paramIndex));
2168 }
2169 guidArray[i].Data1 = (static_cast<uint32_t>(uuid_bytes[3]) << 24) |
2170 (static_cast<uint32_t>(uuid_bytes[2]) << 16) |

Lines 2183-2191

2183 bufferLength = sizeof(SQLGUID);
2184break;
2185 }
2186default: {
! 2187LOG("BindParameterArray: Unsupported C type - param_index=%d, C_type=%d", paramIndex, info.paramCType);
2188ThrowStdException("BindParameterArray: Unsupported C type: " + std::to_string(info.paramCType));
2189 }
2190 }
2191LOG("BindParameterArray: Calling SQLBindParameter - param_index=%d, buffer_length=%lld", 

Lines 2202-2215

2202 bufferLength,
2203 strLenOrIndArray
2204 );
2205if (!SQL_SUCCEEDED(rc)) {
! 2206LOG("BindParameterArray: SQLBindParameter failed - param_index=%d, SQLRETURN=%d", paramIndex, rc);
2207return rc;
2208 }
2209 }
2210 } catch (...) {
! 2211LOG("BindParameterArray: Exception during binding, cleaning up buffers");
2212throw;
2213 }
2214 paramBuffers.insert(paramBuffers.end(), tempBuffers.begin(), tempBuffers.end());
2215LOG("BindParameterArray: Successfully bound all parameters - total_params=%zu, buffer_count=%zu",

Lines 2236-2246

2236LOG("SQLExecuteMany: Using wide string query directly");
2237 #endif
2238RETCODE rc = SQLPrepare_ptr(hStmt, queryPtr, SQL_NTS);
2239if (!SQL_SUCCEEDED(rc)) {
! 2240LOG("SQLExecuteMany: SQLPrepare failed - rc=%d", rc);
! 2241return rc;
! 2242 }
2243LOG("SQLExecuteMany: Query prepared successfully");
22442245bool hasDAE = false;
2246for (constauto& p : paramInfos) {

Lines 2254-2270

2254LOG("SQLExecuteMany: Using array binding (non-DAE) - calling BindParameterArray");
2255 std::vector<std::shared_ptr<void>> paramBuffers;
2256 rc = BindParameterArray(hStmt, columnwise_params, paramInfos, paramSetSize, paramBuffers);
2257if (!SQL_SUCCEEDED(rc)) {
! 2258LOG("SQLExecuteMany: BindParameterArray failed - rc=%d", rc);
! 2259return rc;
! 2260 }
22612262 rc = SQLSetStmtAttr_ptr(hStmt, SQL_ATTR_PARAMSET_SIZE, (SQLPOINTER)paramSetSize, 0);
2263if (!SQL_SUCCEEDED(rc)) {
! 2264LOG("SQLExecuteMany: SQLSetStmtAttr(PARAMSET_SIZE) failed - rc=%d", rc);
! 2265return rc;
! 2266 }
2267LOG("SQLExecuteMany: PARAMSET_SIZE set to %zu", paramSetSize);
22682269 rc = SQLExecute_ptr(hStmt);
2270LOG("SQLExecuteMany: SQLExecute completed - rc=%d", rc);

Lines 2269-2342

2269 rc = SQLExecute_ptr(hStmt);
2270LOG("SQLExecuteMany: SQLExecute completed - rc=%d", rc);
2271return rc;
2272 } else {
! 2273LOG("SQLExecuteMany: Using DAE (data-at-execution) - row_count=%zu", columnwise_params.size());
2274size_t rowCount = columnwise_params.size();
2275for (size_t rowIndex = 0; rowIndex < rowCount; ++rowIndex) {
! 2276LOG("SQLExecuteMany: Processing DAE row %zu of %zu", rowIndex + 1, rowCount);
2277 py::list rowParams = columnwise_params[rowIndex];
22782279 std::vector<std::shared_ptr<void>> paramBuffers;
2280 rc = BindParameters(hStmt, rowParams, const_cast<std::vector<ParamInfo>&>(paramInfos), paramBuffers);
! 2281if (!SQL_SUCCEEDED(rc)) {
! 2282LOG("SQLExecuteMany: BindParameters failed for row %zu - rc=%d", rowIndex, rc);
! 2283return rc;
! 2284 }
! 2285LOG("SQLExecuteMany: Parameters bound for row %zu", rowIndex);
22862287 rc = SQLExecute_ptr(hStmt);
! 2288LOG("SQLExecuteMany: SQLExecute for row %zu - initial_rc=%d", rowIndex, rc);
! 2289size_t dae_chunk_count = 0;
2290while (rc == SQL_NEED_DATA) {
2291SQLPOINTER token;
2292 rc = SQLParamData_ptr(hStmt, &token);
! 2293LOG("SQLExecuteMany: SQLParamData called - chunk=%zu, rc=%d, token=%p", ! 2294 dae_chunk_count, rc, token);
! 2295if (!SQL_SUCCEEDED(rc) && rc != SQL_NEED_DATA) {
! 2296LOG("SQLExecuteMany: SQLParamData failed - chunk=%zu, rc=%d", dae_chunk_count, rc);
! 2297return rc;
! 2298 }
22992300 py::object* py_obj_ptr = reinterpret_cast<py::object*>(token);
! 2301if (!py_obj_ptr) {
! 2302LOG("SQLExecuteMany: NULL token pointer in DAE - chunk=%zu", dae_chunk_count);
! 2303returnSQL_ERROR;
! 2304 }
23052306if (py::isinstance<py::str>(*py_obj_ptr)) {
2307 std::string data = py_obj_ptr->cast<std::string>();
2308SQLLEN data_len = static_cast<SQLLEN>(data.size());
! 2309LOG("SQLExecuteMany: Sending string DAE data - chunk=%zu, length=%lld", ! 2310 dae_chunk_count, static_cast<longlong>(data_len));
2311 rc = SQLPutData_ptr(hStmt, (SQLPOINTER)data.c_str(), data_len);
! 2312if (!SQL_SUCCEEDED(rc) && rc != SQL_NEED_DATA) {
! 2313LOG("SQLExecuteMany: SQLPutData(string) failed - chunk=%zu, rc=%d", dae_chunk_count, rc);
! 2314 }
2315 } elseif (py::isinstance<py::bytes>(*py_obj_ptr) || py::isinstance<py::bytearray>(*py_obj_ptr)) {
2316 std::string data = py_obj_ptr->cast<std::string>();
2317SQLLEN data_len = static_cast<SQLLEN>(data.size());
! 2318LOG("SQLExecuteMany: Sending bytes/bytearray DAE data - chunk=%zu, length=%lld", ! 2319 dae_chunk_count, static_cast<longlong>(data_len));
2320 rc = SQLPutData_ptr(hStmt, (SQLPOINTER)data.c_str(), data_len);
! 2321if (!SQL_SUCCEEDED(rc) && rc != SQL_NEED_DATA) {
! 2322LOG("SQLExecuteMany: SQLPutData(bytes) failed - chunk=%zu, rc=%d", dae_chunk_count, rc);
! 2323 }
2324 } else {
! 2325LOG("SQLExecuteMany: Unsupported DAE data type - chunk=%zu", dae_chunk_count);
2326returnSQL_ERROR;
2327 }
! 2328 dae_chunk_count++;
2329 }
! 2330LOG("SQLExecuteMany: DAE completed for row %zu - total_chunks=%zu, final_rc=%d", ! 2331 rowIndex, dae_chunk_count, rc);
2332 ! 2333if (!SQL_SUCCEEDED(rc)) {
! 2334LOG("SQLExecuteMany: DAE row %zu failed - rc=%d", rowIndex, rc);
! 2335return rc;
! 2336 }
2337 }
! 2338LOG("SQLExecuteMany: All DAE rows processed successfully - total_rows=%zu", rowCount);
2339returnSQL_SUCCESS;
2340 }
2341 }

Lines 2344-2352

2344// Wrap SQLNumResultCols2345SQLSMALLINTSQLNumResultCols_wrap(SqlHandlePtr statementHandle) {
2346LOG("SQLNumResultCols: Getting number of columns in result set for statement_handle=%p", (void*)statementHandle->get());
2347if (!SQLNumResultCols_ptr) {
! 2348LOG("SQLNumResultCols: Function pointer not initialized, loading driver");
2349DriverLoader::getInstance().loadDriver(); // Load the driver2350 }
23512352SQLSMALLINT columnCount;

Lines 2358-2366

2358// Wrap SQLDescribeCol2359SQLRETURNSQLDescribeCol_wrap(SqlHandlePtr StatementHandle, py::list& ColumnMetadata) {
2360LOG("SQLDescribeCol: Getting column descriptions for statement_handle=%p", (void*)StatementHandle->get());
2361if (!SQLDescribeCol_ptr) {
! 2362LOG("SQLDescribeCol: Function pointer not initialized, loading driver");
2363DriverLoader::getInstance().loadDriver(); // Load the driver2364 }
23652366SQLSMALLINT ColumnCount;

Lines 2366-2374

2366SQLSMALLINT ColumnCount;
2367SQLRETURN retcode =
2368SQLNumResultCols_ptr(StatementHandle->get(), &ColumnCount);
2369if (!SQL_SUCCEEDED(retcode)) {
! 2370LOG("SQLDescribeCol: Failed to get number of columns - SQLRETURN=%d", retcode);
2371return retcode;
2372 }
23732374for (SQLUSMALLINT i = 1; i <= ColumnCount; ++i) {

Lines 2450-2460

2450 }
24512452// Wrap SQLFetch to retrieve rows2453SQLRETURNSQLFetch_wrap(SqlHandlePtr StatementHandle) {
! 2454LOG("SQLFetch: Fetching next row for statement_handle=%p", (void*)StatementHandle->get());
2455if (!SQLFetch_ptr) {
! 2456LOG("SQLFetch: Function pointer not initialized, loading driver");
2457DriverLoader::getInstance().loadDriver(); // Load the driver2458 }
24592460returnSQLFetch_ptr(StatementHandle->get());

Lines 2487-2495

2487 oss << "Error fetching LOB for column " << colIndex
2488 << ", cType=" << cType
2489 << ", loop=" << loopCount
2490 << ", SQLGetData return=" << ret;
! 2491LOG("FetchLobColumnData: %s", oss.str().c_str());
2492ThrowStdException(oss.str());
2493 }
2494if (actualRead == SQL_NULL_DATA) {
2495LOG("FetchLobColumnData: Column %d is NULL at loop %d", colIndex, loopCount);

Lines 2581-2589

2581// Helper function to retrieve column data2582SQLRETURNSQLGetData_wrap(SqlHandlePtr StatementHandle, SQLUSMALLINT colCount, py::list& row) {
2583LOG("SQLGetData: Getting data from %d columns for statement_handle=%p", colCount, (void*)StatementHandle->get());
2584if (!SQLGetData_ptr) {
! 2585LOG("SQLGetData: Function pointer not initialized, loading driver");
2586DriverLoader::getInstance().loadDriver(); // Load the driver2587 }
25882589SQLRETURN ret = SQL_SUCCESS;

Lines 2602-2610

26022603 ret = SQLDescribeCol_ptr(hStmt, i, columnName, sizeof(columnName) / sizeof(SQLWCHAR),
2604 &columnNameLen, &dataType, &columnSize, &decimalDigits, &nullable);
2605if (!SQL_SUCCEEDED(ret)) {
! 2606LOG("SQLGetData: Error retrieving metadata for column %d - SQLDescribeCol SQLRETURN=%d", i, ret);
2607 row.append(py::none());
2608continue;
2609 }

Lines 2634-2642

2634 row.append(std::string(reinterpret_cast<char*>(dataBuffer.data())));
2635 #endif
2636 } else {
2637// Buffer too small, fallback to streaming
! 2638LOG("SQLGetData: CHAR column %d data truncated (buffer_size=%zu), using streaming LOB", i, dataBuffer.size());
2639 row.append(FetchLobColumnData(hStmt, i, SQL_C_CHAR, false, false));
2640 }
2641 } elseif (dataLen == SQL_NULL_DATA) {
2642LOG("SQLGetData: Column %d is NULL (CHAR)", i);

Lines 2643-2658

2643 row.append(py::none());
2644 } elseif (dataLen == 0) {
2645 row.append(py::str(""));
2646 } elseif (dataLen == SQL_NO_TOTAL) {
! 2647LOG("SQLGetData: Cannot determine data length (SQL_NO_TOTAL) for column %d (SQL_CHAR), returning NULL", i);
2648 row.append(py::none());
2649 } elseif (dataLen < 0) {
! 2650LOG("SQLGetData: Unexpected negative data length for column %d - dataType=%d, dataLen=%ld", i, dataType, (long)dataLen);
2651ThrowStdException("SQLGetData returned an unexpected negative data length");
2652 }
2653 } else {
! 2654LOG("SQLGetData: Error retrieving data for column %d (SQL_CHAR) - SQLRETURN=%d, returning NULL", i, ret);
2655 row.append(py::none());
2656 }
2657 }
2658break;

Lines 2697-2712

2697 row.append(py::none());
2698 } elseif (dataLen == 0) {
2699 row.append(py::str(""));
2700 } elseif (dataLen == SQL_NO_TOTAL) {
! 2701LOG("SQLGetData: Cannot determine NVARCHAR data length (SQL_NO_TOTAL) for column %d, returning NULL", i);
2702 row.append(py::none());
2703 } elseif (dataLen < 0) {
! 2704LOG("SQLGetData: Unexpected negative data length for column %d (NVARCHAR) - dataLen=%ld", i, (long)dataLen);
2705ThrowStdException("SQLGetData returned an unexpected negative data length");
2706 }
2707 } else {
! 2708LOG("SQLGetData: Error retrieving data for column %d (NVARCHAR) - SQLRETURN=%d", i, ret);
2709 row.append(py::none());
2710 }
2711 }
2712break;

Lines 2726-2734

2726 ret = SQLGetData_ptr(hStmt, i, SQL_C_SHORT, &smallIntValue, 0, NULL);
2727if (SQL_SUCCEEDED(ret)) {
2728 row.append(static_cast<int>(smallIntValue));
2729 } else {
! 2730LOG("SQLGetData: Error retrieving SQL_SMALLINT for column %d - SQLRETURN=%d", i, ret);
2731 row.append(py::none());
2732 }
2733break;
2734 }

Lines 2737-2745

2737 ret = SQLGetData_ptr(hStmt, i, SQL_C_FLOAT, &realValue, 0, NULL);
2738if (SQL_SUCCEEDED(ret)) {
2739 row.append(realValue);
2740 } else {
! 2741LOG("SQLGetData: Error retrieving SQL_REAL for column %d - SQLRETURN=%d", i, ret);
2742 row.append(py::none());
2743 }
2744break;
2745 }

Lines 2780-2793

2780 py::object decimalObj = PythonObjectCache::get_decimal_class()(py::str(cnum, safeLen));
2781 row.append(decimalObj);
2782 } catch (const py::error_already_set& e) {
2783// If conversion fails, append None
! 2784LOG("SQLGetData: Error converting to decimal for column %d - %s", i, e.what());
2785 row.append(py::none());
2786 }
2787 }
2788else {
! 2789LOG("SQLGetData: Error retrieving SQL_NUMERIC/DECIMAL for column %d - SQLRETURN=%d", i, ret);
2790 row.append(py::none());
2791 }
2792break;
2793 }

Lines 2798-2806

2798 ret = SQLGetData_ptr(hStmt, i, SQL_C_DOUBLE, &doubleValue, 0, NULL);
2799if (SQL_SUCCEEDED(ret)) {
2800 row.append(doubleValue);
2801 } else {
! 2802LOG("SQLGetData: Error retrieving SQL_DOUBLE/FLOAT for column %d - SQLRETURN=%d", i, ret);
2803 row.append(py::none());
2804 }
2805break;
2806 }

Lines 2809-2817

2809 ret = SQLGetData_ptr(hStmt, i, SQL_C_SBIGINT, &bigintValue, 0, NULL);
2810if (SQL_SUCCEEDED(ret)) {
2811 row.append(static_cast<longlong>(bigintValue));
2812 } else {
! 2813LOG("SQLGetData: Error retrieving SQL_BIGINT for column %d - SQLRETURN=%d", i, ret);
2814 row.append(py::none());
2815 }
2816break;
2817 }

Lines 2846-2854

2846 timeValue.second
2847 )
2848 );
2849 } else {
! 2850LOG("SQLGetData: Error retrieving SQL_TYPE_TIME for column %d - SQLRETURN=%d", i, ret);
2851 row.append(py::none());
2852 }
2853break;
2854 }

Lines 2870-2878

2870 timestampValue.fraction / 1000// Convert back ns to µs2871 )
2872 );
2873 } else {
! 2874LOG("SQLGetData: Error retrieving SQL_TYPE_TIMESTAMP for column %d - SQLRETURN=%d", i, ret);
2875 row.append(py::none());
2876 }
2877break;
2878 }

Lines 2919-2927

2919 tzinfo
2920 );
2921 row.append(py_dt);
2922 } else {
! 2923LOG("SQLGetData: Error fetching DATETIMEOFFSET for column %d - SQLRETURN=%d, indicator=%ld", i, ret, (long)indicator);
2924 row.append(py::none());
2925 }
2926break;
2927 }

Lines 2952-2964

2952 } else {
2953 std::ostringstream oss;
2954 oss << "Unexpected negative length (" << dataLen << ") returned by SQLGetData. ColumnID="2955 << i << ", dataType=" << dataType << ", bufferSize=" << columnSize;
! 2956LOG("SQLGetData: %s", oss.str().c_str());
2957ThrowStdException(oss.str());
2958 }
2959 } else {
! 2960LOG("SQLGetData: Error retrieving VARBINARY data for column %d - SQLRETURN=%d", i, ret);
2961 row.append(py::none());
2962 }
2963 }
2964break;

Lines 2968-2976

2968 ret = SQLGetData_ptr(hStmt, i, SQL_C_TINYINT, &tinyIntValue, 0, NULL);
2969if (SQL_SUCCEEDED(ret)) {
2970 row.append(static_cast<int>(tinyIntValue));
2971 } else {
! 2972LOG("SQLGetData: Error retrieving SQL_TINYINT for column %d - SQLRETURN=%d", i, ret);
2973 row.append(py::none());
2974 }
2975break;
2976 }

Lines 2979-2987

2979 ret = SQLGetData_ptr(hStmt, i, SQL_C_BIT, &bitValue, 0, NULL);
2980if (SQL_SUCCEEDED(ret)) {
2981 row.append(static_cast<bool>(bitValue));
2982 } else {
! 2983LOG("SQLGetData: Error retrieving SQL_BIT for column %d - SQLRETURN=%d", i, ret);
2984 row.append(py::none());
2985 }
2986break;
2987 }

Lines 3008-3016

3008 row.append(uuid_obj);
3009 } elseif (indicator == SQL_NULL_DATA) {
3010 row.append(py::none());
3011 } else {
! 3012LOG("SQLGetData: Error retrieving SQL_GUID for column %d - SQLRETURN=%d, indicator=%ld", i, ret, (long)indicator);
3013 row.append(py::none());
3014 }
3015break;
3016 }

Lines 3018-3026

3018default:
3019 std::ostringstream errorString;
3020 errorString << "Unsupported data type for column - " << columnName << ", Type - "3021 << dataType << ", column ID - " << i;
! 3022LOG("SQLGetData: %s", errorString.str().c_str());
3023ThrowStdException(errorString.str());
3024break;
3025 }
3026 }

Lines 3029-3037

30293030SQLRETURNSQLFetchScroll_wrap(SqlHandlePtr StatementHandle, SQLSMALLINT FetchOrientation, SQLLEN FetchOffset, py::list& row_data) {
3031LOG("SQLFetchScroll_wrap: Fetching with scroll orientation=%d, offset=%ld", FetchOrientation, (long)FetchOffset);
3032if (!SQLFetchScroll_ptr) {
! 3033LOG("SQLFetchScroll_wrap: Function pointer not initialized. Loading the driver.");
3034DriverLoader::getInstance().loadDriver(); // Load the driver3035 }
30363037// Unbind any columns from previous fetch operations to avoid memory corruption

Lines 3191-3199

3191 std::wstring columnName = columnMeta["ColumnName"].cast<std::wstring>();
3192 std::ostringstream errorString;
3193 errorString << "Unsupported data type for column - " << columnName.c_str()
3194 << ", Type - " << dataType << ", column ID - " << col;
! 3195LOG("SQLBindColums: %s", errorString.str().c_str());
3196ThrowStdException(errorString.str());
3197break;
3198 }
3199if (!SQL_SUCCEEDED(ret)) {

Lines 3200-3208

3200 std::wstring columnName = columnMeta["ColumnName"].cast<std::wstring>();
3201 std::ostringstream errorString;
3202 errorString << "Failed to bind column - " << columnName.c_str() << ", Type - "3203 << dataType << ", column ID - " << col;
! 3204LOG("SQLBindColums: %s", errorString.str().c_str());
3205ThrowStdException(errorString.str());
3206return ret;
3207 }
3208 }

Lines 3370-3378

3370PyList_SET_ITEM(row, col - 1, Py_None);
3371continue;
3372 } elseif (dataLen < 0) {
3373// Negative value is unexpected, log column index, SQL type & raise exception
! 3374LOG("FetchBatchData: Unexpected negative data length - column=%d, SQL_type=%d, dataLen=%ld", col, dataType, (long)dataLen);
3375ThrowStdException("Unexpected negative data length, check logs for details");
3376 }
3377assert(dataLen > 0 && "Data length must be > 0");

Lines 3481-3489

3481 std::wstring columnName = columnMeta["ColumnName"].cast<std::wstring>();
3482 std::ostringstream errorString;
3483 errorString << "Unsupported data type for column - " << columnName.c_str()
3484 << ", Type - " << dataType << ", column ID - " << col;
! 3485LOG("FetchBatchData: %s", errorString.str().c_str());
3486ThrowStdException(errorString.str());
3487break;
3488 }
3489 }

Lines 3579-3587

3579 std::wstring columnName = columnMeta["ColumnName"].cast<std::wstring>();
3580 std::ostringstream errorString;
3581 errorString << "Unsupported data type for column - " << columnName.c_str()
3582 << ", Type - " << dataType << ", column ID - " << col;
! 3583LOG("calculateRowSize: %s", errorString.str().c_str());
3584ThrowStdException(errorString.str());
3585break;
3586 }
3587 }

Lines 3611-3619

3611// Retrieve column metadata3612 py::list columnNames;
3613 ret = SQLDescribeCol_wrap(StatementHandle, columnNames);
3614if (!SQL_SUCCEEDED(ret)) {
! 3615LOG("FetchMany_wrap: Failed to get column descriptions - SQLRETURN=%d", ret);
3616return ret;
3617 }
36183619 std::vector<SQLUSMALLINT> lobColumns;

Lines 3650-3658

36503651// Bind columns3652 ret = SQLBindColums(hStmt, buffers, columnNames, numCols, fetchSize);
3653if (!SQL_SUCCEEDED(ret)) {
! 3654LOG("FetchMany_wrap: Error when binding columns - SQLRETURN=%d", ret);
3655return ret;
3656 }
36573658SQLULEN numRowsFetched;

Lines 3660-3668

3660SQLSetStmtAttr_ptr(hStmt, SQL_ATTR_ROWS_FETCHED_PTR, &numRowsFetched, 0);
36613662 ret = FetchBatchData(hStmt, buffers, columnNames, rows, numCols, numRowsFetched, lobColumns);
3663if (!SQL_SUCCEEDED(ret) && ret != SQL_NO_DATA) {
! 3664LOG("FetchMany_wrap: Error when fetching data - SQLRETURN=%d", ret);
3665return ret;
3666 }
36673668// Reset attributes before returning to avoid using stack pointers later

Lines 3693-3701

3693// Retrieve column metadata3694 py::list columnNames;
3695 ret = SQLDescribeCol_wrap(StatementHandle, columnNames);
3696if (!SQL_SUCCEEDED(ret)) {
! 3697LOG("FetchAll_wrap: Failed to get column descriptions - SQLRETURN=%d", ret);
3698return ret;
3699 }
37003701// Define a memory limit (1 GB)

Lines 3770-3778

37703771// Bind columns3772 ret = SQLBindColums(hStmt, buffers, columnNames, numCols, fetchSize);
3773if (!SQL_SUCCEEDED(ret)) {
! 3774LOG("FetchAll_wrap: Error when binding columns - SQLRETURN=%d", ret);
3775return ret;
3776 }
37773778SQLULEN numRowsFetched;

Lines 3826-3834

3826// Wrap SQLMoreResults3827SQLRETURNSQLMoreResults_wrap(SqlHandlePtr StatementHandle) {
3828LOG("SQLMoreResults_wrap: Check for more results");
3829if (!SQLMoreResults_ptr) {
! 3830LOG("SQLMoreResults_wrap: Function pointer not initialized. Loading the driver.");
3831DriverLoader::getInstance().loadDriver(); // Load the driver3832 }
38333834returnSQLMoreResults_ptr(StatementHandle->get());

Lines 3835-3845

3835 }
38363837// Wrap SQLFreeHandle3838SQLRETURNSQLFreeHandle_wrap(SQLSMALLINT HandleType, SqlHandlePtr Handle) {
! 3839LOG("SQLFreeHandle_wrap: Free SQL handle type=%d", HandleType);
3840if (!SQLAllocHandle_ptr) {
! 3841LOG("SQLFreeHandle_wrap: Function pointer not initialized. Loading the driver.");
3842DriverLoader::getInstance().loadDriver(); // Load the driver3843 }
38443845SQLRETURN ret = SQLFreeHandle_ptr(HandleType, Handle->get());

Lines 3843-3851

3843 }
38443845SQLRETURN ret = SQLFreeHandle_ptr(HandleType, Handle->get());
3846if (!SQL_SUCCEEDED(ret)) {
! 3847LOG("SQLFreeHandle_wrap: SQLFreeHandle failed with error code - %d", ret);
3848return ret;
3849 }
3850return ret;
3851 }

Lines 3853-3861

3853// Wrap SQLRowCount3854SQLLENSQLRowCount_wrap(SqlHandlePtr StatementHandle) {
3855LOG("SQLRowCount_wrap: Get number of rows affected by last execute");
3856if (!SQLRowCount_ptr) {
! 3857LOG("SQLRowCount_wrap: Function pointer not initialized. Loading the driver.");
3858DriverLoader::getInstance().loadDriver(); // Load the driver3859 }
38603861SQLLEN rowCount;

Lines 3860-3868

38603861SQLLEN rowCount;
3862SQLRETURN ret = SQLRowCount_ptr(StatementHandle->get(), &rowCount);
3863if (!SQL_SUCCEEDED(ret)) {
! 3864LOG("SQLRowCount_wrap: SQLRowCount failed with error code - %d", ret);
3865return ret;
3866 }
3867LOG("SQLRowCount_wrap: SQLRowCount returned %ld", (long)rowCount);
3868return rowCount;

Lines 4041-4050

4041mssql_python::logging::LoggerBridge::initialize();
4042 } catch (const std::exception& e) {
4043// Log initialization failure but don't throw4044// Use std::cerr instead of fprintf for type-safe output
! 4045 std::cerr << "Logger bridge initialization failed: " << e.what() << std::endl;
! 4046 }
40474048try {
4049// Try loading the ODBC driver when the module is imported4050LOG("Module initialization: Loading ODBC driver");

Lines 4050-4057

4050LOG("Module initialization: Loading ODBC driver");
4051DriverLoader::getInstance().loadDriver(); // Load the driver4052 } catch (const std::exception& e) {
4053// Log the error but don't throw - let the error happen when functions are called
! 4054LOG("Module initialization: Failed to load ODBC driver - %s", e.what());
4055 }
4056 }

mssql_python/pybind/logger_bridge.cpp

Lines 26-35

26 std::lock_guard<std::mutex> lock(mutex_);
2728// Skip if already initialized (check inside lock to prevent TOCTOU race)29if (initialized_) {
! 30return;
! 31 }
3233try {
34// Acquire GIL for Python API calls35 py::gil_scoped_acquire gil;

Lines 55-68

5556 } catch (const py::error_already_set& e) {
57// Failed to initialize - log to stderr and continue58// (logging will be disabled but won't crash)
! 59 std::cerr << "LoggerBridge initialization failed: " << e.what() << std::endl;
! 60 initialized_ = false;
! 61 } catch (const std::exception& e) {
! 62 std::cerr << "LoggerBridge initialization failed: " << e.what() << std::endl;
! 63 initialized_ = false;
! 64 }
65 }
6667voidLoggerBridge::updateLevel(int level) {
68// Update the cached level atomically

Lines 69-83

69// This is lock-free and can be called from any thread70 cached_level_.store(level, std::memory_order_relaxed);
71 }
72 ! 73intLoggerBridge::getLevel() {
! 74return cached_level_.load(std::memory_order_relaxed);
! 75 }
76 ! 77boolLoggerBridge::isInitialized() {
! 78return initialized_;
! 79 }
8081 std::string LoggerBridge::formatMessage(constchar* format, va_list args) {
82// Use a stack buffer for most messages (4KB should be enough)83char buffer[4096];

Lines 92-101

92va_end(args_copy);
9394if (result < 0) {
95// Error during formatting
! 96return"[Formatting error]";
! 97 }
9899if (result < static_cast<int>(sizeof(buffer))) {
100// Message fit in buffer (vsnprintf guarantees null-termination)101returnstd::string(buffer, std::min(static_cast<size_t>(result), sizeof(buffer) - 1));

Lines 102-123

102 }
103104// Message was truncated - allocate larger buffer105// (This should be rare for typical log messages)
! 106 std::vector<char> large_buffer(result + 1);
! 107va_copy(args_copy, args);
108// Use std::vsnprintf with explicit size for safety (C++11 standard)109// This is the recommended safe alternative to vsprintf110// DevSkim: ignore DS185832 - std::vsnprintf with size is safe
! 111int final_result = std::vsnprintf(large_buffer.data(), large_buffer.size(), format, args_copy);
! 112va_end(args_copy);
113114// Ensure null termination even if formatting fails
! 115if (final_result < 0 || final_result >= static_cast<int>(large_buffer.size())) {
! 116 large_buffer[large_buffer.size() - 1] = '\0';
! 117 }
118 ! 119return std::string(large_buffer.data());
120 }
121122constchar* LoggerBridge::extractFilename(constchar* path) {
123// Extract just the filename from full path using safer C++ string search

Lines 121-130

121122constchar* LoggerBridge::extractFilename(constchar* path) {
123// Extract just the filename from full path using safer C++ string search124if (!path) {
! 125return"";
! 126 }
127128// Find last occurrence of Unix path separator129constchar* filename = std::strrchr(path, '/');
130if (filename) {

Lines 131-146

131return filename + 1;
132 }
133134// Try Windows path separator
! 135 filename = std::strrchr(path, '\\');
! 136if (filename) {
! 137return filename + 1;
! 138 }
139140// No path separator found, return the whole string
! 141return path;
! 142 }
143144voidLoggerBridge::log(int level, constchar* file, int line, 145constchar* format, ...) {
146// Fast level check (should already be done by macro, but double-check)

Lines 144-158

144voidLoggerBridge::log(int level, constchar* file, int line, 145constchar* format, ...) {
146// Fast level check (should already be done by macro, but double-check)147if (!isLoggable(level)) {
! 148return;
! 149 }
150151// Check if initialized152if (!initialized_ || !cached_logger_) {
! 153return;
! 154 }
155156// Format the message157 va_list args;
158va_start(args, format);

Lines 172-184

172// Warn if message exceeds reasonable size (critical for troubleshooting)173constexprsize_tMAX_LOG_SIZE = 4095; // Keep same limit for consistency174if (complete_message.size() > MAX_LOG_SIZE) {
175// Use stderr to notify about truncation (logging may be the truncated call itself)
! 176 std::cerr << "[MSSQL-Python] Warning: Log message truncated from " ! 177 << complete_message.size() << " bytes to " << MAX_LOG_SIZE ! 178 << " bytes at " << file << ":" << line << std::endl;
! 179 complete_message.resize(MAX_LOG_SIZE);
! 180 }
181182// Lock for Python call (minimize critical section)183 std::lock_guard<std::mutex> lock(mutex_);
184

Lines 211-226

211212 } catch (const py::error_already_set& e) {
213// Python error during logging - ignore to prevent cascading failures214// (Logging errors should not crash the application)
! 215 (void)e; // Suppress unused variable warning
! 216 } catch (const std::exception& e) {
217// Standard C++ exception - ignore
! 218 (void)e;
! 219 } catch (...) {
220// Catch-all for unknown exceptions (non-standard exceptions, corrupted state, etc.)221// Logging must NEVER crash the application
! 222 }
223 }
224225 } // namespace logging226 } // namespace mssql_python


📋 Files Needing Attention

📉 Files with overall lowest coverage (click to expand)
mssql_python.pybind.logger_bridge.cpp: 59.2%
mssql_python.helpers.py: 66.6%
mssql_python.row.py: 67.4%
mssql_python.pybind.ddbc_bindings.cpp: 70.4%
mssql_python.pybind.connection.connection.cpp: 76.3%
mssql_python.ddbc_bindings.py: 79.6%
mssql_python.pybind.connection.connection_pool.cpp: 79.6%
mssql_python.pybind.ddbc_bindings.h: 79.7%
mssql_python.connection.py: 82.5%
mssql_python.cursor.py: 83.5%

🔗 Quick Links

⚙️ Build Summary📋 Coverage Details

View Azure DevOps Build

Browse Full Coverage Report

Comment threadmssql_python/pybind/logger_bridge.cpp Dismissed
Comment threadmssql_python/pybind/logger_bridge.cpp Fixed
…nflict and include logger_bridge.hpp in ddbc_bindings.h
- Changed log filename format: timestamp now has no separator (YYYYMMDDHHMMSS)
- Added CSV header with metadata: script name, PID, Python version, OS info
- Converted log format to CSV: Timestamp, ThreadID, Level, Location, Source, Message
- Replaced trace ID with OS native thread ID for debugger compatibility
- Updated Python formatter to parse [Python]/[DDBC] tags into Source column
- Updated C++ logger_bridge to use makeRecord() for proper file/line attribution
- Logs are now easily parseable as CSV for analysis in Excel/pandas
- Counter logic for connection/cursor tracking kept internally but not displayed
- Updated LOGGING.md:
* Changed log format examples from trace ID to CSV format
* Updated filename format (YYYYMMDDHHMMSS with no separators)
* Replaced trace ID section with Thread Tracking section
* Added CSV parsing examples with pandas
* Updated all log output samples to show CSV columns
- Updated MSSQL-Python-Logging-Design.md:
* Changed file handler config to describe CSV format
* Replaced Trace ID System with Thread Tracking System
* Updated architecture to reflect OS native thread IDs
* Added CSV formatter implementation details
* Updated all code examples to use setup_logging() API
* Changed log output examples to CSV format
- Thread tracking now uses OS native thread IDs (threading.get_native_id())
- CSV columns: Timestamp, ThreadID, Level, Location, Source, Message
- File header includes metadata (PID, script name, Python version, etc.)
- Easy analysis with pandas/Excel/CSV tools
- CSV format now mentioned only once as optional import capability
- Focus on log structure and content, not format
- Removed repetitive CSV parsing examples
- Single section 'Importing Logs as CSV (Optional)' in LOGGING.md
- Brief mention in design doc that format is importable as CSV
Features:
- Whitelist log file extensions: .txt, .log, .csv only
- Raise ValueError for invalid extensions
- Export driver_logger for use in application code
- Allow apps to use mssql-python's logger: from mssql_python.logging import driver_logger
- Updated documentation with usage examples
- Added validation in _setLevel() method
Benefits:
- Prevents accidental use of wrong file types
- Clear error messages for invalid extensions
- Unified logging - apps can use same logger as driver
- Same format and thread tracking for app logs
- Added 'Executing query:' log at start of execute() method
- Removed duplicate log statement that was causing queries to appear twice
- executemany() uses existing detailed log (shows parameter set count)
- Each query now logged exactly once at DEBUG level
- Parameters excluded from basic query log (pending PII review)
… (Linux/macOS)
Problem:
- Linux/macOS performed double conversion for NVARCHAR columns
- SQLWCHAR → std::wstring (via SQLWCHARToWString) → Python unicode
- Created unnecessary intermediate std::wstring allocation
Solution:
- Use PyUnicode_DecodeUTF16() to convert UTF-16 directly to Python unicode
- Single-step conversion eliminates intermediate allocation
- Platform-specific optimization (Linux/macOS only)
Impact:
- Reduces memory allocations for wide-character string columns
- Eliminates one full conversion step per NVARCHAR cell
- Regular VARCHAR/CHAR columns unchanged (already optimal)
… (Linux/macOS)
Problem:
- Linux/macOS performed double conversion for NVARCHAR columns
- SQLWCHAR → std::wstring (via SQLWCHARToWString) → Python unicode
- Created unnecessary intermediate std::wstring allocation
Solution:
- Use PyUnicode_DecodeUTF16() to convert UTF-16 directly to Python unicode
- Single-step conversion eliminates intermediate allocation
- Platform-specific optimization (Linux/macOS only)
Impact:
- Reduces memory allocations for wide-character string columns
- Eliminates one full conversion step per NVARCHAR cell
- Regular VARCHAR/CHAR columns unchanged (already optimal)
Problem:
- All numeric conversions used pybind11 wrappers with overhead:
* Type detection, wrapper object creation, bounds checking
* ~20-40 CPU cycles overhead per cell
Solution:
- Use direct Python C API calls:
* PyLong_FromLong/PyLong_FromLongLong for integers
* PyFloat_FromDouble for floats
* PyBool_FromLong for booleans
* PyList_SET_ITEM macro (no bounds check - list pre-sized)
Changes:
- SQL_INTEGER, SQL_SMALLINT, SQL_BIGINT, SQL_TINYINT → PyLong_*
- SQL_BIT → PyBool_FromLong
- SQL_REAL, SQL_DOUBLE, SQL_FLOAT → PyFloat_FromDouble
- Added explicit NULL handling for each type
Impact:
- Eliminates pybind11 wrapper overhead for simple numeric types
- Direct array access via PyList_SET_ITEM macro
- Affects 7 common numeric SQL types
Problem:
--------
Column metadata (dataType, columnSize, isLob, fetchBufferSize) was accessed
from the columnInfos vector inside the hot row processing loop. For a query
with 1,000 rows × 10 columns, this resulted in 10,000 struct field accesses.
Each access involves:
- Vector bounds checking
- Large struct loading (~50+ bytes per ColumnInfo)
- Poor cache locality (struct fields scattered in memory)
- Cost: ~10-15 CPU cycles per access (L2 cache misses likely)
Solution:
---------
Prefetch metadata into tightly-packed local arrays before the row loop:
- std::vector<SQLSMALLINT> dataTypes (2 bytes per element)
- std::vector<SQLULEN> columnSizes (8 bytes per element)
- std::vector<uint64_t> fetchBufferSizes (8 bytes per element)
- std::vector<bool> isLobs (1 byte per element)
Total: ~190 bytes for 10 columns vs 500+ bytes with structs.
These arrays stay hot in L1 cache for the entire batch processing,
eliminating repeated struct access overhead.
Changes:
--------
- Added 4 prefetch vectors before row processing loop
- Added prefetch loop to populate metadata arrays (read columnInfos once)
- Replaced all columnInfos[col-1].field accesses with array lookups
- Updated SQL_CHAR/SQL_VARCHAR cases
- Updated SQL_WCHAR/SQL_WVARCHAR cases
- Updated SQL_BINARY/SQL_VARBINARY cases
Impact:
-------
- Eliminates O(rows × cols) metadata lookups
- 10,000 array accesses @ 3-5 cycles vs 10,000 struct accesses @ 10-15 cycles
- ~70% reduction in metadata access overhead
- Better L1 cache utilization (190 bytes vs 500+ bytes)
- Expected 15-25% overall performance improvement on large result sets
Comment threadmssql_python/pybind/logger_bridge.cpp Outdated
Comment threadmssql_python/pybind/logger_bridge.cpp
Comment threadmssql_python/pybind/logger_bridge.cpp
Comment threadtests/test_007_logging.py Dismissed
Comment threadtests/test_007_logging.py Dismissed
Comment threadmssql_python/pybind/logger_bridge.cpp Dismissed
@bewithgaurav

Gaurav Sharma (bewithgaurav) commented Nov 13, 2025

Copy link
Copy Markdown
CollaboratorAuthor

Coverage Note: LOG Statements Not Covered

Current coverage: ~77% excludes ~150 C++ LOG() statements

Why? Tests don't initialize logging (disabled by default for performance), so LOG statements return early without executing:

if (!s_pythonLogger) return; // Early exit when logging disabled

Attempted solutions:

  1. Exclude via LCOV markers - Add // LCOV_EXCL_LINE to each LOG statement

    • ❌ Requires 150+ manual code changes, clutters codebase
    • ❌ Compromises coverage integrity (hides executable code paths)
    • Note: .coveragerc works for Python but not applicable to C++ LCOV data
  2. Enable logging in coverage tests - pytest.ini flag + conftest setup

    • ❌ Tests got massively slower (CI checks got stuck for >30 mins) - not viable
    • ❌ File I/O overhead from 30k+ log writes unacceptable for CI/CD

Decision: Accept the gap. LOG statements work correctly in production when enabled. Can revisit in future PRs dedicated to increase coverage.

@bewithgauravGaurav Sharma (bewithgaurav) changed the title FEAT: Logging FrameworkFEAT: Comprehensive Logging Framework with Python-C++ Bridge & InstrumentationNov 13, 2025
@bewithgaurav
Gaurav Sharma (bewithgaurav) merged commit 2f309cd into mainNov 13, 2025
25 checks passed
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

pr-size: largeSubstantial code update

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants

@bewithgaurav@github-advanced-security@subrata-ms@sumitmsft