Uh oh!
There was an error while loading. Please reload this page.
FEAT: Comprehensive Logging Framework with Python-C++ Bridge & Instrumentation - #312
Conversation
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
📊 Code Coverage Report
Diff CoverageDiff: main...HEAD, staged and unstaged changes
Summary
mssql_python/auth.pyLines 166-174 166try:
167token=AADAuth.get_token(auth_type)
168logger.info('get_auth_token: Token acquired successfully - auth_type=%s', auth_type)
169returntoken
! 170except (ValueError, RuntimeError) ase:
171logger.warning('get_auth_token: Token acquisition failed - auth_type=%s, error=%s', auth_type, str(e))
172returnNone173mssql_python/helpers.pyLines 39-50 39connection_attributes=connection_str.split(";")
40final_connection_attributes= []
4142# Iterate through the attributes and exclude any existing driver attribute
! 43driver_found=False44forattributeinconnection_attributes:
45ifattribute.lower().split("=")[0] =="driver":
! 46driver_found=True47logger.debug('add_driver_to_connection_str: Existing driver attribute found, removing')
48continue49final_connection_attributes.append(attribute)mssql_python/logging.pyLines 38-48 38"""Add thread_id (OS native) attribute to log record."""39# Use OS native thread ID for debugging compatibility40try:
41thread_id=threading.get_native_id()
! 42exceptAttributeError:
43# Fallback for Python < 3.8
! 44thread_id=threading.current_thread().ident45record.thread_id=thread_id46returnTrue47Lines 128-137 128# Flush and close each handler while holding its lock129forhandlerinold_handlers:
130try:
131handler.flush() # Flush BEFORE close
! 132except:
! 133pass# Ignore flush errors134handler.close()
135self._logger.removeHandler(handler)
136finally:
137# Release locks on old handlersLines 137-146 137# Release locks on old handlers138forhandlerinold_handlers:
139try:
140handler.release()
! 141except:
! 142pass# Handler might already be closed143144self._file_handler=None145self._stdout_handler=None146Lines 154-163 154end_bracket=msg.index(']')
155source=msg[1:end_bracket]
156message=msg[end_bracket+2:].strip() # Skip '] '157else:
! 158source='Unknown'
! 159message=msg160161# Format timestamp with milliseconds using period separator162timestamp=self.formatTime(record, '%Y-%m-%d %H:%M:%S')
163timestamp_with_ms=f"{timestamp}.{int(record.msecs):03d}"Lines 226-234 226""" 227 Reconfigure handlers when output mode changes. 228 Closes existing handlers and creates new ones based on current output mode. 229 """
! 230self._setup_handlers()
231232def_cleanup_handlers(self):
233 """
234Cleanupallhandlersonprocessexit.Lines 250-259 250forhandlerinhandlers:
251try:
252handler.flush()
253handler.close()
! 254except:
! 255pass# Ignore errors during cleanup256self._logger.removeHandler(handler)
257finally:
258forhandlerinhandlers:
259try:Lines 257-266 257finally:
258forhandlerinhandlers:
259try:
260handler.release()
! 261except:
! 262pass263264def_validate_log_file_extension(self, file_path: str) ->None:
265 """
266Validatethatthelogfilehasanallowedextension.Lines 286-294 286WriteCSVheaderandmetadatatothelogfile.
287Calledoncewhenlogfileiscreated.
288 """
289ifnotself._log_fileornotself._file_handler:
! 290return291292try:
293# Get script name from sys.argv or __main__294script_name=os.path.basename(sys.argv[0]) ifsys.argvelse'<interactive>'Lines 298-306 298299# Get driver version (try to import from package)300try:
301frommssql_pythonimport__version__
! 302driver_version=__version__303except:
304driver_version='unknown'305306# Get current timeLines 322-336 322withopen(self._log_file, 'a') asf:
323f.write(header_line)
324f.write(csv_header)
325 ! 326exceptExceptionase:
327# Notify on stderr so user knows why header is missing
! 328try:
! 329sys.stderr.write(f"[MSSQL-Python] Warning: Failed to write log header to {self._log_file}: {type(e).__name__}\n")
! 330sys.stderr.flush()
! 331except:
! 332pass# Even stderr notification failed333# Don't crash - logging continues without header334335def_log(self, level: int, msg: str, add_prefix: bool=True, *args, **kwargs):
336 """Lines 379-388 379importsys380level_name=logging.getLevelName(level)
381sys.stderr.write(f"[MSSQL-Python Logging Failed - {level_name}] {msgif'msg'inlocals() else'Unable to format message'}\n")
382sys.stderr.flush()
! 383except:
! 384pass# Even stderr failed - give up silently385386# Convenience methods for logging387388defdebug(self, msg: str, *args, **kwargs):Lines 470-484 470# Handler management471472defaddHandler(self, handler: logging.Handler):
473"""Add a handler to the logger (thread-safe)"""
! 474withself._handler_lock:
! 475self._logger.addHandler(handler)
476477defremoveHandler(self, handler: logging.Handler):
478"""Remove a handler from the logger (thread-safe)"""
! 479withself._handler_lock:
! 480self._logger.removeHandler(handler)
481482 @property483defhandlers(self) ->list:
484"""Get list of handlers attached to the logger (thread-safe)"""Lines 489-497 489""" 490 Reset/recreate handlers. 491 Useful when log file has been deleted or needs to be recreated. 492 """
! 493self._setup_handlers()
494495def_notify_cpp_level_change(self, level: int):
496 """
497NotifyC++bridgethatloglevelhaschanged.Lines 504-514 504# Import here to avoid circular dependency505from . importddbc_bindings506ifhasattr(ddbc_bindings, 'update_log_level'):
507ddbc_bindings.update_log_level(level)
! 508except (ImportError, AttributeError):
509# C++ bindings not available or not yet initialized
! 510pass511512# Properties513514 @propertyLines 526-543 526527Raises:
528ValueError: IfmodeisnotavalidOutputModevalue529"""! 530 if mode not in (FILE, STDOUT, BOTH):! 531 raise ValueError( 532 f"Invalid output mode: {mode}. " 533 f"Must be one of: {FILE}, {STDOUT}, {BOTH}" 534 )! 535 self._output_mode = mode 536 537 # Only reconfigure if handlers were already initialized! 538 if self._handlers_initialized:! 539 self._reconfigure_handlers() 540 541 @property 542 def log_file(self) -> Optional[str]: 543 """Getthecurrentlogfilepath (Noneiffileoutputisdisabled)"""mssql_python/pybind/connection/connection.cppLines 214-222 214215// Convert to wide string216 std::wstring wstr = Utf8ToWString(utf8_str);
217if (wstr.empty() && !utf8_str.empty()) {
! 218LOG("Failed to convert string value to wide string for attribute=%d", attribute);
219returnSQL_ERROR;
220 }
221this->wstrStringBuffer.clear();
222this->wstrStringBuffer = std::move(wstr);Lines 227-235 227 #ifdefined(__APPLE__) || defined(__linux__)
228 // For macOS/Linux, convert wstring to SQLWCHAR buffer
229 std::vector<SQLWCHAR> sqlwcharBuffer = WStringToSQLWCHAR(this->wstrStringBuffer);
230if (sqlwcharBuffer.empty() && !this->wstrStringBuffer.empty()) {
! 231LOG("Failed to convert wide string to SQLWCHAR buffer for attribute=%d", attribute);
232returnSQL_ERROR;
233 }
234235 ptr = sqlwcharBuffer.data();Lines 243-251 243244SQLRETURN ret = SQLSetConnectAttr_ptr(_dbcHandle->get(),
245 attribute, ptr, length);
246if (!SQL_SUCCEEDED(ret)) {
! 247LOG("Failed to set string attribute=%d, ret=%d", attribute, ret);
248 } else {
249LOG("Set string attribute=%d successfully", attribute);
250 }
251return ret;Lines 249-257 249LOG("Set string attribute=%d successfully", attribute);
250 }
251return ret;
252 } catch (const std::exception& e) {
! 253LOG("Exception during string attribute=%d setting: %s", attribute, e.what());
254returnSQL_ERROR;
255 }
256 } elseif (py::isinstance<py::bytes>(value) ||
257 py::isinstance<py::bytearray>(value)) {Lines 264-282 264265SQLRETURN ret = SQLSetConnectAttr_ptr(_dbcHandle->get(),
266 attribute, ptr, length);
267if (!SQL_SUCCEEDED(ret)) {
! 268LOG("Failed to set binary attribute=%d, ret=%d", attribute, ret);
269 } else {
! 270LOG("Set binary attribute=%d successfully (length=%d)", attribute, length);
271 }
272return ret;
273 } catch (const std::exception& e) {
! 274LOG("Exception during binary attribute=%d setting: %s", attribute, e.what());
275returnSQL_ERROR;
276 }
277 } else {
! 278LOG("Unsupported attribute value type for attribute=%d", attribute);
279returnSQL_ERROR;
280 }
281 }Lines 322-330 322SQL_ATTR_RESET_CONNECTION,
323 (SQLPOINTER)SQL_RESET_CONNECTION_YES,
324SQL_IS_INTEGER);
325if (!SQL_SUCCEEDED(ret)) {
! 326LOG("Failed to reset connection (ret=%d). Marking as dead.", ret);
327disconnect();
328returnfalse;
329 }
330updateLastUsed();mssql_python/pybind/connection/connection_pool.cppLines 71-79 71for (auto& conn : to_disconnect) {
72try {
73 conn->disconnect();
74 } catch (const std::exception& ex) {
! 75LOG("Disconnect bad/expired connections failed: %s", ex.what());
76 }
77 }
78return valid_conn;
79 }Lines 102-110 102for (auto& conn : to_close) {
103try {
104 conn->disconnect();
105 } catch (const std::exception& ex) {
! 106LOG("ConnectionPool::close: disconnect failed: %s", ex.what());
107 }
108 }
109 }mssql_python/pybind/ddbc_bindings.cppLines 293-301 293 !py::isinstance<py::bytes>(param)) {
294ThrowStdException(MakeParamMismatchErrorStr(paramInfo.paramCType, paramIndex));
295 }
296if (paramInfo.isDAE) {
! 297LOG("BindParameters: param[%d] SQL_C_CHAR - Using DAE (Data-At-Execution) for large string streaming", paramIndex);
298 dataPtr = const_cast<void*>(reinterpret_cast<constvoid*>(¶mInfos[paramIndex]));
299 strLenOrIndPtr = AllocateParamBuffer<SQLLEN>(paramBuffers);
300 *strLenOrIndPtr = SQL_LEN_DATA_AT_EXEC(0);
301 bufferLength = 0;Lines 395-403 395 &describedDigits,
396 &nullable
397 );
398if (!SQL_SUCCEEDED(rc)) {
! 399LOG("BindParameters: SQLDescribeParam failed for param[%d] (NULL parameter) - SQLRETURN=%d", paramIndex, rc);
400return rc;
401 }
402 sqlType = describedType;
403 columnSize = describedSize;Lines 606-615 606 }
607 py::bytes uuid_bytes = param.cast<py::bytes>();
608constunsignedchar* uuid_data = reinterpret_cast<constunsignedchar*>(PyBytes_AS_STRING(uuid_bytes.ptr()));
609if (PyBytes_GET_SIZE(uuid_bytes.ptr()) != 16) {
! 610LOG("BindParameters: param[%d] SQL_C_GUID - Invalid UUID length: expected 16 bytes, got %ld bytes", ! 611 paramIndex, PyBytes_GET_SIZE(uuid_bytes.ptr()));
612ThrowStdException("UUID binary data must be exactly 16 bytes long.");
613 }
614SQLGUID* guid_data_ptr = AllocateParamBuffer<SQLGUID>(paramBuffers);
615 guid_data_ptr->Data1 =Lines 655-668 655if (paramInfo.paramCType == SQL_C_NUMERIC) {
656SQLHDESC hDesc = nullptr;
657 rc = SQLGetStmtAttr_ptr(hStmt, SQL_ATTR_APP_PARAM_DESC, &hDesc, 0, NULL);
658if(!SQL_SUCCEEDED(rc)) {
! 659LOG("BindParameters: SQLGetStmtAttr(SQL_ATTR_APP_PARAM_DESC) failed for param[%d] - SQLRETURN=%d", paramIndex, rc);
660return rc;
661 }
662 rc = SQLSetDescField_ptr(hDesc, 1, SQL_DESC_TYPE, (SQLPOINTER) SQL_C_NUMERIC, 0);
663if(!SQL_SUCCEEDED(rc)) {
! 664LOG("BindParameters: SQLSetDescField(SQL_DESC_TYPE) failed for param[%d] - SQLRETURN=%d", paramIndex, rc);
665return rc;
666 }
667SQL_NUMERIC_STRUCT* numericPtr = reinterpret_cast<SQL_NUMERIC_STRUCT*>(dataPtr);
668 rc = SQLSetDescField_ptr(hDesc, 1, SQL_DESC_PRECISION,Lines 667-675 667SQL_NUMERIC_STRUCT* numericPtr = reinterpret_cast<SQL_NUMERIC_STRUCT*>(dataPtr);
668 rc = SQLSetDescField_ptr(hDesc, 1, SQL_DESC_PRECISION,
669 (SQLPOINTER) numericPtr->precision, 0);
670if(!SQL_SUCCEEDED(rc)) {
! 671LOG("BindParameters: SQLSetDescField(SQL_DESC_PRECISION) failed for param[%d] - SQLRETURN=%d", paramIndex, rc);
672return rc;
673 }
674675 rc = SQLSetDescField_ptr(hDesc, 1, SQL_DESC_SCALE,Lines 674-682 674675 rc = SQLSetDescField_ptr(hDesc, 1, SQL_DESC_SCALE,
676 (SQLPOINTER) numericPtr->scale, 0);
677if(!SQL_SUCCEEDED(rc)) {
! 678LOG("BindParameters: SQLSetDescField(SQL_DESC_SCALE) failed for param[%d] - SQLRETURN=%d", paramIndex, rc);
679return rc;
680 }
681682 rc = SQLSetDescField_ptr(hDesc, 1, SQL_DESC_DATA_PTR, (SQLPOINTER) numericPtr, 0);Lines 680-688 680 }
681682 rc = SQLSetDescField_ptr(hDesc, 1, SQL_DESC_DATA_PTR, (SQLPOINTER) numericPtr, 0);
683if(!SQL_SUCCEEDED(rc)) {
! 684LOG("BindParameters: SQLSetDescField(SQL_DESC_DATA_PTR) failed for param[%d] - SQLRETURN=%d", paramIndex, rc);
685return rc;
686 }
687 }
688 }Lines 760-768 760if (pos != std::string::npos) {
761 std::string dir = module_file.substr(0, pos);
762return dir;
763 }
! 764LOG("GetModuleDirectory: Could not extract directory from module path - path='%s'", module_file.c_str());
765return module_file;
766 #endif
767 }Lines 783-792 783 #else784// macOS/Unix: Use dlopen785void* handle = dlopen(driverPath.c_str(), RTLD_LAZY);
786if (!handle) {
! 787LOG("LoadDriverLibrary: dlopen failed for path='%s' - %s", ! 788 driverPath.c_str(), dlerror() ? dlerror() : "unknown error");
789 }
790return handle;
791 #endif
792 }Lines 928-937 928 }
929930 DriverHandle handle = LoadDriverLibrary(driverPath.string());
931if (!handle) {
! 932LOG("LoadDriverOrThrowException: Failed to load ODBC driver - path='%s', error='%s'", ! 933 driverPath.string().c_str(), GetLastErrorMessage().c_str());
934ThrowStdException("Failed to load the driver. Please read the documentation (https://github.com/microsoft/mssql-python#installation) to install the required dependencies.");
935 }
936LOG("LoadDriverOrThrowException: ODBC driver library loaded successfully from '%s'", driverPath.string().c_str());Lines 1311-1319 1311 ErrorInfo SQLCheckError_Wrap(SQLSMALLINT handleType, SqlHandlePtr handle, SQLRETURN retcode) {
1312LOG("SQLCheckError: Checking ODBC errors - handleType=%d, retcode=%d", handleType, retcode);
1313 ErrorInfo errorInfo;
1314if (retcode == SQL_INVALID_HANDLE) {
! 1315LOG("SQLCheckError: SQL_INVALID_HANDLE detected - handle is invalid");
1316 errorInfo.ddbcErrorMsg = std::wstring( L"Invalid handle!");
1317return errorInfo;
1318 }
1319assert(handle != 0);Lines 1319-1327 1319assert(handle != 0);
1320SQLHANDLE rawHandle = handle->get();
1321if (!SQL_SUCCEEDED(retcode)) {
1322if (!SQLGetDiagRec_ptr) {
! 1323LOG("SQLCheckError: SQLGetDiagRec function pointer not initialized, loading driver");
1324DriverLoader::getInstance().loadDriver(); // Load the driver1325 }
13261327SQLWCHAR sqlState[6], message[SQL_MAX_MESSAGE_LENGTH];Lines 1350-1358 1350 py::list SQLGetAllDiagRecords(SqlHandlePtr handle) {
1351LOG("SQLGetAllDiagRecords: Retrieving all diagnostic records for handle %p, handleType=%d", 1352 (void*)handle->get(), handle->type());
1353if (!SQLGetDiagRec_ptr) {
! 1354LOG("SQLGetAllDiagRecords: SQLGetDiagRec function pointer not initialized, loading driver");
1355DriverLoader::getInstance().loadDriver();
1356 }
13571358 py::list records;Lines 1414-1426 1414 }
14151416// Wrap SQLExecDirect1417SQLRETURNSQLExecDirect_wrap(SqlHandlePtr StatementHandle, const std::wstring& Query) {
! 1418 std::string queryUtf8 = WideToUTF8(Query);
! 1419LOG("SQLExecDirect: Executing query directly - statement_handle=%p, query_length=%zu chars", ! 1420 (void*)StatementHandle->get(), Query.length());
1421if (!SQLExecDirect_ptr) {
! 1422LOG("SQLExecDirect: Function pointer not initialized, loading driver");
1423DriverLoader::getInstance().loadDriver(); // Load the driver1424 }
14251426// Configure forward-only cursorLines 1443-1451 1443 queryPtr = const_cast<SQLWCHAR*>(Query.c_str());
1444 #endif
1445SQLRETURN ret = SQLExecDirect_ptr(StatementHandle->get(), queryPtr, SQL_NTS);
1446if (!SQL_SUCCEEDED(ret)) {
! 1447LOG("SQLExecDirect: Query execution failed - SQLRETURN=%d", ret);
1448 }
1449return ret;
1450 }Lines 1456-1464 1456const std::wstring& table,
1457const std::wstring& tableType) {
14581459if (!SQLTables_ptr) {
! 1460LOG("SQLTables: Function pointer not initialized, loading driver");
1461DriverLoader::getInstance().loadDriver();
1462 }
14631464SQLWCHAR* catalogPtr = nullptr;Lines 1541-1549 1541 py::list& isStmtPrepared, constbool usePrepare = true) {
1542LOG("SQLExecute: Executing %s query - statement_handle=%p, param_count=%zu, query_length=%zu chars", 1543 (params.size() > 0 ? "parameterized" : "direct"), (void*)statementHandle->get(), params.size(), query.length());
1544if (!SQLPrepare_ptr) {
! 1545LOG("SQLExecute: Function pointer not initialized, loading driver");
1546DriverLoader::getInstance().loadDriver(); // Load the driver1547 }
1548assert(SQLPrepare_ptr && SQLBindParameter_ptr && SQLExecute_ptr && SQLExecDirect_ptr);Lines 1554-1562 15541555RETCODE rc;
1556SQLHANDLE hStmt = statementHandle->get();
1557if (!statementHandle || !statementHandle->get()) {
! 1558LOG("SQLExecute: Statement handle is null or invalid");
1559 }
15601561// Configure forward-only cursor1562if (SQLSetStmtAttr_ptr && hStmt) {Lines 1594-1602 1594assert(isStmtPrepared.size() == 1);
1595if (usePrepare) {
1596 rc = SQLPrepare_ptr(hStmt, queryPtr, SQL_NTS);
1597if (!SQL_SUCCEEDED(rc)) {
! 1598LOG("SQLExecute: SQLPrepare failed - SQLRETURN=%d, statement_handle=%p", rc, (void*)hStmt);
1599return rc;
1600 }
1601 isStmtPrepared[0] = py::cast(true);
1602 } else {Lines 1659-1668 1659ThrowStdException("Chunk size exceeds maximum allowed by SQLLEN");
1660 }
1661 rc = SQLPutData_ptr(hStmt, (SQLPOINTER)(dataPtr + offset), static_cast<SQLLEN>(lenBytes));
1662if (!SQL_SUCCEEDED(rc)) {
! 1663LOG("SQLExecute: SQLPutData failed for SQL_C_WCHAR chunk - offset=%zu", ! 1664 offset, totalChars, lenBytes, rc);
1665return rc;
1666 }
1667 offset += len;
1668 }Lines 1676-1685 1676size_t len = std::min(chunkBytes, totalBytes - offset);
16771678 rc = SQLPutData_ptr(hStmt, (SQLPOINTER)(dataPtr + offset), static_cast<SQLLEN>(len));
1679if (!SQL_SUCCEEDED(rc)) {
! 1680LOG("SQLExecute: SQLPutData failed for SQL_C_CHAR chunk - offset=%zu", ! 1681 offset, totalBytes, len, rc);
1682return rc;
1683 }
1684 offset += len;
1685 }Lines 1695-1704 1695for (size_t offset = 0; offset < totalBytes; offset += chunkSize) {
1696size_t len = std::min(chunkSize, totalBytes - offset);
1697 rc = SQLPutData_ptr(hStmt, (SQLPOINTER)(dataPtr + offset), static_cast<SQLLEN>(len));
1698if (!SQL_SUCCEEDED(rc)) {
! 1699LOG("SQLExecute: SQLPutData failed for binary/bytes chunk - offset=%zu", ! 1700 offset, totalBytes, len, rc);
1701return rc;
1702 }
1703 }
1704 } else {Lines 1705-1714 1705ThrowStdException("DAE only supported for str or bytes");
1706 }
1707 }
1708if (!SQL_SUCCEEDED(rc)) {
! 1709LOG("SQLExecute: SQLParamData final call %s - SQLRETURN=%d", ! 1710 (rc == SQL_NO_DATA ? "completed with no data" : "failed"), rc);
1711return rc;
1712 }
1713LOG("SQLExecute: DAE streaming completed successfully, SQLExecute resumed");
1714 }Lines 1740-1749 1740const ParamInfo& info = paramInfos[paramIndex];
1741LOG("BindParameterArray: Processing param_index=%d, C_type=%d, SQL_type=%d, column_size=%zu, decimal_digits=%d", 1742 paramIndex, info.paramCType, info.paramSQLType, info.columnSize, info.decimalDigits);
1743if (columnValues.size() != paramSetSize) {
! 1744LOG("BindParameterArray: Size mismatch - param_index=%d, expected=%zu, actual=%zu", ! 1745 paramIndex, paramSetSize, columnValues.size());
1746ThrowStdException("Column " + std::to_string(paramIndex) + " has mismatched size.");
1747 }
1748void* dataPtr = nullptr;
1749SQLLEN* strLenOrIndArray = nullptr;Lines 1767-1775 1767 dataPtr = dataArray;
1768break;
1769 }
1770caseSQL_C_DOUBLE: {
! 1771LOG("BindParameterArray: Binding SQL_C_DOUBLE array - param_index=%d, count=%zu", paramIndex, paramSetSize);
1772double* dataArray = AllocateParamBufferArray<double>(tempBuffers, paramSetSize);
1773for (size_t i = 0; i < paramSetSize; ++i) {
1774if (columnValues[i].is_none()) {
1775if (!strLenOrIndArray)Lines 1780-1788 1780 dataArray[i] = columnValues[i].cast<double>();
1781if (strLenOrIndArray) strLenOrIndArray[i] = 0;
1782 }
1783 }
! 1784LOG("BindParameterArray: SQL_C_DOUBLE bound - param_index=%d", paramIndex);
1785 dataPtr = dataArray;
1786break;
1787 }
1788caseSQL_C_WCHAR: {Lines 1802-1813 1802size_t utf16_len = utf16Buf.size() > 0 ? utf16Buf.size() - 1 : 0;
1803// Check UTF-16 length (excluding null terminator) against column size1804if (utf16Buf.size() > 0 && utf16_len > info.columnSize) {
1805 std::string offending = WideToUTF8(wstr);
! 1806LOG("BindParameterArray: SQL_C_WCHAR string too long - param_index=%d, row=%zu, utf16_length=%zu, max=%zu",
! 1807 paramIndex, i, utf16_len, info.columnSize);
1808ThrowStdException("Input string UTF-16 length exceeds allowed column size at parameter index " + std::to_string(paramIndex) + ! 1809". UTF-16 length: " + std::to_string(utf16_len) + ", Column size: " + std::to_string(info.columnSize));
1810 }
1811// If we reach here, the UTF-16 string fits - copy it completely1812std::memcpy(wcharArray + i * (info.columnSize + 1), utf16Buf.data(), utf16Buf.size() * sizeof(SQLWCHAR));
1813 #elseLines 1838-1847 1838 strLenOrIndArray[i] = SQL_NULL_DATA;
1839 } else {
1840int intVal = columnValues[i].cast<int>();
1841if (intVal < 0 || intVal > 255) {
! 1842LOG("BindParameterArray: TINYINT value out of range - param_index=%d, row=%zu, value=%d",
! 1843 paramIndex, i, intVal);
1844ThrowStdException("UTINYINT value out of range at rowIndex " + std::to_string(i));
1845 }
1846 dataArray[i] = static_cast<unsignedchar>(intVal);
1847if (strLenOrIndArray) strLenOrIndArray[i] = 0;Lines 1864-1873 1864 } else {
1865int intVal = columnValues[i].cast<int>();
1866if (intVal < std::numeric_limits<short>::min() ||
1867 intVal > std::numeric_limits<short>::max()) {
! 1868LOG("BindParameterArray: SHORT value out of range - param_index=%d, row=%zu, value=%d",
! 1869 paramIndex, i, intVal);
1870ThrowStdException("SHORT value out of range at rowIndex " + std::to_string(i));
1871 }
1872 dataArray[i] = static_cast<short>(intVal);
1873if (strLenOrIndArray) strLenOrIndArray[i] = 0;Lines 1890-1901 1890std::memset(charArray + i * (info.columnSize + 1), 0, info.columnSize + 1);
1891 } else {
1892 std::string str = columnValues[i].cast<std::string>();
1893if (str.size() > info.columnSize) {
! 1894LOG("BindParameterArray: String/binary too long - param_index=%d, row=%zu, size=%zu, max=%zu",
! 1895 paramIndex, i, str.size(), info.columnSize);
1896ThrowStdException("Input exceeds column size at index " + std::to_string(i));
! 1897 }
1898std::memcpy(charArray + i * (info.columnSize + 1), str.c_str(), str.size());
1899 strLenOrIndArray[i] = static_cast<SQLLEN>(str.size());
1900 }
1901 }Lines 1904-1912 1904 bufferLength = info.columnSize + 1;
1905break;
1906 }
1907caseSQL_C_BIT: {
! 1908LOG("BindParameterArray: Binding SQL_C_BIT array - param_index=%d, count=%zu", paramIndex, paramSetSize);
1909char* boolArray = AllocateParamBufferArray<char>(tempBuffers, paramSetSize);
1910 strLenOrIndArray = AllocateParamBufferArray<SQLLEN>(tempBuffers, paramSetSize);
1911for (size_t i = 0; i < paramSetSize; ++i) {
1912if (columnValues[i].is_none()) {Lines 1912-1925 1912if (columnValues[i].is_none()) {
1913 boolArray[i] = 0;
1914 strLenOrIndArray[i] = SQL_NULL_DATA;
1915 } else {
! 1916bool val = columnValues[i].cast<bool>();
! 1917 boolArray[i] = val ? 1 : 0;
1918 strLenOrIndArray[i] = 0;
1919 }
1920 }
! 1921LOG("BindParameterArray: SQL_C_BIT bound - param_index=%d", paramIndex);
1922 dataPtr = boolArray;
1923 bufferLength = sizeof(char);
1924break;
1925 }Lines 1924-1932 1924break;
1925 }
1926caseSQL_C_STINYINT:
1927caseSQL_C_USHORT: {
! 1928LOG("BindParameterArray: Binding SQL_C_USHORT/STINYINT array - param_index=%d, count=%zu", paramIndex, paramSetSize);
1929unsignedshort* dataArray = AllocateParamBufferArray<unsignedshort>(tempBuffers, paramSetSize);
1930 strLenOrIndArray = AllocateParamBufferArray<SQLLEN>(tempBuffers, paramSetSize);
1931for (size_t i = 0; i < paramSetSize; ++i) {
1932if (columnValues[i].is_none()) {Lines 1936-1944 1936 dataArray[i] = columnValues[i].cast<unsignedshort>();
1937 strLenOrIndArray[i] = 0;
1938 }
1939 }
! 1940LOG("BindParameterArray: SQL_C_USHORT bound - param_index=%d", paramIndex);
1941 dataPtr = dataArray;
1942 bufferLength = sizeof(unsignedshort);
1943break;
1944 }Lines 1981-1989 1981 bufferLength = sizeof(float);
1982break;
1983 }
1984caseSQL_C_TYPE_DATE: {
! 1985LOG("BindParameterArray: Binding SQL_C_TYPE_DATE array - param_index=%d, count=%zu", paramIndex, paramSetSize);
1986SQL_DATE_STRUCT* dateArray = AllocateParamBufferArray<SQL_DATE_STRUCT>(tempBuffers, paramSetSize);
1987 strLenOrIndArray = AllocateParamBufferArray<SQLLEN>(tempBuffers, paramSetSize);
1988for (size_t i = 0; i < paramSetSize; ++i) {
1989if (columnValues[i].is_none()) {Lines 1996-2004 1996 dateArray[i].day = dateObj.attr("day").cast<SQLUSMALLINT>();
1997 strLenOrIndArray[i] = 0;
1998 }
1999 }
! 2000LOG("BindParameterArray: SQL_C_TYPE_DATE bound - param_index=%d", paramIndex);
2001 dataPtr = dateArray;
2002 bufferLength = sizeof(SQL_DATE_STRUCT);
2003break;
2004 }Lines 2002-2010 2002 bufferLength = sizeof(SQL_DATE_STRUCT);
2003break;
2004 }
2005caseSQL_C_TYPE_TIME: {
! 2006LOG("BindParameterArray: Binding SQL_C_TYPE_TIME array - param_index=%d, count=%zu", paramIndex, paramSetSize);
2007SQL_TIME_STRUCT* timeArray = AllocateParamBufferArray<SQL_TIME_STRUCT>(tempBuffers, paramSetSize);
2008 strLenOrIndArray = AllocateParamBufferArray<SQLLEN>(tempBuffers, paramSetSize);
2009for (size_t i = 0; i < paramSetSize; ++i) {
2010if (columnValues[i].is_none()) {Lines 2017-2025 2017 timeArray[i].second = timeObj.attr("second").cast<SQLUSMALLINT>();
2018 strLenOrIndArray[i] = 0;
2019 }
2020 }
! 2021LOG("BindParameterArray: SQL_C_TYPE_TIME bound - param_index=%d", paramIndex);
2022 dataPtr = timeArray;
2023 bufferLength = sizeof(SQL_TIME_STRUCT);
2024break;
2025 }Lines 2023-2031 2023 bufferLength = sizeof(SQL_TIME_STRUCT);
2024break;
2025 }
2026caseSQL_C_TYPE_TIMESTAMP: {
! 2027LOG("BindParameterArray: Binding SQL_C_TYPE_TIMESTAMP array - param_index=%d, count=%zu", paramIndex, paramSetSize);
2028SQL_TIMESTAMP_STRUCT* tsArray = AllocateParamBufferArray<SQL_TIMESTAMP_STRUCT>(tempBuffers, paramSetSize);
2029 strLenOrIndArray = AllocateParamBufferArray<SQLLEN>(tempBuffers, paramSetSize);
2030for (size_t i = 0; i < paramSetSize; ++i) {
2031if (columnValues[i].is_none()) {Lines 2042-2050 2042 tsArray[i].fraction = static_cast<SQLUINTEGER>(dtObj.attr("microsecond").cast<int>() * 1000); // µs to ns2043 strLenOrIndArray[i] = 0;
2044 }
2045 }
! 2046LOG("BindParameterArray: SQL_C_TYPE_TIMESTAMP bound - param_index=%d", paramIndex);
2047 dataPtr = tsArray;
2048 bufferLength = sizeof(SQL_TIMESTAMP_STRUCT);
2049break;
2050 }Lines 2097-2105 2097 bufferLength = sizeof(DateTimeOffset);
2098break;
2099 }
2100caseSQL_C_NUMERIC: {
! 2101LOG("BindParameterArray: Binding SQL_C_NUMERIC array - param_index=%d, count=%zu", paramIndex, paramSetSize);
2102SQL_NUMERIC_STRUCT* numericArray = AllocateParamBufferArray<SQL_NUMERIC_STRUCT>(tempBuffers, paramSetSize);
2103 strLenOrIndArray = AllocateParamBufferArray<SQLLEN>(tempBuffers, paramSetSize);
2104for (size_t i = 0; i < paramSetSize; ++i) {
2105const py::handle& element = columnValues[i];Lines 2108-2121 2108std::memset(&numericArray[i], 0, sizeof(SQL_NUMERIC_STRUCT));
2109continue;
2110 }
2111if (!py::isinstance<NumericData>(element)) {
! 2112LOG("BindParameterArray: NUMERIC type mismatch - param_index=%d, row=%zu", paramIndex, i);
2113throwstd::runtime_error(MakeParamMismatchErrorStr(info.paramCType, paramIndex));
2114 }
2115 NumericData decimalParam = element.cast<NumericData>();
! 2116LOG("BindParameterArray: NUMERIC value - param_index=%d, row=%zu, precision=%d, scale=%d, sign=%d",
! 2117 paramIndex, i, decimalParam.precision, decimalParam.scale, decimalParam.sign);
2118SQL_NUMERIC_STRUCT& target = numericArray[i];
2119std::memset(&target, 0, sizeof(SQL_NUMERIC_STRUCT));
2120 target.precision = decimalParam.precision;
2121 target.scale = decimalParam.scale;Lines 2125-2133 2125std::memcpy(target.val, decimalParam.val.data(), copyLen);
2126 }
2127 strLenOrIndArray[i] = sizeof(SQL_NUMERIC_STRUCT);
2128 }
! 2129LOG("BindParameterArray: SQL_C_NUMERIC bound - param_index=%d", paramIndex);
2130 dataPtr = numericArray;
2131 bufferLength = sizeof(SQL_NUMERIC_STRUCT);
2132break;
2133 }Lines 2151-2160 2151 }
2152elseif (py::isinstance<py::bytes>(element)) {
2153 py::bytes b = element.cast<py::bytes>();
2154if (PyBytes_GET_SIZE(b.ptr()) != 16) {
! 2155LOG("BindParameterArray: GUID bytes wrong length - param_index=%d, row=%zu, length=%d",
! 2156 paramIndex, i, PyBytes_GET_SIZE(b.ptr()));
2157ThrowStdException("UUID binary data must be exactly 16 bytes long.");
2158 }
2159std::memcpy(uuid_bytes.data(), PyBytes_AS_STRING(b.ptr()), 16);
2160 }Lines 2162-2170 2162 py::bytes b = element.attr("bytes_le").cast<py::bytes>();
2163std::memcpy(uuid_bytes.data(), PyBytes_AS_STRING(b.ptr()), 16);
2164 }
2165else {
! 2166LOG("BindParameterArray: GUID type mismatch - param_index=%d, row=%zu", paramIndex, i);
2167ThrowStdException(MakeParamMismatchErrorStr(info.paramCType, paramIndex));
2168 }
2169 guidArray[i].Data1 = (static_cast<uint32_t>(uuid_bytes[3]) << 24) |
2170 (static_cast<uint32_t>(uuid_bytes[2]) << 16) |Lines 2183-2191 2183 bufferLength = sizeof(SQLGUID);
2184break;
2185 }
2186default: {
! 2187LOG("BindParameterArray: Unsupported C type - param_index=%d, C_type=%d", paramIndex, info.paramCType);
2188ThrowStdException("BindParameterArray: Unsupported C type: " + std::to_string(info.paramCType));
2189 }
2190 }
2191LOG("BindParameterArray: Calling SQLBindParameter - param_index=%d, buffer_length=%lld", Lines 2202-2215 2202 bufferLength,
2203 strLenOrIndArray
2204 );
2205if (!SQL_SUCCEEDED(rc)) {
! 2206LOG("BindParameterArray: SQLBindParameter failed - param_index=%d, SQLRETURN=%d", paramIndex, rc);
2207return rc;
2208 }
2209 }
2210 } catch (...) {
! 2211LOG("BindParameterArray: Exception during binding, cleaning up buffers");
2212throw;
2213 }
2214 paramBuffers.insert(paramBuffers.end(), tempBuffers.begin(), tempBuffers.end());
2215LOG("BindParameterArray: Successfully bound all parameters - total_params=%zu, buffer_count=%zu",Lines 2236-2246 2236LOG("SQLExecuteMany: Using wide string query directly");
2237 #endif
2238RETCODE rc = SQLPrepare_ptr(hStmt, queryPtr, SQL_NTS);
2239if (!SQL_SUCCEEDED(rc)) {
! 2240LOG("SQLExecuteMany: SQLPrepare failed - rc=%d", rc);
! 2241return rc;
! 2242 }
2243LOG("SQLExecuteMany: Query prepared successfully");
22442245bool hasDAE = false;
2246for (constauto& p : paramInfos) {Lines 2254-2270 2254LOG("SQLExecuteMany: Using array binding (non-DAE) - calling BindParameterArray");
2255 std::vector<std::shared_ptr<void>> paramBuffers;
2256 rc = BindParameterArray(hStmt, columnwise_params, paramInfos, paramSetSize, paramBuffers);
2257if (!SQL_SUCCEEDED(rc)) {
! 2258LOG("SQLExecuteMany: BindParameterArray failed - rc=%d", rc);
! 2259return rc;
! 2260 }
22612262 rc = SQLSetStmtAttr_ptr(hStmt, SQL_ATTR_PARAMSET_SIZE, (SQLPOINTER)paramSetSize, 0);
2263if (!SQL_SUCCEEDED(rc)) {
! 2264LOG("SQLExecuteMany: SQLSetStmtAttr(PARAMSET_SIZE) failed - rc=%d", rc);
! 2265return rc;
! 2266 }
2267LOG("SQLExecuteMany: PARAMSET_SIZE set to %zu", paramSetSize);
22682269 rc = SQLExecute_ptr(hStmt);
2270LOG("SQLExecuteMany: SQLExecute completed - rc=%d", rc);Lines 2269-2342 2269 rc = SQLExecute_ptr(hStmt);
2270LOG("SQLExecuteMany: SQLExecute completed - rc=%d", rc);
2271return rc;
2272 } else {
! 2273LOG("SQLExecuteMany: Using DAE (data-at-execution) - row_count=%zu", columnwise_params.size());
2274size_t rowCount = columnwise_params.size();
2275for (size_t rowIndex = 0; rowIndex < rowCount; ++rowIndex) {
! 2276LOG("SQLExecuteMany: Processing DAE row %zu of %zu", rowIndex + 1, rowCount);
2277 py::list rowParams = columnwise_params[rowIndex];
22782279 std::vector<std::shared_ptr<void>> paramBuffers;
2280 rc = BindParameters(hStmt, rowParams, const_cast<std::vector<ParamInfo>&>(paramInfos), paramBuffers);
! 2281if (!SQL_SUCCEEDED(rc)) {
! 2282LOG("SQLExecuteMany: BindParameters failed for row %zu - rc=%d", rowIndex, rc);
! 2283return rc;
! 2284 }
! 2285LOG("SQLExecuteMany: Parameters bound for row %zu", rowIndex);
22862287 rc = SQLExecute_ptr(hStmt);
! 2288LOG("SQLExecuteMany: SQLExecute for row %zu - initial_rc=%d", rowIndex, rc);
! 2289size_t dae_chunk_count = 0;
2290while (rc == SQL_NEED_DATA) {
2291SQLPOINTER token;
2292 rc = SQLParamData_ptr(hStmt, &token);
! 2293LOG("SQLExecuteMany: SQLParamData called - chunk=%zu, rc=%d, token=%p", ! 2294 dae_chunk_count, rc, token);
! 2295if (!SQL_SUCCEEDED(rc) && rc != SQL_NEED_DATA) {
! 2296LOG("SQLExecuteMany: SQLParamData failed - chunk=%zu, rc=%d", dae_chunk_count, rc);
! 2297return rc;
! 2298 }
22992300 py::object* py_obj_ptr = reinterpret_cast<py::object*>(token);
! 2301if (!py_obj_ptr) {
! 2302LOG("SQLExecuteMany: NULL token pointer in DAE - chunk=%zu", dae_chunk_count);
! 2303returnSQL_ERROR;
! 2304 }
23052306if (py::isinstance<py::str>(*py_obj_ptr)) {
2307 std::string data = py_obj_ptr->cast<std::string>();
2308SQLLEN data_len = static_cast<SQLLEN>(data.size());
! 2309LOG("SQLExecuteMany: Sending string DAE data - chunk=%zu, length=%lld", ! 2310 dae_chunk_count, static_cast<longlong>(data_len));
2311 rc = SQLPutData_ptr(hStmt, (SQLPOINTER)data.c_str(), data_len);
! 2312if (!SQL_SUCCEEDED(rc) && rc != SQL_NEED_DATA) {
! 2313LOG("SQLExecuteMany: SQLPutData(string) failed - chunk=%zu, rc=%d", dae_chunk_count, rc);
! 2314 }
2315 } elseif (py::isinstance<py::bytes>(*py_obj_ptr) || py::isinstance<py::bytearray>(*py_obj_ptr)) {
2316 std::string data = py_obj_ptr->cast<std::string>();
2317SQLLEN data_len = static_cast<SQLLEN>(data.size());
! 2318LOG("SQLExecuteMany: Sending bytes/bytearray DAE data - chunk=%zu, length=%lld", ! 2319 dae_chunk_count, static_cast<longlong>(data_len));
2320 rc = SQLPutData_ptr(hStmt, (SQLPOINTER)data.c_str(), data_len);
! 2321if (!SQL_SUCCEEDED(rc) && rc != SQL_NEED_DATA) {
! 2322LOG("SQLExecuteMany: SQLPutData(bytes) failed - chunk=%zu, rc=%d", dae_chunk_count, rc);
! 2323 }
2324 } else {
! 2325LOG("SQLExecuteMany: Unsupported DAE data type - chunk=%zu", dae_chunk_count);
2326returnSQL_ERROR;
2327 }
! 2328 dae_chunk_count++;
2329 }
! 2330LOG("SQLExecuteMany: DAE completed for row %zu - total_chunks=%zu, final_rc=%d", ! 2331 rowIndex, dae_chunk_count, rc);
2332 ! 2333if (!SQL_SUCCEEDED(rc)) {
! 2334LOG("SQLExecuteMany: DAE row %zu failed - rc=%d", rowIndex, rc);
! 2335return rc;
! 2336 }
2337 }
! 2338LOG("SQLExecuteMany: All DAE rows processed successfully - total_rows=%zu", rowCount);
2339returnSQL_SUCCESS;
2340 }
2341 }Lines 2344-2352 2344// Wrap SQLNumResultCols2345SQLSMALLINTSQLNumResultCols_wrap(SqlHandlePtr statementHandle) {
2346LOG("SQLNumResultCols: Getting number of columns in result set for statement_handle=%p", (void*)statementHandle->get());
2347if (!SQLNumResultCols_ptr) {
! 2348LOG("SQLNumResultCols: Function pointer not initialized, loading driver");
2349DriverLoader::getInstance().loadDriver(); // Load the driver2350 }
23512352SQLSMALLINT columnCount;Lines 2358-2366 2358// Wrap SQLDescribeCol2359SQLRETURNSQLDescribeCol_wrap(SqlHandlePtr StatementHandle, py::list& ColumnMetadata) {
2360LOG("SQLDescribeCol: Getting column descriptions for statement_handle=%p", (void*)StatementHandle->get());
2361if (!SQLDescribeCol_ptr) {
! 2362LOG("SQLDescribeCol: Function pointer not initialized, loading driver");
2363DriverLoader::getInstance().loadDriver(); // Load the driver2364 }
23652366SQLSMALLINT ColumnCount;Lines 2366-2374 2366SQLSMALLINT ColumnCount;
2367SQLRETURN retcode =
2368SQLNumResultCols_ptr(StatementHandle->get(), &ColumnCount);
2369if (!SQL_SUCCEEDED(retcode)) {
! 2370LOG("SQLDescribeCol: Failed to get number of columns - SQLRETURN=%d", retcode);
2371return retcode;
2372 }
23732374for (SQLUSMALLINT i = 1; i <= ColumnCount; ++i) {Lines 2450-2460 2450 }
24512452// Wrap SQLFetch to retrieve rows2453SQLRETURNSQLFetch_wrap(SqlHandlePtr StatementHandle) {
! 2454LOG("SQLFetch: Fetching next row for statement_handle=%p", (void*)StatementHandle->get());
2455if (!SQLFetch_ptr) {
! 2456LOG("SQLFetch: Function pointer not initialized, loading driver");
2457DriverLoader::getInstance().loadDriver(); // Load the driver2458 }
24592460returnSQLFetch_ptr(StatementHandle->get());Lines 2487-2495 2487 oss << "Error fetching LOB for column " << colIndex
2488 << ", cType=" << cType
2489 << ", loop=" << loopCount
2490 << ", SQLGetData return=" << ret;
! 2491LOG("FetchLobColumnData: %s", oss.str().c_str());
2492ThrowStdException(oss.str());
2493 }
2494if (actualRead == SQL_NULL_DATA) {
2495LOG("FetchLobColumnData: Column %d is NULL at loop %d", colIndex, loopCount);Lines 2581-2589 2581// Helper function to retrieve column data2582SQLRETURNSQLGetData_wrap(SqlHandlePtr StatementHandle, SQLUSMALLINT colCount, py::list& row) {
2583LOG("SQLGetData: Getting data from %d columns for statement_handle=%p", colCount, (void*)StatementHandle->get());
2584if (!SQLGetData_ptr) {
! 2585LOG("SQLGetData: Function pointer not initialized, loading driver");
2586DriverLoader::getInstance().loadDriver(); // Load the driver2587 }
25882589SQLRETURN ret = SQL_SUCCESS;Lines 2602-2610 26022603 ret = SQLDescribeCol_ptr(hStmt, i, columnName, sizeof(columnName) / sizeof(SQLWCHAR),
2604 &columnNameLen, &dataType, &columnSize, &decimalDigits, &nullable);
2605if (!SQL_SUCCEEDED(ret)) {
! 2606LOG("SQLGetData: Error retrieving metadata for column %d - SQLDescribeCol SQLRETURN=%d", i, ret);
2607 row.append(py::none());
2608continue;
2609 }Lines 2634-2642 2634 row.append(std::string(reinterpret_cast<char*>(dataBuffer.data())));
2635 #endif
2636 } else {
2637// Buffer too small, fallback to streaming
! 2638LOG("SQLGetData: CHAR column %d data truncated (buffer_size=%zu), using streaming LOB", i, dataBuffer.size());
2639 row.append(FetchLobColumnData(hStmt, i, SQL_C_CHAR, false, false));
2640 }
2641 } elseif (dataLen == SQL_NULL_DATA) {
2642LOG("SQLGetData: Column %d is NULL (CHAR)", i);Lines 2643-2658 2643 row.append(py::none());
2644 } elseif (dataLen == 0) {
2645 row.append(py::str(""));
2646 } elseif (dataLen == SQL_NO_TOTAL) {
! 2647LOG("SQLGetData: Cannot determine data length (SQL_NO_TOTAL) for column %d (SQL_CHAR), returning NULL", i);
2648 row.append(py::none());
2649 } elseif (dataLen < 0) {
! 2650LOG("SQLGetData: Unexpected negative data length for column %d - dataType=%d, dataLen=%ld", i, dataType, (long)dataLen);
2651ThrowStdException("SQLGetData returned an unexpected negative data length");
2652 }
2653 } else {
! 2654LOG("SQLGetData: Error retrieving data for column %d (SQL_CHAR) - SQLRETURN=%d, returning NULL", i, ret);
2655 row.append(py::none());
2656 }
2657 }
2658break;Lines 2697-2712 2697 row.append(py::none());
2698 } elseif (dataLen == 0) {
2699 row.append(py::str(""));
2700 } elseif (dataLen == SQL_NO_TOTAL) {
! 2701LOG("SQLGetData: Cannot determine NVARCHAR data length (SQL_NO_TOTAL) for column %d, returning NULL", i);
2702 row.append(py::none());
2703 } elseif (dataLen < 0) {
! 2704LOG("SQLGetData: Unexpected negative data length for column %d (NVARCHAR) - dataLen=%ld", i, (long)dataLen);
2705ThrowStdException("SQLGetData returned an unexpected negative data length");
2706 }
2707 } else {
! 2708LOG("SQLGetData: Error retrieving data for column %d (NVARCHAR) - SQLRETURN=%d", i, ret);
2709 row.append(py::none());
2710 }
2711 }
2712break;Lines 2726-2734 2726 ret = SQLGetData_ptr(hStmt, i, SQL_C_SHORT, &smallIntValue, 0, NULL);
2727if (SQL_SUCCEEDED(ret)) {
2728 row.append(static_cast<int>(smallIntValue));
2729 } else {
! 2730LOG("SQLGetData: Error retrieving SQL_SMALLINT for column %d - SQLRETURN=%d", i, ret);
2731 row.append(py::none());
2732 }
2733break;
2734 }Lines 2737-2745 2737 ret = SQLGetData_ptr(hStmt, i, SQL_C_FLOAT, &realValue, 0, NULL);
2738if (SQL_SUCCEEDED(ret)) {
2739 row.append(realValue);
2740 } else {
! 2741LOG("SQLGetData: Error retrieving SQL_REAL for column %d - SQLRETURN=%d", i, ret);
2742 row.append(py::none());
2743 }
2744break;
2745 }Lines 2780-2793 2780 py::object decimalObj = PythonObjectCache::get_decimal_class()(py::str(cnum, safeLen));
2781 row.append(decimalObj);
2782 } catch (const py::error_already_set& e) {
2783// If conversion fails, append None
! 2784LOG("SQLGetData: Error converting to decimal for column %d - %s", i, e.what());
2785 row.append(py::none());
2786 }
2787 }
2788else {
! 2789LOG("SQLGetData: Error retrieving SQL_NUMERIC/DECIMAL for column %d - SQLRETURN=%d", i, ret);
2790 row.append(py::none());
2791 }
2792break;
2793 }Lines 2798-2806 2798 ret = SQLGetData_ptr(hStmt, i, SQL_C_DOUBLE, &doubleValue, 0, NULL);
2799if (SQL_SUCCEEDED(ret)) {
2800 row.append(doubleValue);
2801 } else {
! 2802LOG("SQLGetData: Error retrieving SQL_DOUBLE/FLOAT for column %d - SQLRETURN=%d", i, ret);
2803 row.append(py::none());
2804 }
2805break;
2806 }Lines 2809-2817 2809 ret = SQLGetData_ptr(hStmt, i, SQL_C_SBIGINT, &bigintValue, 0, NULL);
2810if (SQL_SUCCEEDED(ret)) {
2811 row.append(static_cast<longlong>(bigintValue));
2812 } else {
! 2813LOG("SQLGetData: Error retrieving SQL_BIGINT for column %d - SQLRETURN=%d", i, ret);
2814 row.append(py::none());
2815 }
2816break;
2817 }Lines 2846-2854 2846 timeValue.second
2847 )
2848 );
2849 } else {
! 2850LOG("SQLGetData: Error retrieving SQL_TYPE_TIME for column %d - SQLRETURN=%d", i, ret);
2851 row.append(py::none());
2852 }
2853break;
2854 }Lines 2870-2878 2870 timestampValue.fraction / 1000// Convert back ns to µs2871 )
2872 );
2873 } else {
! 2874LOG("SQLGetData: Error retrieving SQL_TYPE_TIMESTAMP for column %d - SQLRETURN=%d", i, ret);
2875 row.append(py::none());
2876 }
2877break;
2878 }Lines 2919-2927 2919 tzinfo
2920 );
2921 row.append(py_dt);
2922 } else {
! 2923LOG("SQLGetData: Error fetching DATETIMEOFFSET for column %d - SQLRETURN=%d, indicator=%ld", i, ret, (long)indicator);
2924 row.append(py::none());
2925 }
2926break;
2927 }Lines 2952-2964 2952 } else {
2953 std::ostringstream oss;
2954 oss << "Unexpected negative length (" << dataLen << ") returned by SQLGetData. ColumnID="2955 << i << ", dataType=" << dataType << ", bufferSize=" << columnSize;
! 2956LOG("SQLGetData: %s", oss.str().c_str());
2957ThrowStdException(oss.str());
2958 }
2959 } else {
! 2960LOG("SQLGetData: Error retrieving VARBINARY data for column %d - SQLRETURN=%d", i, ret);
2961 row.append(py::none());
2962 }
2963 }
2964break;Lines 2968-2976 2968 ret = SQLGetData_ptr(hStmt, i, SQL_C_TINYINT, &tinyIntValue, 0, NULL);
2969if (SQL_SUCCEEDED(ret)) {
2970 row.append(static_cast<int>(tinyIntValue));
2971 } else {
! 2972LOG("SQLGetData: Error retrieving SQL_TINYINT for column %d - SQLRETURN=%d", i, ret);
2973 row.append(py::none());
2974 }
2975break;
2976 }Lines 2979-2987 2979 ret = SQLGetData_ptr(hStmt, i, SQL_C_BIT, &bitValue, 0, NULL);
2980if (SQL_SUCCEEDED(ret)) {
2981 row.append(static_cast<bool>(bitValue));
2982 } else {
! 2983LOG("SQLGetData: Error retrieving SQL_BIT for column %d - SQLRETURN=%d", i, ret);
2984 row.append(py::none());
2985 }
2986break;
2987 }Lines 3008-3016 3008 row.append(uuid_obj);
3009 } elseif (indicator == SQL_NULL_DATA) {
3010 row.append(py::none());
3011 } else {
! 3012LOG("SQLGetData: Error retrieving SQL_GUID for column %d - SQLRETURN=%d, indicator=%ld", i, ret, (long)indicator);
3013 row.append(py::none());
3014 }
3015break;
3016 }Lines 3018-3026 3018default:
3019 std::ostringstream errorString;
3020 errorString << "Unsupported data type for column - " << columnName << ", Type - "3021 << dataType << ", column ID - " << i;
! 3022LOG("SQLGetData: %s", errorString.str().c_str());
3023ThrowStdException(errorString.str());
3024break;
3025 }
3026 }Lines 3029-3037 30293030SQLRETURNSQLFetchScroll_wrap(SqlHandlePtr StatementHandle, SQLSMALLINT FetchOrientation, SQLLEN FetchOffset, py::list& row_data) {
3031LOG("SQLFetchScroll_wrap: Fetching with scroll orientation=%d, offset=%ld", FetchOrientation, (long)FetchOffset);
3032if (!SQLFetchScroll_ptr) {
! 3033LOG("SQLFetchScroll_wrap: Function pointer not initialized. Loading the driver.");
3034DriverLoader::getInstance().loadDriver(); // Load the driver3035 }
30363037// Unbind any columns from previous fetch operations to avoid memory corruptionLines 3191-3199 3191 std::wstring columnName = columnMeta["ColumnName"].cast<std::wstring>();
3192 std::ostringstream errorString;
3193 errorString << "Unsupported data type for column - " << columnName.c_str()
3194 << ", Type - " << dataType << ", column ID - " << col;
! 3195LOG("SQLBindColums: %s", errorString.str().c_str());
3196ThrowStdException(errorString.str());
3197break;
3198 }
3199if (!SQL_SUCCEEDED(ret)) {Lines 3200-3208 3200 std::wstring columnName = columnMeta["ColumnName"].cast<std::wstring>();
3201 std::ostringstream errorString;
3202 errorString << "Failed to bind column - " << columnName.c_str() << ", Type - "3203 << dataType << ", column ID - " << col;
! 3204LOG("SQLBindColums: %s", errorString.str().c_str());
3205ThrowStdException(errorString.str());
3206return ret;
3207 }
3208 }Lines 3370-3378 3370PyList_SET_ITEM(row, col - 1, Py_None);
3371continue;
3372 } elseif (dataLen < 0) {
3373// Negative value is unexpected, log column index, SQL type & raise exception
! 3374LOG("FetchBatchData: Unexpected negative data length - column=%d, SQL_type=%d, dataLen=%ld", col, dataType, (long)dataLen);
3375ThrowStdException("Unexpected negative data length, check logs for details");
3376 }
3377assert(dataLen > 0 && "Data length must be > 0");Lines 3481-3489 3481 std::wstring columnName = columnMeta["ColumnName"].cast<std::wstring>();
3482 std::ostringstream errorString;
3483 errorString << "Unsupported data type for column - " << columnName.c_str()
3484 << ", Type - " << dataType << ", column ID - " << col;
! 3485LOG("FetchBatchData: %s", errorString.str().c_str());
3486ThrowStdException(errorString.str());
3487break;
3488 }
3489 }Lines 3579-3587 3579 std::wstring columnName = columnMeta["ColumnName"].cast<std::wstring>();
3580 std::ostringstream errorString;
3581 errorString << "Unsupported data type for column - " << columnName.c_str()
3582 << ", Type - " << dataType << ", column ID - " << col;
! 3583LOG("calculateRowSize: %s", errorString.str().c_str());
3584ThrowStdException(errorString.str());
3585break;
3586 }
3587 }Lines 3611-3619 3611// Retrieve column metadata3612 py::list columnNames;
3613 ret = SQLDescribeCol_wrap(StatementHandle, columnNames);
3614if (!SQL_SUCCEEDED(ret)) {
! 3615LOG("FetchMany_wrap: Failed to get column descriptions - SQLRETURN=%d", ret);
3616return ret;
3617 }
36183619 std::vector<SQLUSMALLINT> lobColumns;Lines 3650-3658 36503651// Bind columns3652 ret = SQLBindColums(hStmt, buffers, columnNames, numCols, fetchSize);
3653if (!SQL_SUCCEEDED(ret)) {
! 3654LOG("FetchMany_wrap: Error when binding columns - SQLRETURN=%d", ret);
3655return ret;
3656 }
36573658SQLULEN numRowsFetched;Lines 3660-3668 3660SQLSetStmtAttr_ptr(hStmt, SQL_ATTR_ROWS_FETCHED_PTR, &numRowsFetched, 0);
36613662 ret = FetchBatchData(hStmt, buffers, columnNames, rows, numCols, numRowsFetched, lobColumns);
3663if (!SQL_SUCCEEDED(ret) && ret != SQL_NO_DATA) {
! 3664LOG("FetchMany_wrap: Error when fetching data - SQLRETURN=%d", ret);
3665return ret;
3666 }
36673668// Reset attributes before returning to avoid using stack pointers laterLines 3693-3701 3693// Retrieve column metadata3694 py::list columnNames;
3695 ret = SQLDescribeCol_wrap(StatementHandle, columnNames);
3696if (!SQL_SUCCEEDED(ret)) {
! 3697LOG("FetchAll_wrap: Failed to get column descriptions - SQLRETURN=%d", ret);
3698return ret;
3699 }
37003701// Define a memory limit (1 GB)Lines 3770-3778 37703771// Bind columns3772 ret = SQLBindColums(hStmt, buffers, columnNames, numCols, fetchSize);
3773if (!SQL_SUCCEEDED(ret)) {
! 3774LOG("FetchAll_wrap: Error when binding columns - SQLRETURN=%d", ret);
3775return ret;
3776 }
37773778SQLULEN numRowsFetched;Lines 3826-3834 3826// Wrap SQLMoreResults3827SQLRETURNSQLMoreResults_wrap(SqlHandlePtr StatementHandle) {
3828LOG("SQLMoreResults_wrap: Check for more results");
3829if (!SQLMoreResults_ptr) {
! 3830LOG("SQLMoreResults_wrap: Function pointer not initialized. Loading the driver.");
3831DriverLoader::getInstance().loadDriver(); // Load the driver3832 }
38333834returnSQLMoreResults_ptr(StatementHandle->get());Lines 3835-3845 3835 }
38363837// Wrap SQLFreeHandle3838SQLRETURNSQLFreeHandle_wrap(SQLSMALLINT HandleType, SqlHandlePtr Handle) {
! 3839LOG("SQLFreeHandle_wrap: Free SQL handle type=%d", HandleType);
3840if (!SQLAllocHandle_ptr) {
! 3841LOG("SQLFreeHandle_wrap: Function pointer not initialized. Loading the driver.");
3842DriverLoader::getInstance().loadDriver(); // Load the driver3843 }
38443845SQLRETURN ret = SQLFreeHandle_ptr(HandleType, Handle->get());Lines 3843-3851 3843 }
38443845SQLRETURN ret = SQLFreeHandle_ptr(HandleType, Handle->get());
3846if (!SQL_SUCCEEDED(ret)) {
! 3847LOG("SQLFreeHandle_wrap: SQLFreeHandle failed with error code - %d", ret);
3848return ret;
3849 }
3850return ret;
3851 }Lines 3853-3861 3853// Wrap SQLRowCount3854SQLLENSQLRowCount_wrap(SqlHandlePtr StatementHandle) {
3855LOG("SQLRowCount_wrap: Get number of rows affected by last execute");
3856if (!SQLRowCount_ptr) {
! 3857LOG("SQLRowCount_wrap: Function pointer not initialized. Loading the driver.");
3858DriverLoader::getInstance().loadDriver(); // Load the driver3859 }
38603861SQLLEN rowCount;Lines 3860-3868 38603861SQLLEN rowCount;
3862SQLRETURN ret = SQLRowCount_ptr(StatementHandle->get(), &rowCount);
3863if (!SQL_SUCCEEDED(ret)) {
! 3864LOG("SQLRowCount_wrap: SQLRowCount failed with error code - %d", ret);
3865return ret;
3866 }
3867LOG("SQLRowCount_wrap: SQLRowCount returned %ld", (long)rowCount);
3868return rowCount;Lines 4041-4050 4041mssql_python::logging::LoggerBridge::initialize();
4042 } catch (const std::exception& e) {
4043// Log initialization failure but don't throw4044// Use std::cerr instead of fprintf for type-safe output
! 4045 std::cerr << "Logger bridge initialization failed: " << e.what() << std::endl;
! 4046 }
40474048try {
4049// Try loading the ODBC driver when the module is imported4050LOG("Module initialization: Loading ODBC driver");Lines 4050-4057 4050LOG("Module initialization: Loading ODBC driver");
4051DriverLoader::getInstance().loadDriver(); // Load the driver4052 } catch (const std::exception& e) {
4053// Log the error but don't throw - let the error happen when functions are called
! 4054LOG("Module initialization: Failed to load ODBC driver - %s", e.what());
4055 }
4056 }mssql_python/pybind/logger_bridge.cppLines 26-35 26 std::lock_guard<std::mutex> lock(mutex_);
2728// Skip if already initialized (check inside lock to prevent TOCTOU race)29if (initialized_) {
! 30return;
! 31 }
3233try {
34// Acquire GIL for Python API calls35 py::gil_scoped_acquire gil;Lines 55-68 5556 } catch (const py::error_already_set& e) {
57// Failed to initialize - log to stderr and continue58// (logging will be disabled but won't crash)
! 59 std::cerr << "LoggerBridge initialization failed: " << e.what() << std::endl;
! 60 initialized_ = false;
! 61 } catch (const std::exception& e) {
! 62 std::cerr << "LoggerBridge initialization failed: " << e.what() << std::endl;
! 63 initialized_ = false;
! 64 }
65 }
6667voidLoggerBridge::updateLevel(int level) {
68// Update the cached level atomicallyLines 69-83 69// This is lock-free and can be called from any thread70 cached_level_.store(level, std::memory_order_relaxed);
71 }
72 ! 73intLoggerBridge::getLevel() {
! 74return cached_level_.load(std::memory_order_relaxed);
! 75 }
76 ! 77boolLoggerBridge::isInitialized() {
! 78return initialized_;
! 79 }
8081 std::string LoggerBridge::formatMessage(constchar* format, va_list args) {
82// Use a stack buffer for most messages (4KB should be enough)83char buffer[4096];Lines 92-101 92va_end(args_copy);
9394if (result < 0) {
95// Error during formatting
! 96return"[Formatting error]";
! 97 }
9899if (result < static_cast<int>(sizeof(buffer))) {
100// Message fit in buffer (vsnprintf guarantees null-termination)101returnstd::string(buffer, std::min(static_cast<size_t>(result), sizeof(buffer) - 1));Lines 102-123 102 }
103104// Message was truncated - allocate larger buffer105// (This should be rare for typical log messages)
! 106 std::vector<char> large_buffer(result + 1);
! 107va_copy(args_copy, args);
108// Use std::vsnprintf with explicit size for safety (C++11 standard)109// This is the recommended safe alternative to vsprintf110// DevSkim: ignore DS185832 - std::vsnprintf with size is safe
! 111int final_result = std::vsnprintf(large_buffer.data(), large_buffer.size(), format, args_copy);
! 112va_end(args_copy);
113114// Ensure null termination even if formatting fails
! 115if (final_result < 0 || final_result >= static_cast<int>(large_buffer.size())) {
! 116 large_buffer[large_buffer.size() - 1] = '\0';
! 117 }
118 ! 119return std::string(large_buffer.data());
120 }
121122constchar* LoggerBridge::extractFilename(constchar* path) {
123// Extract just the filename from full path using safer C++ string searchLines 121-130 121122constchar* LoggerBridge::extractFilename(constchar* path) {
123// Extract just the filename from full path using safer C++ string search124if (!path) {
! 125return"";
! 126 }
127128// Find last occurrence of Unix path separator129constchar* filename = std::strrchr(path, '/');
130if (filename) {Lines 131-146 131return filename + 1;
132 }
133134// Try Windows path separator
! 135 filename = std::strrchr(path, '\\');
! 136if (filename) {
! 137return filename + 1;
! 138 }
139140// No path separator found, return the whole string
! 141return path;
! 142 }
143144voidLoggerBridge::log(int level, constchar* file, int line, 145constchar* format, ...) {
146// Fast level check (should already be done by macro, but double-check)Lines 144-158 144voidLoggerBridge::log(int level, constchar* file, int line, 145constchar* format, ...) {
146// Fast level check (should already be done by macro, but double-check)147if (!isLoggable(level)) {
! 148return;
! 149 }
150151// Check if initialized152if (!initialized_ || !cached_logger_) {
! 153return;
! 154 }
155156// Format the message157 va_list args;
158va_start(args, format);Lines 172-184 172// Warn if message exceeds reasonable size (critical for troubleshooting)173constexprsize_tMAX_LOG_SIZE = 4095; // Keep same limit for consistency174if (complete_message.size() > MAX_LOG_SIZE) {
175// Use stderr to notify about truncation (logging may be the truncated call itself)
! 176 std::cerr << "[MSSQL-Python] Warning: Log message truncated from " ! 177 << complete_message.size() << " bytes to " << MAX_LOG_SIZE ! 178 << " bytes at " << file << ":" << line << std::endl;
! 179 complete_message.resize(MAX_LOG_SIZE);
! 180 }
181182// Lock for Python call (minimize critical section)183 std::lock_guard<std::mutex> lock(mutex_);
184Lines 211-226 211212 } catch (const py::error_already_set& e) {
213// Python error during logging - ignore to prevent cascading failures214// (Logging errors should not crash the application)
! 215 (void)e; // Suppress unused variable warning
! 216 } catch (const std::exception& e) {
217// Standard C++ exception - ignore
! 218 (void)e;
! 219 } catch (...) {
220// Catch-all for unknown exceptions (non-standard exceptions, corrupted state, etc.)221// Logging must NEVER crash the application
! 222 }
223 }
224225 } // namespace logging226 } // namespace mssql_python📋 Files Needing Attention📉 Files with overall lowest coverage (click to expand)mssql_python.pybind.logger_bridge.cpp: 59.2%
mssql_python.helpers.py: 66.6%
mssql_python.row.py: 67.4%
mssql_python.pybind.ddbc_bindings.cpp: 70.4%
mssql_python.pybind.connection.connection.cpp: 76.3%
mssql_python.ddbc_bindings.py: 79.6%
mssql_python.pybind.connection.connection_pool.cpp: 79.6%
mssql_python.pybind.ddbc_bindings.h: 79.7%
mssql_python.connection.py: 82.5%
mssql_python.cursor.py: 83.5%🔗 Quick Links
|
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
…nflict and include logger_bridge.hpp in ddbc_bindings.h
- Changed log filename format: timestamp now has no separator (YYYYMMDDHHMMSS) - Added CSV header with metadata: script name, PID, Python version, OS info - Converted log format to CSV: Timestamp, ThreadID, Level, Location, Source, Message - Replaced trace ID with OS native thread ID for debugger compatibility - Updated Python formatter to parse [Python]/[DDBC] tags into Source column - Updated C++ logger_bridge to use makeRecord() for proper file/line attribution - Logs are now easily parseable as CSV for analysis in Excel/pandas - Counter logic for connection/cursor tracking kept internally but not displayed
- Updated LOGGING.md: * Changed log format examples from trace ID to CSV format * Updated filename format (YYYYMMDDHHMMSS with no separators) * Replaced trace ID section with Thread Tracking section * Added CSV parsing examples with pandas * Updated all log output samples to show CSV columns - Updated MSSQL-Python-Logging-Design.md: * Changed file handler config to describe CSV format * Replaced Trace ID System with Thread Tracking System * Updated architecture to reflect OS native thread IDs * Added CSV formatter implementation details * Updated all code examples to use setup_logging() API * Changed log output examples to CSV format - Thread tracking now uses OS native thread IDs (threading.get_native_id()) - CSV columns: Timestamp, ThreadID, Level, Location, Source, Message - File header includes metadata (PID, script name, Python version, etc.) - Easy analysis with pandas/Excel/CSV tools
- CSV format now mentioned only once as optional import capability - Focus on log structure and content, not format - Removed repetitive CSV parsing examples - Single section 'Importing Logs as CSV (Optional)' in LOGGING.md - Brief mention in design doc that format is importable as CSV
Features: - Whitelist log file extensions: .txt, .log, .csv only - Raise ValueError for invalid extensions - Export driver_logger for use in application code - Allow apps to use mssql-python's logger: from mssql_python.logging import driver_logger - Updated documentation with usage examples - Added validation in _setLevel() method Benefits: - Prevents accidental use of wrong file types - Clear error messages for invalid extensions - Unified logging - apps can use same logger as driver - Same format and thread tracking for app logs
- Added 'Executing query:' log at start of execute() method - Removed duplicate log statement that was causing queries to appear twice - executemany() uses existing detailed log (shows parameter set count) - Each query now logged exactly once at DEBUG level - Parameters excluded from basic query log (pending PII review)
… (Linux/macOS) Problem: - Linux/macOS performed double conversion for NVARCHAR columns - SQLWCHAR → std::wstring (via SQLWCHARToWString) → Python unicode - Created unnecessary intermediate std::wstring allocation Solution: - Use PyUnicode_DecodeUTF16() to convert UTF-16 directly to Python unicode - Single-step conversion eliminates intermediate allocation - Platform-specific optimization (Linux/macOS only) Impact: - Reduces memory allocations for wide-character string columns - Eliminates one full conversion step per NVARCHAR cell - Regular VARCHAR/CHAR columns unchanged (already optimal)
… (Linux/macOS) Problem: - Linux/macOS performed double conversion for NVARCHAR columns - SQLWCHAR → std::wstring (via SQLWCHARToWString) → Python unicode - Created unnecessary intermediate std::wstring allocation Solution: - Use PyUnicode_DecodeUTF16() to convert UTF-16 directly to Python unicode - Single-step conversion eliminates intermediate allocation - Platform-specific optimization (Linux/macOS only) Impact: - Reduces memory allocations for wide-character string columns - Eliminates one full conversion step per NVARCHAR cell - Regular VARCHAR/CHAR columns unchanged (already optimal)
Problem: - All numeric conversions used pybind11 wrappers with overhead: * Type detection, wrapper object creation, bounds checking * ~20-40 CPU cycles overhead per cell Solution: - Use direct Python C API calls: * PyLong_FromLong/PyLong_FromLongLong for integers * PyFloat_FromDouble for floats * PyBool_FromLong for booleans * PyList_SET_ITEM macro (no bounds check - list pre-sized) Changes: - SQL_INTEGER, SQL_SMALLINT, SQL_BIGINT, SQL_TINYINT → PyLong_* - SQL_BIT → PyBool_FromLong - SQL_REAL, SQL_DOUBLE, SQL_FLOAT → PyFloat_FromDouble - Added explicit NULL handling for each type Impact: - Eliminates pybind11 wrapper overhead for simple numeric types - Direct array access via PyList_SET_ITEM macro - Affects 7 common numeric SQL types
Problem: -------- Column metadata (dataType, columnSize, isLob, fetchBufferSize) was accessed from the columnInfos vector inside the hot row processing loop. For a query with 1,000 rows × 10 columns, this resulted in 10,000 struct field accesses. Each access involves: - Vector bounds checking - Large struct loading (~50+ bytes per ColumnInfo) - Poor cache locality (struct fields scattered in memory) - Cost: ~10-15 CPU cycles per access (L2 cache misses likely) Solution: --------- Prefetch metadata into tightly-packed local arrays before the row loop: - std::vector<SQLSMALLINT> dataTypes (2 bytes per element) - std::vector<SQLULEN> columnSizes (8 bytes per element) - std::vector<uint64_t> fetchBufferSizes (8 bytes per element) - std::vector<bool> isLobs (1 byte per element) Total: ~190 bytes for 10 columns vs 500+ bytes with structs. These arrays stay hot in L1 cache for the entire batch processing, eliminating repeated struct access overhead. Changes: -------- - Added 4 prefetch vectors before row processing loop - Added prefetch loop to populate metadata arrays (read columnInfos once) - Replaced all columnInfos[col-1].field accesses with array lookups - Updated SQL_CHAR/SQL_VARCHAR cases - Updated SQL_WCHAR/SQL_WVARCHAR cases - Updated SQL_BINARY/SQL_VARBINARY cases Impact: ------- - Eliminates O(rows × cols) metadata lookups - 10,000 array accesses @ 3-5 cycles vs 10,000 struct accesses @ 10-15 cycles - ~70% reduction in metadata access overhead - Better L1 cache utilization (190 bytes vs 500+ bytes) - Expected 15-25% overall performance improvement on large result sets
…icrosoft/mssql-python into bewithgaurav/logging_framework
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Coverage Note: LOG Statements Not CoveredCurrent coverage: ~77% excludes ~150 C++ LOG() statements Why? Tests don't initialize logging (disabled by default for performance), so LOG statements return early without executing: if (!s_pythonLogger) return; // Early exit when logging disabledAttempted solutions:
Decision: Accept the gap. LOG statements work correctly in production when enabled. Can revisit in future PRs dedicated to increase coverage. |
Uh oh!
There was an error while loading. Please reload this page.
Work Item / Issue Reference
Summary
Logging Framework Implementation
Overview
Implements a comprehensive logging system for mssql-python with both Python and C++ components, replacing the previous fragmented approach with a unified, performance-optimized solution.
Key Changes
🎯 Core Logging System
mssql_python/logging.py): Single DEBUG-level logger with file/stdout/both output modesmssql_python/pybind/logger_bridge.cpp): High-performance bridge from C++ to Python logging with zero overhead when disabledsetup_logging(output='file'|'stdout'|'both')- all-or-nothing DEBUG logging for troubleshooting🔧 Technical Improvements
📝 Logging Coverage
warnings.warn()calls with proper logging🧪 Testing & Quality
rownumberassertion (was expectingNone, should be-1)📚 Documentation
learnings/directoryPerformance Impact
Breaking Changes
logging_config.pyandget_logger()APIsetup_logging(output='file')for DEBUG loggingTesting
Related Work