PERF: Handle utf16 strings using simdutf and std::u16string

[ffelixg]

Work Item / Issue Reference

GitHub Issue: #514

---

Summary

• Use external dependency simdutf to decode utf16 when decoding in c++
• Use pybind11 support for std::u16string to decode utf16 when interacting with python
• Remove SQLWCHARToWString, WideToUTF8, Utf8ToWString and WStringToSQLWCHAR; refactor call sites

[ffelixg]

I have been evaluating the external library simdutf as a high performance replacement for utf16 -> utf8 conversions, i.e. the functions SQLWCHARToWString, WideToUTF8 and Utf8ToWString. Rather than only using it for the arrow fetch path, I have been trying to make the switch for every location where one of these three functions is used, as the applications follow similar patterns. I didn't use simdutf in every case, another way to eliminate these function calls was to use std::u16string instead of std::wstring when passing strings to/from python. I think this avoids the whole issue where wchars are defined as 32 bit on some OSes but SQLWCHARs are always 16 bit.

This brings arrow performance on linux for nvarchars in line with what it should be.

If the std::u16string type works as well as I hope it does (will have to see what CI says about mac), there are some more spots where it could be used, for example to replace WStringToSQLWCHAR

[Copilot]

Pull request overview

This PR introduces a higher-performance and more platform-consistent UTF-16LE → UTF-8 conversion path in the pybind ODBC layer by adopting simdutf and shifting several wide-string interfaces from std::wstring to std::u16string (to avoid wchar_t width differences across OSes).

Changes:

• Add simdutf (via find_package or FetchContent) and use it for UTF-16LE → UTF-8 conversions in diagnostics and data fetch paths.
• Replace a number of std::wstring usages with std::u16string for connection strings, queries, and SQL_C_WCHAR parameter buffers.
• Remove legacy SQLWCHAR→wstring conversion utilities that are no longer used.

Reviewed changes

Copilot reviewed 9 out of 9 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
mssql_python/pybind/CMakeLists.txt	Adds simdutf dependency resolution and links it into the ddbc_bindings module.
mssql_python/pybind/ddbc_bindings.h	Introduces UTF-16 helpers (dupeSqlWCharAsUtf16Le, utf16LeToUtf8Alloc) and changes ErrorInfo to store UTF-8 std::string.
mssql_python/pybind/ddbc_bindings.cpp	Switches parameter binding, diagnostics, query execution, and fetch conversions to UTF-16 + simdutf.
mssql_python/pybind/connection/connection.h	Updates connection APIs/state to store connection strings as std::u16string.
mssql_python/pybind/connection/connection.cpp	Uses UTF-16 connection string/query handling and returns UTF-8 error messages directly.
mssql_python/pybind/connection/connection_pool.h	Updates pooling key types and APIs to use std::u16string.
mssql_python/pybind/connection/connection_pool.cpp	Implements pooling with std::u16string connection string keys.
mssql_python/pybind/unix_utils.h	Removes the SQLWCHARToWString declaration.
mssql_python/pybind/unix_utils.cpp	Removes the SQLWCHARToWString implementation.

Comments suppressed due to low confidence (1)

mssql_python/pybind/ddbc_bindings.cpp:583

• In the SQL_C_WCHAR non-DAE path, the bound buffer is a std::u16string but the bind call uses data() + SQL_NTS and sets bufferLength to size() * sizeof(SQLWCHAR). For SQL_NTS, BufferLength should include the null terminator, and the pointer should be a null-terminated buffer (prefer c_str()). As written, drivers that validate BufferLength may treat this as truncated or read past the provided length.

Suggested fix: use sqlwcharBuffer->c_str() and set bufferLength to (sqlwcharBuffer->size() + 1) * sizeof(SQLWCHAR), or alternatively set *strLenOrIndPtr to the explicit byte length (excluding terminator) and keep BufferLength consistent.

 std::u16string* sqlwcharBuffer = AllocateParamBuffer<std::u16string>(
paramBuffers, param.cast<std::u16string>());
LOG("BindParameters: param[%d] SQL_C_WCHAR - String "
"length=%zu characters, buffer=%zu bytes",
paramIndex, sqlwcharBuffer->size(), sqlwcharBuffer->size() * sizeof(SQLWCHAR));
dataPtr = sqlwcharBuffer->data();
bufferLength = sqlwcharBuffer->size() * sizeof(SQLWCHAR);
strLenOrIndPtr = AllocateParamBuffer<SQLLEN>(paramBuffers);
*strLenOrIndPtr = SQL_NTS;

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[gargsaumya]

/azp run

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[github-actions]

📊 Code Coverage Report

🔥 Diff Coverage 94%

🎯 Overall Coverage 25%

📈 Total Lines Covered:6961 out of 27068 📁 Project:mssql-python

---

Diff Coverage

Diff: main...HEAD, staged and unstaged changes

• mssql_python/pybind/connection/connection.cpp (100%)
• mssql_python/pybind/ddbc_bindings.cpp (94.9%): Missing lines 1576,1655,1672,3896,3905,4252,4354
• mssql_python/pybind/utf_utils.h (92.0%): Missing lines 15-16

Summary

• Total: 169 lines
• Missing: 9 lines
• Coverage: 94%

mssql_python/pybind/ddbc_bindings.cpp

Lines 1572-1580

1572LOG("SQLCheckError: Checking ODBC errors - handleType=%d, retcode=%d", handleType, retcode);
1573 ErrorInfo errorInfo;
1574if (retcode == SQL_INVALID_HANDLE) {
1575LOG("SQLCheckError: SQL_INVALID_HANDLE detected - handle is invalid");
! 1576 errorInfo.ddbcErrorMsg = "Invalid handle!";
1577return errorInfo;
1578 }
1579assert(handle != 0);
1580SQLHANDLE rawHandle = handle->get();

---

Lines 1651-1659

1651return records;
1652 }
16531654// Wrap SQLExecDirect
! 1655SQLRETURNSQLExecDirect_wrap(SqlHandlePtr StatementHandle, const std::u16string& Query) {
1656LOG("SQLExecDirect: Executing query directly - statement_handle=%p, "1657"query_length=%zu chars",
1658 (void*)StatementHandle->get(), Query.length());
1659if (!SQLExecDirect_ptr) {

---

Lines 1668-1676

1668SQLSetStmtAttr_ptr(StatementHandle->get(), SQL_ATTR_CONCURRENCY,
1669 (SQLPOINTER)SQL_CONCUR_READ_ONLY, 0);
1670 }
1671 ! 1672SQLWCHAR* queryPtr = reinterpretU16stringAsSqlWChar(Query);
1673SQLRETURN ret;
1674 {
1675// Release the GIL during the blocking ODBC call so that other Python1676// threads (e.g. asyncio event loop, heartbeat threads) can run while

---

Lines 3892-3900

3892sizeof(DateTimeOffset) * fetchSize,
3893 buffers.indicators[col - 1].data());
3894break;
3895default:
! 3896 std::string columnName = columnMeta["ColumnName"].cast<std::string>();
3897 std::ostringstream errorString;
3898 errorString << "Unsupported data type for column - " << columnName.c_str()
3899 << ", Type - " << dataType << ", column ID - " << col;
3900LOG("SQLBindColums: %s", errorString.str().c_str());

---

Lines 3901-3909

3901ThrowStdException(errorString.str());
3902break;
3903 }
3904if (!SQL_SUCCEEDED(ret)) {
! 3905 std::string columnName = columnMeta["ColumnName"].cast<std::string>();
3906 std::ostringstream errorString;
3907 errorString << "Failed to bind column - " << columnName.c_str() << ", Type - "3908 << dataType << ", column ID - " << col;
3909LOG("SQLBindColums: %s", errorString.str().c_str());

---

Lines 4248-4256

4248break;
4249 }
4250default: {
4251constauto& columnMeta = columnNames[col - 1].cast<py::dict>();
! 4252 std::string columnName = columnMeta["ColumnName"].cast<std::string>();
4253 std::ostringstream errorString;
4254 errorString << "Unsupported data type for column - " << columnName.c_str()
4255 << ", Type - " << dataType << ", column ID - " << col;
4256LOG("FetchBatchData: %s", errorString.str().c_str());

---

Lines 4350-4358

4350caseSQL_SS_TIMESTAMPOFFSET:
4351 rowSize += sizeof(DateTimeOffset);
4352break;
4353default:
! 4354 std::string columnName = columnMeta["ColumnName"].cast<std::string>();
4355 std::ostringstream errorString;
4356 errorString << "Unsupported data type for column - " << columnName.c_str()
4357 << ", Type - " << dataType << ", column ID - " << col;
4358LOG("calculateRowSize: %s", errorString.str().c_str());

---

mssql_python/pybind/utf_utils.h

 11 #include <string>
12 13 inline std::string utf16LeToUtf8Alloc(const std::u16string& utf16) {
14 if (utf16.empty()) {
! 15 return {};
! 16 }
17 18 std::string utf8(utf16.size() * 3, '\0');
19 size_t n = simdutf::convert_utf16le_to_utf8_with_replacement(
20 utf16.data(), utf16.size(), utf8.data());

---

---

📋 Files Needing Attention

📉 Files with overall lowest coverage (click to expand)

mssql_python.pybind.build._deps.simdutf-src.src.haswell.implementation.cpp: 0.4%
mssql_python.pybind.build._deps.simdutf-src.src.implementation.cpp: 6.7%
mssql_python.pybind.build._deps.simdutf-src.include.simdutf.implementation.h: 10.4%
mssql_python.pybind.build._deps.simdutf-src.include.simdutf.scalar.utf16_to_utf8.utf16_to_utf8.h: 25.3%
mssql_python.pybind.logger_bridge.cpp: 59.2%
mssql_python.pybind.ddbc_bindings.h: 59.7%
mssql_python.pybind.build._deps.simdutf-src.include.simdutf.internal.isadetection.h: 65.3%
mssql_python.row.py: 70.5%
mssql_python.pybind.logger_bridge.hpp: 70.8%
mssql_python.pybind.ddbc_bindings.cpp: 74.2%

---

🔗 Quick Links

⚙️ Build Summary	📋 Coverage Details
View Azure DevOps Build	Browse Full Coverage Report

[ffelixg]

Only issue seems to be that some Linux CI containers don't have git. I'm trying to fetch simdutf via url instead.

[gargsaumya]

/azp run

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[ffelixg]

gargsaumya The second CI error was due to the ubuntu image using an older version of cmake. Should be fine now I hope.

I have also replaced the remaining std::wstring occurences with std::u16string and eliminated helper functions as well as platform specific code accordingly. Let me know if you are happy with this holistic update to utf16 string handling or if you would prefer to keep it contained to the arrow fetch path.

[gargsaumya]

Thanks for the update ffelixg. I actually prefer this and it LGTM overall. ButI’ll still take a closer look at the changes and get back to you.
I’ve approved the workflow for now so we can run the pipeline and get a passing test run.

[gargsaumya]

/azp run

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[ffelixg]

I think that this latest CI failure was due to a flaky test. The same test / platform passed in a past CI run and I don't think my changes affected that test. Could you rerun that one gargsaumya?

[gargsaumya]

/azp run

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[gargsaumya]

Really appreciate the effort here, this is a very meaningful contribution ffelixg. I've reviewed the PR and left 2 inline comments, please take a look!

[Gaurav Sharma (bewithgaurav)]

one real bug (diag message overflow, stack memory leaking into exception text on long error messages, repro inline), one accidental README typo, one perf nit on the new helper. only the diag bug actually blocks, the other two can ride along or land separately.

[gargsaumya]

/azp run

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[Gaurav Sharma (bewithgaurav)]

ffelixg - thanks for your responses - requesting you to please resolve conflicts and we'll do a last set of review

[ffelixg]

Gaurav Sharma (@bewithgaurav) Done, I also noticed that a couple utf16LeToUtf8Alloc calls could be replaced by std::u16string + pybind11 while I was doing the conflict resolution.

[gargsaumya]

/azp run

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[Gaurav Sharma (bewithgaurav)]

lgtm, great stuff - thanks ffelixg !