CHORE: Add standalone mssql-python-odbc package for ODBC driver binaries

[Jahnvi Thakkar (jahnvi480)]

Work Item / Issue Reference

AB#46217

---

Summary

Splits the ODBC driver binaries out of mssql-python into a new standalone, pure-data package mssql-python-odbc (import name mssql_python_odbc, version 18.6.0). This is the code / packaging half of the split; the ADO release pipelines are in companion PR #664.

What is included

• mssql_python_odbc/__init__.py - pure-Python package (no native extension) that ships the ODBC driver libs/ tree and exposes get_driver_path() and get_libs_dir() with the same per-platform layout the C++ loader expects.
• setup_odbc.py - builds the mssql-python-odbc wheels. Each wheel is platform-specific but Python-agnostic (py3-none-<platform>), so we ship 7 wheels total (Windows amd64/arm64, macOS universal2, manylinux_2_28 x86_64/aarch64, musllinux_1_2 x86_64/aarch64) instead of a per-Python matrix. On Windows the wheel intentionally ships the VC++ runtime (vcredist/msvcp140.dll + license) alongside the driver binaries - see "VC++ runtime" below.
• mssql_python/pybind/ddbc_bindings.cpp - LoadDriverOrThrowException() now resolves the driver / libs base directory via a new GetOdbcLibsBaseDir() helper, which imports mssql_python_odbc when present and falls back to the bundled mssql_python libs when it is not. The fallback is GIL-safe and Alpine/musl-safe.
• tests/test_024_odbc_package_split.py - verifies Python vs C++ driver-path parity.
• .gitignore - ignores local mssql_python_odbc/libs/ build copies and egg-info.

VC++ runtime (vcredist) on Windows

The Windows wheel ships libs/windows/<arch>/vcredist/msvcp140.dll (with its license). This is a deliberate, documented choice - not "parity" with the main wheel's file layout, and not a reversal of a licensing/size decision:

• The main mssql-python wheel also ships the VC++ runtime, just relocated: build.bat copies msvcp140.dll out of vcredist/ to the package root next to the compiled ddbc_bindings extension, and setup.py then excludes the now-duplicate vcredist/ folder. That exclusion is de-duplication, not a removal of the runtime.
• This package has no compiled extension, so there is no package-root .pyd to place the runtime beside. Keeping msvcp140.dll in its original vcredist/ folder keeps the driver binaries and the runtime they were built against together, and keeps the wheel self-contained.
• No Phase-2 runtime skew: msodbcsql18.dll's dependency on msvcp140.dll is satisfied in-process by the mssql-python extension (built /MD, which loads msvcp140.dll from beside the .pyd), so the external-package and bundled-libs paths behave identically. This copy is completeness + attribution, which is why GetOdbcLibsBaseDir()'s completeness check verifies the driver + mssql-auth.dll but not vcredist.

Phasing (no breaking change in this PR)

• Phase 2 (this PR): mssql-python prefers the external package but keeps bundling libs/ as a fallback, so existing installs keep working.
• Phase 3 (future, v2.0.0): remove the bundled libs/ and depend on mssql-python-odbc exclusively.

Testing

• test_024 parity suite: 7 passed, 1 skipped.
• Built mssql_python_odbc-18.6.0-py3-none-win_amd64.whl locally: correct wheel tag, Root-Is-Purelib: false, driver DLLs present, vcredist (msvcp140.dll + license) present, twine check passes.

Notes

Companion PR (release pipelines): #664. Infra registration (ESRP onboarding of mssql-python-odbc, ADO build-definition id) is tracked there.

[Copilot]

Pull request overview

Splits the bundled ODBC driver binaries out of mssql-python into a new standalone data-only package (mssql-python-odbc / mssql_python_odbc), updates the native loader to prefer the external package with a fallback, and adds parity tests to ensure Python and C++ resolve identical driver paths.

Changes:

• Add new mssql_python_odbc package with get_driver_path() / get_libs_dir() and platform/arch/distro detection mirroring the C++ resolver.
• Add setup_odbc.py to build platform-specific, Python-agnostic wheels for the ODBC-binaries package, including a local “sync libs” convenience.
• Update ddbc_bindings.cpp to resolve the ODBC libs base dir via mssql_python_odbc when installed, with fallback to bundled mssql_python libs; add tests validating Python/C++ path parity.

Reviewed changes

Copilot reviewed 4 out of 5 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
tests/test_024_odbc_package_split.py	Adds test coverage validating the new package’s API and Python/C++ driver-path parity.
setup_odbc.py	Introduces a dedicated wheel build script for the new ODBC-binaries package and a transition-time libs sync.
mssql_python/pybind/ddbc_bindings.cpp	Switches native driver/libs base-dir resolution to prefer mssql_python_odbc, falling back to bundled libs.
mssql_python_odbc/__init__.py	Implements the new Python API for locating driver binaries within the standalone package.
.gitignore	Ignores transition-time generated/copy artifacts for the new package (libs/, egg-info).

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[github-actions]

📊 Code Coverage Report

🔥 Diff Coverage 51%

🎯 Overall Coverage 80%

📈 Total Lines Covered:6759 out of 8369 📁 Project:mssql-python

---

Diff Coverage

Diff: main...HEAD, staged and unstaged changes

• mssql_python/pybind/ddbc_bindings.cpp (51.4%): Missing lines 1136-1138,1142,1146-1148,1151-1153,1175-1176,1182,1195-1198,1331

Summary

• Total: 37 lines
• Missing: 18 lines
• Coverage: 51%

mssql_python/pybind/ddbc_bindings.cpp

Lines 1132-1157

1132// its directory as the base that `GetDriverPathCpp` (and the Windows1133// `mssql-auth.dll` lookup) append `libs` to.1134//1135// During the Phase-2 transition we fall back to the bundled `mssql_python`
! 1136// directory when the external package is not installed, so a wheel that still
! 1137// bundles `libs/` keeps working. Importing `mssql_python_odbc` here is
! 1138// Alpine/musl-safe precisely because it is a separate pure package: it cannot1139// trigger the partially-initialized-module circular import that motivated1140// resolving these paths in C++ in the first place.1141//
! 1142// (`GetDriverPathCpp` is defined further below; forward-declared here so we can1143// verify the external package actually ships this platform's driver binary.)1144 std::string GetDriverPathCpp(const std::string& moduleDir);
1145 ! 1146 std::string GetOdbcLibsBaseDir() {
! 1147namespacefs= std::filesystem;
! 1148// This function calls into the Python C-API (py::module::import, attribute1149// access, casts), so it must run with the GIL held. It is a no-op when the1150// GIL is already held — which it is at the sole current call site, during
! 1151// module initialization — but acquiring it here self-documents the C-API
! 1152// dependency and keeps a future GIL-released caller from turning this into a
! 1153// hard crash.1154 py::gil_scoped_acquire gil;
1155try {
1156 py::object module = py::module::import("mssql_python_odbc");
1157 py::object module_path = module.attr("__file__");

---

Lines 1171-1180

1171// unconditionally. Verifying both here keeps this resolver's notion of a1172// usable base dir consistent with what the loader below actually needs,1173// so we never select a directory that would later make the loader throw1174// (e.g. a dir that has msodbcsql18.dll but is missing mssql-auth.dll).
! 1175 std::error_code ec;
! 1176 fs::path externalDriver(GetDriverPathCpp(parentDir.string()));
1177bool externalComplete = fs::exists(externalDriver, ec);
1178 #ifdef _WIN32
1179if (externalComplete) {
1180 fs::path externalAuthDll = externalDriver.parent_path() / "mssql-auth.dll";

---

Lines 1178-1186

1178 #ifdef _WIN32
1179if (externalComplete) {
1180 fs::path externalAuthDll = externalDriver.parent_path() / "mssql-auth.dll";
1181 externalComplete = fs::exists(externalAuthDll, ec);
! 1182 }
1183 #endif
1184if (externalComplete) {
1185LOG("GetOdbcLibsBaseDir: Using external mssql_python_odbc package - directory='%s'",
1186 parentDir.string().c_str());

---

Lines 1191-1202

1191 parentDir.string().c_str());
1192return GetModuleDirectory();
1193 } catch (const py::error_already_set& e) {
1194if (e.matches(PyExc_ModuleNotFoundError)) {
! 1195// Expected in Phase 2 when the standalone package is not installed.
! 1196// pybind11 has already fetched and cleared the CPython error
! 1197// indicator, so re-importing `mssql_python` below is safe.
! 1198LOG("GetOdbcLibsBaseDir: mssql_python_odbc not installed (%s); "1199"falling back to bundled libs in mssql_python",
1200 e.what());
1201returnGetModuleDirectory();
1202 }

---

Lines 1327-1335

1327 platform = "windows";
1328// Normalize x86_64 to x64 for Windows naming1329if (arch == "x86_64")
1330 arch = "x64";
! 1331 fs::path driverPath =
1332 basePath / "libs" / platform / arch / ("msodbcsql"MSODBCSQL_VERSION_MAJOR".dll");
1333return driverPath.string();
13341335 #else

---

---

📋 Files Needing Attention

📉 Files with overall lowest coverage (click to expand)

mssql_python.pybind.logger_bridge.cpp: 59.2%
mssql_python.pybind.ddbc_bindings.h: 59.9%
mssql_python.pybind.logger_bridge.hpp: 70.8%
mssql_python.pybind.ddbc_bindings.cpp: 76.1%
mssql_python.pybind.connection.connection.cpp: 76.2%
mssql_python.__init__.py: 77.3%
mssql_python.row.py: 77.6%
mssql_python.ddbc_bindings.py: 79.6%
mssql_python.connection.py: 83.6%
mssql_python.logging.py: 85.5%

---

🔗 Quick Links

⚙️ Build Summary	📋 Coverage Details
View Azure DevOps Build	Browse Full Coverage Report

[Gaurav Sharma (bewithgaurav)]

some duplicacies are there & need to be resolved, comments added inline
requesting changes

[Gaurav Sharma (bewithgaurav)]

lgtm, lets do the version pinning once we're published and ready to link