Installing `msal[broker]` fails on Python 3.12: unresolved external symbol PyUnicode_AsUnicode, PyUnicode_FromUnicode

[jiasli]

Describe the bug
Even though #604 declared the support for Python 3.12, installing msal[broker] fails on Python 3.12.

As pymsalruntime doesn't have a GitHub repo, the MSAL repo is the closest to pymsalruntime.

To Reproduce

C:\Users\xxx\AppData\Local\Programs\Python\Python312\python.exe -m venv py312
.\py312\Scripts\Activate.ps1
python.exe -m pip install --upgrade pip
pip install msal[broker]

Error:

Building wheels for collected packages: pymsalruntime
  Building wheel for pymsalruntime (pyproject.toml) ... error
  error: subprocess-exited-with-error

  × Building wheel for pymsalruntime (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [50 lines of output]
      running bdist_wheel
      running build
      running build_py
      creating build
      creating build\lib.win-amd64-cpython-312
      creating build\lib.win-amd64-cpython-312\pymsalruntime
      copying pymsalruntime\__init__.py -> build\lib.win-amd64-cpython-312\pymsalruntime
      copying pymsalruntime\msalruntime.dll -> build\lib.win-amd64-cpython-312\pymsalruntime
      copying pymsalruntime\PyMsalRuntime.pyi -> build\lib.win-amd64-cpython-312\pymsalruntime
      running build_ext
      building 'pymsalruntime.pymsalruntime' extension
      creating build\temp.win-amd64-cpython-312
      creating build\temp.win-amd64-cpython-312\Release
      creating build\temp.win-amd64-cpython-312\Release\pymsalruntime
      "C:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\VC\Tools\MSVC\14.29.30133\bin\HostX86\x64\cl.exe" /c /nologo /O2 /W3 /GL /DNDEBUG /MD -Ibuild_resources -ID:\cli\py312\include -IC:\Users\xxx\AppData\Local\Programs\Python\Python312\include -IC:\Users\xxx\AppData\Local\Programs\Python\Python312\Include "-IC:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\VC\Tools\MSVC\14.29.30133\include" "-IC:\Program Files (x86)\Windows Kits\NETFXSDK\4.8\include\um" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.18362.0\ucrt" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.18362.0\shared" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.18362.0\um" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.18362.0\winrt" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.18362.0\cppwinrt" /Tcpymsalruntime/PyMsalRuntime.c /Fobuild\temp.win-amd64-cpython-312\Release\pymsalruntime/PyMsalRuntime.obj
      PyMsalRuntime.c
      pymsalruntime/PyMsalRuntime.c(2888): warning C4013: 'PyUnicode_FromUnicode' undefined; assuming extern returning int
      pymsalruntime/PyMsalRuntime.c(2888): warning C4047: '=': 'PyObject *' differs in levels of indirection from 'int'
      pymsalruntime/PyMsalRuntime.c(3230): warning C4013: 'PyUnicode_AsUnicode' undefined; assuming extern returning int
      pymsalruntime/PyMsalRuntime.c(3230): warning C4047: '=': 'os_char *' differs in levels of indirection from 'int'
      pymsalruntime/PyMsalRuntime.c(3267): warning C4047: '=': 'os_char *' differs in levels of indirection from 'int'
      pymsalruntime/PyMsalRuntime.c(3616): warning C4047: '=': 'os_char *' differs in levels of indirection from 'int'
      pymsalruntime/PyMsalRuntime.c(3662): warning C4047: '=': 'os_char *' differs in levels of indirection from 'int'
      pymsalruntime/PyMsalRuntime.c(3976): warning C4047: '=': 'os_char *' differs in levels of indirection from 'int'
      pymsalruntime/PyMsalRuntime.c(4325): warning C4047: '=': 'os_char *' differs in levels of indirection from 'int'
      pymsalruntime/PyMsalRuntime.c(4371): warning C4047: '=': 'os_char *' differs in levels of indirection from 'int'
      pymsalruntime/PyMsalRuntime.c(4689): warning C4047: '=': 'os_char *' differs in levels of indirection from 'int'
      pymsalruntime/PyMsalRuntime.c(4726): warning C4047: '=': 'os_char *' differs in levels of indirection from 'int'
      pymsalruntime/PyMsalRuntime.c(5040): warning C4047: '=': 'os_char *' differs in levels of indirection from 'int'
      pymsalruntime/PyMsalRuntime.c(5077): warning C4047: '=': 'os_char *' differs in levels of indirection from 'int'
      pymsalruntime/PyMsalRuntime.c(5402): warning C4047: '=': 'os_char *' differs in levels of indirection from 'int'
      pymsalruntime/PyMsalRuntime.c(5759): warning C4047: '=': 'os_char *' differs in levels of indirection from 'int'
      pymsalruntime/PyMsalRuntime.c(8287): warning C4047: '=': 'os_char *' differs in levels of indirection from 'int'
      pymsalruntime/PyMsalRuntime.c(8324): warning C4047: '=': 'os_char *' differs in levels of indirection from 'int'
      pymsalruntime/PyMsalRuntime.c(8673): warning C4047: '=': 'os_char *' differs in levels of indirection from 'int'
      pymsalruntime/PyMsalRuntime.c(8844): warning C4047: '=': 'os_char *' differs in levels of indirection from 'int'
      pymsalruntime/PyMsalRuntime.c(9014): warning C4047: '=': 'os_char *' differs in levels of indirection from 'int'
      pymsalruntime/PyMsalRuntime.c(9184): warning C4047: '=': 'os_char *' differs in levels of indirection from 'int'
      pymsalruntime/PyMsalRuntime.c(9435): warning C4047: '=': 'os_char *' differs in levels of indirection from 'int'
      pymsalruntime/PyMsalRuntime.c(9472): warning C4047: '=': 'os_char *' differs in levels of indirection from 'int'
      pymsalruntime/PyMsalRuntime.c(9518): warning C4047: '=': 'os_char *' differs in levels of indirection from 'int'
      pymsalruntime/PyMsalRuntime.c(9564): warning C4047: '=': 'os_char *' differs in levels of indirection from 'int'
      pymsalruntime/PyMsalRuntime.c(9789): warning C4047: '=': 'os_char *' differs in levels of indirection from 'int'
      pymsalruntime/PyMsalRuntime.c(9826): warning C4047: '=': 'os_char *' differs in levels of indirection from 'int'
      "C:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\VC\Tools\MSVC\14.29.30133\bin\HostX86\x64\link.exe" /nologo /INCREMENTAL:NO /LTCG /DLL /MANIFEST:EMBED,ID=2 /MANIFESTUAC:NO /LIBPATH:build_resources /LIBPATH:D:\cli\py312\libs /LIBPATH:C:\Users\xxx\AppData\Local\Programs\Python\Python312\libs /LIBPATH:C:\Users\xxx\AppData\Local\Programs\Python\Python312 /LIBPATH:D:\cli\py312\PCbuild\amd64 "/LIBPATH:C:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\VC\Tools\MSVC\14.29.30133\lib\x64" "/LIBPATH:C:\Program Files (x86)\Windows Kits\NETFXSDK\4.8\lib\um\x64" "/LIBPATH:C:\Program Files (x86)\Windows Kits\10\lib\10.0.18362.0\ucrt\x64" "/LIBPATH:C:\Program Files (x86)\Windows Kits\10\lib\10.0.18362.0\um\x64" msalruntime.lib user32.lib /EXPORT:PyInit_pymsalruntime build\temp.win-amd64-cpython-312\Release\pymsalruntime/PyMsalRuntime.obj /OUT:build\lib.win-amd64-cpython-312\pymsalruntime\pymsalruntime.cp312-win_amd64.pyd /IMPLIB:build\temp.win-amd64-cpython-312\Release\pymsalruntime\pymsalruntime.cp312-win_amd64.lib
         Creating library build\temp.win-amd64-cpython-312\Release\pymsalruntime\pymsalruntime.cp312-win_amd64.lib and object build\temp.win-amd64-cpython-312\Release\pymsalruntime\pymsalruntime.cp312-win_amd64.exp
      PyMsalRuntime.obj : error LNK2001: unresolved external symbol PyUnicode_AsUnicode
      PyMsalRuntime.obj : error LNK2001: unresolved external symbol PyUnicode_FromUnicode
      build\lib.win-amd64-cpython-312\pymsalruntime\pymsalruntime.cp312-win_amd64.pyd : fatal error LNK1120: 2 unresolved externals
      error: command 'C:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\BuildTools\\VC\\Tools\\MSVC\\14.29.30133\\bin\\HostX86\\x64\\link.exe' failed with exit code 1120
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for pymsalruntime
Failed to build pymsalruntime
ERROR: Could not build wheels for pymsalruntime, which is required to install pyproject.toml-based projects

[rayluo]

As pymsalruntime doesn't have a GitHub repo, the MSAL repo is the closest to pymsalruntime.

Fair enough. I'll assign this issue to Ashok Kumar Ramakrishnan (@ashok672) , though. :-)

Describe the bug Even though #604 declared the support for Python 3.12, installing msal[broker] fails on Python 3.12.

#604 was about MSAL Python itself being cross-Python-versions. But the msal[broker] package's dependency pymsalruntime does not currently have a build artifact for Python 3.12.

Iulian C. (@iulico-1) , Ashok Kumar Ramakrishnan (@ashok672) and I happened to mention about this known issue earlier today.

[jiasli]

#604 was about MSAL Python itself being cross-Python-versions. But the msal[broker] package's dependency pymsalruntime does not currently have a build artifact for Python 3.12.

Well, this is only part of the reason. Even though pymsalruntime doesn't have a built artifact for Python 3.12, it should be able to compile on Python 3.12 with a C compiler (VC 2019 on my machine). The fact is it does not because of LNK2001:

      PyMsalRuntime.obj : error LNK2001: unresolved external symbol PyUnicode_AsUnicode
      PyMsalRuntime.obj : error LNK2001: unresolved external symbol PyUnicode_FromUnicode

[rayluo]

#604 was about MSAL Python itself being cross-Python-versions. But the msal[broker] package's dependency pymsalruntime does not currently have a build artifact for Python 3.12.

Well, this is only part of the reason. Even though pymsalruntime doesn't have a built artifact for Python 3.12, it should be able to compile on Python 3.12 with a C compiler (VC 2019 on my machine). The fact is it does not because of LNK2001:

      PyMsalRuntime.obj : error LNK2001: unresolved external symbol PyUnicode_AsUnicode
      PyMsalRuntime.obj : error LNK2001: unresolved external symbol PyUnicode_FromUnicode

I heard that we ran into similar issues while attempting to build it with Python 3.12. We will work on it. By the time that we eventually solve this, we shall be able to ship an already-built whl file to PyPI, so that customers do not have to deal with all these.

[AkechiShiro]

Any news on this downstream issue since November 2023, just following up on this comment here : Azure/azure-cli#27673 (comment)

[vairakkumaar-svs-hf]

It does not seems like this was uploaded to pip or the package itself fixed. Its failing in Visual Studio 2022 PY3.12 windows server 2022 x64
What is the working version combination?

[rayluo]

Samy Lahfa (@AkechiShiro) , Vairakkumaar SVS (@vairakkumaar-svs-hf) ,

For now, instead of pip install msal[broker], try this as a workaround.

pip install msal
pip install pymsalruntime

[rayluo]

UPDATE:

• This has technically been fixed in MSAL Python 1.27
• However, the latest PyMsalRuntime 0.14.1 happens to be not quite ready. Downstream apps would need to manually do pip install pymsalruntime==0.14.0 as a short term workaround, until newer patch of pymsalruntime becomes available.