P1502R1 Standard Library Header Units

[StephanTLavavej]

📄 Paper

• P1502R1 Standard Library Header Units

🚧 Status

This feature will be ready for production use in VS 2019 16.10 Preview 2, for MSVC only. See #1694 for more details.

[StephanTLavavej]

It's unclear to me whether any changes to library product code will be necessary. When compiler support is available, we should presumably add tests similar to the Standard's example:

import <vector>;     // imports the <vector> header unit
std::vector<int> vi; // OK

[StephanTLavavej]

After talking with Cameron DaCamara (@cdacamar), I learned that users won't be able to control standard library header units via macros defined in source code (or separate headers), but they'll still be able to define macros on the command line. Additionally, Standard Library headers will be able to define macros for internal and external use.

We still need to determine how the standard library header units will be built - we may need to build and distribute them for our default configurations (static/dynamic release/debug), or the VS IDE and user build systems may build them on demand (and will need to, for any custom configurations, including warning suppression).

[StephanTLavavej]

Cameron DaCamara (@cdacamar) found one issue involving macros that the STL drags in. Quoted with his permission:

Using header units has produced an interesting interaction between macro definitions in a header unit vs. header inclusion order in translation units.
One example: io.hxx (header unit) and io.cxx (TU) produces:
'_Outptr_': macro redefinition
'_Outptr_result_maybenull_': macro redefinition
...
'__fallthrough': macro redefinition
'SIZE_MAX': macro redefinition
'_At_': macro redefinition
...
'__analysis_assume': macro redefinition
Just grepping through the Windows SDK and our own headers, it looks like there is a difference in files like Windows/10/sdk/inc/sal.h and Windows/10/sdk/inc/specstrings_strict.h. Because the .cpp file compiles with <windows.h> included before the header unit import I suspect that the compiler sees a different definition for the majority of these macros.

I'm not sure what the STL can do about this.

[StephanTLavavej]

Repro:

C:\Temp>type main.cpp
import <vector>;
import <iostream>;
int main() { }

C:\Temp>cl /EHsc /nologo /W4 /std:c++latest /MD /experimental:module /module:showResolvedHeader /module:exportHeader "<iostream>" /Fo:iostream.obj
<iostream>
Note: resolved <iostream> to 'S:\GitHub\STL\out\build\x86\out\inc\iostream'

C:\Temp>cl /EHsc /nologo /W4 /std:c++latest /MD /experimental:module /module:showResolvedHeader /module:exportHeader "<vector>" /Fo:vector.obj
<vector>
Note: resolved <vector> to 'S:\GitHub\STL\out\build\x86\out\inc\vector'

C:\Temp>cl /EHsc /nologo /W4 /std:c++latest /MD /experimental:module /headerUnit S:\GitHub\STL\out\build\x86\out\inc\iostream=iostream.ifc /headerUnit S:\GitHub\STL\out\build\x86\out\inc\vector=vector.ifc main.cpp
main.cpp
main.cpp(2): warning C4005: 'SIZE_MAX': macro redefinition
main.cpp(1): note: see previous definition of 'SIZE_MAX'

VCRuntime's limits.h defines:

#ifndef SIZE_MAX
    #ifdef _WIN64
        #define SIZE_MAX _UI64_MAX
    #else
        #define SIZE_MAX UINT_MAX
    #endif
#endif

While VCRuntime's stdint.h defines:

#ifndef SIZE_MAX
    #define SIZE_MAX     UINTPTR_MAX
#endif

In the modules scenario, the #ifndef SIZE_MAX aren't sufficient to prevent the conflict.

[StephanTLavavej]

Testing all Standard headers (I excluded <strstream> due to deprecation) indicates that SIZE_MAX is the only such macro:

C:\Temp>type main.cpp
import <algorithm>;
import <any>;
import <array>;
import <atomic>;
import <bit>;
import <bitset>;
import <charconv>;
import <chrono>;
import <codecvt>;
import <compare>;
import <complex>;
import <concepts>;
import <condition_variable>;
import <coroutine>;
import <deque>;
import <exception>;
import <execution>;
import <filesystem>;
import <forward_list>;
import <fstream>;
import <functional>;
import <future>;
import <initializer_list>;
import <iomanip>;
import <ios>;
import <iosfwd>;
import <iostream>;
import <istream>;
import <iterator>;
import <limits>;
import <list>;
import <locale>;
import <map>;
import <memory_resource>;
import <memory>;
import <mutex>;
import <new>;
import <numbers>;
import <numeric>;
import <optional>;
import <ostream>;
import <queue>;
import <random>;
import <ranges>;
import <ratio>;
import <regex>;
import <scoped_allocator>;
import <set>;
import <shared_mutex>;
import <span>;
import <sstream>;
import <stack>;
import <stdexcept>;
import <streambuf>;
import <string_view>;
import <string>;
import <system_error>;
import <thread>;
import <tuple>;
import <type_traits>;
import <typeindex>;
import <typeinfo>;
import <unordered_map>;
import <unordered_set>;
import <utility>;
import <valarray>;
import <variant>;
import <vector>;
import <version>;

int main() { }

C:\Temp>type cl.rsp
/EHsc /nologo /W4 /std:c++latest /MD /experimental:module /headerUnit S:\GitHub\STL\out\build\x86\out\inc\algorithm=algorithm.ifc /headerUnit S:\GitHub\STL\out\build\x86\out\inc\any=any.ifc /headerUnit S:\GitHub\STL\out\build\x86\out\inc\array=array.ifc /headerUnit S:\GitHub\STL\out\build\x86\out\inc\atomic=atomic.ifc /headerUnit S:\GitHub\STL\out\build\x86\out\inc\bit=bit.ifc /headerUnit S:\GitHub\STL\out\build\x86\out\inc\bitset=bitset.ifc /headerUnit S:\GitHub\STL\out\build\x86\out\inc\charconv=charconv.ifc /headerUnit S:\GitHub\STL\out\build\x86\out\inc\chrono=chrono.ifc /headerUnit S:\GitHub\STL\out\build\x86\out\inc\codecvt=codecvt.ifc /headerUnit S:\GitHub\STL\out\build\x86\out\inc\compare=compare.ifc /headerUnit S:\GitHub\STL\out\build\x86\out\inc\complex=complex.ifc /headerUnit S:\GitHub\STL\out\build\x86\out\inc\concepts=concepts.ifc /headerUnit S:\GitHub\STL\out\build\x86\out\inc\condition_variable=condition_variable.ifc /headerUnit S:\GitHub\STL\out\build\x86\out\inc\coroutine=coroutine.ifc /headerUnit S:\GitHub\STL\out\build\x86\out\inc\deque=deque.ifc /headerUnit S:\GitHub\STL\out\build\x86\out\inc\exception=exception.ifc /headerUnit S:\GitHub\STL\out\build\x86\out\inc\execution=execution.ifc /headerUnit S:\GitHub\STL\out\build\x86\out\inc\filesystem=filesystem.ifc /headerUnit S:\GitHub\STL\out\build\x86\out\inc\forward_list=forward_list.ifc /headerUnit S:\GitHub\STL\out\build\x86\out\inc\fstream=fstream.ifc /headerUnit S:\GitHub\STL\out\build\x86\out\inc\functional=functional.ifc /headerUnit S:\GitHub\STL\out\build\x86\out\inc\future=future.ifc /headerUnit S:\GitHub\STL\out\build\x86\out\inc\initializer_list=initializer_list.ifc /headerUnit S:\GitHub\STL\out\build\x86\out\inc\iomanip=iomanip.ifc /headerUnit S:\GitHub\STL\out\build\x86\out\inc\ios=ios.ifc /headerUnit S:\GitHub\STL\out\build\x86\out\inc\iosfwd=iosfwd.ifc /headerUnit S:\GitHub\STL\out\build\x86\out\inc\iostream=iostream.ifc /headerUnit S:\GitHub\STL\out\build\x86\out\inc\istream=istream.ifc /headerUnit S:\GitHub\STL\out\build\x86\out\inc\iterator=iterator.ifc /headerUnit S:\GitHub\STL\out\build\x86\out\inc\limits=limits.ifc /headerUnit S:\GitHub\STL\out\build\x86\out\inc\list=list.ifc /headerUnit S:\GitHub\STL\out\build\x86\out\inc\locale=locale.ifc /headerUnit S:\GitHub\STL\out\build\x86\out\inc\map=map.ifc /headerUnit S:\GitHub\STL\out\build\x86\out\inc\memory_resource=memory_resource.ifc /headerUnit S:\GitHub\STL\out\build\x86\out\inc\memory=memory.ifc /headerUnit S:\GitHub\STL\out\build\x86\out\inc\mutex=mutex.ifc /headerUnit S:\GitHub\STL\out\build\x86\out\inc\new=new.ifc /headerUnit S:\GitHub\STL\out\build\x86\out\inc\numbers=numbers.ifc /headerUnit S:\GitHub\STL\out\build\x86\out\inc\numeric=numeric.ifc /headerUnit S:\GitHub\STL\out\build\x86\out\inc\optional=optional.ifc /headerUnit S:\GitHub\STL\out\build\x86\out\inc\ostream=ostream.ifc /headerUnit S:\GitHub\STL\out\build\x86\out\inc\queue=queue.ifc /headerUnit S:\GitHub\STL\out\build\x86\out\inc\random=random.ifc /headerUnit S:\GitHub\STL\out\build\x86\out\inc\ranges=ranges.ifc /headerUnit S:\GitHub\STL\out\build\x86\out\inc\ratio=ratio.ifc /headerUnit S:\GitHub\STL\out\build\x86\out\inc\regex=regex.ifc /headerUnit S:\GitHub\STL\out\build\x86\out\inc\scoped_allocator=scoped_allocator.ifc /headerUnit S:\GitHub\STL\out\build\x86\out\inc\set=set.ifc /headerUnit S:\GitHub\STL\out\build\x86\out\inc\shared_mutex=shared_mutex.ifc /headerUnit S:\GitHub\STL\out\build\x86\out\inc\span=span.ifc /headerUnit S:\GitHub\STL\out\build\x86\out\inc\sstream=sstream.ifc /headerUnit S:\GitHub\STL\out\build\x86\out\inc\stack=stack.ifc /headerUnit S:\GitHub\STL\out\build\x86\out\inc\stdexcept=stdexcept.ifc /headerUnit S:\GitHub\STL\out\build\x86\out\inc\streambuf=streambuf.ifc /headerUnit S:\GitHub\STL\out\build\x86\out\inc\string_view=string_view.ifc /headerUnit S:\GitHub\STL\out\build\x86\out\inc\string=string.ifc /headerUnit S:\GitHub\STL\out\build\x86\out\inc\system_error=system_error.ifc /headerUnit S:\GitHub\STL\out\build\x86\out\inc\thread=thread.ifc /headerUnit S:\GitHub\STL\out\build\x86\out\inc\tuple=tuple.ifc /headerUnit S:\GitHub\STL\out\build\x86\out\inc\type_traits=type_traits.ifc /headerUnit S:\GitHub\STL\out\build\x86\out\inc\typeindex=typeindex.ifc /headerUnit S:\GitHub\STL\out\build\x86\out\inc\typeinfo=typeinfo.ifc /headerUnit S:\GitHub\STL\out\build\x86\out\inc\unordered_map=unordered_map.ifc /headerUnit S:\GitHub\STL\out\build\x86\out\inc\unordered_set=unordered_set.ifc /headerUnit S:\GitHub\STL\out\build\x86\out\inc\utility=utility.ifc /headerUnit S:\GitHub\STL\out\build\x86\out\inc\valarray=valarray.ifc /headerUnit S:\GitHub\STL\out\build\x86\out\inc\variant=variant.ifc /headerUnit S:\GitHub\STL\out\build\x86\out\inc\vector=vector.ifc /headerUnit S:\GitHub\STL\out\build\x86\out\inc\version=version.ifc main.cpp

C:\Temp>for %I in (algorithm any array atomic bit bitset charconv chrono codecvt compare complex concepts condition_variable coroutine deque exception execution filesystem forward_list fstream functional future initializer_list iomanip ios iosfwd iostream istream iterator limits list locale map memory_resource memory mutex new numbers numeric optional ostream queue random ranges ratio regex scoped_allocator set shared_mutex span sstream stack stdexcept streambuf string_view string system_error thread tuple type_traits typeindex typeinfo unordered_map unordered_set utility valarray variant vector version) do @cl /EHsc /nologo /W4 /std:c++latest /MD /experimental:module /module:exportHeader "<%I>" /Fo:%I.obj
<algorithm>
<any>
<array>
<atomic>
<bit>
<bitset>
<charconv>
<chrono>
<codecvt>
<compare>
<complex>
<concepts>
<condition_variable>
<coroutine>
The contents of <coroutine> are available only with C++20 or later.
<deque>
<exception>
<execution>
<filesystem>
<forward_list>
<fstream>
<functional>
<future>
<initializer_list>
<iomanip>
<ios>
<iosfwd>
<iostream>
<istream>
<iterator>
<limits>
<list>
<locale>
<map>
<memory_resource>
<memory>
<mutex>
<new>
<numbers>
<numeric>
<optional>
<ostream>
<queue>
<random>
<ranges>
<ratio>
<regex>
<scoped_allocator>
<set>
<shared_mutex>
<span>
<sstream>
<stack>
<stdexcept>
<streambuf>
<string_view>
<string>
<system_error>
<thread>
<tuple>
<type_traits>
<typeindex>
<typeinfo>
<unordered_map>
<unordered_set>
<utility>
<valarray>
<variant>
<vector>
<version>

C:\Temp>cl @cl.rsp
main.cpp
main.cpp(2): warning C4005: 'SIZE_MAX': macro redefinition
main.cpp(1): note: see previous definition of 'SIZE_MAX'
main.cpp(4): warning C4005: 'SIZE_MAX': macro redefinition
main.cpp(2): note: see previous definition of 'SIZE_MAX'
main.cpp(5): warning C4005: 'SIZE_MAX': macro redefinition
main.cpp(4): note: see previous definition of 'SIZE_MAX'
main.cpp(7): warning C4005: 'SIZE_MAX': macro redefinition
main.cpp(5): note: see previous definition of 'SIZE_MAX'
main.cpp(8): warning C4005: 'SIZE_MAX': macro redefinition
main.cpp(7): note: see previous definition of 'SIZE_MAX'
main.cpp(15): warning C4005: 'SIZE_MAX': macro redefinition
main.cpp(8): note: see previous definition of 'SIZE_MAX'
main.cpp(16): warning C4005: 'SIZE_MAX': macro redefinition
main.cpp(15): note: see previous definition of 'SIZE_MAX'
main.cpp(17): warning C4005: 'SIZE_MAX': macro redefinition
main.cpp(16): note: see previous definition of 'SIZE_MAX'
main.cpp(20): warning C4005: 'SIZE_MAX': macro redefinition
main.cpp(17): note: see previous definition of 'SIZE_MAX'
main.cpp(31): warning C4005: 'SIZE_MAX': macro redefinition
main.cpp(20): note: see previous definition of 'SIZE_MAX'
main.cpp(32): warning C4005: 'SIZE_MAX': macro redefinition
main.cpp(31): note: see previous definition of 'SIZE_MAX'
main.cpp(34): warning C4005: 'SIZE_MAX': macro redefinition
main.cpp(32): note: see previous definition of 'SIZE_MAX'
main.cpp(35): warning C4005: 'SIZE_MAX': macro redefinition
main.cpp(34): note: see previous definition of 'SIZE_MAX'
main.cpp(42): warning C4005: 'SIZE_MAX': macro redefinition
main.cpp(35): note: see previous definition of 'SIZE_MAX'
main.cpp(44): warning C4005: 'SIZE_MAX': macro redefinition
main.cpp(42): note: see previous definition of 'SIZE_MAX'
main.cpp(45): warning C4005: 'SIZE_MAX': macro redefinition
main.cpp(44): note: see previous definition of 'SIZE_MAX'
main.cpp(47): warning C4005: 'SIZE_MAX': macro redefinition
main.cpp(45): note: see previous definition of 'SIZE_MAX'
main.cpp(48): warning C4005: 'SIZE_MAX': macro redefinition
main.cpp(47): note: see previous definition of 'SIZE_MAX'
main.cpp(49): warning C4005: 'SIZE_MAX': macro redefinition
main.cpp(48): note: see previous definition of 'SIZE_MAX'
main.cpp(52): warning C4005: 'SIZE_MAX': macro redefinition
main.cpp(49): note: see previous definition of 'SIZE_MAX'
main.cpp(53): warning C4005: 'SIZE_MAX': macro redefinition
main.cpp(52): note: see previous definition of 'SIZE_MAX'
main.cpp(68): warning C4005: 'SIZE_MAX': macro redefinition
main.cpp(53): note: see previous definition of 'SIZE_MAX'

C:\Temp>

[CaseyCarter]

The SIZE_MAX issue is being fixed in vcruntime in internal MSVC-PR-257059. Once those changes are live in the STL toolset compiler (probably the first VS 2019 16.8 preview) we can add test coverage to prevent regression and declare victory by closing this issue.

[AlexGuteniev]

After talking with Cameron DaCamara (@cdacamar), I learned that users won't be able to control standard library header units via macros defined in source code (or separate headers), but they'll still be able to define macros on the command line.

That's great. I though modules would not support macros at all. Sensitivity to potential conflicts is also good.

[StephanTLavavej]

We should have test coverage for EDG too. Russell Johnston (@rpjohnst) says that the command lines should be the same as for MSVC, with /BE added. @merlynoppenheim has confirmed that this should work in 16.8 Preview 3.

[StephanTLavavej]

We'll eventually need Clang coverage, although they don't support modules yet: https://clang.llvm.org/cxx_status.html#cxx20 (This will probably need a separate issue after this one is closed).