general crayCC compilation failure

[cpranger]

I am very interested to use this amazing JSON library for my project. However, I am virtually tied to using the Cray Compiling Environment (CCE) and the Cray C++ compiler (crayCC). I realise that what I am reporting below may well be outside the scope of the developer(s), and that moreover, CrayCC is not on the list of supported compilers. Yet I want to make mention of this, and leave it up to the author to decide what to do with this information. Any help would be appreciated!

I have constructed a minimal working example (hello world) which does absolutely nothing with JSON except include the single header file:

#include <iostream>
#include "nlohmann/json.hpp"

int main()
	{ std::cout << "hello world!" << std::endl; }

this program is compiled with either gcc 7.2.0 producing expected behaviour (hello world!), and with crayCC 8.6.5 (https://pubs.cray.com/content/S-2179/8.6/cray-c-and-c++-reference-manual-s-2179-86/about-the-cray-and-c++-reference-manual).

> CC hello_json.cpp

the output is:

/tmp/pe_38754/hello_json_1.o: In function `std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release()':
/opt/gcc/6.1.0/snos/include/g++/bits/shared_ptr_base.h:147: undefined reference to `_ZN23_INTERNAL_b13fd261_main9__gnu_cxx27__exchange_and_add_dispatchEPii$$CFE_id_b13fd261_main'
/opt/gcc/6.1.0/snos/include/g++/bits/shared_ptr_base.h:162: undefined reference to `_ZN23_INTERNAL_b13fd261_main9__gnu_cxx27__exchange_and_add_dispatchEPii$$CFE_id_b13fd261_main'
/tmp/pe_38754/hello_json_1.o: In function `std::__shared_ptr<nlohmann::detail::input_adapter_protocol, (__gnu_cxx::_Lock_policy)2>::~__shared_ptr()':
/users/cpranger/garnet/experiments/json_compilation/hello_json.cpp:1: undefined reference to `_ZN23_INTERNAL_b13fd261_main9__gnu_cxx27__exchange_and_add_dispatchEPii$$CFE_id_b13fd261_main'
/users/cpranger/garnet/experiments/json_compilation/hello_json.cpp:1: undefined reference to `_ZN23_INTERNAL_b13fd261_main9__gnu_cxx27__exchange_and_add_dispatchEPii$$CFE_id_b13fd261_main'
/tmp/pe_38754/hello_json_1.o: In function `std::__shared_ptr<nlohmann::detail::output_adapter_protocol<char>, (__gnu_cxx::_Lock_policy)2>::~__shared_ptr()':
/users/cpranger/garnet/experiments/json_compilation/hello_json.cpp:7488: undefined reference to `_ZN23_INTERNAL_b13fd261_main9__gnu_cxx27__exchange_and_add_dispatchEPii$$CFE_id_b13fd261_main'
/tmp/pe_38754/hello_json_1.o:/users/cpranger/garnet/experiments/json_compilation/hello_json.cpp:7488: more undefined references to `_ZN23_INTERNAL_b13fd261_main9__gnu_cxx27__exchange_and_add_dispatchEPii$$CFE_id_b13fd261_main' follow
/tmp/pe_38754/hello_json_1.o: In function `std::unique_ptr<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >* nlohmann::basic_json<std::map, std::vector, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool, long, unsigned long, double, std::allocator, nlohmann::adl_serializer>::create<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&>(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)::{lambda(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*)#1}>::~unique_ptr()':
/opt/gcc/6.1.0/snos/include/g++/bits/unique_ptr.h:236: undefined reference to `_ZZN8nlohmann10basic_jsonISt3mapSt6vectorNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEblmdSaNS_14adl_serializerEE6createIS8_JRKS8_EEEPT_DpOT0_ENKUlPS8_E_clESJ_$$CFE_id_b13fd261_main'
/opt/cray/pe/cce/8.6.5/binutils/x86_64/x86_64-pc-linux-gnu/bin/ld: link errors found, deleting executable `a.out'
/usr/bin/X11/sha1sum: a.out: No such file or directory

FYI, _ZN23_INTERNAL_b13fd261_main9__gnu_cxx27__exchange_and_add_dispatchEPii$$CFE_id_b13fd261_main demangles to

_INTERNAL_b13fd261_main::__gnu_cxx::__exchange_and_add_dispatch(int*, int)$$CFE_id_b13fd261_main

and _ZZN8nlohmann10basic_jsonISt3mapSt6vectorNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEblmdSaNS_14adl_serializerEE6createIS8_JRKS8_EEEPT_DpOT0_ENKUlPS8_E_clESJ_$$CFE_id_b13fd261_main demangles to

std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >* nlohmann::basic_json<std::map, std::vector, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool, long, unsigned long, double, std::allocator, nlohmann::adl_serializer>::create<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&>(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)::{lambda(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*)#1}::operator()(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*) const$$CFE_id_b13fd261_main

Please let me know if any more information is required!

Casper

[cpranger]

It seems I cannot edit my own issue (older browser?), so I will add here that I am using the version I cloned this morning from the development branch.

[OvermindDL1]

That doesn't look like a json library issue, that looks like that the included system headers for the C++ standard std::shared_ptr is using compare-and-swap intrinsics (which is a good thing) that the compiler itself is not resolving. This looks like an issue either with the compiler (less likely though to be honest) or with the standard libraries being used (this is the most likely issue).

[OvermindDL1]

Try compiling something like this using the same steps as you used above and tell what happens (don't run it, just compile it):

#include <memory>

void main() {
  std::shared_ptr<void*> test;
  test = &test;
  test.reset();
}

[cpranger]

Thank you for your comment. You may well be right that this is a compiler issue. shared_ptr and unique_prt are used extensively throughout my project without any issues. Your provided test compiles neither in gcc version 7.2.0 nor with crayCC version 8.6.5 (nor in my head, to be honest), with crayCC complaining that:

  No operator "=" matches these operands.
            The operand types are:  std::shared_ptr<void *> =
                      std::shared_ptr<void *> *.
    test = &test;

and gcc making the same complaint in different wording. FYI, I believe the crayCC compiler is using the STL of gcc 6.1.0

Thanks again for your input,

Casper

[OvermindDL1]

Eh ignore the test = &test part, it was just a quick add to test equality but feel free to assign 0 to it. ^.^;

I believe the crayCC compiler is using the STL of gcc 6.1.0

Yeah which is why it makes me thing that the json library is using shared_ptr more completely than your project...

It's definitely hard to tell though, I've never even heard of that compiler, I thought all other systems have basically migrated to clang backends by now instead of inventing their own compilers... ^.^;

[cpranger]

Both compilers accept your code example with the assignment test = 0;. I'm sorry this issue can be reproduced by only few people due to the exotic compiler! I presume Cray uses their own compilers for optimal performance on their proprietary hardware (in this case Cray XC40 and XC50). If you can propose further tests, I'd be happy to try them! In the mean time, I can try compiling against another STL version.

Thanks again,

Casper

[OvermindDL1]

Is the cray compiler a modified gcc? If so then do you know what version it is based on? Any updated one? If it is not then another standard library might help? Maybe a command-line flag to disable atomic operations?

[cpranger]

These are hard questions! The compiling environment is kind of opaque to me, so I can not really answer them at this moment. Since you are essentially suggesting that the issue is likely not with the JSON library, I have contacted support of the machine in question and will report back as soon as there is news! In the meantime, thank your for helping!

Casper

[OvermindDL1]

I'm exceptionally curious about this. ^.^
Whatever you find out, please report back! It 'could' indeed be a bug in this library but based on all information given thus far I'd bet hard money that it's not. :-)

[cpranger]

A helpful computational scientist working at the site of the machine came up with an interesting solution. While my original example didn't compile with crayCC, the following does:

#include <iostream>
#include "nlohmann/json.hpp"

int main()
{
    using nlohmann::json;
    auto a = std::make_shared<double>(5.0);
    std::stringstream ss1;
    ss1 << "[\"foo\",1,2,3,false,{\"one\":1}]";
    json j = json::parse(ss1);
    std::cout << "hello world!" << std::endl;
    std::cout << j << std::endl;
}

It is not clear to me why this would be necessary, and what the general underlying idea is of this fix (though I have asked for clarification on that). It seems that simply using the shared_ptr gets rid of the first few compilation errors, and that actually using JSON gets rid of the last. I'll have to see how this works out in my larger project since JSON will be an optional component, but this is a good start!

[OvermindDL1]

That... is very odd! I'm quite curious in this bug... What is that compiler doing...

[cpranger]

@OvermindDL1, thank you very much for thinking with/for me! I have just implemented the fix posted above in my bigger project and it compiles fine now. Since at the moment I have no further information sources that could lead to a more general solution, I will close this issue. If I get any more feedback (from e.g. technical support of the machine) I will be sure to share!

All the best,

Casper