Does it support chinese character?

[JueLance]

I try to input some chinese characters, the application is crashed. The code as following below:

json j;

j["chinese"] = "中文";

string s = j.dump();

cout << s.c_str() << endl;

Can you tell me how to make it works?

[nlohmann]

Yes, the library full supports UTF-8. You need to make sure whether your IDE actually uses UTF-8, which is sometimes an issue with MSVC.

See also #694.

[nlohmann]

You say the application crashed. Could you please provide more information: compiler version, concrete example, library version?

[JueLance]

Thank you for your patience. During servual hours search from internet, I tried some methods on MSVC, but all can't get correct result. I guess the reason is that MSVC treat the "中文" as non-utf-8 character(maybe ASCII, Unicode-16, I'm not sure.) while compile code or/and runing the program. As your metion, I tried to compile the code with MingW, it works fine due to MingW using Utf-8 as default setting.

My full code is as below:

#include <iostream>  
#include <string> 
#include "json.hpp"

using namespace std;

using json = nlohmann::json;

int main() {

    try
    {
        json j;

        j["chinese"] = "中文";

        string s = j.dump();

        cout << s.c_str() << endl;
    }
    catch (const std::exception& e)
    {
        cout << e.what() << endl;
    }

    return 0;
}

when I run it in Visual Studio 2017, the command line will show as following:
[json.exception.type_error.316] invalid UTF-8 byte at index 1: 0xD0
Press any key to continue . . .

Addtional Infomation(by typing 'systeminfo' in command line):
OS Version: Microsoft Windows [Version 10.0.16299.309]
System Locale: zh-cn;Chinese (China)
Input Locale: zh-cn;Chinese (China)
Time Zone: (UTC+08:00) Beijing, Chongqing, Hong Kong, Urumqi

Visual Studio Project Setting:
Windows SDK version: 10.0.16299.0
Platform Toolset: Visual Studio 2017 (v141)
Character set: Use Unicode Character Set
json.hpp version: 3.1.2

[nlohmann]

The error message indicates that "中文" is not encoded with UTF-8. Could you execute the following program?

#include <string>
#include <iostream>

int main() {
    std::string s = "中文";
    for (size_t i = 0; i < s.size(); ++i)
    {
        std::cout << i << " " << std::hex << static_cast<int>(static_cast<uint8_t>(s[i])) << std::endl;
    }
}

With UTF-8, this should yield:

0 e4
1 b8
2 ad
3 e6
4 96
5 87

• https://unicode-table.com/de/4E2D/ (0-2)
• https://unicode-table.com/de/6587/ (3-5)

[JueLance]

I have to add a flag "/utf-8" to compile the program, otherwise I still get the same error - invalid UTF-8 byte. Reference here: https://msdn.microsoft.com/zh-cn/library/mt708819.aspx.

`#include
#include
#include "json.hpp"

using namespace std;

using json = nlohmann::json;

int main() {
try
{
std::string s = "中文";
for (size_t i = 0; i < s.size(); ++i)
{
std::cout << i << " " << std::hex << static_cast(static_cast<uint8_t>(s[i])) << std::endl;
}

    json j;
    j["chinese"] = s;

    string jsonStr = j.dump();

    cout << jsonStr << endl;

}
catch (const std::exception& e)
{
    cout << e.what() << endl;
}

return 0;

}`

the result as following:
`
0 e4

1 b8

2 ad

3 e6

4 96

5 87

{"chinese":"涓枃"}
Press any key to continue . . .
`
I also tried to run "chcp 65001"(change the windows active page to utf-8) and "chcp 936"(Chinese) in command line first, then run the exe file, still get incorrect character. when windows active page set to utf-8, can't print all message in command line; When set to Chinese, same with above.

[OvermindDL1]

You should encode UTF-8 out of source. C++ does not define a text mapping format for the source files, hence why you should keep it in resource files. Adding that utf-8 switch is not really compliant. The standard states:

Physical source file characters are mapped, in an implementation-defined manner, to the basic source character set (introducing new-line characters for end-of-line indicators) if necessary. Trigraph sequences (2.3) are replaced by corresponding single-character internal representations. Any source file character not in the basic source character set (2.2) is replaced by the universal-character-name that des- ignates that character. (An implementation may use any internal encoding, so long as an actual extended character encountered in the source file, and the same extended character expressed in the source file as a universal-character-name (i.e. using the \uXXXX notation), are handled equivalently.)

As such C++ does not have chinese defined in the basic source character set so you cannot ever rely on it working, thus you either need to encode it in the source or use an external mapping file (like a resource file for windows).

But yes, this library supports chinese fine in JSON, but the source code is not under this library's control for the character set, it can only rely on the basic source character set as per the C++ standard.

(EDIT: 'encode it' meaning use the \uXXXX syntax)

[JueLance]

Thanks for all, I found the root case: I try to write the json string to a file(following below code), "data.json" encoding with UTF-8, and can display chinese character correctly. So, the problem is caused by windows 10 bug or command line's bug.

string json_file = "data.json";

ofstream json_output(json_file);

json_output << j;// the json object

json_output.close();

@OvermindDL1 Thank you for your tips. Yes, I did try to change the source code file encoding.

[OvermindDL1]

Yes, I did try to change the source code file encoding.

Even that is not guaranteed to work depending on the compiler (MSVC is especially bad at it). It is better to not depend on encoding at all and instead follow the spec to the letter.

So, the problem is caused by windows 10 bug or command line's bug.

Windows is not UTF-8, it is USC-16 or whatever it's called (essentially a garbage format to be honest).

But yes, store UTF-8 in a non-source file and read it in should always work or it is a bug.

[nlohmann]

Can we close the issue?

[JueLance]

sure