Neovim macros can use control characters, which are detected as invalid utf8

[matu3ba]

As example, take minifile.lua

-- stylua: ignore start
-- convert
--   QFETCH(std::string>("received_message");
-- to
--   QFETCH(std::string, received_message);
--f(f(xxhr,a �ü�C$hhx^j
-- stylua: ignore end

Running stylua minifile.lua, in my case with stylua.toml, but this should not be essential

# use default column_width = 120
line_endings = "Unix"
indent_type = "Spaces"
indent_width = 2
quote_style = "AutoPreferSingle"
# remove parenthesis around function arguments, if possible
no_call_parentheses = true

yields in error: failed to read lua/my_cmds.lua: stream did not contain valid UTF-8.

I would expect that stylua does not try to touch the content in between stylua: ignore start and stylua: ignore end.

If you disagree and think that the user must conform utf-8, can you provide or link a brief example/guideline how to load file content on a by-line base to associate it to another table and brief reasoning?

Thanks a lot for your work.

UPDATE: the thing after a is <80> and after ü is ^D. Unfortunately is github unable or unwilling to render those properly.

[JohnnyMorganz]

The UTF-8 restriction is unfortunately a restriction on the string type I believe, which is what the file op, parser etc. all use. We can't detect the ignore comments because we don't even reach the parsing stage, this error occurs on file read.

I don't know of any workaround right now. I can look into it, but not sure exactly

[JohnnyMorganz]

Unfortunately, my current thoughts is that this is something we most likely cannot support.

Invalid UTF-8 present within standard text is not typically common. As you saw when you posted this, GitHub is not happy with it, neither are most web browsers in general. You may also find other tools don't accept it too.

I highly recommend, if possible, to have a proper encoding for macros here which is valid UTF8. This isn't just for stylua, but for other tools as well.

If you disagree with this though, let me know. I'm not sure what I can do though, because this opens up a lot of complexity. Simply, there is a String::from_utf8_lossy function, which can work, but I don't think a lossy conversion is suitable for a formatter which will output reformatted code, and lose this info.

[matu3ba]

Unfortunately, my current thoughts is that this is something we most likely cannot support.

That is fine. I will try some form of basic macro to lua translator then, which would enable debugging with existing tooling.

This is a good reason to fix https://github.com/linty-org/key-menu.nvim/issues/10 or redesign the bad vim parts vim/vim#10503 as firing events from key combinations are merely functions getting called (by arguments) and which should be catchable inside anything claiming to be modal.