v0.1.13 breaks semver

[joshka]

In ratatui, running cargo update from v0.1.12 to v0.1.13 results in the latter failing (something has changed in the way that unicode-width handles zero width strings

Error from our tests

--- STDOUT: ratatui buffer::buffer::tests::set_string_zero_width ---

running 1 test
test buffer::buffer::tests::set_string_zero_width ... FAILED

failures:

failures:
buffer::buffer::tests::set_string_zero_width

test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 1557 filtered out; finished in 0.00s

--- STDERR: ratatui buffer::buffer::tests::set_string_zero_width ---
thread 'buffer::buffer::tests::set_string_zero_width' panicked at src/buffer/buffer.rs:618:9:
assertion left == right failed
left: Buffer {
area: Rect { x: 0, y: 0, width: 1, height: 1 },
content: [
"",
],
styles: [
x: 0, y: 0, fg: Reset, bg: Reset, modifier: NONE,
]
}
right: Buffer {
area: Rect { x: 0, y: 0, width: 1, height: 1 },
content: [
"a",
],
styles: [
x: 0, y: 0, fg: Reset, bg: Reset, modifier: NONE,
]
}

I'd suggest yanking 0.1.13 and publishing the latest release as 0.2.0 instead.

[Manishearth]

This is a behavior change and not considered breaking by this crate.

[joshka]

This is a behavior change

That's literally what semver suggests. You broke the implied contract that 0.1.12 should be compatible with 0.1.13 and in doing so broke our tests, which implies that downstream user's apps will be rendered differently.

[Manishearth]

"should be compatible with" has an extremely broad range of meanings. I'm careful to make sure that code will compile; but changing the code to have more accurate answers is not a breakage.

The Rust stdlib's semver policy (which is roughly what the ecosystem follows as well) talks about behavioral changes as a notion of contract. It really depends on the situation, and I am thoroughly unconvinced that this crate already has or needs such a strong stability contract.

This crate says it attempts to approximate the displayed width of characters. That's not a guarantee of stability; and we've (well, @Jules-Bertholet) been trying to improve the accuracy of these heuristics.

You are free to rely on such behavior in tests, but you do have to expect that tests will break some time due to this kind of thing. If you want to pin the behavior, pin the dependency.

---

There is an argument to be made for an more stable (not 100% stable: the unicode standard and data this relies on itself changes!) API that uses the east asian width properties only (@Jules-Bertholet: would this be easy to add again? I was somewhat uncomfortable about expanding the scope of the crate, but UAX 11 is pretty vague about this stuff anyway)

[joshka]

The Rust stdlib's semver policy (which is roughly what the ecosystem follows as well) talks about behavioral changes as a notion of contract. It really depends on the situation, and I am thoroughly unconvinced that this crate already has or needs such a strong stability contract.

from the link:

In general, APIs are expected to provide explicit contracts for their behavior via documentation, and behavior that is not part of this contract is permitted to change in minor revisions.

A changelog is the exact form of documentation that would have helped notify this change in behavior, but you've rejected adding one in #56

the behavior change here is that the width of `"\u{1}" is now 1 instead of 0. It is however rendered as taking up no space as evidenced by the test output.

Working out what the change is caused by and why is next to impossible with the current approach (I have to look at every single commit that was made to work out relevance, and then backtrack to find a related PR). This is not ideal and there are easy ways that help consumers of this library work understand the changes more easily.

[Manishearth]

It is however rendered as taking up no space as evidenced by the test output.

This is highly highly dependent on the exact software stack used and is why this crate is explicitly documented as not necessarily matching actual rendered output.

Yes, the documentation could be clearer about how it approximates "displayed width", and why it is an ultimately futile effort to try and accurately predict display width from just codepoints. You've been lucky so far that your tests were dealing with cases that matched up.

If you want such an operation, this is not the crate for you, and furthermore, that operation does not exist.

(I've been meaning to write documentation on why that is the case for a while, but haven't had the time)

A changelog is the exact form of documentation that would have helped notify this change in behavior, but you've rejected adding one in #56

I don't think that's what the RFC means by "documentation".

But seriously, this is absolutely not my experience with semver in the Rust community: this kind of behavior change is almost always fine. Of course making a function do something entirely different would be breaking, but changing functions to still continue to do what they are documented as doing. This function still does what it is documented as doing.

(I have to look at every single commit that was made to work out relevance, and then backtrack to find a related PR)

That's ... what you have to do with the changelog too? I rejected the changelog because it produces more or less the same output as scrolling through commits.

[workingjubilee]

That's literally what semver suggests.

SemVer disagrees with the Rust community on what SemVer means..

[joshka]

To be clear - I'm using the Rust definition of semver, which treats 0.x.y the same way semver would treat x.y.0

https://doc.rust-lang.org/cargo/reference/resolver.html#semver-compatibility

This notion of “compatibility” is important because Cargo assumes it should be safe to update a dependency within a compatibility range without breaking the build.

Versions are considered compatible if their left-most non-zero major/minor/patch component is the same. ... This convention also applies to versions with leading zeros. For example, 0.1.0 and 0.1.2 are compatible, but 0.1.0 and 0.2.0 are not. Similarly, 0.0.1 and 0.0.2 are not compatible.

https://doc.rust-lang.org/cargo/reference/resolver.html#semver-breaking-patch-release-breaks-the-build

Sometimes a project may inadvertently publish a point release with a SemVer-breaking change. When users update with cargo update, they will pick up this new release, and then their build may break. In this situation, it is recommended that the project should yank the release, and either remove the SemVer-breaking change, or publish it as a new SemVer-major version increase.

...

If it looks like the third-party project is unable or unwilling to yank the release, then one option is to update your code to be compatible with the changes, and update the dependency requirement to set the minimum version to the new release.

This is the path that we are taking in ratatui/ratatui#1171, but it's a poor one given the blast radius of this sort of breaking change. The changes here are:

• silent - cargo update brings the changes in as 0.1.12 is rust semver compatible with 0.1.13
• undocumented - there is no changelog
• potentially affect millions of consumers

The right thing to do is yank and publish 0.2, and document breaking changes like this. But as this issue is closed and we've worked around it for our particular use case, I'll leave pushing this issue forward to any of the 584 other direct dependencies of unicode-width that have actual breaks cause by this.

Our reliance on \0 and \u{1} were just as being known zero-width characters to use in rendering tests that had to deal with how to render zero-width characters at various places in terminal UI. These particular characters changed width inbetween 0.1.12 and 0.1.13 for some reason.

[Manishearth]

Yeah, it changed because the Unicode standard recommends they be rendered unless they are interpretable in that context (which these days is almost never).

#45 (comment)

This crate implements Unicode standards.