Support UTF-8 in metric creation, parsing, and exposition by ywwg · Pull Request #1070 · prometheus/client_python

ywwg · 2024-11-06T14:36:59Z

Adding support for UTF-8 required substantial reworking of the parsing code because the previous code relied on regexes and .index calls that would incorrectly find characters inside quote marks. Note that this means the previous code was already broken for parsing exposition text where characters like braces and hashes were in label values. I have added tests to exercise the edge cases I can think of.

Does not address content negotiation, which will be a followon.

part of #1013

ywwg · 2024-11-07T20:39:24Z

I'll squash commits when we're done

csmarchbanks

I gave this a first runthrough. Generally looking ok to me. A couple thoughts:

If you want to split it up I think refactoring validation + removing regex parsing could be good opportunities. Plus have some tests that catch the incorrect behavior you mentioned (e.g. curly braces in a label value).
- Tests for this would be nice in this PR too
Some more tests for various permutations would be nice, specifically there are not many tests for escaped label names/values right now

csmarchbanks · 2024-11-12T22:20:45Z

+ _legacy_validation = True
+
+
+def validate_metric_name(name: str) -> None:


Do we want these to be public now? Might be worth having them still prefixed with _ but I also think they could be useful general functions for other libraries to use.

the functions are used by both prometheus_client and prometheus_client/openmetrics so I thought they had to be public

Ahh, yeah Python doesn't really do public/private, it's just a convention.

I am confused what change you want me to make, then -- leave it? prepend _? something else?

Ahh sorry, yeah if you could prepend with _ I think that would be safest to begin with to avoid others depending on the methods.

ywwg · 2024-11-15T16:28:48Z

Added a test, fixed an issue using .split

ywwg · 2024-11-15T20:08:02Z

+ return tokens
+
+
+def unquote_unescape(text):


maybe these should be private too?

I would say yes, thanks!

csmarchbanks

Just a bit of prepending with more _ and I think this is good to go! Plus cleaning up the commit history and such.

csmarchbanks · 2024-11-19T22:07:32Z

+ return tokens
+
+
+def unquote_unescape(text):


I would say yes, thanks!

part of #1013 Signed-off-by: Owen Williams <owen.williams@grafana.com>

vesari

Thanks for this great work, LGTM!

ywwg commented Nov 6, 2024
View reviewed changes

Comment threadtests/test_core.py

ywwg marked this pull request as draft November 6, 2024 14:42

ywwg marked this pull request as ready for review November 7, 2024 20:47

ywwg marked this pull request as draft November 7, 2024 20:55

ywwg marked this pull request as ready for review November 12, 2024 15:42

ywwg requested a review from csmarchbanks November 12, 2024 15:59

csmarchbanks reviewed Nov 12, 2024
View reviewed changes

ywwg commented Nov 15, 2024
View reviewed changes

ywwg requested a review from csmarchbanks November 18, 2024 15:13

csmarchbanks reviewed Nov 19, 2024
View reviewed changes

ywwg changed the title ~~WIP: Support UTF-8 in metric creation, parsing, and exposition~~Support UTF-8 in metric creation, parsing, and expositionNov 20, 2024

ywwg force-pushed the owilliams/utf8-01 branch from 303ff91 to d50068fCompare November 20, 2024 16:38

ywwg requested a review from csmarchbanks November 20, 2024 16:39

ywwg force-pushed the owilliams/utf8-01 branch from d50068f to cf5bdc0Compare November 20, 2024 18:22

Support UTF-8 in metric creation, parsing, and exposition
b21a7a1
part of #1013 Signed-off-by: Owen Williams <owen.williams@grafana.com>

ywwg force-pushed the owilliams/utf8-01 branch from cf5bdc0 to b21a7a1Compare November 20, 2024 18:23

ywwg requested a review from vesari November 27, 2024 14:59

vesari approved these changes Nov 30, 2024
View reviewed changes

Comment threadprometheus_client/openmetrics/exposition.py
Comment threadprometheus_client/openmetrics/exposition.py

csmarchbanks approved these changes Dec 2, 2024
View reviewed changes

csmarchbanks merged commit 33e6828 into masterDec 2, 2024

csmarchbanks deleted the owilliams/utf8-01 branch December 2, 2024 18:18

dg98 mentioned this pull request Feb 24, 2025
Fix order-dependent flaky tests related to UTF-8 support #1093
Merged

csmarchbanks mentioned this pull request Jun 24, 2025
CPU performance is degraded on version 0.22.1 #1114
Open

renovateBot mentioned this pull request Jul 26, 2025
Update all non-major dependencies IBM/detect-secrets-stream#38
Open
1 task

Nevon mentioned this pull request Aug 14, 2025
Missing OpenMetrics/Prometheus metrics after upgrade to 7.68.0 DataDog/integrations-core#21054
Closed

		_legacy_validation = True


		def validate_metric_name(name: str) -> None:

Uh oh!

Conversation

ywwg commented Nov 6, 2024• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

ywwg commented Nov 7, 2024

Uh oh!

csmarchbanks left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ywwg commented Nov 15, 2024

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

csmarchbanks left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

vesari left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

ywwg commented Nov 6, 2024•
edited
Loading