Skip to content

Update dotstrings parser to handle entries with multi-line values#12

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
Salman (fla-t) wants to merge 4 commits into
base:master
Choose a base branch
from
Open

Update dotstrings parser to handle entries with multi-line values #12

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
b00d8d0
feat: Update dotstrings parser to handle multi-line entries
fla-t Jun 28, 2024
94cef1c
tests: Add tests to make sure that everything is working fine
fla-t Jun 28, 2024
507b8de
suggestion: revert formatting change
fla-t Jun 28, 2024
84ff6d3
revert: keep the dotstring import as is
fla-t Jun 28, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion dotstrings/parser.py
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -14,7 +14,7 @@ class Patterns:

comment = re.compile(r"(\'(?:[^\'\\]|\\[\s\S])*\')|//.*|/\*(?:[^*]|\*(?!/))*\*/", re.MULTILINE)
whitespace = re.compile(r"\s*", re.MULTILINE)
entry = re.compile(r'"(.*)"\s*=\s*"(.*)";')
entry = re.compile(r'"([^"]*?)"\s*=\s*"((?:[^";]|"(?!\s*;))*?)";', re.DOTALL)

@dalemyersDale (dalemyers)Jun 28, 2024

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unfortunately this will break on strings that contain quotes. e.g. NSLocalizedString("Hello \"World\"", "Some Comment") gets turned into "Hello \"World\"" = "Hello \"World\"";.

I think the second change on this line will also stop the string containing ;.

@fla-tSalman (fla-t)Jun 28, 2024•
edited
Loading

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The first change is there so that the first match group doesn't end up matching the whole string (even matching the "value" part).

But yeah it wouldn't work when there are more than two quotes, in the "key" part of the string (the two quotes around the key itself). This was intentional because I didn't thought there would be any "key" that would have more quotes than two.

If we really want to cater keys that have quotes inside of them, we can replicate the regex that we have for value part, but without the semicolon

entry = re.compile(r'"((?:[^";]|"(?!\s*;))*?)"\s*=\s*"((?:[^";]|"(?!\s*;))*?)";')

@fla-tSalman (fla-t)Jun 28, 2024

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The second change is finding a ";" to match the value part of the string.

@fla-tSalman (fla-t)Jun 28, 2024

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you change the test file I have here to contain edge cases?



class Scanner:
Expand Down
13 changes: 13 additions & 0 deletions tests/strings_files/string_with_multiline_value.strings
View file
Open in desktop
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,13 @@
"key-case1" = "value
that
has
newlines";

"key-case2" = "Major text in one line and just the closing in next line
";

"key-case3" = "A paragraph like text structure

that goes like this and goes on and on

and ends here";
31 changes: 30 additions & 1 deletion tests/test_simple.py
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -88,7 +88,7 @@ def test_strings_format(self):
self.assertEqual(entries[0].strings_format(), '/* This is a\n multiline comment */\n"carry" = "breathe";')
self.assertEqual(entries[1].strings_format(), '/* This is another\n multiline comment, but for a dupe */\n"condition" = "outgoing";')
self.assertEqual(entries[2].strings_format(), '/* precious */\n"condition" = "outgoing";')

def test_string_with_uneven_whitespace(self):
"""Test that string with uneven whitespaces work"""
string_with_uneven_whitespace_path = os.path.join(self.strings_path, "string_with_uneven_whitespace.strings")
Expand All@@ -103,3 +103,32 @@ def test_string_with_uneven_whitespace(self):

self.assertEqual(entries[2].key, "This is again a key")
self.assertEqual(entries[2].value, "This is again a value")

def test_string_with_multiline_value(self) -> None:
"""Test the strings that have values spanning multiple lines"""

multiline_value_path = os.path.join(self.strings_path, "string_with_multiline_value.strings")
entries = dotstrings.load(multiline_value_path)
self.assertEqual(len(entries), 3)

self.assertEqual(entries[0].key, "key-case1")
self.assertEqual(entries[0].value, """value
that
has
newlines""")
self.assertEqual(entries[0].value, "value\nthat\nhas\nnewlines")

self.assertEqual(entries[1].key, "key-case2")
self.assertEqual(entries[1].value, """Major text in one line and just the closing in next line
""")
self.assertEqual(entries[1].value, "Major text in one line and just the closing in next line\n")

self.assertEqual(entries[2].key, "key-case3")
self.assertEqual(entries[2].value, """A paragraph like text structure

that goes like this and goes on and on

and ends here""")

self.assertEqual(
entries[2].value, "A paragraph like text structure\n\nthat goes like this and goes on and on\n\nand ends here")