Comments at the end of line are treated as messages and integrated single line comments#53

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
igorkulman merged 4 commits into from
May 25, 2019
Merged

Comments at the end of line are treated as messages and integrated single line comments #53

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
cd718d5
Adapted parser for message at the end of a line instead of before an …
Tantalum73 May 24, 2019
d0426a4
Introduced singleLineMessageOpen and singleLineMessageClose as a star…
Tantalum73 May 25, 2019
ddf95d5
Working integration of single line comments.
Tantalum73 May 25, 2019
9db3dcc
Some cleanup.
Tantalum73 May 25, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion sources/LocalizationEditor/Info.plist
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -19,7 +19,7 @@
<key>CFBundleShortVersionString</key>
<string>2.1</string>
<key>CFBundleVersion</key>
<string>191</string>
<string>198</string>
<key>LSMinimumSystemVersion</key>
<string>$(MACOSX_DEPLOYMENT_TARGET)</string>
<key>NSHumanReadableCopyright</key>
Expand Down
147 changes: 85 additions & 62 deletions sources/LocalizationEditor/Providers/Parser.swift
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -26,7 +26,7 @@ class Parser {
fileprivate enum ParserState {
case readingKey
case readingValue
case readingMessage
case readingMessage(isSingleLine: Bool)
case other
}
/// The current state of the parser.
Expand All@@ -53,23 +53,6 @@ class Parser {
return results
}

/**
Special handling for single line comments to turn them into message tokens. Should only be called when state is other so // in a middle of value does not get caught
*/
private func skipAndProcessSingleLineComments() {
while !input.isEmpty, let character = String(input[input.startIndex]).unicodeScalars.first, CharacterSet.whitespacesAndNewlines.contains(character) {
input.remove(at: input.index(input.startIndex, offsetBy: 0))
}

if input.hasPrefix("//"), let endIndex = input.index(of: "\n") {
let messageRange = input.index(input.startIndex, offsetBy: 2) ..< endIndex
tokens.append(.message(String(input[messageRange])))

let rangeForRemoving = input.startIndex ..< endIndex
input.removeSubrange(rangeForRemoving)
}
}

/**
This function reads through the input and populates an array of tokens.

Expand All@@ -82,49 +65,56 @@ class Parser {
// Actions depend on the current state.
switch state {
case .other:
skipAndProcessSingleLineComments()

// Extract the upcoming control character, also switch the current state and append the extracted token, if any.
if let extractedToken = try prepareNextState() {
tokens.append(extractedToken)
}
case .readingKey:
// Until the key-end marker is reached, the text should be interpreted as key.
let currentKeyText = extractText(until: .quote)
let potentialNewToken: Token = .key(currentKeyText)
// If the prior token was also a key, append it.
let newToken = tokenByConcatinatingwithPriorToken(potentialNewToken, seperatingString: EnclosingControlCharacters.quote.rawValue)
tokens.append(newToken)
// If the upcoming control character is also a key, do not stop reading a key. Otherwise a unescaped quote may exclude text from the key. Otherwise the state may be anything else.
if let nextControlCharacter = findNextControlCharacter(andExtractFromSource: false), case EnclosingControlCharacters.quote = nextControlCharacter {
state = .readingKey
} else {
state = .other
}
extractAndAppendIfPossible(for: .key(""), until: .quote)
case .readingValue:
// Text until value-end marker is a value.
// If the prior token as also a value, append it.
let currentValueText = extractText(until: .quote)
let potentialNewToken: Token = .value(currentValueText)
// If the prior token was also a key, append it.
let newToken = tokenByConcatinatingwithPriorToken(potentialNewToken, seperatingString: EnclosingControlCharacters.quote.rawValue)
tokens.append(newToken)
// If the upcoming control character is also a key, do not stop reading a value. Otherwise a unescaped quote may exclude text from the value. Otherwise the state may be anything else.
if let nextControlCharacter = findNextControlCharacter(andExtractFromSource: false), case EnclosingControlCharacters.quote = nextControlCharacter {
state = .readingValue
} else {
state = .other
}
case .readingMessage:
// Text until value-end marker is a message.
extractAndAppendIfPossible(for: .value(""), until: .quote)
case .readingMessage(let isReadingSingleLine):
// If the prior token as also a message, DO NOT append it since the prior message could be a license header.
let currentMessageText = extractText(until: .messageBoundaryClose)
let endMarker: EnclosingControlCharacters = isReadingSingleLine ? .singleLineMessageClose : .messageBoundaryClose
let currentMessageText = extractText(until: endMarker)
let newToken: Token = .message(currentMessageText)
tokens.append(newToken)
state = .other
}
}
}
/// Extracts text from the input until the end marker is reached. Uses that text to create a new token and appends it to a prior extracted token if possible. In any case it updates the current list of extracted tokens.
///
/// - Parameters:
/// - token: The type of token that should be created from the text before the end marker. The associated value of the input is ignored.
/// - endMarker: Marks the end of the tokens content.
private func extractAndAppendIfPossible(for token: Token, until endMarker: EnclosingControlCharacters) {
let currentText = extractText(until: endMarker)
let potentialNewToken: Token
switch token {
case .key:
potentialNewToken = .key(currentText)
case .value:
potentialNewToken = .value(currentText)
default:
assertionFailure("Currently, only the .key and .value support joining.")
return
}
// Append to the prior token if possible.
let newToken = tokenByConcatinatingwithPriorToken(potentialNewToken, seperatingString: endMarker.rawValue)
tokens.append(newToken)
// Do not stop reading when a newline or a quote is the next control character. Otherwise an unescaped quote may exclude text from the value. Keep the state unchanged if any other control character follows.
if let nextControlCharacter = findNextControlCharacter(andExtractFromSource: false) {
switch nextControlCharacter {
case SeperatingControlCharacters.newline, EnclosingControlCharacters.singleLineMessageClose, EnclosingControlCharacters.quote:
// Do not change the state and just continue.
return
default:
break
}
}
state = .other
}
/// Call this method when the list of tokens is ready and model object can be created. It will iterate through the tokens and try to map their values into model objects. Whe the mapping failed, an error is thrown.
///
/// - Returns: The extracted model values.
Expand All@@ -134,6 +124,25 @@ class Parser {
var currentKey: String?
var currentValue: String?
var results = [LocalizationString]()
// The token that delimits an entry.
guard let endToken = entriesEndToken(for: tokens) else {
throw ParserError.malformattedInput
}
// Generates a result and appends it to the list of results if possible.
func generateResultIfPossible(from processedToken: Token) {
guard processedToken.isCaseEqual(to: endToken) else { return }
// Done with that line. Check if values are populated and append them to the results.
guard let key = currentKey, let value = currentValue else {
return
}
let correctedMessage = removeLeadingTrailingSpaces(from: currentMessage)
let entry = LocalizationString(key: key, value: value.replacingOccurrences(of: "\\\"", with: "\""), message: correctedMessage)
results.append(entry)
// Reset the properties to be ready for the next line.
currentValue = nil
currentKey = nil
currentMessage = nil
}
// Iterate through the tokens and transform them into model objects.
for token in tokens {
switch token {
Expand All@@ -143,28 +152,36 @@ class Parser {
currentKey = containedText
case .value(let containedText):
currentValue = containedText
case .semicolon:
// Done with that line. Check if values are populated and append them to the results.
guard let key = currentKey, let value = currentValue else {
throw ParserError.malformattedInput
}
let correctedMessage = removeLeadingTrailingSpaces(from: currentMessage)
let entry = LocalizationString(key: key, value: value.replacingOccurrences(of: "\\\"", with: "\""), message: correctedMessage)
results.append(entry)
// Reset the properties to be ready for the next line.
currentValue = nil
currentKey = nil
currentMessage = nil
default:
()
}
generateResultIfPossible(from: token)
}
// Throw an execption to indicate that something went wront when tokens are extracted but they could not be transferred into model objects:
if !tokens.isEmpty && results.isEmpty {
throw ParserError.malformattedInput
}
return results
}
/// Determines the token that ends an entry. An entry can either be ended by a semicolon (if no comment was provided or the comment is above the entry) or a comment located at the end of a line. In the second case the `.message` token marks the end of the entry.
///
/// - Parameter tokens: The tokens that were extracted during tokenization.
/// - Returns: The token that ends an entry.
private func entriesEndToken(for tokens: [Token]) -> Token? {
// Assumption: after the first semicolon comes a new line -> semicolon delimits entry
// After first semicolon comes a message, followed by a new line -> message delimits entry
guard let semicolonIndex = tokens.firstIndex(where: { $0.isCaseEqual(to: .semicolon) }) else {
return nil
}
guard let indexAfterSemicolon = tokens.index(semicolonIndex, offsetBy: 1, limitedBy: tokens.endIndex - 1) else { return nil }
let elementAfterSemicolon = tokens[indexAfterSemicolon]
switch elementAfterSemicolon {
case .newline:
return .semicolon
default:
return elementAfterSemicolon
}
}
/// This function removes leading and trailing spaces from the input.
///
/// - Parameter input: The string whose leading and trailing spaces should be removed.
Expand DownExpand Up@@ -265,7 +282,7 @@ class Parser {
// Extract the given range and remove it from the input string.

let lengthOfControlCharacter: Int = includingControlCharacter.skippingLength
let endIndexOfExtraction = input.index(endindex, offsetBy: lengthOfControlCharacter)
let endIndexOfExtraction = input.index(endindex, offsetBy: lengthOfControlCharacter, limitedBy: input.endIndex) ?? input.endIndex
// Remove the range that includes the control character. The input range is used for extracting the text before it.
let rangeForRemoving = input.startIndex ..< endIndexOfExtraction
let rangeForExtraction = input.startIndex ..< endindex
Expand DownExpand Up@@ -354,7 +371,7 @@ extension Parser {
case EnclosingControlCharacters.messageBoundaryOpen:
// A new message begins.
// Set the state to expect a message.
state = .readingMessage
state = .readingMessage(isSingleLine: false)
case EnclosingControlCharacters.messageBoundaryClose:
// Message-end markers should only be detected when the lexer is reading a message. If they occure 'in the wild' the input must be ill formatted.
break
Expand All@@ -366,6 +383,12 @@ extension Parser {
// Extract semicolon as token. A quote or message-start mark will follow as next control character but for now the state remains .other in order to detect that quote.
returnToken = .semicolon
state = .other
case SeperatingControlCharacters.newline:
returnToken = .newline
case EnclosingControlCharacters.singleLineMessageOpen:
state = .readingMessage(isSingleLine: true)
case EnclosingControlCharacters.singleLineMessageClose:
returnToken = .newline
default:
// New types need to be registered.
throw ParserError.notParsable
Expand Down
46 changes: 41 additions & 5 deletions sources/LocalizationEditor/Providers/ParserTypes.swift
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -15,12 +15,36 @@ import Foundation
/// - key: The key and its text.
/// - equal: The equal sign that maps a key to a value: ".
/// - semicolon: The semicolon that ends a line: ;
/// - newline: A new line \n.
enum Token {
case message(String)
case value(String)
case key(String)
case equal
case semicolon
case newline
/// Checks if `self` is of the same type as `other` without taking the associated values into account.
///
/// - Parameter other: The token to which self should be compared to.
/// - Returns: `true` when the type of `other` matches the type of `self` without taking associated values into account.
func isCaseEqual(to other: Token) -> Bool {
switch (self, other) {
case (.message, .message):
return true
case (.value, .value):
return true
case (.key, .key):
return true
case (.equal, .equal):
return true
case (.semicolon, .semicolon):
return true
case (.newline, .newline):
return true
default:
return false
}
}
}

/// Control characters define starting and end points of tokens. They can be for example ", /* or ;
Expand All@@ -35,13 +59,17 @@ protocol EnclosingType: ControlCharacterType {}
protocol SeperatingType: ControlCharacterType {}

/// Enclosing control characters that wrapp text. They may start or end a message or contain a value/key.
/// - messageBoundaryOpen Opens a message.
/// - messageBoundaryClose Ends a message.
/// - quote Wraps a key or a value.
/// - messageBoundaryOpen: Opens a message.
/// - messageBoundaryClose: Ends a message.
/// - quote: Wraps a key or a value.
/// - singleLineMessageOpen: Opens a single line message.
/// - singleLineMessageClose: Closes the single line message.
enum EnclosingControlCharacters: String, EnclosingType, CaseIterable {
case messageBoundaryOpen = "/*"
case messageBoundaryClose = "*/"
case quote = "\""
case singleLineMessageOpen = "//"
case singleLineMessageClose = "\n"

var skippingLength: Int {
switch self {
Expand All@@ -51,25 +79,33 @@ enum EnclosingControlCharacters: String, EnclosingType, CaseIterable {
return EnclosingControlCharacters.messageBoundaryClose.rawValue.count
case .quote:
return EnclosingControlCharacters.quote.rawValue.count
case .singleLineMessageOpen:
return EnclosingControlCharacters.singleLineMessageOpen.rawValue.count
case .singleLineMessageClose:
return EnclosingControlCharacters.singleLineMessageClose.rawValue.count
}
}
}

/// Seperating control characters do not wrap text. They function as position markers. For example they seperate a key from its value or end the line.
/// - equal The equal sign that seperates a key from its value.
/// - semicolon The semicolon that end a line.
/// - equal: The equal sign that seperates a key from its value.
/// - semicolon: The semicolon that end a line.
/// - newline: A new line.
enum SeperatingControlCharacters: String, SeperatingType, CaseIterable {
var skippingLength: Int {
switch self {
case .equal:
return SeperatingControlCharacters.equal.rawValue.count
case .semicolon:
return SeperatingControlCharacters.semicolon.rawValue.count
case .newline:
return SeperatingControlCharacters.newline.rawValue.count
}
}

case equal = "="
case semicolon = ";"
case newline = "\n"
}

/// Errors that may occure during parsing.
Expand Down
Loading
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Add copy buttons to all 
 blocks\n(function() {\n function addCopyButtons() {\n document.querySelectorAll('pre code').forEach(function(codeBlock) {\n if (codeBlock.parentElement.hasAttribute('data-copy-added')) return;\n codeBlock.parentElement.setAttribute('data-copy-added', 'true');\n \n var btn = document.createElement('button');\n btn.textContent = 'Copy';\n btn.style.cssText = 'position:absolute;top:4px;right:4px;padding:2px 8px;font-size:11px;background:#4ecdc4;border:none;border-radius:4px;color:#1a1a2e;cursor:pointer;opacity:0.7;transition:opacity 0.2s;';\n btn.onmouseover = function() { this.style.opacity = '1'; };\n btn.onmouseout = function() { this.style.opacity = '0.7'; };\n btn.onclick = function() {\n navigator.clipboard.writeText(codeBlock.textContent).then(function() {\n btn.textContent = 'Copied!';\n setTimeout(function() { btn.textContent = 'Copy'; }, 1500);\n });\n };\n codeBlock.parentElement.style.position = 'relative';\n codeBlock.parentElement.appendChild(btn);\n });\n }\n \n addCopyButtons();\n \n // Re-run on dynamic content\n var observer = new MutationObserver(addCopyButtons);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Add Copy Buttons to Code Blocks");
}
} catch(__e) { console.warn('[Userscript:Add Copy Buttons to Code Blocks]', __e); }
})();
(function(){
try {
var __m = "github.com";
var __re = new RegExp('^' + "github\\.com" + '

Skip to content
Comments at the end of line are treated as messages and integrated single line comments#53

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

 Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and
privacy statement. We’ll occasionally send you account related emails.

Already on GitHub?
Sign in
to your account

 Merged

igorkulman
merged 4 commits into

from

May 25, 2019
 Merged


Comments at the end of line are treated as messages and integrated single line comments
#53
Show file tree
Hide file tree
 Changes from all commits
Commits

Show all changes


4 commits


Select commit
Hold shift + click to select a range
cd718d5

Adapted parser for message at the end of a line instead of before an …


Tantalum73 May 24, 2019
d0426a4

Introduced singleLineMessageOpen and singleLineMessageClose as a star…


Tantalum73 May 25, 2019
ddf95d5

Working integration of single line comments.


Tantalum73 May 25, 2019
9db3dcc

Some cleanup.


Tantalum73 May 25, 2019

File filter

Filter by extension
Filter by extension
Conversations

Failed to load comments. 
Loading
Jump to

Jump to file


Failed to load files. 
Loading
 Diff view

 Diff view


2 changes: 1 addition & 1 deletion
sources/LocalizationEditor/Info.plist

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -19,7 +19,7 @@
	<key>CFBundleShortVersionString</key>
	<string>2.1</string>
	<key>CFBundleVersion</key>
	<string>191</string>
	<string>198</string>
	<key>LSMinimumSystemVersion</key>
	<string>$(MACOSX_DEPLOYMENT_TARGET)</string>
	<key>NSHumanReadableCopyright</key>
Expand Down

147 changes: 85 additions & 62 deletions
sources/LocalizationEditor/Providers/Parser.swift

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -26,7 +26,7 @@ class Parser {
 fileprivate enum ParserState {
 case readingKey
 case readingValue
 case readingMessage
 case readingMessage(isSingleLine: Bool)
 case other
 }
 /// The current state of the parser.
Expand All@@ -53,23 +53,6 @@ class Parser {
 return results
 }

 /**
 Special handling for single line comments to turn them into message tokens. Should only be called when state is other so // in a middle of value does not get caught
 */
 private func skipAndProcessSingleLineComments() {
 while !input.isEmpty, let character = String(input[input.startIndex]).unicodeScalars.first, CharacterSet.whitespacesAndNewlines.contains(character) {
 input.remove(at: input.index(input.startIndex, offsetBy: 0))
 }

 if input.hasPrefix("//"), let endIndex = input.index(of: "\n") {
 let messageRange = input.index(input.startIndex, offsetBy: 2) ..< endIndex
 tokens.append(.message(String(input[messageRange])))

 let rangeForRemoving = input.startIndex ..< endIndex
 input.removeSubrange(rangeForRemoving)
 }
 }

 /**
 This function reads through the input and populates an array of tokens.

Expand All@@ -82,49 +65,56 @@ class Parser {
 // Actions depend on the current state.
 switch state {
 case .other:
 skipAndProcessSingleLineComments()

 // Extract the upcoming control character, also switch the current state and append the extracted token, if any.
 if let extractedToken = try prepareNextState() {
 tokens.append(extractedToken)
 }
 case .readingKey:
 // Until the key-end marker is reached, the text should be interpreted as key.
 let currentKeyText = extractText(until: .quote)
 let potentialNewToken: Token = .key(currentKeyText)
 // If the prior token was also a key, append it.
 let newToken = tokenByConcatinatingwithPriorToken(potentialNewToken, seperatingString: EnclosingControlCharacters.quote.rawValue)
 tokens.append(newToken)
 // If the upcoming control character is also a key, do not stop reading a key. Otherwise a unescaped quote may exclude text from the key. Otherwise the state may be anything else.
 if let nextControlCharacter = findNextControlCharacter(andExtractFromSource: false), case EnclosingControlCharacters.quote = nextControlCharacter {
 state = .readingKey
 } else {
 state = .other
 }
 extractAndAppendIfPossible(for: .key(""), until: .quote)
 case .readingValue:
 // Text until value-end marker is a value.
 // If the prior token as also a value, append it.
 let currentValueText = extractText(until: .quote)
 let potentialNewToken: Token = .value(currentValueText)
 // If the prior token was also a key, append it.
 let newToken = tokenByConcatinatingwithPriorToken(potentialNewToken, seperatingString: EnclosingControlCharacters.quote.rawValue)
 tokens.append(newToken)
 // If the upcoming control character is also a key, do not stop reading a value. Otherwise a unescaped quote may exclude text from the value. Otherwise the state may be anything else.
 if let nextControlCharacter = findNextControlCharacter(andExtractFromSource: false), case EnclosingControlCharacters.quote = nextControlCharacter {
 state = .readingValue
 } else {
 state = .other
 }
 case .readingMessage:
 // Text until value-end marker is a message.
 extractAndAppendIfPossible(for: .value(""), until: .quote)
 case .readingMessage(let isReadingSingleLine):
 // If the prior token as also a message, DO NOT append it since the prior message could be a license header.
 let currentMessageText = extractText(until: .messageBoundaryClose)
 let endMarker: EnclosingControlCharacters = isReadingSingleLine ? .singleLineMessageClose : .messageBoundaryClose
 let currentMessageText = extractText(until: endMarker)
 let newToken: Token = .message(currentMessageText)
 tokens.append(newToken)
 state = .other
 }
 }
 }
 /// Extracts text from the input until the end marker is reached. Uses that text to create a new token and appends it to a prior extracted token if possible. In any case it updates the current list of extracted tokens.
 ///
 /// - Parameters:
 /// - token: The type of token that should be created from the text before the end marker. The associated value of the input is ignored.
 /// - endMarker: Marks the end of the tokens content.
 private func extractAndAppendIfPossible(for token: Token, until endMarker: EnclosingControlCharacters) {
 let currentText = extractText(until: endMarker)
 let potentialNewToken: Token
 switch token {
 case .key:
 potentialNewToken = .key(currentText)
 case .value:
 potentialNewToken = .value(currentText)
 default:
 assertionFailure("Currently, only the .key and .value support joining.")
 return
 }
 // Append to the prior token if possible.
 let newToken = tokenByConcatinatingwithPriorToken(potentialNewToken, seperatingString: endMarker.rawValue)
 tokens.append(newToken)
 // Do not stop reading when a newline or a quote is the next control character. Otherwise an unescaped quote may exclude text from the value. Keep the state unchanged if any other control character follows.
 if let nextControlCharacter = findNextControlCharacter(andExtractFromSource: false) {
 switch nextControlCharacter {
 case SeperatingControlCharacters.newline, EnclosingControlCharacters.singleLineMessageClose, EnclosingControlCharacters.quote:
 // Do not change the state and just continue.
 return
 default:
 break
 }
 }
 state = .other
 }
 /// Call this method when the list of tokens is ready and model object can be created. It will iterate through the tokens and try to map their values into model objects. Whe the mapping failed, an error is thrown.
 ///
 /// - Returns: The extracted model values.
Expand All@@ -134,6 +124,25 @@ class Parser {
 var currentKey: String?
 var currentValue: String?
 var results = [LocalizationString]()
 // The token that delimits an entry.
 guard let endToken = entriesEndToken(for: tokens) else {
 throw ParserError.malformattedInput
 }
 // Generates a result and appends it to the list of results if possible.
 func generateResultIfPossible(from processedToken: Token) {
 guard processedToken.isCaseEqual(to: endToken) else { return }
 // Done with that line. Check if values are populated and append them to the results.
 guard let key = currentKey, let value = currentValue else {
 return
 }
 let correctedMessage = removeLeadingTrailingSpaces(from: currentMessage)
 let entry = LocalizationString(key: key, value: value.replacingOccurrences(of: "\\\"", with: "\""), message: correctedMessage)
 results.append(entry)
 // Reset the properties to be ready for the next line.
 currentValue = nil
 currentKey = nil
 currentMessage = nil
 }
 // Iterate through the tokens and transform them into model objects.
 for token in tokens {
 switch token {
Expand All@@ -143,28 +152,36 @@ class Parser {
 currentKey = containedText
 case .value(let containedText):
 currentValue = containedText
 case .semicolon:
 // Done with that line. Check if values are populated and append them to the results.
 guard let key = currentKey, let value = currentValue else {
 throw ParserError.malformattedInput
 }
 let correctedMessage = removeLeadingTrailingSpaces(from: currentMessage)
 let entry = LocalizationString(key: key, value: value.replacingOccurrences(of: "\\\"", with: "\""), message: correctedMessage)
 results.append(entry)
 // Reset the properties to be ready for the next line.
 currentValue = nil
 currentKey = nil
 currentMessage = nil
 default:
 ()
 }
 generateResultIfPossible(from: token)
 }
 // Throw an execption to indicate that something went wront when tokens are extracted but they could not be transferred into model objects:
 if !tokens.isEmpty && results.isEmpty {
 throw ParserError.malformattedInput
 }
 return results
 }
 /// Determines the token that ends an entry. An entry can either be ended by a semicolon (if no comment was provided or the comment is above the entry) or a comment located at the end of a line. In the second case the `.message` token marks the end of the entry.
 ///
 /// - Parameter tokens: The tokens that were extracted during tokenization.
 /// - Returns: The token that ends an entry.
 private func entriesEndToken(for tokens: [Token]) -> Token? {
 // Assumption: after the first semicolon comes a new line -> semicolon delimits entry
 // After first semicolon comes a message, followed by a new line -> message delimits entry
 guard let semicolonIndex = tokens.firstIndex(where: { $0.isCaseEqual(to: .semicolon) }) else {
 return nil
 }
 guard let indexAfterSemicolon = tokens.index(semicolonIndex, offsetBy: 1, limitedBy: tokens.endIndex - 1) else { return nil }
 let elementAfterSemicolon = tokens[indexAfterSemicolon]
 switch elementAfterSemicolon {
 case .newline:
 return .semicolon
 default:
 return elementAfterSemicolon
 }
 }
 /// This function removes leading and trailing spaces from the input.
 ///
 /// - Parameter input: The string whose leading and trailing spaces should be removed.
Expand DownExpand Up@@ -265,7 +282,7 @@ class Parser {
 // Extract the given range and remove it from the input string.

 let lengthOfControlCharacter: Int = includingControlCharacter.skippingLength
 let endIndexOfExtraction = input.index(endindex, offsetBy: lengthOfControlCharacter)
 let endIndexOfExtraction = input.index(endindex, offsetBy: lengthOfControlCharacter, limitedBy: input.endIndex) ?? input.endIndex
 // Remove the range that includes the control character. The input range is used for extracting the text before it.
 let rangeForRemoving = input.startIndex ..< endIndexOfExtraction
 let rangeForExtraction = input.startIndex ..< endindex
Expand DownExpand Up@@ -354,7 +371,7 @@ extension Parser {
 case EnclosingControlCharacters.messageBoundaryOpen:
 // A new message begins.
 // Set the state to expect a message.
 state = .readingMessage
 state = .readingMessage(isSingleLine: false)
 case EnclosingControlCharacters.messageBoundaryClose:
 // Message-end markers should only be detected when the lexer is reading a message. If they occure 'in the wild' the input must be ill formatted.
 break
Expand All@@ -366,6 +383,12 @@ extension Parser {
 // Extract semicolon as token. A quote or message-start mark will follow as next control character but for now the state remains .other in order to detect that quote.
 returnToken = .semicolon
 state = .other
 case SeperatingControlCharacters.newline:
 returnToken = .newline
 case EnclosingControlCharacters.singleLineMessageOpen:
 state = .readingMessage(isSingleLine: true)
 case EnclosingControlCharacters.singleLineMessageClose:
 returnToken = .newline
 default:
 // New types need to be registered.
 throw ParserError.notParsable
Expand Down

46 changes: 41 additions & 5 deletions
sources/LocalizationEditor/Providers/ParserTypes.swift

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -15,12 +15,36 @@ import Foundation
/// - key: The key and its text.
/// - equal: The equal sign that maps a key to a value: ".
/// - semicolon: The semicolon that ends a line: ;
/// - newline: A new line \n.
enum Token {
 case message(String)
 case value(String)
 case key(String)
 case equal
 case semicolon
 case newline
 /// Checks if `self` is of the same type as `other` without taking the associated values into account.
 ///
 /// - Parameter other: The token to which self should be compared to.
 /// - Returns: `true` when the type of `other` matches the type of `self` without taking associated values into account.
 func isCaseEqual(to other: Token) -> Bool {
 switch (self, other) {
 case (.message, .message):
 return true
 case (.value, .value):
 return true
 case (.key, .key):
 return true
 case (.equal, .equal):
 return true
 case (.semicolon, .semicolon):
 return true
 case (.newline, .newline):
 return true
 default:
 return false
 }
 }
}

/// Control characters define starting and end points of tokens. They can be for example ", /* or ;
Expand All@@ -35,13 +59,17 @@ protocol EnclosingType: ControlCharacterType {}
protocol SeperatingType: ControlCharacterType {}

/// Enclosing control characters that wrapp text. They may start or end a message or contain a value/key.
/// - messageBoundaryOpen Opens a message.
/// - messageBoundaryClose Ends a message.
/// - quote Wraps a key or a value.
/// - messageBoundaryOpen: Opens a message.
/// - messageBoundaryClose: Ends a message.
/// - quote: Wraps a key or a value.
/// - singleLineMessageOpen: Opens a single line message.
/// - singleLineMessageClose: Closes the single line message.
enum EnclosingControlCharacters: String, EnclosingType, CaseIterable {
 case messageBoundaryOpen = "/*"
 case messageBoundaryClose = "*/"
 case quote = "\""
 case singleLineMessageOpen = "//"
 case singleLineMessageClose = "\n"

 var skippingLength: Int {
 switch self {
Expand All@@ -51,25 +79,33 @@ enum EnclosingControlCharacters: String, EnclosingType, CaseIterable {
 return EnclosingControlCharacters.messageBoundaryClose.rawValue.count
 case .quote:
 return EnclosingControlCharacters.quote.rawValue.count
 case .singleLineMessageOpen:
 return EnclosingControlCharacters.singleLineMessageOpen.rawValue.count
 case .singleLineMessageClose:
 return EnclosingControlCharacters.singleLineMessageClose.rawValue.count
 }
 }
}

/// Seperating control characters do not wrap text. They function as position markers. For example they seperate a key from its value or end the line.
/// - equal The equal sign that seperates a key from its value.
/// - semicolon The semicolon that end a line.
/// - equal: The equal sign that seperates a key from its value.
/// - semicolon: The semicolon that end a line.
/// - newline: A new line.
enum SeperatingControlCharacters: String, SeperatingType, CaseIterable {
 var skippingLength: Int {
 switch self {
 case .equal:
 return SeperatingControlCharacters.equal.rawValue.count
 case .semicolon:
 return SeperatingControlCharacters.semicolon.rawValue.count
 case .newline:
 return SeperatingControlCharacters.newline.rawValue.count
 }
 }

 case equal = "="
 case semicolon = ";"
 case newline = "\n"
}

/// Errors that may occure during parsing.
Expand Down
Loading

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Force GitHub README to respect dark mode\n(function() {\n var style = document.createElement('style');\n style.textContent = '\n .markdown-body {\n color-scheme: dark light;\n }\n .markdown-body pre { background: #161b22 !important; }\n .markdown-body code { background: rgba(110, 118, 129, 0.4) !important; }\n .markdown-body table th, .markdown-body table td { border-color: #30363d !important; }\n .markdown-body img { background: #0d1117; }\n .markdown-body blockquote { border-left-color: #8b949e; }\n .markdown-body hr { border-color: #30363d; }\n ';\n document.head.appendChild(style);\n})();", "GitHub Dark Mode README Fix");
}
} catch(__e) { console.warn('[Userscript:GitHub Dark Mode README Fix]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
Comments at the end of line are treated as messages and integrated single line comments#53

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

 Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and
privacy statement. We’ll occasionally send you account related emails.

Already on GitHub?
Sign in
to your account

 Merged

igorkulman
merged 4 commits into

from

May 25, 2019
 Merged


Comments at the end of line are treated as messages and integrated single line comments
#53
Show file tree
Hide file tree
 Changes from all commits
Commits

Show all changes


4 commits


Select commit
Hold shift + click to select a range
cd718d5

Adapted parser for message at the end of a line instead of before an …


Tantalum73 May 24, 2019
d0426a4

Introduced singleLineMessageOpen and singleLineMessageClose as a star…


Tantalum73 May 25, 2019
ddf95d5

Working integration of single line comments.


Tantalum73 May 25, 2019
9db3dcc

Some cleanup.


Tantalum73 May 25, 2019

File filter

Filter by extension
Filter by extension
Conversations

Failed to load comments. 
Loading
Jump to

Jump to file


Failed to load files. 
Loading
 Diff view

 Diff view


2 changes: 1 addition & 1 deletion
sources/LocalizationEditor/Info.plist

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -19,7 +19,7 @@
	<key>CFBundleShortVersionString</key>
	<string>2.1</string>
	<key>CFBundleVersion</key>
	<string>191</string>
	<string>198</string>
	<key>LSMinimumSystemVersion</key>
	<string>$(MACOSX_DEPLOYMENT_TARGET)</string>
	<key>NSHumanReadableCopyright</key>
Expand Down

147 changes: 85 additions & 62 deletions
sources/LocalizationEditor/Providers/Parser.swift

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -26,7 +26,7 @@ class Parser {
 fileprivate enum ParserState {
 case readingKey
 case readingValue
 case readingMessage
 case readingMessage(isSingleLine: Bool)
 case other
 }
 /// The current state of the parser.
Expand All@@ -53,23 +53,6 @@ class Parser {
 return results
 }

 /**
 Special handling for single line comments to turn them into message tokens. Should only be called when state is other so // in a middle of value does not get caught
 */
 private func skipAndProcessSingleLineComments() {
 while !input.isEmpty, let character = String(input[input.startIndex]).unicodeScalars.first, CharacterSet.whitespacesAndNewlines.contains(character) {
 input.remove(at: input.index(input.startIndex, offsetBy: 0))
 }

 if input.hasPrefix("//"), let endIndex = input.index(of: "\n") {
 let messageRange = input.index(input.startIndex, offsetBy: 2) ..< endIndex
 tokens.append(.message(String(input[messageRange])))

 let rangeForRemoving = input.startIndex ..< endIndex
 input.removeSubrange(rangeForRemoving)
 }
 }

 /**
 This function reads through the input and populates an array of tokens.

Expand All@@ -82,49 +65,56 @@ class Parser {
 // Actions depend on the current state.
 switch state {
 case .other:
 skipAndProcessSingleLineComments()

 // Extract the upcoming control character, also switch the current state and append the extracted token, if any.
 if let extractedToken = try prepareNextState() {
 tokens.append(extractedToken)
 }
 case .readingKey:
 // Until the key-end marker is reached, the text should be interpreted as key.
 let currentKeyText = extractText(until: .quote)
 let potentialNewToken: Token = .key(currentKeyText)
 // If the prior token was also a key, append it.
 let newToken = tokenByConcatinatingwithPriorToken(potentialNewToken, seperatingString: EnclosingControlCharacters.quote.rawValue)
 tokens.append(newToken)
 // If the upcoming control character is also a key, do not stop reading a key. Otherwise a unescaped quote may exclude text from the key. Otherwise the state may be anything else.
 if let nextControlCharacter = findNextControlCharacter(andExtractFromSource: false), case EnclosingControlCharacters.quote = nextControlCharacter {
 state = .readingKey
 } else {
 state = .other
 }
 extractAndAppendIfPossible(for: .key(""), until: .quote)
 case .readingValue:
 // Text until value-end marker is a value.
 // If the prior token as also a value, append it.
 let currentValueText = extractText(until: .quote)
 let potentialNewToken: Token = .value(currentValueText)
 // If the prior token was also a key, append it.
 let newToken = tokenByConcatinatingwithPriorToken(potentialNewToken, seperatingString: EnclosingControlCharacters.quote.rawValue)
 tokens.append(newToken)
 // If the upcoming control character is also a key, do not stop reading a value. Otherwise a unescaped quote may exclude text from the value. Otherwise the state may be anything else.
 if let nextControlCharacter = findNextControlCharacter(andExtractFromSource: false), case EnclosingControlCharacters.quote = nextControlCharacter {
 state = .readingValue
 } else {
 state = .other
 }
 case .readingMessage:
 // Text until value-end marker is a message.
 extractAndAppendIfPossible(for: .value(""), until: .quote)
 case .readingMessage(let isReadingSingleLine):
 // If the prior token as also a message, DO NOT append it since the prior message could be a license header.
 let currentMessageText = extractText(until: .messageBoundaryClose)
 let endMarker: EnclosingControlCharacters = isReadingSingleLine ? .singleLineMessageClose : .messageBoundaryClose
 let currentMessageText = extractText(until: endMarker)
 let newToken: Token = .message(currentMessageText)
 tokens.append(newToken)
 state = .other
 }
 }
 }
 /// Extracts text from the input until the end marker is reached. Uses that text to create a new token and appends it to a prior extracted token if possible. In any case it updates the current list of extracted tokens.
 ///
 /// - Parameters:
 /// - token: The type of token that should be created from the text before the end marker. The associated value of the input is ignored.
 /// - endMarker: Marks the end of the tokens content.
 private func extractAndAppendIfPossible(for token: Token, until endMarker: EnclosingControlCharacters) {
 let currentText = extractText(until: endMarker)
 let potentialNewToken: Token
 switch token {
 case .key:
 potentialNewToken = .key(currentText)
 case .value:
 potentialNewToken = .value(currentText)
 default:
 assertionFailure("Currently, only the .key and .value support joining.")
 return
 }
 // Append to the prior token if possible.
 let newToken = tokenByConcatinatingwithPriorToken(potentialNewToken, seperatingString: endMarker.rawValue)
 tokens.append(newToken)
 // Do not stop reading when a newline or a quote is the next control character. Otherwise an unescaped quote may exclude text from the value. Keep the state unchanged if any other control character follows.
 if let nextControlCharacter = findNextControlCharacter(andExtractFromSource: false) {
 switch nextControlCharacter {
 case SeperatingControlCharacters.newline, EnclosingControlCharacters.singleLineMessageClose, EnclosingControlCharacters.quote:
 // Do not change the state and just continue.
 return
 default:
 break
 }
 }
 state = .other
 }
 /// Call this method when the list of tokens is ready and model object can be created. It will iterate through the tokens and try to map their values into model objects. Whe the mapping failed, an error is thrown.
 ///
 /// - Returns: The extracted model values.
Expand All@@ -134,6 +124,25 @@ class Parser {
 var currentKey: String?
 var currentValue: String?
 var results = [LocalizationString]()
 // The token that delimits an entry.
 guard let endToken = entriesEndToken(for: tokens) else {
 throw ParserError.malformattedInput
 }
 // Generates a result and appends it to the list of results if possible.
 func generateResultIfPossible(from processedToken: Token) {
 guard processedToken.isCaseEqual(to: endToken) else { return }
 // Done with that line. Check if values are populated and append them to the results.
 guard let key = currentKey, let value = currentValue else {
 return
 }
 let correctedMessage = removeLeadingTrailingSpaces(from: currentMessage)
 let entry = LocalizationString(key: key, value: value.replacingOccurrences(of: "\\\"", with: "\""), message: correctedMessage)
 results.append(entry)
 // Reset the properties to be ready for the next line.
 currentValue = nil
 currentKey = nil
 currentMessage = nil
 }
 // Iterate through the tokens and transform them into model objects.
 for token in tokens {
 switch token {
Expand All@@ -143,28 +152,36 @@ class Parser {
 currentKey = containedText
 case .value(let containedText):
 currentValue = containedText
 case .semicolon:
 // Done with that line. Check if values are populated and append them to the results.
 guard let key = currentKey, let value = currentValue else {
 throw ParserError.malformattedInput
 }
 let correctedMessage = removeLeadingTrailingSpaces(from: currentMessage)
 let entry = LocalizationString(key: key, value: value.replacingOccurrences(of: "\\\"", with: "\""), message: correctedMessage)
 results.append(entry)
 // Reset the properties to be ready for the next line.
 currentValue = nil
 currentKey = nil
 currentMessage = nil
 default:
 ()
 }
 generateResultIfPossible(from: token)
 }
 // Throw an execption to indicate that something went wront when tokens are extracted but they could not be transferred into model objects:
 if !tokens.isEmpty && results.isEmpty {
 throw ParserError.malformattedInput
 }
 return results
 }
 /// Determines the token that ends an entry. An entry can either be ended by a semicolon (if no comment was provided or the comment is above the entry) or a comment located at the end of a line. In the second case the `.message` token marks the end of the entry.
 ///
 /// - Parameter tokens: The tokens that were extracted during tokenization.
 /// - Returns: The token that ends an entry.
 private func entriesEndToken(for tokens: [Token]) -> Token? {
 // Assumption: after the first semicolon comes a new line -> semicolon delimits entry
 // After first semicolon comes a message, followed by a new line -> message delimits entry
 guard let semicolonIndex = tokens.firstIndex(where: { $0.isCaseEqual(to: .semicolon) }) else {
 return nil
 }
 guard let indexAfterSemicolon = tokens.index(semicolonIndex, offsetBy: 1, limitedBy: tokens.endIndex - 1) else { return nil }
 let elementAfterSemicolon = tokens[indexAfterSemicolon]
 switch elementAfterSemicolon {
 case .newline:
 return .semicolon
 default:
 return elementAfterSemicolon
 }
 }
 /// This function removes leading and trailing spaces from the input.
 ///
 /// - Parameter input: The string whose leading and trailing spaces should be removed.
Expand DownExpand Up@@ -265,7 +282,7 @@ class Parser {
 // Extract the given range and remove it from the input string.

 let lengthOfControlCharacter: Int = includingControlCharacter.skippingLength
 let endIndexOfExtraction = input.index(endindex, offsetBy: lengthOfControlCharacter)
 let endIndexOfExtraction = input.index(endindex, offsetBy: lengthOfControlCharacter, limitedBy: input.endIndex) ?? input.endIndex
 // Remove the range that includes the control character. The input range is used for extracting the text before it.
 let rangeForRemoving = input.startIndex ..< endIndexOfExtraction
 let rangeForExtraction = input.startIndex ..< endindex
Expand DownExpand Up@@ -354,7 +371,7 @@ extension Parser {
 case EnclosingControlCharacters.messageBoundaryOpen:
 // A new message begins.
 // Set the state to expect a message.
 state = .readingMessage
 state = .readingMessage(isSingleLine: false)
 case EnclosingControlCharacters.messageBoundaryClose:
 // Message-end markers should only be detected when the lexer is reading a message. If they occure 'in the wild' the input must be ill formatted.
 break
Expand All@@ -366,6 +383,12 @@ extension Parser {
 // Extract semicolon as token. A quote or message-start mark will follow as next control character but for now the state remains .other in order to detect that quote.
 returnToken = .semicolon
 state = .other
 case SeperatingControlCharacters.newline:
 returnToken = .newline
 case EnclosingControlCharacters.singleLineMessageOpen:
 state = .readingMessage(isSingleLine: true)
 case EnclosingControlCharacters.singleLineMessageClose:
 returnToken = .newline
 default:
 // New types need to be registered.
 throw ParserError.notParsable
Expand Down

46 changes: 41 additions & 5 deletions
sources/LocalizationEditor/Providers/ParserTypes.swift

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -15,12 +15,36 @@ import Foundation
/// - key: The key and its text.
/// - equal: The equal sign that maps a key to a value: ".
/// - semicolon: The semicolon that ends a line: ;
/// - newline: A new line \n.
enum Token {
 case message(String)
 case value(String)
 case key(String)
 case equal
 case semicolon
 case newline
 /// Checks if `self` is of the same type as `other` without taking the associated values into account.
 ///
 /// - Parameter other: The token to which self should be compared to.
 /// - Returns: `true` when the type of `other` matches the type of `self` without taking associated values into account.
 func isCaseEqual(to other: Token) -> Bool {
 switch (self, other) {
 case (.message, .message):
 return true
 case (.value, .value):
 return true
 case (.key, .key):
 return true
 case (.equal, .equal):
 return true
 case (.semicolon, .semicolon):
 return true
 case (.newline, .newline):
 return true
 default:
 return false
 }
 }
}

/// Control characters define starting and end points of tokens. They can be for example ", /* or ;
Expand All@@ -35,13 +59,17 @@ protocol EnclosingType: ControlCharacterType {}
protocol SeperatingType: ControlCharacterType {}

/// Enclosing control characters that wrapp text. They may start or end a message or contain a value/key.
/// - messageBoundaryOpen Opens a message.
/// - messageBoundaryClose Ends a message.
/// - quote Wraps a key or a value.
/// - messageBoundaryOpen: Opens a message.
/// - messageBoundaryClose: Ends a message.
/// - quote: Wraps a key or a value.
/// - singleLineMessageOpen: Opens a single line message.
/// - singleLineMessageClose: Closes the single line message.
enum EnclosingControlCharacters: String, EnclosingType, CaseIterable {
 case messageBoundaryOpen = "/*"
 case messageBoundaryClose = "*/"
 case quote = "\""
 case singleLineMessageOpen = "//"
 case singleLineMessageClose = "\n"

 var skippingLength: Int {
 switch self {
Expand All@@ -51,25 +79,33 @@ enum EnclosingControlCharacters: String, EnclosingType, CaseIterable {
 return EnclosingControlCharacters.messageBoundaryClose.rawValue.count
 case .quote:
 return EnclosingControlCharacters.quote.rawValue.count
 case .singleLineMessageOpen:
 return EnclosingControlCharacters.singleLineMessageOpen.rawValue.count
 case .singleLineMessageClose:
 return EnclosingControlCharacters.singleLineMessageClose.rawValue.count
 }
 }
}

/// Seperating control characters do not wrap text. They function as position markers. For example they seperate a key from its value or end the line.
/// - equal The equal sign that seperates a key from its value.
/// - semicolon The semicolon that end a line.
/// - equal: The equal sign that seperates a key from its value.
/// - semicolon: The semicolon that end a line.
/// - newline: A new line.
enum SeperatingControlCharacters: String, SeperatingType, CaseIterable {
 var skippingLength: Int {
 switch self {
 case .equal:
 return SeperatingControlCharacters.equal.rawValue.count
 case .semicolon:
 return SeperatingControlCharacters.semicolon.rawValue.count
 case .newline:
 return SeperatingControlCharacters.newline.rawValue.count
 }
 }

 case equal = "="
 case semicolon = ";"
 case newline = "\n"
}

/// Errors that may occure during parsing.
Expand Down
Loading

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Highlight search terms from Google/DuckDuckGo/Bing referrer\n(function() {\n var ref = document.referrer;\n var terms = [];\n \n if (ref.includes('google.com') || ref.includes('duckduckgo.com') || ref.includes('bing.com')) {\n var url = new URL(ref);\n var q = url.searchParams.get('q') || url.searchParams.get('p');\n if (q) {\n terms = q.split(/\\s+/).filter(function(t) { return t.length > 2; });\n }\n }\n \n if (terms.length === 0) return;\n \n var style = document.createElement('style');\n style.textContent = '.userscript-highlight { background: #fbbf24; color: #1a1a2e; padding: 1px 3px; border-radius: 2px; }';\n document.head.appendChild(style);\n \n function highlight(node) {\n if (node.nodeType === 3) { // text node\n var text = node.textContent;\n var found = false;\n terms.forEach(function(term) {\n var regex = new RegExp('(' + term.replace(/[.*+?^${}()|[\\]\\\\]/g, '\\\\') + ')', 'gi');\n if (regex.test(text)) {\n found = true;\n var frag = document.createDocumentFragment();\n var parts = text.split(regex);\n parts.forEach(function(part, i) {\n if (i % 2 === 0) {\n frag.appendChild(document.createTextNode(part));\n } else {\n var span = document.createElement('span');\n span.className = 'userscript-highlight';\n span.textContent = part;\n frag.appendChild(span);\n }\n });\n node.parentNode.replaceChild(frag, node);\n }\n });\n } else if (node.nodeType === 1 && node.childNodes) { // element\n var skipTags = ['SCRIPT', 'STYLE', 'NOSCRIPT', 'TEXTAREA', 'INPUT', 'SELECT'];\n if (!skipTags.includes(node.tagName)) {\n Array.from(node.childNodes).forEach(highlight);\n }\n }\n }\n \n highlight(document.body);\n \n // Re-highlight on dynamic content\n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1 || node.nodeType === 3) highlight(node);\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Highlight Search Terms");
}
} catch(__e) { console.warn('[Userscript:Highlight Search Terms]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
Comments at the end of line are treated as messages and integrated single line comments#53

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

 Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and
privacy statement. We’ll occasionally send you account related emails.

Already on GitHub?
Sign in
to your account

 Merged

igorkulman
merged 4 commits into

from

May 25, 2019
 Merged


Comments at the end of line are treated as messages and integrated single line comments
#53
Show file tree
Hide file tree
 Changes from all commits
Commits

Show all changes


4 commits


Select commit
Hold shift + click to select a range
cd718d5

Adapted parser for message at the end of a line instead of before an …


Tantalum73 May 24, 2019
d0426a4

Introduced singleLineMessageOpen and singleLineMessageClose as a star…


Tantalum73 May 25, 2019
ddf95d5

Working integration of single line comments.


Tantalum73 May 25, 2019
9db3dcc

Some cleanup.


Tantalum73 May 25, 2019

File filter

Filter by extension
Filter by extension
Conversations

Failed to load comments. 
Loading
Jump to

Jump to file


Failed to load files. 
Loading
 Diff view

 Diff view


2 changes: 1 addition & 1 deletion
sources/LocalizationEditor/Info.plist

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -19,7 +19,7 @@
	<key>CFBundleShortVersionString</key>
	<string>2.1</string>
	<key>CFBundleVersion</key>
	<string>191</string>
	<string>198</string>
	<key>LSMinimumSystemVersion</key>
	<string>$(MACOSX_DEPLOYMENT_TARGET)</string>
	<key>NSHumanReadableCopyright</key>
Expand Down

147 changes: 85 additions & 62 deletions
sources/LocalizationEditor/Providers/Parser.swift

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -26,7 +26,7 @@ class Parser {
 fileprivate enum ParserState {
 case readingKey
 case readingValue
 case readingMessage
 case readingMessage(isSingleLine: Bool)
 case other
 }
 /// The current state of the parser.
Expand All@@ -53,23 +53,6 @@ class Parser {
 return results
 }

 /**
 Special handling for single line comments to turn them into message tokens. Should only be called when state is other so // in a middle of value does not get caught
 */
 private func skipAndProcessSingleLineComments() {
 while !input.isEmpty, let character = String(input[input.startIndex]).unicodeScalars.first, CharacterSet.whitespacesAndNewlines.contains(character) {
 input.remove(at: input.index(input.startIndex, offsetBy: 0))
 }

 if input.hasPrefix("//"), let endIndex = input.index(of: "\n") {
 let messageRange = input.index(input.startIndex, offsetBy: 2) ..< endIndex
 tokens.append(.message(String(input[messageRange])))

 let rangeForRemoving = input.startIndex ..< endIndex
 input.removeSubrange(rangeForRemoving)
 }
 }

 /**
 This function reads through the input and populates an array of tokens.

Expand All@@ -82,49 +65,56 @@ class Parser {
 // Actions depend on the current state.
 switch state {
 case .other:
 skipAndProcessSingleLineComments()

 // Extract the upcoming control character, also switch the current state and append the extracted token, if any.
 if let extractedToken = try prepareNextState() {
 tokens.append(extractedToken)
 }
 case .readingKey:
 // Until the key-end marker is reached, the text should be interpreted as key.
 let currentKeyText = extractText(until: .quote)
 let potentialNewToken: Token = .key(currentKeyText)
 // If the prior token was also a key, append it.
 let newToken = tokenByConcatinatingwithPriorToken(potentialNewToken, seperatingString: EnclosingControlCharacters.quote.rawValue)
 tokens.append(newToken)
 // If the upcoming control character is also a key, do not stop reading a key. Otherwise a unescaped quote may exclude text from the key. Otherwise the state may be anything else.
 if let nextControlCharacter = findNextControlCharacter(andExtractFromSource: false), case EnclosingControlCharacters.quote = nextControlCharacter {
 state = .readingKey
 } else {
 state = .other
 }
 extractAndAppendIfPossible(for: .key(""), until: .quote)
 case .readingValue:
 // Text until value-end marker is a value.
 // If the prior token as also a value, append it.
 let currentValueText = extractText(until: .quote)
 let potentialNewToken: Token = .value(currentValueText)
 // If the prior token was also a key, append it.
 let newToken = tokenByConcatinatingwithPriorToken(potentialNewToken, seperatingString: EnclosingControlCharacters.quote.rawValue)
 tokens.append(newToken)
 // If the upcoming control character is also a key, do not stop reading a value. Otherwise a unescaped quote may exclude text from the value. Otherwise the state may be anything else.
 if let nextControlCharacter = findNextControlCharacter(andExtractFromSource: false), case EnclosingControlCharacters.quote = nextControlCharacter {
 state = .readingValue
 } else {
 state = .other
 }
 case .readingMessage:
 // Text until value-end marker is a message.
 extractAndAppendIfPossible(for: .value(""), until: .quote)
 case .readingMessage(let isReadingSingleLine):
 // If the prior token as also a message, DO NOT append it since the prior message could be a license header.
 let currentMessageText = extractText(until: .messageBoundaryClose)
 let endMarker: EnclosingControlCharacters = isReadingSingleLine ? .singleLineMessageClose : .messageBoundaryClose
 let currentMessageText = extractText(until: endMarker)
 let newToken: Token = .message(currentMessageText)
 tokens.append(newToken)
 state = .other
 }
 }
 }
 /// Extracts text from the input until the end marker is reached. Uses that text to create a new token and appends it to a prior extracted token if possible. In any case it updates the current list of extracted tokens.
 ///
 /// - Parameters:
 /// - token: The type of token that should be created from the text before the end marker. The associated value of the input is ignored.
 /// - endMarker: Marks the end of the tokens content.
 private func extractAndAppendIfPossible(for token: Token, until endMarker: EnclosingControlCharacters) {
 let currentText = extractText(until: endMarker)
 let potentialNewToken: Token
 switch token {
 case .key:
 potentialNewToken = .key(currentText)
 case .value:
 potentialNewToken = .value(currentText)
 default:
 assertionFailure("Currently, only the .key and .value support joining.")
 return
 }
 // Append to the prior token if possible.
 let newToken = tokenByConcatinatingwithPriorToken(potentialNewToken, seperatingString: endMarker.rawValue)
 tokens.append(newToken)
 // Do not stop reading when a newline or a quote is the next control character. Otherwise an unescaped quote may exclude text from the value. Keep the state unchanged if any other control character follows.
 if let nextControlCharacter = findNextControlCharacter(andExtractFromSource: false) {
 switch nextControlCharacter {
 case SeperatingControlCharacters.newline, EnclosingControlCharacters.singleLineMessageClose, EnclosingControlCharacters.quote:
 // Do not change the state and just continue.
 return
 default:
 break
 }
 }
 state = .other
 }
 /// Call this method when the list of tokens is ready and model object can be created. It will iterate through the tokens and try to map their values into model objects. Whe the mapping failed, an error is thrown.
 ///
 /// - Returns: The extracted model values.
Expand All@@ -134,6 +124,25 @@ class Parser {
 var currentKey: String?
 var currentValue: String?
 var results = [LocalizationString]()
 // The token that delimits an entry.
 guard let endToken = entriesEndToken(for: tokens) else {
 throw ParserError.malformattedInput
 }
 // Generates a result and appends it to the list of results if possible.
 func generateResultIfPossible(from processedToken: Token) {
 guard processedToken.isCaseEqual(to: endToken) else { return }
 // Done with that line. Check if values are populated and append them to the results.
 guard let key = currentKey, let value = currentValue else {
 return
 }
 let correctedMessage = removeLeadingTrailingSpaces(from: currentMessage)
 let entry = LocalizationString(key: key, value: value.replacingOccurrences(of: "\\\"", with: "\""), message: correctedMessage)
 results.append(entry)
 // Reset the properties to be ready for the next line.
 currentValue = nil
 currentKey = nil
 currentMessage = nil
 }
 // Iterate through the tokens and transform them into model objects.
 for token in tokens {
 switch token {
Expand All@@ -143,28 +152,36 @@ class Parser {
 currentKey = containedText
 case .value(let containedText):
 currentValue = containedText
 case .semicolon:
 // Done with that line. Check if values are populated and append them to the results.
 guard let key = currentKey, let value = currentValue else {
 throw ParserError.malformattedInput
 }
 let correctedMessage = removeLeadingTrailingSpaces(from: currentMessage)
 let entry = LocalizationString(key: key, value: value.replacingOccurrences(of: "\\\"", with: "\""), message: correctedMessage)
 results.append(entry)
 // Reset the properties to be ready for the next line.
 currentValue = nil
 currentKey = nil
 currentMessage = nil
 default:
 ()
 }
 generateResultIfPossible(from: token)
 }
 // Throw an execption to indicate that something went wront when tokens are extracted but they could not be transferred into model objects:
 if !tokens.isEmpty && results.isEmpty {
 throw ParserError.malformattedInput
 }
 return results
 }
 /// Determines the token that ends an entry. An entry can either be ended by a semicolon (if no comment was provided or the comment is above the entry) or a comment located at the end of a line. In the second case the `.message` token marks the end of the entry.
 ///
 /// - Parameter tokens: The tokens that were extracted during tokenization.
 /// - Returns: The token that ends an entry.
 private func entriesEndToken(for tokens: [Token]) -> Token? {
 // Assumption: after the first semicolon comes a new line -> semicolon delimits entry
 // After first semicolon comes a message, followed by a new line -> message delimits entry
 guard let semicolonIndex = tokens.firstIndex(where: { $0.isCaseEqual(to: .semicolon) }) else {
 return nil
 }
 guard let indexAfterSemicolon = tokens.index(semicolonIndex, offsetBy: 1, limitedBy: tokens.endIndex - 1) else { return nil }
 let elementAfterSemicolon = tokens[indexAfterSemicolon]
 switch elementAfterSemicolon {
 case .newline:
 return .semicolon
 default:
 return elementAfterSemicolon
 }
 }
 /// This function removes leading and trailing spaces from the input.
 ///
 /// - Parameter input: The string whose leading and trailing spaces should be removed.
Expand DownExpand Up@@ -265,7 +282,7 @@ class Parser {
 // Extract the given range and remove it from the input string.

 let lengthOfControlCharacter: Int = includingControlCharacter.skippingLength
 let endIndexOfExtraction = input.index(endindex, offsetBy: lengthOfControlCharacter)
 let endIndexOfExtraction = input.index(endindex, offsetBy: lengthOfControlCharacter, limitedBy: input.endIndex) ?? input.endIndex
 // Remove the range that includes the control character. The input range is used for extracting the text before it.
 let rangeForRemoving = input.startIndex ..< endIndexOfExtraction
 let rangeForExtraction = input.startIndex ..< endindex
Expand DownExpand Up@@ -354,7 +371,7 @@ extension Parser {
 case EnclosingControlCharacters.messageBoundaryOpen:
 // A new message begins.
 // Set the state to expect a message.
 state = .readingMessage
 state = .readingMessage(isSingleLine: false)
 case EnclosingControlCharacters.messageBoundaryClose:
 // Message-end markers should only be detected when the lexer is reading a message. If they occure 'in the wild' the input must be ill formatted.
 break
Expand All@@ -366,6 +383,12 @@ extension Parser {
 // Extract semicolon as token. A quote or message-start mark will follow as next control character but for now the state remains .other in order to detect that quote.
 returnToken = .semicolon
 state = .other
 case SeperatingControlCharacters.newline:
 returnToken = .newline
 case EnclosingControlCharacters.singleLineMessageOpen:
 state = .readingMessage(isSingleLine: true)
 case EnclosingControlCharacters.singleLineMessageClose:
 returnToken = .newline
 default:
 // New types need to be registered.
 throw ParserError.notParsable
Expand Down

46 changes: 41 additions & 5 deletions
sources/LocalizationEditor/Providers/ParserTypes.swift

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -15,12 +15,36 @@ import Foundation
/// - key: The key and its text.
/// - equal: The equal sign that maps a key to a value: ".
/// - semicolon: The semicolon that ends a line: ;
/// - newline: A new line \n.
enum Token {
 case message(String)
 case value(String)
 case key(String)
 case equal
 case semicolon
 case newline
 /// Checks if `self` is of the same type as `other` without taking the associated values into account.
 ///
 /// - Parameter other: The token to which self should be compared to.
 /// - Returns: `true` when the type of `other` matches the type of `self` without taking associated values into account.
 func isCaseEqual(to other: Token) -> Bool {
 switch (self, other) {
 case (.message, .message):
 return true
 case (.value, .value):
 return true
 case (.key, .key):
 return true
 case (.equal, .equal):
 return true
 case (.semicolon, .semicolon):
 return true
 case (.newline, .newline):
 return true
 default:
 return false
 }
 }
}

/// Control characters define starting and end points of tokens. They can be for example ", /* or ;
Expand All@@ -35,13 +59,17 @@ protocol EnclosingType: ControlCharacterType {}
protocol SeperatingType: ControlCharacterType {}

/// Enclosing control characters that wrapp text. They may start or end a message or contain a value/key.
/// - messageBoundaryOpen Opens a message.
/// - messageBoundaryClose Ends a message.
/// - quote Wraps a key or a value.
/// - messageBoundaryOpen: Opens a message.
/// - messageBoundaryClose: Ends a message.
/// - quote: Wraps a key or a value.
/// - singleLineMessageOpen: Opens a single line message.
/// - singleLineMessageClose: Closes the single line message.
enum EnclosingControlCharacters: String, EnclosingType, CaseIterable {
 case messageBoundaryOpen = "/*"
 case messageBoundaryClose = "*/"
 case quote = "\""
 case singleLineMessageOpen = "//"
 case singleLineMessageClose = "\n"

 var skippingLength: Int {
 switch self {
Expand All@@ -51,25 +79,33 @@ enum EnclosingControlCharacters: String, EnclosingType, CaseIterable {
 return EnclosingControlCharacters.messageBoundaryClose.rawValue.count
 case .quote:
 return EnclosingControlCharacters.quote.rawValue.count
 case .singleLineMessageOpen:
 return EnclosingControlCharacters.singleLineMessageOpen.rawValue.count
 case .singleLineMessageClose:
 return EnclosingControlCharacters.singleLineMessageClose.rawValue.count
 }
 }
}

/// Seperating control characters do not wrap text. They function as position markers. For example they seperate a key from its value or end the line.
/// - equal The equal sign that seperates a key from its value.
/// - semicolon The semicolon that end a line.
/// - equal: The equal sign that seperates a key from its value.
/// - semicolon: The semicolon that end a line.
/// - newline: A new line.
enum SeperatingControlCharacters: String, SeperatingType, CaseIterable {
 var skippingLength: Int {
 switch self {
 case .equal:
 return SeperatingControlCharacters.equal.rawValue.count
 case .semicolon:
 return SeperatingControlCharacters.semicolon.rawValue.count
 case .newline:
 return SeperatingControlCharacters.newline.rawValue.count
 }
 }

 case equal = "="
 case semicolon = ";"
 case newline = "\n"
}

/// Errors that may occure during parsing.
Expand Down
Loading

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Strip utm_, fbclid, gclid, etc. from all links on page\n(function() {\n var trackingParams = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content',\n 'fbclid', 'gclid', 'dclid', 'msclkid', 'yclid',\n 'ref', 'ref_src', 'source', 'medium', 'campaign'];\n \n function cleanUrl(url) {\n try {\n var u = new URL(url, window.location.origin);\n var changed = false;\n trackingParams.forEach(function(p) {\n if (u.searchParams.has(p)) {\n u.searchParams.delete(p);\n changed = true;\n }\n });\n return changed ? u.toString() : url;\n } catch (e) {\n return url;\n }\n }\n \n function cleanLinks() {\n document.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n \n cleanLinks();\n \n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1) {\n if (node.tagName === 'A') cleanLinks();\n node.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Remove Tracking Parameters from Links");
}
} catch(__e) { console.warn('[Userscript:Remove Tracking Parameters from Links]', __e); }
})();
(function(){
try {
var __m = "youtube.com";
var __re = new RegExp('^' + "youtube\\.com" + '

Skip to content
Comments at the end of line are treated as messages and integrated single line comments#53

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

 Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and
privacy statement. We’ll occasionally send you account related emails.

Already on GitHub?
Sign in
to your account

 Merged

igorkulman
merged 4 commits into

from

May 25, 2019
 Merged


Comments at the end of line are treated as messages and integrated single line comments
#53
Show file tree
Hide file tree
 Changes from all commits
Commits

Show all changes


4 commits


Select commit
Hold shift + click to select a range
cd718d5

Adapted parser for message at the end of a line instead of before an …


Tantalum73 May 24, 2019
d0426a4

Introduced singleLineMessageOpen and singleLineMessageClose as a star…


Tantalum73 May 25, 2019
ddf95d5

Working integration of single line comments.


Tantalum73 May 25, 2019
9db3dcc

Some cleanup.


Tantalum73 May 25, 2019

File filter

Filter by extension
Filter by extension
Conversations

Failed to load comments. 
Loading
Jump to

Jump to file


Failed to load files. 
Loading
 Diff view

 Diff view


2 changes: 1 addition & 1 deletion
sources/LocalizationEditor/Info.plist

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -19,7 +19,7 @@
	<key>CFBundleShortVersionString</key>
	<string>2.1</string>
	<key>CFBundleVersion</key>
	<string>191</string>
	<string>198</string>
	<key>LSMinimumSystemVersion</key>
	<string>$(MACOSX_DEPLOYMENT_TARGET)</string>
	<key>NSHumanReadableCopyright</key>
Expand Down

147 changes: 85 additions & 62 deletions
sources/LocalizationEditor/Providers/Parser.swift

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -26,7 +26,7 @@ class Parser {
 fileprivate enum ParserState {
 case readingKey
 case readingValue
 case readingMessage
 case readingMessage(isSingleLine: Bool)
 case other
 }
 /// The current state of the parser.
Expand All@@ -53,23 +53,6 @@ class Parser {
 return results
 }

 /**
 Special handling for single line comments to turn them into message tokens. Should only be called when state is other so // in a middle of value does not get caught
 */
 private func skipAndProcessSingleLineComments() {
 while !input.isEmpty, let character = String(input[input.startIndex]).unicodeScalars.first, CharacterSet.whitespacesAndNewlines.contains(character) {
 input.remove(at: input.index(input.startIndex, offsetBy: 0))
 }

 if input.hasPrefix("//"), let endIndex = input.index(of: "\n") {
 let messageRange = input.index(input.startIndex, offsetBy: 2) ..< endIndex
 tokens.append(.message(String(input[messageRange])))

 let rangeForRemoving = input.startIndex ..< endIndex
 input.removeSubrange(rangeForRemoving)
 }
 }

 /**
 This function reads through the input and populates an array of tokens.

Expand All@@ -82,49 +65,56 @@ class Parser {
 // Actions depend on the current state.
 switch state {
 case .other:
 skipAndProcessSingleLineComments()

 // Extract the upcoming control character, also switch the current state and append the extracted token, if any.
 if let extractedToken = try prepareNextState() {
 tokens.append(extractedToken)
 }
 case .readingKey:
 // Until the key-end marker is reached, the text should be interpreted as key.
 let currentKeyText = extractText(until: .quote)
 let potentialNewToken: Token = .key(currentKeyText)
 // If the prior token was also a key, append it.
 let newToken = tokenByConcatinatingwithPriorToken(potentialNewToken, seperatingString: EnclosingControlCharacters.quote.rawValue)
 tokens.append(newToken)
 // If the upcoming control character is also a key, do not stop reading a key. Otherwise a unescaped quote may exclude text from the key. Otherwise the state may be anything else.
 if let nextControlCharacter = findNextControlCharacter(andExtractFromSource: false), case EnclosingControlCharacters.quote = nextControlCharacter {
 state = .readingKey
 } else {
 state = .other
 }
 extractAndAppendIfPossible(for: .key(""), until: .quote)
 case .readingValue:
 // Text until value-end marker is a value.
 // If the prior token as also a value, append it.
 let currentValueText = extractText(until: .quote)
 let potentialNewToken: Token = .value(currentValueText)
 // If the prior token was also a key, append it.
 let newToken = tokenByConcatinatingwithPriorToken(potentialNewToken, seperatingString: EnclosingControlCharacters.quote.rawValue)
 tokens.append(newToken)
 // If the upcoming control character is also a key, do not stop reading a value. Otherwise a unescaped quote may exclude text from the value. Otherwise the state may be anything else.
 if let nextControlCharacter = findNextControlCharacter(andExtractFromSource: false), case EnclosingControlCharacters.quote = nextControlCharacter {
 state = .readingValue
 } else {
 state = .other
 }
 case .readingMessage:
 // Text until value-end marker is a message.
 extractAndAppendIfPossible(for: .value(""), until: .quote)
 case .readingMessage(let isReadingSingleLine):
 // If the prior token as also a message, DO NOT append it since the prior message could be a license header.
 let currentMessageText = extractText(until: .messageBoundaryClose)
 let endMarker: EnclosingControlCharacters = isReadingSingleLine ? .singleLineMessageClose : .messageBoundaryClose
 let currentMessageText = extractText(until: endMarker)
 let newToken: Token = .message(currentMessageText)
 tokens.append(newToken)
 state = .other
 }
 }
 }
 /// Extracts text from the input until the end marker is reached. Uses that text to create a new token and appends it to a prior extracted token if possible. In any case it updates the current list of extracted tokens.
 ///
 /// - Parameters:
 /// - token: The type of token that should be created from the text before the end marker. The associated value of the input is ignored.
 /// - endMarker: Marks the end of the tokens content.
 private func extractAndAppendIfPossible(for token: Token, until endMarker: EnclosingControlCharacters) {
 let currentText = extractText(until: endMarker)
 let potentialNewToken: Token
 switch token {
 case .key:
 potentialNewToken = .key(currentText)
 case .value:
 potentialNewToken = .value(currentText)
 default:
 assertionFailure("Currently, only the .key and .value support joining.")
 return
 }
 // Append to the prior token if possible.
 let newToken = tokenByConcatinatingwithPriorToken(potentialNewToken, seperatingString: endMarker.rawValue)
 tokens.append(newToken)
 // Do not stop reading when a newline or a quote is the next control character. Otherwise an unescaped quote may exclude text from the value. Keep the state unchanged if any other control character follows.
 if let nextControlCharacter = findNextControlCharacter(andExtractFromSource: false) {
 switch nextControlCharacter {
 case SeperatingControlCharacters.newline, EnclosingControlCharacters.singleLineMessageClose, EnclosingControlCharacters.quote:
 // Do not change the state and just continue.
 return
 default:
 break
 }
 }
 state = .other
 }
 /// Call this method when the list of tokens is ready and model object can be created. It will iterate through the tokens and try to map their values into model objects. Whe the mapping failed, an error is thrown.
 ///
 /// - Returns: The extracted model values.
Expand All@@ -134,6 +124,25 @@ class Parser {
 var currentKey: String?
 var currentValue: String?
 var results = [LocalizationString]()
 // The token that delimits an entry.
 guard let endToken = entriesEndToken(for: tokens) else {
 throw ParserError.malformattedInput
 }
 // Generates a result and appends it to the list of results if possible.
 func generateResultIfPossible(from processedToken: Token) {
 guard processedToken.isCaseEqual(to: endToken) else { return }
 // Done with that line. Check if values are populated and append them to the results.
 guard let key = currentKey, let value = currentValue else {
 return
 }
 let correctedMessage = removeLeadingTrailingSpaces(from: currentMessage)
 let entry = LocalizationString(key: key, value: value.replacingOccurrences(of: "\\\"", with: "\""), message: correctedMessage)
 results.append(entry)
 // Reset the properties to be ready for the next line.
 currentValue = nil
 currentKey = nil
 currentMessage = nil
 }
 // Iterate through the tokens and transform them into model objects.
 for token in tokens {
 switch token {
Expand All@@ -143,28 +152,36 @@ class Parser {
 currentKey = containedText
 case .value(let containedText):
 currentValue = containedText
 case .semicolon:
 // Done with that line. Check if values are populated and append them to the results.
 guard let key = currentKey, let value = currentValue else {
 throw ParserError.malformattedInput
 }
 let correctedMessage = removeLeadingTrailingSpaces(from: currentMessage)
 let entry = LocalizationString(key: key, value: value.replacingOccurrences(of: "\\\"", with: "\""), message: correctedMessage)
 results.append(entry)
 // Reset the properties to be ready for the next line.
 currentValue = nil
 currentKey = nil
 currentMessage = nil
 default:
 ()
 }
 generateResultIfPossible(from: token)
 }
 // Throw an execption to indicate that something went wront when tokens are extracted but they could not be transferred into model objects:
 if !tokens.isEmpty && results.isEmpty {
 throw ParserError.malformattedInput
 }
 return results
 }
 /// Determines the token that ends an entry. An entry can either be ended by a semicolon (if no comment was provided or the comment is above the entry) or a comment located at the end of a line. In the second case the `.message` token marks the end of the entry.
 ///
 /// - Parameter tokens: The tokens that were extracted during tokenization.
 /// - Returns: The token that ends an entry.
 private func entriesEndToken(for tokens: [Token]) -> Token? {
 // Assumption: after the first semicolon comes a new line -> semicolon delimits entry
 // After first semicolon comes a message, followed by a new line -> message delimits entry
 guard let semicolonIndex = tokens.firstIndex(where: { $0.isCaseEqual(to: .semicolon) }) else {
 return nil
 }
 guard let indexAfterSemicolon = tokens.index(semicolonIndex, offsetBy: 1, limitedBy: tokens.endIndex - 1) else { return nil }
 let elementAfterSemicolon = tokens[indexAfterSemicolon]
 switch elementAfterSemicolon {
 case .newline:
 return .semicolon
 default:
 return elementAfterSemicolon
 }
 }
 /// This function removes leading and trailing spaces from the input.
 ///
 /// - Parameter input: The string whose leading and trailing spaces should be removed.
Expand DownExpand Up@@ -265,7 +282,7 @@ class Parser {
 // Extract the given range and remove it from the input string.

 let lengthOfControlCharacter: Int = includingControlCharacter.skippingLength
 let endIndexOfExtraction = input.index(endindex, offsetBy: lengthOfControlCharacter)
 let endIndexOfExtraction = input.index(endindex, offsetBy: lengthOfControlCharacter, limitedBy: input.endIndex) ?? input.endIndex
 // Remove the range that includes the control character. The input range is used for extracting the text before it.
 let rangeForRemoving = input.startIndex ..< endIndexOfExtraction
 let rangeForExtraction = input.startIndex ..< endindex
Expand DownExpand Up@@ -354,7 +371,7 @@ extension Parser {
 case EnclosingControlCharacters.messageBoundaryOpen:
 // A new message begins.
 // Set the state to expect a message.
 state = .readingMessage
 state = .readingMessage(isSingleLine: false)
 case EnclosingControlCharacters.messageBoundaryClose:
 // Message-end markers should only be detected when the lexer is reading a message. If they occure 'in the wild' the input must be ill formatted.
 break
Expand All@@ -366,6 +383,12 @@ extension Parser {
 // Extract semicolon as token. A quote or message-start mark will follow as next control character but for now the state remains .other in order to detect that quote.
 returnToken = .semicolon
 state = .other
 case SeperatingControlCharacters.newline:
 returnToken = .newline
 case EnclosingControlCharacters.singleLineMessageOpen:
 state = .readingMessage(isSingleLine: true)
 case EnclosingControlCharacters.singleLineMessageClose:
 returnToken = .newline
 default:
 // New types need to be registered.
 throw ParserError.notParsable
Expand Down

46 changes: 41 additions & 5 deletions
sources/LocalizationEditor/Providers/ParserTypes.swift

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -15,12 +15,36 @@ import Foundation
/// - key: The key and its text.
/// - equal: The equal sign that maps a key to a value: ".
/// - semicolon: The semicolon that ends a line: ;
/// - newline: A new line \n.
enum Token {
 case message(String)
 case value(String)
 case key(String)
 case equal
 case semicolon
 case newline
 /// Checks if `self` is of the same type as `other` without taking the associated values into account.
 ///
 /// - Parameter other: The token to which self should be compared to.
 /// - Returns: `true` when the type of `other` matches the type of `self` without taking associated values into account.
 func isCaseEqual(to other: Token) -> Bool {
 switch (self, other) {
 case (.message, .message):
 return true
 case (.value, .value):
 return true
 case (.key, .key):
 return true
 case (.equal, .equal):
 return true
 case (.semicolon, .semicolon):
 return true
 case (.newline, .newline):
 return true
 default:
 return false
 }
 }
}

/// Control characters define starting and end points of tokens. They can be for example ", /* or ;
Expand All@@ -35,13 +59,17 @@ protocol EnclosingType: ControlCharacterType {}
protocol SeperatingType: ControlCharacterType {}

/// Enclosing control characters that wrapp text. They may start or end a message or contain a value/key.
/// - messageBoundaryOpen Opens a message.
/// - messageBoundaryClose Ends a message.
/// - quote Wraps a key or a value.
/// - messageBoundaryOpen: Opens a message.
/// - messageBoundaryClose: Ends a message.
/// - quote: Wraps a key or a value.
/// - singleLineMessageOpen: Opens a single line message.
/// - singleLineMessageClose: Closes the single line message.
enum EnclosingControlCharacters: String, EnclosingType, CaseIterable {
 case messageBoundaryOpen = "/*"
 case messageBoundaryClose = "*/"
 case quote = "\""
 case singleLineMessageOpen = "//"
 case singleLineMessageClose = "\n"

 var skippingLength: Int {
 switch self {
Expand All@@ -51,25 +79,33 @@ enum EnclosingControlCharacters: String, EnclosingType, CaseIterable {
 return EnclosingControlCharacters.messageBoundaryClose.rawValue.count
 case .quote:
 return EnclosingControlCharacters.quote.rawValue.count
 case .singleLineMessageOpen:
 return EnclosingControlCharacters.singleLineMessageOpen.rawValue.count
 case .singleLineMessageClose:
 return EnclosingControlCharacters.singleLineMessageClose.rawValue.count
 }
 }
}

/// Seperating control characters do not wrap text. They function as position markers. For example they seperate a key from its value or end the line.
/// - equal The equal sign that seperates a key from its value.
/// - semicolon The semicolon that end a line.
/// - equal: The equal sign that seperates a key from its value.
/// - semicolon: The semicolon that end a line.
/// - newline: A new line.
enum SeperatingControlCharacters: String, SeperatingType, CaseIterable {
 var skippingLength: Int {
 switch self {
 case .equal:
 return SeperatingControlCharacters.equal.rawValue.count
 case .semicolon:
 return SeperatingControlCharacters.semicolon.rawValue.count
 case .newline:
 return SeperatingControlCharacters.newline.rawValue.count
 }
 }

 case equal = "="
 case semicolon = ";"
 case newline = "\n"
}

/// Errors that may occure during parsing.
Expand Down
Loading

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Auto-enable theater mode on YouTube\n(function() {\n function tryTheater() {\n var btn = document.querySelector('button[aria-label=\"Theater mode\"], ytd-player #player button[title=\"Theater mode\"]');\n if (btn && !btn.classList.contains('activated')) {\n btn.click();\n }\n }\n \n // Try immediately\n tryTheater();\n \n // Try after navigation (SPA)\n var lastUrl = location.href;\n setInterval(function() {\n if (location.href !== lastUrl) {\n lastUrl = location.href;\n setTimeout(tryTheater, 500);\n }\n }, 1000);\n \n // Also try on player load\n var observer = new MutationObserver(tryTheater);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "YouTube Theater Mode Default");
}
} catch(__e) { console.warn('[Userscript:YouTube Theater Mode Default]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
Comments at the end of line are treated as messages and integrated single line comments#53

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

 Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and
privacy statement. We’ll occasionally send you account related emails.

Already on GitHub?
Sign in
to your account

 Merged

igorkulman
merged 4 commits into

from

May 25, 2019
 Merged


Comments at the end of line are treated as messages and integrated single line comments
#53
Show file tree
Hide file tree
 Changes from all commits
Commits

Show all changes


4 commits


Select commit
Hold shift + click to select a range
cd718d5

Adapted parser for message at the end of a line instead of before an …


Tantalum73 May 24, 2019
d0426a4

Introduced singleLineMessageOpen and singleLineMessageClose as a star…


Tantalum73 May 25, 2019
ddf95d5

Working integration of single line comments.


Tantalum73 May 25, 2019
9db3dcc

Some cleanup.


Tantalum73 May 25, 2019

File filter

Filter by extension
Filter by extension
Conversations

Failed to load comments. 
Loading
Jump to

Jump to file


Failed to load files. 
Loading
 Diff view

 Diff view


2 changes: 1 addition & 1 deletion
sources/LocalizationEditor/Info.plist

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -19,7 +19,7 @@
	<key>CFBundleShortVersionString</key>
	<string>2.1</string>
	<key>CFBundleVersion</key>
	<string>191</string>
	<string>198</string>
	<key>LSMinimumSystemVersion</key>
	<string>$(MACOSX_DEPLOYMENT_TARGET)</string>
	<key>NSHumanReadableCopyright</key>
Expand Down

147 changes: 85 additions & 62 deletions
sources/LocalizationEditor/Providers/Parser.swift

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -26,7 +26,7 @@ class Parser {
 fileprivate enum ParserState {
 case readingKey
 case readingValue
 case readingMessage
 case readingMessage(isSingleLine: Bool)
 case other
 }
 /// The current state of the parser.
Expand All@@ -53,23 +53,6 @@ class Parser {
 return results
 }

 /**
 Special handling for single line comments to turn them into message tokens. Should only be called when state is other so // in a middle of value does not get caught
 */
 private func skipAndProcessSingleLineComments() {
 while !input.isEmpty, let character = String(input[input.startIndex]).unicodeScalars.first, CharacterSet.whitespacesAndNewlines.contains(character) {
 input.remove(at: input.index(input.startIndex, offsetBy: 0))
 }

 if input.hasPrefix("//"), let endIndex = input.index(of: "\n") {
 let messageRange = input.index(input.startIndex, offsetBy: 2) ..< endIndex
 tokens.append(.message(String(input[messageRange])))

 let rangeForRemoving = input.startIndex ..< endIndex
 input.removeSubrange(rangeForRemoving)
 }
 }

 /**
 This function reads through the input and populates an array of tokens.

Expand All@@ -82,49 +65,56 @@ class Parser {
 // Actions depend on the current state.
 switch state {
 case .other:
 skipAndProcessSingleLineComments()

 // Extract the upcoming control character, also switch the current state and append the extracted token, if any.
 if let extractedToken = try prepareNextState() {
 tokens.append(extractedToken)
 }
 case .readingKey:
 // Until the key-end marker is reached, the text should be interpreted as key.
 let currentKeyText = extractText(until: .quote)
 let potentialNewToken: Token = .key(currentKeyText)
 // If the prior token was also a key, append it.
 let newToken = tokenByConcatinatingwithPriorToken(potentialNewToken, seperatingString: EnclosingControlCharacters.quote.rawValue)
 tokens.append(newToken)
 // If the upcoming control character is also a key, do not stop reading a key. Otherwise a unescaped quote may exclude text from the key. Otherwise the state may be anything else.
 if let nextControlCharacter = findNextControlCharacter(andExtractFromSource: false), case EnclosingControlCharacters.quote = nextControlCharacter {
 state = .readingKey
 } else {
 state = .other
 }
 extractAndAppendIfPossible(for: .key(""), until: .quote)
 case .readingValue:
 // Text until value-end marker is a value.
 // If the prior token as also a value, append it.
 let currentValueText = extractText(until: .quote)
 let potentialNewToken: Token = .value(currentValueText)
 // If the prior token was also a key, append it.
 let newToken = tokenByConcatinatingwithPriorToken(potentialNewToken, seperatingString: EnclosingControlCharacters.quote.rawValue)
 tokens.append(newToken)
 // If the upcoming control character is also a key, do not stop reading a value. Otherwise a unescaped quote may exclude text from the value. Otherwise the state may be anything else.
 if let nextControlCharacter = findNextControlCharacter(andExtractFromSource: false), case EnclosingControlCharacters.quote = nextControlCharacter {
 state = .readingValue
 } else {
 state = .other
 }
 case .readingMessage:
 // Text until value-end marker is a message.
 extractAndAppendIfPossible(for: .value(""), until: .quote)
 case .readingMessage(let isReadingSingleLine):
 // If the prior token as also a message, DO NOT append it since the prior message could be a license header.
 let currentMessageText = extractText(until: .messageBoundaryClose)
 let endMarker: EnclosingControlCharacters = isReadingSingleLine ? .singleLineMessageClose : .messageBoundaryClose
 let currentMessageText = extractText(until: endMarker)
 let newToken: Token = .message(currentMessageText)
 tokens.append(newToken)
 state = .other
 }
 }
 }
 /// Extracts text from the input until the end marker is reached. Uses that text to create a new token and appends it to a prior extracted token if possible. In any case it updates the current list of extracted tokens.
 ///
 /// - Parameters:
 /// - token: The type of token that should be created from the text before the end marker. The associated value of the input is ignored.
 /// - endMarker: Marks the end of the tokens content.
 private func extractAndAppendIfPossible(for token: Token, until endMarker: EnclosingControlCharacters) {
 let currentText = extractText(until: endMarker)
 let potentialNewToken: Token
 switch token {
 case .key:
 potentialNewToken = .key(currentText)
 case .value:
 potentialNewToken = .value(currentText)
 default:
 assertionFailure("Currently, only the .key and .value support joining.")
 return
 }
 // Append to the prior token if possible.
 let newToken = tokenByConcatinatingwithPriorToken(potentialNewToken, seperatingString: endMarker.rawValue)
 tokens.append(newToken)
 // Do not stop reading when a newline or a quote is the next control character. Otherwise an unescaped quote may exclude text from the value. Keep the state unchanged if any other control character follows.
 if let nextControlCharacter = findNextControlCharacter(andExtractFromSource: false) {
 switch nextControlCharacter {
 case SeperatingControlCharacters.newline, EnclosingControlCharacters.singleLineMessageClose, EnclosingControlCharacters.quote:
 // Do not change the state and just continue.
 return
 default:
 break
 }
 }
 state = .other
 }
 /// Call this method when the list of tokens is ready and model object can be created. It will iterate through the tokens and try to map their values into model objects. Whe the mapping failed, an error is thrown.
 ///
 /// - Returns: The extracted model values.
Expand All@@ -134,6 +124,25 @@ class Parser {
 var currentKey: String?
 var currentValue: String?
 var results = [LocalizationString]()
 // The token that delimits an entry.
 guard let endToken = entriesEndToken(for: tokens) else {
 throw ParserError.malformattedInput
 }
 // Generates a result and appends it to the list of results if possible.
 func generateResultIfPossible(from processedToken: Token) {
 guard processedToken.isCaseEqual(to: endToken) else { return }
 // Done with that line. Check if values are populated and append them to the results.
 guard let key = currentKey, let value = currentValue else {
 return
 }
 let correctedMessage = removeLeadingTrailingSpaces(from: currentMessage)
 let entry = LocalizationString(key: key, value: value.replacingOccurrences(of: "\\\"", with: "\""), message: correctedMessage)
 results.append(entry)
 // Reset the properties to be ready for the next line.
 currentValue = nil
 currentKey = nil
 currentMessage = nil
 }
 // Iterate through the tokens and transform them into model objects.
 for token in tokens {
 switch token {
Expand All@@ -143,28 +152,36 @@ class Parser {
 currentKey = containedText
 case .value(let containedText):
 currentValue = containedText
 case .semicolon:
 // Done with that line. Check if values are populated and append them to the results.
 guard let key = currentKey, let value = currentValue else {
 throw ParserError.malformattedInput
 }
 let correctedMessage = removeLeadingTrailingSpaces(from: currentMessage)
 let entry = LocalizationString(key: key, value: value.replacingOccurrences(of: "\\\"", with: "\""), message: correctedMessage)
 results.append(entry)
 // Reset the properties to be ready for the next line.
 currentValue = nil
 currentKey = nil
 currentMessage = nil
 default:
 ()
 }
 generateResultIfPossible(from: token)
 }
 // Throw an execption to indicate that something went wront when tokens are extracted but they could not be transferred into model objects:
 if !tokens.isEmpty && results.isEmpty {
 throw ParserError.malformattedInput
 }
 return results
 }
 /// Determines the token that ends an entry. An entry can either be ended by a semicolon (if no comment was provided or the comment is above the entry) or a comment located at the end of a line. In the second case the `.message` token marks the end of the entry.
 ///
 /// - Parameter tokens: The tokens that were extracted during tokenization.
 /// - Returns: The token that ends an entry.
 private func entriesEndToken(for tokens: [Token]) -> Token? {
 // Assumption: after the first semicolon comes a new line -> semicolon delimits entry
 // After first semicolon comes a message, followed by a new line -> message delimits entry
 guard let semicolonIndex = tokens.firstIndex(where: { $0.isCaseEqual(to: .semicolon) }) else {
 return nil
 }
 guard let indexAfterSemicolon = tokens.index(semicolonIndex, offsetBy: 1, limitedBy: tokens.endIndex - 1) else { return nil }
 let elementAfterSemicolon = tokens[indexAfterSemicolon]
 switch elementAfterSemicolon {
 case .newline:
 return .semicolon
 default:
 return elementAfterSemicolon
 }
 }
 /// This function removes leading and trailing spaces from the input.
 ///
 /// - Parameter input: The string whose leading and trailing spaces should be removed.
Expand DownExpand Up@@ -265,7 +282,7 @@ class Parser {
 // Extract the given range and remove it from the input string.

 let lengthOfControlCharacter: Int = includingControlCharacter.skippingLength
 let endIndexOfExtraction = input.index(endindex, offsetBy: lengthOfControlCharacter)
 let endIndexOfExtraction = input.index(endindex, offsetBy: lengthOfControlCharacter, limitedBy: input.endIndex) ?? input.endIndex
 // Remove the range that includes the control character. The input range is used for extracting the text before it.
 let rangeForRemoving = input.startIndex ..< endIndexOfExtraction
 let rangeForExtraction = input.startIndex ..< endindex
Expand DownExpand Up@@ -354,7 +371,7 @@ extension Parser {
 case EnclosingControlCharacters.messageBoundaryOpen:
 // A new message begins.
 // Set the state to expect a message.
 state = .readingMessage
 state = .readingMessage(isSingleLine: false)
 case EnclosingControlCharacters.messageBoundaryClose:
 // Message-end markers should only be detected when the lexer is reading a message. If they occure 'in the wild' the input must be ill formatted.
 break
Expand All@@ -366,6 +383,12 @@ extension Parser {
 // Extract semicolon as token. A quote or message-start mark will follow as next control character but for now the state remains .other in order to detect that quote.
 returnToken = .semicolon
 state = .other
 case SeperatingControlCharacters.newline:
 returnToken = .newline
 case EnclosingControlCharacters.singleLineMessageOpen:
 state = .readingMessage(isSingleLine: true)
 case EnclosingControlCharacters.singleLineMessageClose:
 returnToken = .newline
 default:
 // New types need to be registered.
 throw ParserError.notParsable
Expand Down

46 changes: 41 additions & 5 deletions
sources/LocalizationEditor/Providers/ParserTypes.swift

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -15,12 +15,36 @@ import Foundation
/// - key: The key and its text.
/// - equal: The equal sign that maps a key to a value: ".
/// - semicolon: The semicolon that ends a line: ;
/// - newline: A new line \n.
enum Token {
 case message(String)
 case value(String)
 case key(String)
 case equal
 case semicolon
 case newline
 /// Checks if `self` is of the same type as `other` without taking the associated values into account.
 ///
 /// - Parameter other: The token to which self should be compared to.
 /// - Returns: `true` when the type of `other` matches the type of `self` without taking associated values into account.
 func isCaseEqual(to other: Token) -> Bool {
 switch (self, other) {
 case (.message, .message):
 return true
 case (.value, .value):
 return true
 case (.key, .key):
 return true
 case (.equal, .equal):
 return true
 case (.semicolon, .semicolon):
 return true
 case (.newline, .newline):
 return true
 default:
 return false
 }
 }
}

/// Control characters define starting and end points of tokens. They can be for example ", /* or ;
Expand All@@ -35,13 +59,17 @@ protocol EnclosingType: ControlCharacterType {}
protocol SeperatingType: ControlCharacterType {}

/// Enclosing control characters that wrapp text. They may start or end a message or contain a value/key.
/// - messageBoundaryOpen Opens a message.
/// - messageBoundaryClose Ends a message.
/// - quote Wraps a key or a value.
/// - messageBoundaryOpen: Opens a message.
/// - messageBoundaryClose: Ends a message.
/// - quote: Wraps a key or a value.
/// - singleLineMessageOpen: Opens a single line message.
/// - singleLineMessageClose: Closes the single line message.
enum EnclosingControlCharacters: String, EnclosingType, CaseIterable {
 case messageBoundaryOpen = "/*"
 case messageBoundaryClose = "*/"
 case quote = "\""
 case singleLineMessageOpen = "//"
 case singleLineMessageClose = "\n"

 var skippingLength: Int {
 switch self {
Expand All@@ -51,25 +79,33 @@ enum EnclosingControlCharacters: String, EnclosingType, CaseIterable {
 return EnclosingControlCharacters.messageBoundaryClose.rawValue.count
 case .quote:
 return EnclosingControlCharacters.quote.rawValue.count
 case .singleLineMessageOpen:
 return EnclosingControlCharacters.singleLineMessageOpen.rawValue.count
 case .singleLineMessageClose:
 return EnclosingControlCharacters.singleLineMessageClose.rawValue.count
 }
 }
}

/// Seperating control characters do not wrap text. They function as position markers. For example they seperate a key from its value or end the line.
/// - equal The equal sign that seperates a key from its value.
/// - semicolon The semicolon that end a line.
/// - equal: The equal sign that seperates a key from its value.
/// - semicolon: The semicolon that end a line.
/// - newline: A new line.
enum SeperatingControlCharacters: String, SeperatingType, CaseIterable {
 var skippingLength: Int {
 switch self {
 case .equal:
 return SeperatingControlCharacters.equal.rawValue.count
 case .semicolon:
 return SeperatingControlCharacters.semicolon.rawValue.count
 case .newline:
 return SeperatingControlCharacters.newline.rawValue.count
 }
 }

 case equal = "="
 case semicolon = ";"
 case newline = "\n"
}

/// Errors that may occure during parsing.
Expand Down
Loading

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Remove or un-stick sticky/fixed headers that block content\n(function() {\n function unstick() {\n document.querySelectorAll('header, nav, [role=\"banner\"], .header, .navbar, .sticky, .fixed-top, [style*=\"position: fixed\"], [style*=\"position:sticky\"]').forEach(function(el) {\n if (el.style.position === 'fixed' || el.style.position === 'sticky' || \n getComputedStyle(el).position === 'fixed' || getComputedStyle(el).position === 'sticky') {\n el.style.position = 'static';\n el.style.top = 'auto';\n el.style.zIndex = 'auto';\n }\n });\n }\n \n unstick();\n \n var observer = new MutationObserver(unstick);\n observer.observe(document.body, { childList: true, subtree: true, attributes: true, attributeFilter: ['style', 'class'] });\n})();", "Kill Sticky Headers");
}
} catch(__e) { console.warn('[Userscript:Kill Sticky Headers]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
Comments at the end of line are treated as messages and integrated single line comments#53

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

 Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and
privacy statement. We’ll occasionally send you account related emails.

Already on GitHub?
Sign in
to your account

 Merged

igorkulman
merged 4 commits into

from

May 25, 2019
 Merged


Comments at the end of line are treated as messages and integrated single line comments
#53
Show file tree
Hide file tree
 Changes from all commits
Commits

Show all changes


4 commits


Select commit
Hold shift + click to select a range
cd718d5

Adapted parser for message at the end of a line instead of before an …


Tantalum73 May 24, 2019
d0426a4

Introduced singleLineMessageOpen and singleLineMessageClose as a star…


Tantalum73 May 25, 2019
ddf95d5

Working integration of single line comments.


Tantalum73 May 25, 2019
9db3dcc

Some cleanup.


Tantalum73 May 25, 2019

File filter

Filter by extension
Filter by extension
Conversations

Failed to load comments. 
Loading
Jump to

Jump to file


Failed to load files. 
Loading
 Diff view

 Diff view


2 changes: 1 addition & 1 deletion
sources/LocalizationEditor/Info.plist

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -19,7 +19,7 @@
	<key>CFBundleShortVersionString</key>
	<string>2.1</string>
	<key>CFBundleVersion</key>
	<string>191</string>
	<string>198</string>
	<key>LSMinimumSystemVersion</key>
	<string>$(MACOSX_DEPLOYMENT_TARGET)</string>
	<key>NSHumanReadableCopyright</key>
Expand Down

147 changes: 85 additions & 62 deletions
sources/LocalizationEditor/Providers/Parser.swift

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -26,7 +26,7 @@ class Parser {
 fileprivate enum ParserState {
 case readingKey
 case readingValue
 case readingMessage
 case readingMessage(isSingleLine: Bool)
 case other
 }
 /// The current state of the parser.
Expand All@@ -53,23 +53,6 @@ class Parser {
 return results
 }

 /**
 Special handling for single line comments to turn them into message tokens. Should only be called when state is other so // in a middle of value does not get caught
 */
 private func skipAndProcessSingleLineComments() {
 while !input.isEmpty, let character = String(input[input.startIndex]).unicodeScalars.first, CharacterSet.whitespacesAndNewlines.contains(character) {
 input.remove(at: input.index(input.startIndex, offsetBy: 0))
 }

 if input.hasPrefix("//"), let endIndex = input.index(of: "\n") {
 let messageRange = input.index(input.startIndex, offsetBy: 2) ..< endIndex
 tokens.append(.message(String(input[messageRange])))

 let rangeForRemoving = input.startIndex ..< endIndex
 input.removeSubrange(rangeForRemoving)
 }
 }

 /**
 This function reads through the input and populates an array of tokens.

Expand All@@ -82,49 +65,56 @@ class Parser {
 // Actions depend on the current state.
 switch state {
 case .other:
 skipAndProcessSingleLineComments()

 // Extract the upcoming control character, also switch the current state and append the extracted token, if any.
 if let extractedToken = try prepareNextState() {
 tokens.append(extractedToken)
 }
 case .readingKey:
 // Until the key-end marker is reached, the text should be interpreted as key.
 let currentKeyText = extractText(until: .quote)
 let potentialNewToken: Token = .key(currentKeyText)
 // If the prior token was also a key, append it.
 let newToken = tokenByConcatinatingwithPriorToken(potentialNewToken, seperatingString: EnclosingControlCharacters.quote.rawValue)
 tokens.append(newToken)
 // If the upcoming control character is also a key, do not stop reading a key. Otherwise a unescaped quote may exclude text from the key. Otherwise the state may be anything else.
 if let nextControlCharacter = findNextControlCharacter(andExtractFromSource: false), case EnclosingControlCharacters.quote = nextControlCharacter {
 state = .readingKey
 } else {
 state = .other
 }
 extractAndAppendIfPossible(for: .key(""), until: .quote)
 case .readingValue:
 // Text until value-end marker is a value.
 // If the prior token as also a value, append it.
 let currentValueText = extractText(until: .quote)
 let potentialNewToken: Token = .value(currentValueText)
 // If the prior token was also a key, append it.
 let newToken = tokenByConcatinatingwithPriorToken(potentialNewToken, seperatingString: EnclosingControlCharacters.quote.rawValue)
 tokens.append(newToken)
 // If the upcoming control character is also a key, do not stop reading a value. Otherwise a unescaped quote may exclude text from the value. Otherwise the state may be anything else.
 if let nextControlCharacter = findNextControlCharacter(andExtractFromSource: false), case EnclosingControlCharacters.quote = nextControlCharacter {
 state = .readingValue
 } else {
 state = .other
 }
 case .readingMessage:
 // Text until value-end marker is a message.
 extractAndAppendIfPossible(for: .value(""), until: .quote)
 case .readingMessage(let isReadingSingleLine):
 // If the prior token as also a message, DO NOT append it since the prior message could be a license header.
 let currentMessageText = extractText(until: .messageBoundaryClose)
 let endMarker: EnclosingControlCharacters = isReadingSingleLine ? .singleLineMessageClose : .messageBoundaryClose
 let currentMessageText = extractText(until: endMarker)
 let newToken: Token = .message(currentMessageText)
 tokens.append(newToken)
 state = .other
 }
 }
 }
 /// Extracts text from the input until the end marker is reached. Uses that text to create a new token and appends it to a prior extracted token if possible. In any case it updates the current list of extracted tokens.
 ///
 /// - Parameters:
 /// - token: The type of token that should be created from the text before the end marker. The associated value of the input is ignored.
 /// - endMarker: Marks the end of the tokens content.
 private func extractAndAppendIfPossible(for token: Token, until endMarker: EnclosingControlCharacters) {
 let currentText = extractText(until: endMarker)
 let potentialNewToken: Token
 switch token {
 case .key:
 potentialNewToken = .key(currentText)
 case .value:
 potentialNewToken = .value(currentText)
 default:
 assertionFailure("Currently, only the .key and .value support joining.")
 return
 }
 // Append to the prior token if possible.
 let newToken = tokenByConcatinatingwithPriorToken(potentialNewToken, seperatingString: endMarker.rawValue)
 tokens.append(newToken)
 // Do not stop reading when a newline or a quote is the next control character. Otherwise an unescaped quote may exclude text from the value. Keep the state unchanged if any other control character follows.
 if let nextControlCharacter = findNextControlCharacter(andExtractFromSource: false) {
 switch nextControlCharacter {
 case SeperatingControlCharacters.newline, EnclosingControlCharacters.singleLineMessageClose, EnclosingControlCharacters.quote:
 // Do not change the state and just continue.
 return
 default:
 break
 }
 }
 state = .other
 }
 /// Call this method when the list of tokens is ready and model object can be created. It will iterate through the tokens and try to map their values into model objects. Whe the mapping failed, an error is thrown.
 ///
 /// - Returns: The extracted model values.
Expand All@@ -134,6 +124,25 @@ class Parser {
 var currentKey: String?
 var currentValue: String?
 var results = [LocalizationString]()
 // The token that delimits an entry.
 guard let endToken = entriesEndToken(for: tokens) else {
 throw ParserError.malformattedInput
 }
 // Generates a result and appends it to the list of results if possible.
 func generateResultIfPossible(from processedToken: Token) {
 guard processedToken.isCaseEqual(to: endToken) else { return }
 // Done with that line. Check if values are populated and append them to the results.
 guard let key = currentKey, let value = currentValue else {
 return
 }
 let correctedMessage = removeLeadingTrailingSpaces(from: currentMessage)
 let entry = LocalizationString(key: key, value: value.replacingOccurrences(of: "\\\"", with: "\""), message: correctedMessage)
 results.append(entry)
 // Reset the properties to be ready for the next line.
 currentValue = nil
 currentKey = nil
 currentMessage = nil
 }
 // Iterate through the tokens and transform them into model objects.
 for token in tokens {
 switch token {
Expand All@@ -143,28 +152,36 @@ class Parser {
 currentKey = containedText
 case .value(let containedText):
 currentValue = containedText
 case .semicolon:
 // Done with that line. Check if values are populated and append them to the results.
 guard let key = currentKey, let value = currentValue else {
 throw ParserError.malformattedInput
 }
 let correctedMessage = removeLeadingTrailingSpaces(from: currentMessage)
 let entry = LocalizationString(key: key, value: value.replacingOccurrences(of: "\\\"", with: "\""), message: correctedMessage)
 results.append(entry)
 // Reset the properties to be ready for the next line.
 currentValue = nil
 currentKey = nil
 currentMessage = nil
 default:
 ()
 }
 generateResultIfPossible(from: token)
 }
 // Throw an execption to indicate that something went wront when tokens are extracted but they could not be transferred into model objects:
 if !tokens.isEmpty && results.isEmpty {
 throw ParserError.malformattedInput
 }
 return results
 }
 /// Determines the token that ends an entry. An entry can either be ended by a semicolon (if no comment was provided or the comment is above the entry) or a comment located at the end of a line. In the second case the `.message` token marks the end of the entry.
 ///
 /// - Parameter tokens: The tokens that were extracted during tokenization.
 /// - Returns: The token that ends an entry.
 private func entriesEndToken(for tokens: [Token]) -> Token? {
 // Assumption: after the first semicolon comes a new line -> semicolon delimits entry
 // After first semicolon comes a message, followed by a new line -> message delimits entry
 guard let semicolonIndex = tokens.firstIndex(where: { $0.isCaseEqual(to: .semicolon) }) else {
 return nil
 }
 guard let indexAfterSemicolon = tokens.index(semicolonIndex, offsetBy: 1, limitedBy: tokens.endIndex - 1) else { return nil }
 let elementAfterSemicolon = tokens[indexAfterSemicolon]
 switch elementAfterSemicolon {
 case .newline:
 return .semicolon
 default:
 return elementAfterSemicolon
 }
 }
 /// This function removes leading and trailing spaces from the input.
 ///
 /// - Parameter input: The string whose leading and trailing spaces should be removed.
Expand DownExpand Up@@ -265,7 +282,7 @@ class Parser {
 // Extract the given range and remove it from the input string.

 let lengthOfControlCharacter: Int = includingControlCharacter.skippingLength
 let endIndexOfExtraction = input.index(endindex, offsetBy: lengthOfControlCharacter)
 let endIndexOfExtraction = input.index(endindex, offsetBy: lengthOfControlCharacter, limitedBy: input.endIndex) ?? input.endIndex
 // Remove the range that includes the control character. The input range is used for extracting the text before it.
 let rangeForRemoving = input.startIndex ..< endIndexOfExtraction
 let rangeForExtraction = input.startIndex ..< endindex
Expand DownExpand Up@@ -354,7 +371,7 @@ extension Parser {
 case EnclosingControlCharacters.messageBoundaryOpen:
 // A new message begins.
 // Set the state to expect a message.
 state = .readingMessage
 state = .readingMessage(isSingleLine: false)
 case EnclosingControlCharacters.messageBoundaryClose:
 // Message-end markers should only be detected when the lexer is reading a message. If they occure 'in the wild' the input must be ill formatted.
 break
Expand All@@ -366,6 +383,12 @@ extension Parser {
 // Extract semicolon as token. A quote or message-start mark will follow as next control character but for now the state remains .other in order to detect that quote.
 returnToken = .semicolon
 state = .other
 case SeperatingControlCharacters.newline:
 returnToken = .newline
 case EnclosingControlCharacters.singleLineMessageOpen:
 state = .readingMessage(isSingleLine: true)
 case EnclosingControlCharacters.singleLineMessageClose:
 returnToken = .newline
 default:
 // New types need to be registered.
 throw ParserError.notParsable
Expand Down

46 changes: 41 additions & 5 deletions
sources/LocalizationEditor/Providers/ParserTypes.swift

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -15,12 +15,36 @@ import Foundation
/// - key: The key and its text.
/// - equal: The equal sign that maps a key to a value: ".
/// - semicolon: The semicolon that ends a line: ;
/// - newline: A new line \n.
enum Token {
 case message(String)
 case value(String)
 case key(String)
 case equal
 case semicolon
 case newline
 /// Checks if `self` is of the same type as `other` without taking the associated values into account.
 ///
 /// - Parameter other: The token to which self should be compared to.
 /// - Returns: `true` when the type of `other` matches the type of `self` without taking associated values into account.
 func isCaseEqual(to other: Token) -> Bool {
 switch (self, other) {
 case (.message, .message):
 return true
 case (.value, .value):
 return true
 case (.key, .key):
 return true
 case (.equal, .equal):
 return true
 case (.semicolon, .semicolon):
 return true
 case (.newline, .newline):
 return true
 default:
 return false
 }
 }
}

/// Control characters define starting and end points of tokens. They can be for example ", /* or ;
Expand All@@ -35,13 +59,17 @@ protocol EnclosingType: ControlCharacterType {}
protocol SeperatingType: ControlCharacterType {}

/// Enclosing control characters that wrapp text. They may start or end a message or contain a value/key.
/// - messageBoundaryOpen Opens a message.
/// - messageBoundaryClose Ends a message.
/// - quote Wraps a key or a value.
/// - messageBoundaryOpen: Opens a message.
/// - messageBoundaryClose: Ends a message.
/// - quote: Wraps a key or a value.
/// - singleLineMessageOpen: Opens a single line message.
/// - singleLineMessageClose: Closes the single line message.
enum EnclosingControlCharacters: String, EnclosingType, CaseIterable {
 case messageBoundaryOpen = "/*"
 case messageBoundaryClose = "*/"
 case quote = "\""
 case singleLineMessageOpen = "//"
 case singleLineMessageClose = "\n"

 var skippingLength: Int {
 switch self {
Expand All@@ -51,25 +79,33 @@ enum EnclosingControlCharacters: String, EnclosingType, CaseIterable {
 return EnclosingControlCharacters.messageBoundaryClose.rawValue.count
 case .quote:
 return EnclosingControlCharacters.quote.rawValue.count
 case .singleLineMessageOpen:
 return EnclosingControlCharacters.singleLineMessageOpen.rawValue.count
 case .singleLineMessageClose:
 return EnclosingControlCharacters.singleLineMessageClose.rawValue.count
 }
 }
}

/// Seperating control characters do not wrap text. They function as position markers. For example they seperate a key from its value or end the line.
/// - equal The equal sign that seperates a key from its value.
/// - semicolon The semicolon that end a line.
/// - equal: The equal sign that seperates a key from its value.
/// - semicolon: The semicolon that end a line.
/// - newline: A new line.
enum SeperatingControlCharacters: String, SeperatingType, CaseIterable {
 var skippingLength: Int {
 switch self {
 case .equal:
 return SeperatingControlCharacters.equal.rawValue.count
 case .semicolon:
 return SeperatingControlCharacters.semicolon.rawValue.count
 case .newline:
 return SeperatingControlCharacters.newline.rawValue.count
 }
 }

 case equal = "="
 case semicolon = ";"
 case newline = "\n"
}

/// Errors that may occure during parsing.
Expand Down
Loading

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Universal Dark Mode - works on any site\n(function() {\n var enabled = true;\n \n function applyDarkMode() {\n if (!enabled) return;\n \n // Create style element if it doesn't exist\n var style = document.getElementById('universal-dark-mode-style');\n if (!style) {\n style = document.createElement('style');\n style.id = 'universal-dark-mode-style';\n document.head.appendChild(style);\n }\n \n // Dark mode CSS - inverts colors but preserves images/video\n style.textContent = '\n /* Invert everything except media */\n html {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #1a1a2e !important;\n }\n \n /* Restore images, videos, iframes, canvas */\n img, video, iframe, canvas, svg, picture, [style*=\"background-image\"] {\n filter: invert(1) hue-rotate(180deg) !important;\n }\n \n /* Preserve specific elements that should not be inverted */\n .no-dark-mode, .no-dark-mode *,\n [data-theme=\"light\"], [data-theme=\"light\"],\n .ace_editor, .ace_editor *,\n .CodeMirror, .CodeMirror *,\n .monaco-editor, .monaco-editor *,\n .markdown-body pre, .markdown-body pre *,\n .highlight, .highlight *,\n pre code, pre code * {\n filter: none !important;\n }\n \n /* Fix common UI elements */\n .modal, .popup, .dropdown-menu, .tooltip, .popover {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #2d2d44 !important;\n border-color: #444 !important;\n }\n \n /* Scrollbars */\n ::-webkit-scrollbar { background: #1a1a2e !important; }\n ::-webkit-scrollbar-thumb { background: #444 !important; }\n ::-webkit-scrollbar-thumb:hover { background: #555 !important; }\n \n /* Selection */\n ::selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ::-moz-selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ';\n }\n \n function removeDarkMode() {\n var style = document.getElementById('universal-dark-mode-style');\n if (style) style.remove();\n }\n \n // Toggle with Alt+Shift+D\n document.addEventListener('keydown', function(e) {\n if (e.altKey && e.shiftKey && e.key === 'D') {\n e.preventDefault();\n enabled = !enabled;\n if (enabled) {\n applyDarkMode();\n console.log('[Universal Dark Mode] Enabled');\n } else {\n removeDarkMode();\n console.log('[Universal Dark Mode] Disabled');\n }\n }\n });\n \n // Apply on load\n applyDarkMode();\n \n // Re-apply on dynamic content\n var observer = new MutationObserver(function(mutations) {\n if (enabled && !document.getElementById('universal-dark-mode-style')) {\n applyDarkMode();\n }\n });\n observer.observe(document.head, { childList: true });\n \n console.log('[Universal Dark Mode] Loaded - Press Alt+Shift+D to toggle');\n})();", "Universal Dark Mode");
}
} catch(__e) { console.warn('[Userscript:Universal Dark Mode]', __e); }
})();
})();

Skip to content
Comments at the end of line are treated as messages and integrated single line comments#53

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

 Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and
privacy statement. We’ll occasionally send you account related emails.

Already on GitHub?
Sign in
to your account

 Merged

igorkulman
merged 4 commits into

from

May 25, 2019
 Merged


Comments at the end of line are treated as messages and integrated single line comments
#53
Show file tree
Hide file tree
 Changes from all commits
Commits

Show all changes


4 commits


Select commit
Hold shift + click to select a range
cd718d5

Adapted parser for message at the end of a line instead of before an …


Tantalum73 May 24, 2019
d0426a4

Introduced singleLineMessageOpen and singleLineMessageClose as a star…


Tantalum73 May 25, 2019
ddf95d5

Working integration of single line comments.


Tantalum73 May 25, 2019
9db3dcc

Some cleanup.


Tantalum73 May 25, 2019

File filter

Filter by extension
Filter by extension
Conversations

Failed to load comments. 
Loading
Jump to

Jump to file


Failed to load files. 
Loading
 Diff view

 Diff view


2 changes: 1 addition & 1 deletion
sources/LocalizationEditor/Info.plist

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -19,7 +19,7 @@
	<key>CFBundleShortVersionString</key>
	<string>2.1</string>
	<key>CFBundleVersion</key>
	<string>191</string>
	<string>198</string>
	<key>LSMinimumSystemVersion</key>
	<string>$(MACOSX_DEPLOYMENT_TARGET)</string>
	<key>NSHumanReadableCopyright</key>
Expand Down

147 changes: 85 additions & 62 deletions
sources/LocalizationEditor/Providers/Parser.swift

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -26,7 +26,7 @@ class Parser {
 fileprivate enum ParserState {
 case readingKey
 case readingValue
 case readingMessage
 case readingMessage(isSingleLine: Bool)
 case other
 }
 /// The current state of the parser.
Expand All@@ -53,23 +53,6 @@ class Parser {
 return results
 }

 /**
 Special handling for single line comments to turn them into message tokens. Should only be called when state is other so // in a middle of value does not get caught
 */
 private func skipAndProcessSingleLineComments() {
 while !input.isEmpty, let character = String(input[input.startIndex]).unicodeScalars.first, CharacterSet.whitespacesAndNewlines.contains(character) {
 input.remove(at: input.index(input.startIndex, offsetBy: 0))
 }

 if input.hasPrefix("//"), let endIndex = input.index(of: "\n") {
 let messageRange = input.index(input.startIndex, offsetBy: 2) ..< endIndex
 tokens.append(.message(String(input[messageRange])))

 let rangeForRemoving = input.startIndex ..< endIndex
 input.removeSubrange(rangeForRemoving)
 }
 }

 /**
 This function reads through the input and populates an array of tokens.

Expand All@@ -82,49 +65,56 @@ class Parser {
 // Actions depend on the current state.
 switch state {
 case .other:
 skipAndProcessSingleLineComments()

 // Extract the upcoming control character, also switch the current state and append the extracted token, if any.
 if let extractedToken = try prepareNextState() {
 tokens.append(extractedToken)
 }
 case .readingKey:
 // Until the key-end marker is reached, the text should be interpreted as key.
 let currentKeyText = extractText(until: .quote)
 let potentialNewToken: Token = .key(currentKeyText)
 // If the prior token was also a key, append it.
 let newToken = tokenByConcatinatingwithPriorToken(potentialNewToken, seperatingString: EnclosingControlCharacters.quote.rawValue)
 tokens.append(newToken)
 // If the upcoming control character is also a key, do not stop reading a key. Otherwise a unescaped quote may exclude text from the key. Otherwise the state may be anything else.
 if let nextControlCharacter = findNextControlCharacter(andExtractFromSource: false), case EnclosingControlCharacters.quote = nextControlCharacter {
 state = .readingKey
 } else {
 state = .other
 }
 extractAndAppendIfPossible(for: .key(""), until: .quote)
 case .readingValue:
 // Text until value-end marker is a value.
 // If the prior token as also a value, append it.
 let currentValueText = extractText(until: .quote)
 let potentialNewToken: Token = .value(currentValueText)
 // If the prior token was also a key, append it.
 let newToken = tokenByConcatinatingwithPriorToken(potentialNewToken, seperatingString: EnclosingControlCharacters.quote.rawValue)
 tokens.append(newToken)
 // If the upcoming control character is also a key, do not stop reading a value. Otherwise a unescaped quote may exclude text from the value. Otherwise the state may be anything else.
 if let nextControlCharacter = findNextControlCharacter(andExtractFromSource: false), case EnclosingControlCharacters.quote = nextControlCharacter {
 state = .readingValue
 } else {
 state = .other
 }
 case .readingMessage:
 // Text until value-end marker is a message.
 extractAndAppendIfPossible(for: .value(""), until: .quote)
 case .readingMessage(let isReadingSingleLine):
 // If the prior token as also a message, DO NOT append it since the prior message could be a license header.
 let currentMessageText = extractText(until: .messageBoundaryClose)
 let endMarker: EnclosingControlCharacters = isReadingSingleLine ? .singleLineMessageClose : .messageBoundaryClose
 let currentMessageText = extractText(until: endMarker)
 let newToken: Token = .message(currentMessageText)
 tokens.append(newToken)
 state = .other
 }
 }
 }
 /// Extracts text from the input until the end marker is reached. Uses that text to create a new token and appends it to a prior extracted token if possible. In any case it updates the current list of extracted tokens.
 ///
 /// - Parameters:
 /// - token: The type of token that should be created from the text before the end marker. The associated value of the input is ignored.
 /// - endMarker: Marks the end of the tokens content.
 private func extractAndAppendIfPossible(for token: Token, until endMarker: EnclosingControlCharacters) {
 let currentText = extractText(until: endMarker)
 let potentialNewToken: Token
 switch token {
 case .key:
 potentialNewToken = .key(currentText)
 case .value:
 potentialNewToken = .value(currentText)
 default:
 assertionFailure("Currently, only the .key and .value support joining.")
 return
 }
 // Append to the prior token if possible.
 let newToken = tokenByConcatinatingwithPriorToken(potentialNewToken, seperatingString: endMarker.rawValue)
 tokens.append(newToken)
 // Do not stop reading when a newline or a quote is the next control character. Otherwise an unescaped quote may exclude text from the value. Keep the state unchanged if any other control character follows.
 if let nextControlCharacter = findNextControlCharacter(andExtractFromSource: false) {
 switch nextControlCharacter {
 case SeperatingControlCharacters.newline, EnclosingControlCharacters.singleLineMessageClose, EnclosingControlCharacters.quote:
 // Do not change the state and just continue.
 return
 default:
 break
 }
 }
 state = .other
 }
 /// Call this method when the list of tokens is ready and model object can be created. It will iterate through the tokens and try to map their values into model objects. Whe the mapping failed, an error is thrown.
 ///
 /// - Returns: The extracted model values.
Expand All@@ -134,6 +124,25 @@ class Parser {
 var currentKey: String?
 var currentValue: String?
 var results = [LocalizationString]()
 // The token that delimits an entry.
 guard let endToken = entriesEndToken(for: tokens) else {
 throw ParserError.malformattedInput
 }
 // Generates a result and appends it to the list of results if possible.
 func generateResultIfPossible(from processedToken: Token) {
 guard processedToken.isCaseEqual(to: endToken) else { return }
 // Done with that line. Check if values are populated and append them to the results.
 guard let key = currentKey, let value = currentValue else {
 return
 }
 let correctedMessage = removeLeadingTrailingSpaces(from: currentMessage)
 let entry = LocalizationString(key: key, value: value.replacingOccurrences(of: "\\\"", with: "\""), message: correctedMessage)
 results.append(entry)
 // Reset the properties to be ready for the next line.
 currentValue = nil
 currentKey = nil
 currentMessage = nil
 }
 // Iterate through the tokens and transform them into model objects.
 for token in tokens {
 switch token {
Expand All@@ -143,28 +152,36 @@ class Parser {
 currentKey = containedText
 case .value(let containedText):
 currentValue = containedText
 case .semicolon:
 // Done with that line. Check if values are populated and append them to the results.
 guard let key = currentKey, let value = currentValue else {
 throw ParserError.malformattedInput
 }
 let correctedMessage = removeLeadingTrailingSpaces(from: currentMessage)
 let entry = LocalizationString(key: key, value: value.replacingOccurrences(of: "\\\"", with: "\""), message: correctedMessage)
 results.append(entry)
 // Reset the properties to be ready for the next line.
 currentValue = nil
 currentKey = nil
 currentMessage = nil
 default:
 ()
 }
 generateResultIfPossible(from: token)
 }
 // Throw an execption to indicate that something went wront when tokens are extracted but they could not be transferred into model objects:
 if !tokens.isEmpty && results.isEmpty {
 throw ParserError.malformattedInput
 }
 return results
 }
 /// Determines the token that ends an entry. An entry can either be ended by a semicolon (if no comment was provided or the comment is above the entry) or a comment located at the end of a line. In the second case the `.message` token marks the end of the entry.
 ///
 /// - Parameter tokens: The tokens that were extracted during tokenization.
 /// - Returns: The token that ends an entry.
 private func entriesEndToken(for tokens: [Token]) -> Token? {
 // Assumption: after the first semicolon comes a new line -> semicolon delimits entry
 // After first semicolon comes a message, followed by a new line -> message delimits entry
 guard let semicolonIndex = tokens.firstIndex(where: { $0.isCaseEqual(to: .semicolon) }) else {
 return nil
 }
 guard let indexAfterSemicolon = tokens.index(semicolonIndex, offsetBy: 1, limitedBy: tokens.endIndex - 1) else { return nil }
 let elementAfterSemicolon = tokens[indexAfterSemicolon]
 switch elementAfterSemicolon {
 case .newline:
 return .semicolon
 default:
 return elementAfterSemicolon
 }
 }
 /// This function removes leading and trailing spaces from the input.
 ///
 /// - Parameter input: The string whose leading and trailing spaces should be removed.
Expand DownExpand Up@@ -265,7 +282,7 @@ class Parser {
 // Extract the given range and remove it from the input string.

 let lengthOfControlCharacter: Int = includingControlCharacter.skippingLength
 let endIndexOfExtraction = input.index(endindex, offsetBy: lengthOfControlCharacter)
 let endIndexOfExtraction = input.index(endindex, offsetBy: lengthOfControlCharacter, limitedBy: input.endIndex) ?? input.endIndex
 // Remove the range that includes the control character. The input range is used for extracting the text before it.
 let rangeForRemoving = input.startIndex ..< endIndexOfExtraction
 let rangeForExtraction = input.startIndex ..< endindex
Expand DownExpand Up@@ -354,7 +371,7 @@ extension Parser {
 case EnclosingControlCharacters.messageBoundaryOpen:
 // A new message begins.
 // Set the state to expect a message.
 state = .readingMessage
 state = .readingMessage(isSingleLine: false)
 case EnclosingControlCharacters.messageBoundaryClose:
 // Message-end markers should only be detected when the lexer is reading a message. If they occure 'in the wild' the input must be ill formatted.
 break
Expand All@@ -366,6 +383,12 @@ extension Parser {
 // Extract semicolon as token. A quote or message-start mark will follow as next control character but for now the state remains .other in order to detect that quote.
 returnToken = .semicolon
 state = .other
 case SeperatingControlCharacters.newline:
 returnToken = .newline
 case EnclosingControlCharacters.singleLineMessageOpen:
 state = .readingMessage(isSingleLine: true)
 case EnclosingControlCharacters.singleLineMessageClose:
 returnToken = .newline
 default:
 // New types need to be registered.
 throw ParserError.notParsable
Expand Down

46 changes: 41 additions & 5 deletions
sources/LocalizationEditor/Providers/ParserTypes.swift

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -15,12 +15,36 @@ import Foundation
/// - key: The key and its text.
/// - equal: The equal sign that maps a key to a value: ".
/// - semicolon: The semicolon that ends a line: ;
/// - newline: A new line \n.
enum Token {
 case message(String)
 case value(String)
 case key(String)
 case equal
 case semicolon
 case newline
 /// Checks if `self` is of the same type as `other` without taking the associated values into account.
 ///
 /// - Parameter other: The token to which self should be compared to.
 /// - Returns: `true` when the type of `other` matches the type of `self` without taking associated values into account.
 func isCaseEqual(to other: Token) -> Bool {
 switch (self, other) {
 case (.message, .message):
 return true
 case (.value, .value):
 return true
 case (.key, .key):
 return true
 case (.equal, .equal):
 return true
 case (.semicolon, .semicolon):
 return true
 case (.newline, .newline):
 return true
 default:
 return false
 }
 }
}

/// Control characters define starting and end points of tokens. They can be for example ", /* or ;
Expand All@@ -35,13 +59,17 @@ protocol EnclosingType: ControlCharacterType {}
protocol SeperatingType: ControlCharacterType {}

/// Enclosing control characters that wrapp text. They may start or end a message or contain a value/key.
/// - messageBoundaryOpen Opens a message.
/// - messageBoundaryClose Ends a message.
/// - quote Wraps a key or a value.
/// - messageBoundaryOpen: Opens a message.
/// - messageBoundaryClose: Ends a message.
/// - quote: Wraps a key or a value.
/// - singleLineMessageOpen: Opens a single line message.
/// - singleLineMessageClose: Closes the single line message.
enum EnclosingControlCharacters: String, EnclosingType, CaseIterable {
 case messageBoundaryOpen = "/*"
 case messageBoundaryClose = "*/"
 case quote = "\""
 case singleLineMessageOpen = "//"
 case singleLineMessageClose = "\n"

 var skippingLength: Int {
 switch self {
Expand All@@ -51,25 +79,33 @@ enum EnclosingControlCharacters: String, EnclosingType, CaseIterable {
 return EnclosingControlCharacters.messageBoundaryClose.rawValue.count
 case .quote:
 return EnclosingControlCharacters.quote.rawValue.count
 case .singleLineMessageOpen:
 return EnclosingControlCharacters.singleLineMessageOpen.rawValue.count
 case .singleLineMessageClose:
 return EnclosingControlCharacters.singleLineMessageClose.rawValue.count
 }
 }
}

/// Seperating control characters do not wrap text. They function as position markers. For example they seperate a key from its value or end the line.
/// - equal The equal sign that seperates a key from its value.
/// - semicolon The semicolon that end a line.
/// - equal: The equal sign that seperates a key from its value.
/// - semicolon: The semicolon that end a line.
/// - newline: A new line.
enum SeperatingControlCharacters: String, SeperatingType, CaseIterable {
 var skippingLength: Int {
 switch self {
 case .equal:
 return SeperatingControlCharacters.equal.rawValue.count
 case .semicolon:
 return SeperatingControlCharacters.semicolon.rawValue.count
 case .newline:
 return SeperatingControlCharacters.newline.rawValue.count
 }
 }

 case equal = "="
 case semicolon = ";"
 case newline = "\n"
}

/// Errors that may occure during parsing.
Expand Down
Loading