YAML scanner improvements

[nathany]

YAML scanner has problems with unusual mapping keys

Trans reported:

Found an issue with the YAML syntax highlighting in CodeRay. Try it on the following example.
When it hits mri <1.9 it starts reverse highlighting almost every other line.

 - package: system_timer
 engine : mri <1.9

The problem is that the scanner doesn't allow white space before the colon. That's an easy fix; however, the set of characters allowed in keys is limited and non spec-conform.

See http://yaml.org/spec/1.2/spec.html#id2788859 for further reading.

A patch for the 1.0 trunk:

Index: /Users/murphy/ruby/coderay/lib/coderay/scanners/yaml.rb
===================================================================
--- /Users/murphy/ruby/coderay/lib/coderay/scanners/yaml.rb (revision 580)
+++ /Users/murphy/ruby/coderay/lib/coderay/scanners/yaml.rb (working copy)
@@ -75,20 +75,18 @@
 when match = scan(/[,{}\[\]]/)
 encoder.text_token match, :operator
 next
- when state == :initial && match = scan(/[\w.() ]*\S(?=: |:$)/)
+ when state == :initial && match = scan(/[\w.() ]*\S(?= *:(?: |$))/)
 encoder.text_token match, :key
 key_indent = column(pos - match.size - 1)
- # encoder.text_token key_indent.inspect, :debug
 state = :colon
 next
- when match = scan(/(?:"[^"\n]*"|'[^'\n]*')(?=: |:$)/)
+ when match = scan(/(?:"[^"\n]*"|'[^'\n]*')(?= *:(?: |$))/)
 encoder.begin_group :key
 encoder.text_token match[0,1], :delimiter
 encoder.text_token match[1..-2], :content
 encoder.text_token match[-1,1], :delimiter
 encoder.end_group :key
 key_indent = column(pos - match.size - 1)
- # encoder.text_token key_indent.inspect, :debug
 state = :colon
 next
 when match = scan(/(![\w\/]+)(:([\w:]+))?/)

From Redmine: http://odd-eyed-code.org/issues/231

---

YAML scanner doesn't recognize false and true

It should also recognize -.Inf and such. Some readable part of the spec is about the Core Schema which Ruby seems to use.

Needs more investigation.

Index: lib/coderay/scanners/yaml.rb
===================================================================
--- lib/coderay/scanners/yaml.rb (revision 742)
+++ lib/coderay/scanners/yaml.rb (working copy)
@@ -11,6 +11,11 @@

 KINDS_NOT_LOC = :all

+ CONSTANTS = %w[ true True TRUE false False FALSE null Null NULL ] # :nodoc:
+ 
+ IDENT_KIND = WordList.new(nil).
+ add(CONSTANTS, :pre_constant)
+ 
 protected

 def scan_tokens encoder, options
@@ -59,7 +64,11 @@
 encoder.text_token matched, :content if scan(/(?:\n+ {#{string_indent + 1}}.*)+/)
 encoder.end_group :string
 next
- when match = scan(/(?![!"*&]).+?(?=$|\s+#)/)
+ when match = scan(/(?![!*&\[\],{}]|- ).+?(?=$|\s+#)/)
+ if kind = IDENT_KIND[match]
+ encoder.text_token match, kind
+ next
+ end
 encoder.begin_group :string
 encoder.text_token match, :content
 string_indent = key_indent || column(pos - match.size - 1)
@@ -116,6 +125,9 @@
 when match = scan(/:\w+/)
 encoder.text_token match, :symbol
 next
+ when match = scan(/~|\.(?:inf|Inf|INF|nan|NaN|NAN)/)
+ encoder.text_token match, :pre_constant
+ next
 when match = scan(/[^:\s]+(:(?! |$)[^:\s]*)* .*/)
 encoder.text_token match, :error
 next

From Redmine: http://odd-eyed-code.org/issues/234

---

YAML scanner doesn't recognize - -

required_ruby_version: !ruby/object:Gem::Requirement 
 requirements: 
 - - ">=" 
 - !ruby/object:Gem::Version 
 version: 1.8.2
 version: 

The third line is currently interpreted as operator(- )string(- ">="), which is wrong. It should be operator(- )operator(- )string(">=").

From Redmine: http://odd-eyed-code.org/issues/237

---

The yaml.multiline example is broken

The YAML scanner doesn't seem to tokenize multiline strings correctly.

From Redmine: http://odd-eyed-code.org/issues/238

---

YAML scanner doesn't recognize arrays

...like [1, 2, 3].

From Redmine: http://odd-eyed-code.org/issues/239

[korny]

Not sure how to approach this ticket. It seems to me that the YAML scanner needs to be rewritten. Questions:

• What are the most problematic shortcomings of the current YAML scanner?
• Is the Pygments one any better?
• What's the best approach to scanning such an over-complicated "natural" format? Heuristics which lots of special cases or a more simple/rigid scanner that's optimized for the typical style people use?

[nathany]

I wonder if it would make sense to base this on a subset, such as SafeYAML, treating anything else like an error?

[korny]

I don't think SafeYAML is any easier to parse. The problem is the indentation-based syntax.