A tiny collection of high-performance parsers and dumpers
Written in Nim language
nimble install openparser
OpenParser is a collection of parsers and dumpers (serializers) for various data formats, written in Nim. Each module provides zero-copy parsing via memory-mapped files, direct-to-object deserialization, custom hooks for extending type support, and context-aware error reporting.
Note
Importing openparser directly will produce a compile-time error. Import the specific module you need, e.g. openparser/json for JSON.
Zero-copy JSON parser with SIMD-accelerated tokenization, direct-to-object parsing, and full hook support. Exports std/json for JsonNode compatibility.
import openparser/json
typePerson=object
name: string
age: int
email: string# Parse to JsonNodelet data ="""{"name":"Albush","age":40,"email":"al@ex.com"}"""let node: JsonNode=fromJson(data)
echo node["name"].getStr # Albush# Parse directly into Nim objectslet person: Person=fromJson(data, Person)
echo person.name # Albush# Serialize back to JSONechotoJson(person) # {"name":"Albush","age":40,"email":"al@ex.com"}# Memfile-based parsing for large fileslet bigNode =fromJsonFile("huge.json")Features:parseHook/dumpHook for custom types, renameHook for field name mapping, currentField context, skipValue, toStaticJson compile-time optimization, line-delimited JSON (fromJsonL), Option[T] support, maxDepth DoS protection.
Protect against deeply nested JSON that could cause stack overflow:
import openparser/json
# Limit nesting depth to prevent DoS attackslet opts =JsonOptions(maxDepth: 10)
let data ="""{"a":{"b":{"c":1}}}"""# Raises OpenParserJsonError if depth exceeds 10let node =fromJson(data, opts)
# Also works with typed parsinglet user =fromJson(data, User, opts)Map JSON keys to Nim field names using the {.json: "wireName".} pragma (works for compile-time serialization):
import openparser/json
typeUser=object
name {.json: "username".}: string
age {.json: "user_age".}: int
email: string# no pragma - uses field name# Dump: field "name" outputs as "username" using toStaticJsonlet user =User(name: "Alice", age: 30)
let jsonStr =toStaticJson(user)
echo jsonStr # {"username":"Alice","user_age":30}Note
The {.json: "xx".} pragma currently works for toStaticJson (compile-time dump). Runtime toJson and fromJson support is a TODO.
YAML 1.2 parser with block and inline syntax, comments, block scalars, and the same hook-based API as JSON.
import openparser/yaml
typeConfig=object
host: string
port: int
debug: boollet yaml ="""host: localhostport: 8080debug: true"""# Parse to YAMLObject treelet obj: YAMLObject=parseYAML(yaml)
echo obj["host"].strValue # localhost# Parse directly into Nim objectslet config: Config=parseYAML(yaml, Config)
echo config.port # 8080Features: Inline and block sequences/mappings, nested structures, comments, block scalars (|, >), dot-notation access, direct-to-object parsing via parseHook/dumpHook, renameHook, currentField.
Full-featured XML parser with element/attribute mapping to Nim objects, CDATA, comments, entities, self-closing tags, and memfile support. Same hook-based API as JSON/YAML.
import openparser/xml
typePerson=object
name: string
age: int
email: stringlet xml ="""<person name="Alice" age="30"> <email>alice@example.com</email></person>"""# Parse to XmlNode treelet node: XmlNode=fromXml(xml)
echo node["email"].children[0].text # alice@example.com# Parse directly into Nim objects (attributes + child elements)let person: Person=fromXml(xml, Person)
echo person.name # Alice# Serialize back to XMLechotoXml(person, XmlOptions(rootTag: "person"))
# <person><name>Alice</name><age>30</age><email>alice@example.com</email></person># Memfile-based parsinglet doc =fromXmlFile("large.xml")Features: Attributes and child elements both map to object fields, repeated child tags -> seq[T], enum/discriminator attributes for variant objects, parseHook/dumpHook for custom types, renameHook, xmlAttrHook for attribute vs element control, entity decoding (&, &#xHH;), CDATA, comments, processing instructions, XmlNode DOM tree.
TOML config file parser with datetime support, inline tables, arrays, and the same hook-based direct-to-object API.
import openparser/toml
typeServerConfig=object
host: string
port: intlet toml ="""[server]host = "localhost"port = 8080"""# Parse to TomlDocumentlet doc: TomlDocument=parseTOML(toml)
# Parse directly into Nim objectslet config: ServerConfig=parseTOML(toml, ServerConfig)
echo config.port # 8080Features: Sections, inline tables, arrays, datetime types, parseHook/dumpHook, direct-to-object parsing.
Zero-copy CSV parser using memory-mapped files. Processes rows via callback without loading the entire file into memory.
import openparser/csv
# Stream-parse a large CSV filevar i =0parseFile("data.csv",
proc(fields: openArray[CsvFieldSlice], row: int): bool=inc i
for field in fields:
echo field.toString()
true# return true to continue, false to stop
)
echo"Parsed ", i, " rows"Features: Zero-copy parsing via MemFile, configurable delimiters and quote characters, streaming row callback, handles ~600MB+ files efficiently.
Binary JSON encoding/decoding following the BSON 1.1 spec. Converts between JsonNode and raw BSON bytes.
import openparser/[json, bson]
# Encode JSON to BSONlet json =fromJson("""{"name":"Alice","age":30,"active":true}""")
let bsonBytes: seq[byte] = json.toBson()
# Decode BSON back to JSONlet decoded: JsonNode=fromBson(bsonBytes)
echo decoded["name"].getStr # AliceFeatures: Full BSON type support (ObjectId, Date, Binary, Regex, Timestamp, Code, Decimal128), extended JSON v2 notation, streaming encode/decode.
HTML5 parser with configurable parsing policies, memfile support, and a DOM tree output. Handles real-world HTML gracefully.
import openparser/html
let html ="""<html><body><h1>Hello</h1><p>World</p></body></html>"""# Parse with default policy (tolerant)let doc =parseHtml(html)
# Parse a file with a strict policylet policy =defaulHtmlParsingPolicy()
let doc2 =parseHtmlFile("page.html", policy)Features: Configurable parsing policy (self-closing tags, unclosed tags, comments, CDATA, entities, etc.), memfile-based file parsing, HtmlDocument DOM tree.
Parse, read, fetch, and serialize RSS and Atom feeds.
import openparser/rss
import openparser/feed
# RSSlet feed =parseRss(rssXmlString)
echo feed.title
let xml =toRssXml(feed)
# Atomlet atom =parseAtom(atomXmlString)
echo atom.title
let atomXml =toAtomXml(atom)
# Read from file or fetch from URLlet rssFromFile =readRss("feed.xml")
let rssFromUrl =fetchRss("https://example.com/feed.xml")Features: Parse from string/file/URL, serialize back to XML, full feed metadata and entry access.
Parse and load .env files with variable expansion, command substitution, and environment-specific overrides.
import openparser/dotenv
# Load a .env file into the environmentloadDotenv(".env")
# Parse without loadinglet entries =parseEnv("DB_HOST=localhost\nDB_PORT=5432")
for entry in entries:
echo entry.key, "=", entry.value
# Access loaded valuesechoget("DB_HOST") # localhost# Environment-specific loadingloadDotenvForEnv("production")Features: Variable expansion (${VAR}), command substitution (${CMD:default}), override control, get/set/del/has API.
SQL parser and AST builder supporting PostgreSQL, MySQL, and SQLite dialects.
import openparser/sql
let ast =parseSql("SELECT name, age FROM users WHERE active = true ORDER BY name")
echo ast # select name, age from users where active = true order by nameFeatures: SELECT/INSERT/UPDATE/DELETE, JOINs, subqueries, GROUP BY, HAVING, ORDER BY, LIMIT, AST manipulation, query builder.
SIMD-accelerated regex engine with a full parser, compiler, and VM.
import openparser/regex
# Simple matchletresult=match("hello world", "hello")
echoresult.matched # trueechoresult.start # 0echoresult.stop # 5# Find in stringlet found =find("hello world", r"world")
echo found.matched # trueecho found.start # 6# Find all occurrenceslet all =findAll("aabbcc", r"[a-c]+")
echo all.len # 3# Capture groupslet m =match("2024-01-15", r"(\d{4})-(\d{2})-(\d{2})")
if m.matched:
echo m.groupStr("2024-01-15", 1) # "2024"echo m.groupStr("2024-01-15", 2) # "01"echo m.groupStr("2024-01-15", 3) # "15"# Character classes and quantifierslet email =match("user@example.com", r"[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}")
echo email.matched # trueFeatures: SSE2/AVX2 acceleration, character classes, quantifiers, alternation, capture groups, anchoring.
Parse and compile PO/MO translation files with plural form support.
import openparser/gettext/[po, mo]
# Parse a .po file and compile itlet cat =openPoCatalog("messages.po")
let cc =compilePo(cat)
# Simple translationlet greeting = cc.translate("Hello, world!")
echo greeting
# Plural forms (English: singular vs plural)let msg = cc.ntranslate("apple", "apples", 5)
echo msg # "apples"# Russian plural forms (3 forms)let ruMsg = cc.ntranslate("товар", "товара", 5)
echo ruMsg # "товаров"# Parse headerslet headers =parsePoHeaders(cat)
echo headers["Language"] # "en"# Compile to .mo binary formatwriteMoFile(cc, "messages.mo")
close(cat)
# Or parse .mo directlylet moCat =openMoCatalog("messages.mo")
let moMsg = moCat.translate("Hello")
close(moCat)Features: PO/MO parsing, plural form expressions, header extraction, binary MO compilation.
Encode and decode structured data using Fast Binary Encoding. Supports both a high-level compact API and a low-level field-based API with custom field IDs.
import openparser/fbe
typePerson=object
name: string
age: int32
bio: string# High-level: encodeFinal/decodeFinal (compact, automatic field ordering)let alice =Person(name: "Alice", age: 30, bio: "hello")
let buf =encodeFinal(alice) # -> Buffervar decoded =Person()
decodeFinal(buf, decoded) # round-tripassert decoded.name =="Alice"# Encode/decode sequenceslet people =@[Person(name: "Bob", age: 25), Person(name: "Carol", age: 28)]
let seqBuf =encodeFinal(people)
var decodedPeople: seq[Person]
decodeFinal(seqBuf, decodedPeople)
# Low-level: custom field IDs and versioninglet buf2 =encode(alice, 7'u32, proc (fieldName: string): uint16=if fieldName =="name": 1'u16elif fieldName =="age": 2'u16elif fieldName =="bio": 3'u16else: 0'u16
)Features: Zero-copy buffer-based encoding, custom field IDs, struct versioning, disk round-trip, UUID/timestamp/decimal/vector support, UTF-8 strings, inner structs, benchmarked at 10K+ objects.
| Feature | JSON | YAML | XML | TOML | CSV |
|---|---|---|---|---|---|
| Zero-copy / Memfiles | x | x | x | ||
| Direct-to-object | x | x | x | x | |
parseHook / dumpHook | x | x | x | x | |
renameHook | x | x | x | x | |
currentField context | x | x | x | x | |
skipValue | x | x | x | x | |
XmlNode / JsonNode tree | x | x | x | x | |
| SIMD acceleration | x | x | |||
| Context-aware errors | x | x | x | x | x |
Most parsers provide context-aware error reporting with a snippet of the input around the error location:
<person name="Alice" age="30"/>
^
Error (1:26) Unexpected EOF while parsing `element`
{"name":"Alice","age":"isMember":true}
^
Error (1:33) Unexpected token `:`
- JSON depth/size limit to prevent DoS attacks
- JSON schema validation support
- JSON custom field mapping (compile-time
{.json: "xx".}pragma)
Note
Some implementations (dotenv, fbe, gettext) may be incomplete. Contributions are welcome!
- Found a bug? Create a new Issue
- Want to help? Fork it!
MIT license. Made by Humans from OpenPeeps.
Copyright OpenPeeps & Contributors — All rights reserved.