zek

Zek is a prototype for creating a Go struct from an XML document. The resulting struct works best for reading XML (see also #14), to create XML, you might want to use something else.

It was developed at Leipzig University Library to shorten the time to go from raw XML to a struct that allows to access XML data in Go programs.

Skip the fluff, just the code.

Given some XML, run:

$ curl-s https://raw.githubusercontent.com/miku/zek/master/fixtures/e.xml | zek -e// Rss was generated 2018-08-30 20:24:14 by tir on sol.typeRssstruct {
XMLName xml.Name`xml:"rss"`Textstring`xml:",chardata"`Rdfstring`xml:"rdf,attr"`Dcstring`xml:"dc,attr"`Geoscanstring`xml:"geoscan,attr"`Mediastring`xml:"media,attr"`Gmlstring`xml:"gml,attr"`Taxostring`xml:"taxo,attr"`Georssstring`xml:"georss,attr"`Contentstring`xml:"content,attr"`Geostring`xml:"geo,attr"`Versionstring`xml:"version,attr"`Channelstruct {
Textstring`xml:",chardata"`Titlestring`xml:"title"`// ESS New Releases (Display...Linkstring`xml:"link"`// http://tinyurl.com/ESSNew...Descriptionstring`xml:"description"`// New releases from the Ear...LastBuildDatestring`xml:"lastBuildDate"`// Mon, 27 Nov 2017 00:06:35...Item []struct {
Textstring`xml:",chardata"`Titlestring`xml:"title"`// Surficial geology, Aberde...Linkstring`xml:"link"`// https://geoscan.nrcan.gc....Descriptionstring`xml:"description"`// Geological Survey of Cana...Guidstruct {
Textstring`xml:",chardata"`// 304279, 306212, 306175, 3...IsPermaLinkstring`xml:"isPermaLink,attr"`
} `xml:"guid"`PubDatestring`xml:"pubDate"`// Fri, 24 Nov 2017 00:00:00...Polygon []string`xml:"polygon"`// 64.0000 -98.0000 64.0000 ...Downloadstring`xml:"download"`// https://geoscan.nrcan.gc....Licensestring`xml:"license"`// http://data.gc.ca/eng/ope...Authorstring`xml:"author"`// Geological Survey of Cana...Sourcestring`xml:"source"`// Geological Survey of Cana...SndSeriesstring`xml:"SndSeries"`// Bedford Institute of Ocea...Publisherstring`xml:"publisher"`// Natural Resources Canada,...Editionstring`xml:"edition"`// prelim., surficial data m...Meetingstring`xml:"meeting"`// Geological Association of...Documenttypestring`xml:"documenttype"`// serial, open file, serial...Languagestring`xml:"language"`// English, English, English...Mapsstring`xml:"maps"`// 1 map, 5 maps, Publicatio...Mapinfostring`xml:"mapinfo"`// surficial geology, surfic...Mediumstring`xml:"medium"`// on-line; digital, digital...Provincestring`xml:"province"`// Nunavut, Northwest Territ...Ntsstring`xml:"nts"`// 066B, 095J; 095N; 095O; 0...Areastring`xml:"area"`// Aberdeen Lake, Mackenzie ...Subjectsstring`xml:"subjects"`Programstring`xml:"program"`// GEM2: Geo-mapping for Ene...Projectstring`xml:"project"`// Rae Province Project Mana...Projectnumberstring`xml:"projectnumber"`// 340521, 343202, 340557, 3...Abstractstring`xml:"abstract"`// This new surficial geolog...Linksstring`xml:"links"`// Online - En ligne (PDF, 9...Readmestring`xml:"readme"`// readme | https://geoscan....PPIidstring`xml:"PPIid"`// 34532, 35096, 35438, 2563...
} `xml:"item"`
} `xml:"channel"`
}

Online

try online via WASM: https://xml-to-go.github.io/, thanks YaroslavPodorvanov!
try it online at https://blog.kowalczyk.info/tools/xmltogo/ -- thanks, kjk!

About

Upsides:

it works fine for non-recursive structures,
does not need XSD or DTD,
it is relatively convenient to access attributes, children and text,
will generate a single struct, which make for a quite compact representation,
simple user interface,
comments with examples,
schema inference across multiple files.

Downsides:

experimental, early, buggy, unstable prototype,
no support for recursive types (similar to Russian Doll strategy, [1])
no type inference, everything is accessible as string (without a schema, type inference may fail if the type guess is wrong)

Bugs:

Mapping between XML elements and data structures is inherently flawed: an XML element is an order-dependent collection of anonymous values, while a data structure is an order-independent collection of named values.

https://golang.org/pkg/encoding/xml/#pkg-note-BUG

Related projects:

And other awesome XML utilities.

Presentations:

Lightning talk about struct generators at GoLab 2018

Install

$ go install github.com/miku/zek/cmd/zek@latest

Debian and RPM packages:

https://github.com/miku/zek/releases

It's in AUR, too.

Usage

$ zek -h
-B use a fixed banner string (e.g. for CI)
-C emit less compact struct
-F skip formatting
-I use verbatim innerxml instead of chardata
-P string
if set, write out struct within a package with the given name
-S int
read at most this many tags, approximately (0=unlimited)
-c emit more compact struct (noop, as this is the default since 0.1.7)
-d debug output
-e add comments with example
-j add JSON tags
-m omit empty Text fields
-max-examples int
limit number of examples (default 10)
-n string
use a different name for the top-level struct
-o string
if set, write to output file, not stdout
-p write out an example program
-s strict parsing and writing
-t string
emit struct for tag matching this name
-u filter out duplicated examples
-version
show version
-x int
max chars for example (default 25)

Examples:

$ cat fixtures/a.xml
<a></a>
$ zek -C < fixtures/a.xml
type A struct {
XMLName xml.Name `xml:"a"`
Text string `xml:",chardata"`
}

Debug output dumps the internal tree as JSON to stdout.

$ zek -d < fixtures/a.xml
{"name":{"Space":"","Local":"a"}}

Example program:

package main
import (
"encoding/json""encoding/xml""fmt""log""os"
)
// A was generated 2017-12-05 17:35:21 by tir on apollo.typeAstruct {
XMLName xml.Name`xml:"a"`Textstring`xml:",chardata"`
}
funcmain() {
dec:=xml.NewDecoder(os.Stdin)
vardocAiferr:=dec.Decode(&doc); err!=nil {
log.Fatal(err)
}
b, err:=json.Marshal(doc)
iferr!=nil {
log.Fatal(err)
}
fmt.Println(string(b))
}
$ zek-C-p<fixtures/a.xml>sample.go&&gorunsample.go<fixtures/a.xml|jq . &&rm sample.go
{
"XMLName": {
"Space": "",
"Local": "a"
},
"Text": ""
}

More complex example:

$ zek<fixtures/d.xml// Root was generated 2019-06-11 16:27:04 by tir on hayiti.typeRootstruct {
XMLName xml.Name`xml:"root"`Textstring`xml:",chardata"`A []struct {
Textstring`xml:",chardata"`B []struct {
Textstring`xml:",chardata"`Cstring`xml:"c"`Dstring`xml:"d"`
} `xml:"b"`
} `xml:"a"`
}
$ zek-p<fixtures/d.xml>sample.go&&gorunsample.go<fixtures/d.xml|jq . &&rm sample.go
{
"XMLName": {
"Space": "",
"Local": "root"
},
"Text": "\n\n\n\n",
"A": [
{
"Text": "\n\n\n",
"B": [
{
"Text": "\n\n ",
"C": "Hi",
"D": ""
},
{
"Text": "\n\n\n ",
"C": "World",
"D": ""
}
]
},
{
"Text": "\n\n",
"B": [
{
"Text": "\n\n ",
"C": "Hello",
"D": ""
}
]
},
{
"Text": "\n\n",
"B": [
{
"Text": "\n\n ",
"C": "",
"D": "World"
}
]
}
]
}

Annotate with comments:

$ zek-e<fixtures/l.xml// Records was generated 2019-06-11 16:29:35 by tir on hayiti.typeRecordsstruct {
XMLName xml.Name`xml:"Records"`Textstring`xml:",chardata"`// \nXsistring`xml:"xsi,attr"`Record []struct {
Textstring`xml:",chardata"`Headerstruct {
Textstring`xml:",chardata"`Statusstring`xml:"status,attr"`Identifierstring`xml:"identifier"`// oai:ojs.localhost:article...Datestampstring`xml:"datestamp"`// 2009-06-24T14:48:23Z, 200...SetSpecstring`xml:"setSpec"`// eppp:ART, eppp:ART, eppp:...
} `xml:"header"`Metadatastruct {
Textstring`xml:",chardata"`Rfc1807struct {
Textstring`xml:",chardata"`Xmlnsstring`xml:"xmlns,attr"`Xsistring`xml:"xsi,attr"`SchemaLocationstring`xml:"schemaLocation,attr"`BibVersionstring`xml:"bib-version"`// v2, v2, v2...IDstring`xml:"id"`// http://jou...Entrystring`xml:"entry"`// 2009-06-24...Organization []string`xml:"organization"`// Proceeding...Titlestring`xml:"title"`// Introducti...Typestring`xml:"type"`Author []string`xml:"author"`// KRAMPEN, G..Copyrightstring`xml:"copyright"`// Das Urhebe...OtherAccessstring`xml:"other_access"`// url:http:/...Keywordstring`xml:"keyword"`Period []string`xml:"period"`Monitoringstring`xml:"monitoring"`Languagestring`xml:"language"`// en, en, en, e...Abstractstring`xml:"abstract"`// After a short...Datestring`xml:"date"`// 2009-06-22 12...
} `xml:"rfc1807"`
} `xml:"metadata"`Aboutstring`xml:"about"`
} `xml:"Record"`
}

Only consider a nested element

$ zek-tmetadatafixtures/z.xml// Metadata was generated 2019-06-11 16:33:26 by tir on hayiti.typeMetadatastruct {
XMLName xml.Name`xml:"metadata"`Textstring`xml:",chardata"`Dcstruct {
Textstring`xml:",chardata"`Xmlnsstring`xml:"xmlns,attr"`Titlestruct {
Textstring`xml:",chardata"`Xmlnsstring`xml:"xmlns,attr"`
} `xml:"title"`Identifierstruct {
Textstring`xml:",chardata"`Xmlnsstring`xml:"xmlns,attr"`
} `xml:"identifier"`Rightsstruct {
Textstring`xml:",chardata"`Xmlnsstring`xml:"xmlns,attr"`Langstring`xml:"lang,attr"`
} `xml:"rights"`AccessRightsstruct {
Textstring`xml:",chardata"`Xmlnsstring`xml:"xmlns,attr"`
} `xml:"accessRights"`
} `xml:"dc"`
}

Inference across files

$ zekfixtures/a.xmlfixtures/b.xmlfixtures/c.xml// A was generated 2017-12-05 17:40:14 by tir on apollo.typeAstruct {
XMLName xml.Name`xml:"a"`Textstring`xml:",chardata"`B []struct {
Textstring`xml:",chardata"`
} `xml:"b"`
}

This is also useful, if you deal with archives containing XML files:

$ unzip -p 4082359.zip '*.xml'| zek -e

Given a directory full of zip files, you can combined find, unzip and zek:

$ foriin$(find ftp/b571 -type f -name "*zip");do unzip -p $i'*xml';done| zek -e

Another example (tarball with thousands of XML files, seemingly MARC):

$ tar -xOzf /tmp/20180725.125255.tar.gz | zek -e
// OAIPMH was generated 2018-09-26 15:03:29 by tir on sol.
type OAIPMH struct {
XMLName xml.Name `xml:"OAI-PMH"`
Text string `xml:",chardata"`
Xmlns string `xml:"xmlns,attr"`
Xsi string `xml:"xsi,attr"`
SchemaLocation string `xml:"schemaLocation,attr"`
ListRecords struct {
Text string `xml:",chardata"`
Record struct {
Text string `xml:",chardata"`
Header struct {
Text string `xml:",chardata"`
Identifier struct {
Text string `xml:",chardata"` // aleph-pub:000000001, ...
} `xml:"identifier"`
} `xml:"header"`
Metadata struct {
Text string `xml:",chardata"`
Record struct {
Text string `xml:",chardata"`
Xmlns string `xml:"xmlns,attr"`
Xsi string `xml:"xsi,attr"`
SchemaLocation string `xml:"schemaLocation,attr"`
Leader struct
Text string `xml:",chardata"` // 00001nM2.01200024
} `xml:"leader"`
Controlfield []struct {
Text string `xml:",chardata"` // 00001nM2.01200024
Tag string `xml:"tag,attr"`
} `xml:"controlfield"`
Datafield []struct {
Text string `xml:",chardata"`
Tag string `xml:"tag,attr"`
Ind1 string `xml:"ind1,attr"`
Ind2 string `xml:"ind2,attr"`
Subfield []struct {
Text string `xml:",chardata"` // KM0000002
Code string `xml:"code,attr"`
} `xml:"subfield"`
} `xml:"datafield"`
} `xml:"record"`
} `xml:"metadata"`
} `xml:"record"`
} `xml:"ListRecords"`
}

Generate a package

If you want in include generated file in the build process, e.g. with go generate, you may find -P and -o helpful.

$ cat fixtures/b.xml
<a><b></b></a>

Run on the command line or via go generate:

$ zek -P mypkg -o data.go < fixtures/b.xml

This would write out the following in data.go file:

// Code generated by zek; DO NOT EDIT.package mypkg
import"encoding/xml"// A was generated 2021-09-16 11:23:06 by tir on trieste.typeAstruct {
XMLName xml.Name`xml:"a"`Textstring`xml:",chardata"`Bstring`xml:"b"`
}

Note that any existing file will be overwritten, without any warning.

Use innerxml instead of chardata

You may want chardata or innerxml tag. Default is chardata, to use innerxml use the -I flag.

$ zek -B -I fixtures/d.xml
// Root was generated automatically by zek 0.1.24. DO NOT EDIT.
type Root struct {
XMLName xml.Name `xml:"root"`
Text string `xml:",innerxml"`
A []struct {
Text string `xml:",innerxml"`
B []struct {
Text string `xml:",innerxml"`
C string `xml:"c"`
D string `xml:"d"`
} `xml:"b"`
} `xml:"a"`
}

Misc

As a side effect, zek seems to be a useful for debugging. Example:

https://git.io/vbUNa

This record is emitted from a typical OAI server (OJS, not even uncommon), yet one can quickly spot the flaw in the structure.

Over 30 different struct generated manually in the course of a few hours (around five minutes per source): https://git.io/vbTDo.

-- Current extent leader: 1532 lines struct

Using zek:

Finding Aids Unleashed: Iterative Development of a Portable Publication System, New York University Libraries

Name		Name	Last commit message	Last commit date
.github/workflows		.github/workflows
cmd/zek		cmd/zek
docs		docs
examples		examples
fixtures		fixtures
testdata		testdata
.build.yml		.build.yml
.gitignore		.gitignore
.gitlab-ci.yml		.gitlab-ci.yml
.goreleaser.yml		.goreleaser.yml
LICENSE		LICENSE
Makefile		Makefile
README.md		README.md
go.mod		go.mod
go.sum		go.sum
io.go		io.go
nfpm.yaml		nfpm.yaml
node.go		node.go
node_test.go		node_test.go
stack.go		stack.go
stack_test.go		stack_test.go
structwriter.go		structwriter.go
structwriter_test.go		structwriter_test.go
version.go		version.go

Navigation Menu

Repository files navigation

zek

Online

About

Install

Usage

Only consider a nested element

Inference across files

Generate a package

Use innerxml instead of chardata

Misc

About

Releases

Packages

Used by

Contributors

Languages

Latest commit

History

Folders and files

Repository files navigation

zek

Online

About

Install

Usage

Only consider a nested element

Inference across files

Generate a package

Use innerxml instead of chardata

Misc

About

Topics

Resources

Stars

Watchers

Forks

Releases

Packages

Used by

Contributors

Languages