Skip to content
BranchesTags

Repository files navigation

zek

Zek is a prototype for creating a Go struct from an XML document. The resulting struct works best for reading XML (see also #14), to create XML, you might want to use something else.

It was developed at Leipzig University Library to shorten the time to go from raw XML to a struct that allows to access XML data in Go programs.

Skip the fluff, just the code.

Given some XML, run:

$ curl-s https://raw.githubusercontent.com/miku/zek/master/fixtures/e.xml | zek -e// Rss was generated 2018-08-30 20:24:14 by tir on sol.typeRssstruct {
XMLName xml.Name`xml:"rss"`Textstring`xml:",chardata"`Rdfstring`xml:"rdf,attr"`Dcstring`xml:"dc,attr"`Geoscanstring`xml:"geoscan,attr"`Mediastring`xml:"media,attr"`Gmlstring`xml:"gml,attr"`Taxostring`xml:"taxo,attr"`Georssstring`xml:"georss,attr"`Contentstring`xml:"content,attr"`Geostring`xml:"geo,attr"`Versionstring`xml:"version,attr"`Channelstruct {
Textstring`xml:",chardata"`Titlestring`xml:"title"`// ESS New Releases (Display...Linkstring`xml:"link"`// http://tinyurl.com/ESSNew...Descriptionstring`xml:"description"`// New releases from the Ear...LastBuildDatestring`xml:"lastBuildDate"`// Mon, 27 Nov 2017 00:06:35...Item []struct {
Textstring`xml:",chardata"`Titlestring`xml:"title"`// Surficial geology, Aberde...Linkstring`xml:"link"`// https://geoscan.nrcan.gc....Descriptionstring`xml:"description"`// Geological Survey of Cana...Guidstruct {
Textstring`xml:",chardata"`// 304279, 306212, 306175, 3...IsPermaLinkstring`xml:"isPermaLink,attr"`
} `xml:"guid"`PubDatestring`xml:"pubDate"`// Fri, 24 Nov 2017 00:00:00...Polygon []string`xml:"polygon"`// 64.0000 -98.0000 64.0000 ...Downloadstring`xml:"download"`// https://geoscan.nrcan.gc....Licensestring`xml:"license"`// http://data.gc.ca/eng/ope...Authorstring`xml:"author"`// Geological Survey of Cana...Sourcestring`xml:"source"`// Geological Survey of Cana...SndSeriesstring`xml:"SndSeries"`// Bedford Institute of Ocea...Publisherstring`xml:"publisher"`// Natural Resources Canada,...Editionstring`xml:"edition"`// prelim., surficial data m...Meetingstring`xml:"meeting"`// Geological Association of...Documenttypestring`xml:"documenttype"`// serial, open file, serial...Languagestring`xml:"language"`// English, English, English...Mapsstring`xml:"maps"`// 1 map, 5 maps, Publicatio...Mapinfostring`xml:"mapinfo"`// surficial geology, surfic...Mediumstring`xml:"medium"`// on-line; digital, digital...Provincestring`xml:"province"`// Nunavut, Northwest Territ...Ntsstring`xml:"nts"`// 066B, 095J; 095N; 095O; 0...Areastring`xml:"area"`// Aberdeen Lake, Mackenzie ...Subjectsstring`xml:"subjects"`Programstring`xml:"program"`// GEM2: Geo-mapping for Ene...Projectstring`xml:"project"`// Rae Province Project Mana...Projectnumberstring`xml:"projectnumber"`// 340521, 343202, 340557, 3...Abstractstring`xml:"abstract"`// This new surficial geolog...Linksstring`xml:"links"`// Online - En ligne (PDF, 9...Readmestring`xml:"readme"`// readme | https://geoscan....PPIidstring`xml:"PPIid"`// 34532, 35096, 35438, 2563...
} `xml:"item"`
} `xml:"channel"`
}

Online

About

builds.sr.ht statusProject Status: Active – The project has reached a stable, usable state and is being actively developed.

Upsides:

  • it works fine for non-recursive structures,
  • does not need XSD or DTD,
  • it is relatively convenient to access attributes, children and text,
  • will generate a single struct, which make for a quite compact representation,
  • simple user interface,
  • comments with examples,
  • schema inference across multiple files.

Downsides:

  • experimental, early, buggy, unstable prototype,
  • no support for recursive types (similar to Russian Doll strategy, [1])
  • no type inference, everything is accessible as string (without a schema, type inference may fail if the type guess is wrong)

Bugs:

Mapping between XML elements and data structures is inherently flawed: an XML element is an order-dependent collection of anonymous values, while a data structure is an order-independent collection of named values.

https://golang.org/pkg/encoding/xml/#pkg-note-BUG

Related projects:

And other awesome XML utilities.

Presentations:

Install

$ go install github.com/miku/zek/cmd/zek@latest

Debian and RPM packages:

It's in AUR, too.

Usage

$ zek -h
Usage of zek:
-B use a fixed banner string (e.g. for CI)
-C emit less compact struct
-F skip formatting
-P string
if set, write out struct within a package with the given name
-S int
read at most this many tags, approximately (0=unlimited)
-c emit more compact struct (noop, as this is the default since 0.1.7)
-d debug output
-e add comments with example
-j add JSON tags
-m omit empty Text fields
-max-examples int
limit number of examples (default 10)
-n string
use a different name for the top-level struct
-o string
if set, write to output file, not stdout
-p write out an example program
-s strict parsing and writing
-t string
emit struct for tag matching this name
-u filter out duplicated examples
-version
show version
-x int
max chars for example (default 25)

Examples:

$ cat fixtures/a.xml
<a></a>
$ zek -C < fixtures/a.xml
type A struct {
XMLName xml.Name `xml:"a"`
Text string `xml:",chardata"`
}

Debug output dumps the internal tree as JSON to stdout.

$ zek -d < fixtures/a.xml
{"name":{"Space":"","Local":"a"}}

Example program:

package main
import (
"encoding/json""encoding/xml""fmt""log""os"
)
// A was generated 2017-12-05 17:35:21 by tir on apollo.typeAstruct {
XMLName xml.Name`xml:"a"`Textstring`xml:",chardata"`
}
funcmain() {
dec:=xml.NewDecoder(os.Stdin)
vardocAiferr:=dec.Decode(&doc); err!=nil {
log.Fatal(err)
}
b, err:=json.Marshal(doc)
iferr!=nil {
log.Fatal(err)
}
fmt.Println(string(b))
}
$ zek-C-p<fixtures/a.xml>sample.go&&gorunsample.go<fixtures/a.xml|jq . &&rm sample.go
{
"XMLName": {
"Space": "",
"Local": "a"
},
"Text": ""
}

More complex example:

$ zek<fixtures/d.xml// Root was generated 2019-06-11 16:27:04 by tir on hayiti.typeRootstruct {
XMLName xml.Name`xml:"root"`Textstring`xml:",chardata"`A []struct {
Textstring`xml:",chardata"`B []struct {
Textstring`xml:",chardata"`Cstring`xml:"c"`Dstring`xml:"d"`
} `xml:"b"`
} `xml:"a"`
}
$ zek-p<fixtures/d.xml>sample.go&&gorunsample.go<fixtures/d.xml|jq . &&rm sample.go
{
"XMLName": {
"Space": "",
"Local": "root"
},
"Text": "\n\n\n\n",
"A": [
{
"Text": "\n\n\n",
"B": [
{
"Text": "\n\n ",
"C": "Hi",
"D": ""
},
{
"Text": "\n\n\n ",
"C": "World",
"D": ""
}
]
},
{
"Text": "\n\n",
"B": [
{
"Text": "\n\n ",
"C": "Hello",
"D": ""
}
]
},
{
"Text": "\n\n",
"B": [
{
"Text": "\n\n ",
"C": "",
"D": "World"
}
]
}
]
}

Annotate with comments:

$ zek-e<fixtures/l.xml// Records was generated 2019-06-11 16:29:35 by tir on hayiti.typeRecordsstruct {
XMLName xml.Name`xml:"Records"`Textstring`xml:",chardata"`// \nXsistring`xml:"xsi,attr"`Record []struct {
Textstring`xml:",chardata"`Headerstruct {
Textstring`xml:",chardata"`Statusstring`xml:"status,attr"`Identifierstring`xml:"identifier"`// oai:ojs.localhost:article...Datestampstring`xml:"datestamp"`// 2009-06-24T14:48:23Z, 200...SetSpecstring`xml:"setSpec"`// eppp:ART, eppp:ART, eppp:...
} `xml:"header"`Metadatastruct {
Textstring`xml:",chardata"`Rfc1807struct {
Textstring`xml:",chardata"`Xmlnsstring`xml:"xmlns,attr"`Xsistring`xml:"xsi,attr"`SchemaLocationstring`xml:"schemaLocation,attr"`BibVersionstring`xml:"bib-version"`// v2, v2, v2...IDstring`xml:"id"`// http://jou...Entrystring`xml:"entry"`// 2009-06-24...Organization []string`xml:"organization"`// Proceeding...Titlestring`xml:"title"`// Introducti...Typestring`xml:"type"`Author []string`xml:"author"`// KRAMPEN, G..Copyrightstring`xml:"copyright"`// Das Urhebe...OtherAccessstring`xml:"other_access"`// url:http:/...Keywordstring`xml:"keyword"`Period []string`xml:"period"`Monitoringstring`xml:"monitoring"`Languagestring`xml:"language"`// en, en, en, e...Abstractstring`xml:"abstract"`// After a short...Datestring`xml:"date"`// 2009-06-22 12...
} `xml:"rfc1807"`
} `xml:"metadata"`Aboutstring`xml:"about"`
} `xml:"Record"`
}

Only consider a nested element

$ zek-tmetadatafixtures/z.xml// Metadata was generated 2019-06-11 16:33:26 by tir on hayiti.typeMetadatastruct {
XMLName xml.Name`xml:"metadata"`Textstring`xml:",chardata"`Dcstruct {
Textstring`xml:",chardata"`Xmlnsstring`xml:"xmlns,attr"`Titlestruct {
Textstring`xml:",chardata"`Xmlnsstring`xml:"xmlns,attr"`
} `xml:"title"`Identifierstruct {
Textstring`xml:",chardata"`Xmlnsstring`xml:"xmlns,attr"`
} `xml:"identifier"`Rightsstruct {
Textstring`xml:",chardata"`Xmlnsstring`xml:"xmlns,attr"`Langstring`xml:"lang,attr"`
} `xml:"rights"`AccessRightsstruct {
Textstring`xml:",chardata"`Xmlnsstring`xml:"xmlns,attr"`
} `xml:"accessRights"`
} `xml:"dc"`
}

Inference across files

$ zekfixtures/a.xmlfixtures/b.xmlfixtures/c.xml// A was generated 2017-12-05 17:40:14 by tir on apollo.typeAstruct {
XMLName xml.Name`xml:"a"`Textstring`xml:",chardata"`B []struct {
Textstring`xml:",chardata"`
} `xml:"b"`
}

This is also useful, if you deal with archives containing XML files:

$ unzip -p 4082359.zip '*.xml'| zek -e

Given a directory full of zip files, you can combined find, unzip and zek:

$ foriin$(find ftp/b571 -type f -name "*zip");do unzip -p $i'*xml';done| zek -e

Another example (tarball with thousands of XML files, seemingly MARC):

$ tar -xOzf /tmp/20180725.125255.tar.gz | zek -e
// OAIPMH was generated 2018-09-26 15:03:29 by tir on sol.
type OAIPMH struct {
XMLName xml.Name `xml:"OAI-PMH"`
Text string `xml:",chardata"`
Xmlns string `xml:"xmlns,attr"`
Xsi string `xml:"xsi,attr"`
SchemaLocation string `xml:"schemaLocation,attr"`
ListRecords struct {
Text string `xml:",chardata"`
Record struct {
Text string `xml:",chardata"`
Header struct {
Text string `xml:",chardata"`
Identifier struct {
Text string `xml:",chardata"` // aleph-pub:000000001, ...
} `xml:"identifier"`
} `xml:"header"`
Metadata struct {
Text string `xml:",chardata"`
Record struct {
Text string `xml:",chardata"`
Xmlns string `xml:"xmlns,attr"`
Xsi string `xml:"xsi,attr"`
SchemaLocation string `xml:"schemaLocation,attr"`
Leader struct
Text string `xml:",chardata"` // 00001nM2.01200024
} `xml:"leader"`
Controlfield []struct {
Text string `xml:",chardata"` // 00001nM2.01200024
Tag string `xml:"tag,attr"`
} `xml:"controlfield"`
Datafield []struct {
Text string `xml:",chardata"`
Tag string `xml:"tag,attr"`
Ind1 string `xml:"ind1,attr"`
Ind2 string `xml:"ind2,attr"`
Subfield []struct {
Text string `xml:",chardata"` // KM0000002
Code string `xml:"code,attr"`
} `xml:"subfield"`
} `xml:"datafield"`
} `xml:"record"`
} `xml:"metadata"`
} `xml:"record"`
} `xml:"ListRecords"`
}

Generate a package

If you want in include generated file in the build process, e.g. with go generate, you may find -P and -o helpful.

$ cat fixtures/b.xml
<a><b></b></a>

Run on the command line or via go generate:

$ zek -P mypkg -o data.go < fixtures/b.xml

This would write out the following in data.go file:

// Code generated by zek; DO NOT EDIT.package mypkg
import"encoding/xml"// A was generated 2021-09-16 11:23:06 by tir on trieste.typeAstruct {
XMLName xml.Name`xml:"a"`Textstring`xml:",chardata"`Bstring`xml:"b"`
}

Note that any existing file will be overwritten, without any warning.

Misc

As a side effect, zek seems to be a useful for debugging. Example:

This record is emitted from a typical OAI server (OJS, not even uncommon), yet one can quickly spot the flaw in the structure.

Over 30 different struct generated manually in the course of a few hours (around five minutes per source): https://git.io/vbTDo.

-- Current extent leader: 1532 lines struct

About

Generate a Go struct from XML.

Resources

Readme
License
Activity
Custom properties

Stars

0 stars

Watchers

0 watching

Forks

0 forks
Report repository

Releases

Packages

Contributors

Languages