Generate e-reader compatible dictionaries for your favourite fictional worlds.
Builder is the underlying technology at the heart of Runik, containing the tooling required to parse wikis and generate dictionaries.
Runik builder uses the crowd-sourced info of fan wikis to generate dictionaries with custom definitions for fictional names, places, and items. Builder uses non-proprietary formats to store definitions like json and dictfile, allowing for interop between other tools.
Builder can be used on its own as a CLI tool or as a module import.
git clone git@github.com:Runik-3/builder.git
cd builder
go mod tidy
go run . info [wiki_url]The quickest and easiest way to get started using Runik Builder is via the command line.
Validate a wiki url and retrieve metadata describing the wiki. Takes a wiki url as its argument.
builder info https://red-rising.fandom.com/api.phpThe following will print to the standard output:
Wiki title: Red Rising Wiki
Language: en
Size: 753 entries
Other supported languages:
- Spanish: https://amanecer-rojo.fandom.com/es/wiki/
- Hungarian: https://voros-lazadas.fandom.com/hu/wiki/
- Turkish: http://kizil-yukselis.fandom.com/tr/wiki/
Generates an e-reader dictionary based on the pages of a mediawiki-compatible wiki. The generate command requires a wiki url and takes a series of optional flags.
A mediawiki URL used as the target to parse and build the dictionary (eg. https://stardust.fandom.com/api.php).
builder generate https://stardust.fandom.com/api.php| Flag | Name | Description |
|---|---|---|
-n | name | The file name of the generated dictionary file (extension added automatically). If no name is passed in, the file name will default to the subdomain of the target wiki (eg. red-rising.fandom.com becomes red-rising.json). |
-o | Out directory | The directory where the generated dictionary will be written. If no directory is specified, a file is not written to disk. |
-f | Format | The file format the dictionary is written in. Builder currently supports writing to json and dictfile ('df'). When no format is specified, json is the default. |
-d | Depth | The number of sentences that make up each definition. Builder starts parsing the definition at the first setence of the article's main content, ignoring tables and other formatting. As such, be wary that a greater depth has a higher probability of including spoilers. Values between 1 and 5 are suggested, the default is 1. |
-l | Limit | The maximum number of word entries written to a dictionary. Useful for testing. If no limit is specified, the default is 10,000. |
builder -o ./ -f df generate https://stardust.fandom.com/api.php
Running this command generates a dictionary from the entire Stardust Fandom Wiki and write its contents to the current directory as stardust.df.
...
@ Wall Guard
The Wall Guard was a 97-year-old man who guarded the gap in the wall which was the border between Stormhold and England.
@ Ingrid
Ingrid is the star that fell 400 years before Yvaine did, more precisely at some point (presumably) in the 15th century.
...
A snippet of the resulting dictfile.
Use the cpuprofile flag to generate an output file containing pprof profiling data.
builder -o ./ -f df -cpuprofile cpu.prof generate https://stardust.fandom.com/api.php
View in pprof.
go tool pprof cpu.prof
# Or web view
go tool pprof -http=:8080 cpu.profBuilder can be imported as a module to use in your own projects. Run the following command in the root of your project to add builder.
go get github.com/runik-3/builderThe GetWikiDetails function is exported from /wikiBot. It can act as a tool to validate wiki urls or simply fetch useful metadata about a wiki before it's generated.
import (
"fmt"
wikibot "github.com/runik-3/builder/wikiBot"
)
details, err:=wikibot.GetWikiDetails("htts://stardust.fandom.com")
fmt.Printf("%+v", details)
/*{ ApiUrl:https://kingkiller.fandom.com/api.php SiteName:Kingkiller Chronicle Wiki MainPage:Kingkiller Wiki Lang:en Logo:https://images.wikia.com/nameofthewind/images/b/bc/Wiki.png Pages:11700 Articles:466 Languages: [ { Lang:de LangName:German Autonym:Deutsch Url:https://knigsmrder.fandom.com/de/wiki/ } ... ] RequestOpts: { ForceTLS12:false }}*/The BuildDictionary funciton is exported from /dict. BuildDictionary takes in a wiki url and some options and generates a dictionary based on the content of the wiki.
import (
d "github.com/runik-3/builder/dict"
)
varwikiUrlstring="https://kingkiller.fandom.com"_, err:=d.BuildDictionary(wikiUrl, d.GeneratorOptions{
Name: "kingkiller",
Output: outputDir,
Format: "json",
EntryLimit: 10000,
Depth: 1,
})
/*<outputDir>/kingkiller.json:----------------------------{ "Name":"kingkiller", "ApiUrl":"https://kingkiller.fandom.com/api.php", "Lang":"en", "Lexicon":[ { "Word":"Arwyl", "Definition":"Master Arwyl is the Master Physicker of the University, and presides over the instruction and day-to-day operations of the Medica." }, ... ]}*/| Option | Description |
|---|---|
| Name | The name of the generated dictionary. |
| Output | The directory the dictionary will be written to. If none is defined, no file will be written. |
| Format | The file format the dictionary is written in -- supports JSON and dictfile. |
| Depth | The number of sentences that make up each definition. |
| EntryLimit | The maximum number of word entries written to a dictionary. |
| ProgressHook | A function that runs after every batch is processed -- can be used to report progress or perform some other action that hooks into generation. |