⚠️ Alpha Software - Work in Progress

This project is in early development and rapidly evolving. Expect breaking changes, rough edges, and incomplete documentation.

Help Wanted! If you find this useful, please consider contributing:

• Report bugs and issues you encounter
• Suggest improvements or new features
• Submit pull requests for fixes or enhancements
• Share your configuration patterns and workflows
• Help improve documentation and examples

Your feedback and contributions will help make this tool better for the entire Clojure community!

Overview

xitdb-clj is a embedded database for efficiently storing and retrieving immutable, persistent data structures. The library provides atom-like semantics for working with the database from Clojure.

It is a Clojure interface for xitdb-java, itself a port of xitdb, written in Zig.

Main characteristics

• Embeddable, tiny library.
• Supports writing to a file as well as purely in-memory use.
• Each transaction (done via swap!) efficiently creates a new "copy" of the database, and past copies can still be read from.
• Reading/Writing to the database is efficient, only the necessary nodes are read or written.
• Thread safe. Multiple readers, one writer.
• Append-only. The data you are writing is invisible to any reader until the very last step, when the top-level history header is updated.
• All heavy lifting done by the bare-to-the-jvm java library.
• Database files can be used from other languages, via xitdb Java library or the xitdb Zig library

Quick Start

Add the dependency to your project, start a REPL.

You already know how to use it!

For the programmer, a xitdb database is like a Clojure atom. reset! or swap! to reset or update, deref or @ to read.

(require '[xitdb.db :as xdb])
(defdb (xdb/xit-db"my-app.db"))
;; Use it like an atom
(reset! db {:users {"alice" {:name"Alice":age30}
"bob" {:name"Bob":age25}}})
;; Read the entire database
(xdb/materialize @db)
;; => {:users {"alice" {:name "Alice", :age 30}, "bob" {:name "Bob", :age 25}}}
(get-in @db [:users"alice":age])
;; => 30
(swap! db assoc-in [:users"alice":age] 31)
(get-in @db [:users"alice":age])
;; => 31

Data structures are read lazily from the database

Reading from the database returns wrappers around cursors in the database file:

(type @db) ;; => xitdb.hash_map.XITDBHashMap

The returned value is a XITDBHashMap which is a wrapper around the xitdb-java's ReadHashMap, which itself has a cursor to the tree node in the database file. These wrappers implement the protocols for Clojure collections - vectors, lists, maps and sets, so they behave exactly like the Clojure native data structures. Any read operation on these types is going to return new XITDB types:

(type (get-in @db [:users"alice"])) ;; => xitdb.hash_map.XITDBHashMap

So it will not read the entire nested structure into memory, but return a 'cursor' type, which you can operate upon using Clojure functions.

Use materialize to convert a nested XITDB data structure to a native Clojure data structure:

(xdb/materialize (get-in @db [:users"alice"])) ;; => {:name "Alice" :age 31}

No query language

Use filter, group-by, reduce, etc. If you want a query engine, datascript works out of the box, you can store the datoms as a vector in the db.

Here's a taste of how your queries could look like:

(defntitles-of-songs-for-artist
[db artist]
(->> (get-in db [:songs-indices:artist artist])
(map #(get-in db [:songs % :title]))))
(defnwhat-is-the-most-viewed-song? [db tag]
(let [views (->> (get-in db [:songs-indices:tag tag])
(map (:songs db))
(map (juxt:id:views))
(sort-by #(parse-long (second %))))]
(get-in db [:songs (first (last views))])))

Sorted collections

In addition to (unordered) hash maps and sets, xitdb supports on-disk sorted maps and sets, backed by the engine's rank-augmented B-tree. Store a Clojure sorted-map / sorted-set and it is persisted as a sorted collection that keeps its keys/members ordered on disk:

(reset! db (sorted-map"banana"2"apple"1"cherry"3))
@db
;; => #XITDBSortedMap{"apple" 1, "banana" 2, "cherry" 3}
(swap! db assoc "date"4) ;; inserted in order, not appended

Reading back yields an XITDBSortedMap / XITDBSortedSet that implements Clojure's ordered interfaces, so seq, rseq, nth, subseq and rsubseq all work and read only what they touch from disk:

(reset! db (into (sorted-map) (map vector (range01002) (range))))
(nth @db 10) ;; => [20 10] ;; O(log n), no full scan
(subseq @db >= 90) ;; => ([90 45] [92 46] [94 47] [96 48] [98 49])
(rseq @db) ;; => lazy descending seq of entries

Supported key/member types are strings, keywords, booleans, chars, longs, doubles, UUID, Instant and Date. They are stored with an order-preserving codec, so they iterate in natural order — numeric for numbers, chronological for temporals, lexicographic (by code point) for strings, UUID.compareTo order for UUIDs. Longs and doubles share a single numeric ordering, so they interleave by value (e.g. 1 < 1.5 < 2). Only the default ordering is supported: sorted-map-by / sorted-set-by with a custom comparator is rejected, as are java.util.Date subclasses like java.sql.Timestamp (use Instant instead).

Ranking & pagination

The xitdb.sorted namespace exposes the B-tree's O(log n) superpowers, which are handy for building and paging on-disk secondary indexes:

• (rank coll k) — number of entries strictly less than k (i.e. the index of k, or its would-be insertion index if absent).
• (from-index coll n) — lazy ordered seq starting at rank n.
• (page coll offset limit) — lazy ordered page [offset, offset+limit).

(require '[xitdb.sorted :as xsorted])
;; build a timestamp -> id index; events can arrive out of order
(reset! db (sorted-map))
(doseq [e events]
(swap! db assoc (:ts e) (:id e)))
(xsorted/rank @db some-ts) ;; chronological position of some-ts
(xsorted/page @db 10020) ;; the 20 entries at ranks [100, 120)

History

Since the database is immutable, all previous values are accessed by reading from the respective history index. The root data structure of a xitdb database is a ArrayList, called 'history'. Each transaction adds a new entry into this array, which points to the latest value of the database (usually a map).

(xdb/deref-at db -1) ;; the most recent value, same as @db
(xdb/deref-at db -2) ;; the second most recent value
(xdb/deref-at db 0) ;; the earliest value
(xdb/deref-at db 1) ;; the second value

You can get the latest history index from the count of the database:

(defhistory-index (dec (count db)))

After making further transactions, you can revert back to it simply like this:

(reset! db (xdb/deref-at db history-index))

It is also possible to create a transaction which returns the previous and current values of the database, by setting the *return-history?* binding to true.

;; Work with history tracking
(binding [xdb/*return-history?* true]
(let [[history-index old-value new-value] (swap! db assoc :new-key"value")]
(println"old value:" old-value)
(println"new value:" new-value)))

Freezing

One important distinction from the Clojure atom is that inside a transaction (eg. a swap!), the data is temporarily mutable. This is exactly like Clojure's transients, and it is a very important optimization. However, this can lead to a surprising behavior:

(swap! db (fn [moment]
(let [moment (assoc moment :fruits ["apple""pear""grape"])
moment (assoc moment :food (:fruits moment))
moment (update moment :food conj "eggs""rice""fish")]
moment)))
;; =>
{:fruits ["apple""pear""grape""eggs""rice""fish"]
:food ["apple""pear""grape""eggs""rice""fish"]}
;; the fruits vector was mutated!

If you want to prevent data from being mutated within a transaction, you must freeze! it:

(swap! db (fn [moment]
(let [moment (assoc moment :fruits ["apple""pear""grape"])
moment (assoc moment :food (xdb/freeze! (:fruits moment)))
moment (update moment :food conj "eggs""rice""fish")]
moment)))
;; =>
{:fruits ["apple""pear""grape"]
:food ["apple""pear""grape""eggs""rice""fish"]}

Note that this is not doing an expensive copy of the fruits vector. We are benefitting from structural sharing, just like in-memory Clojure data. The reason we have to freeze! is because the default is different than Clojure; in Clojure, you must opt-in to temporary mutability by using transients, whereas in xitdb you must opt-out of it.

Architecture

xitdb-clj builds on xitdb-java which implements:

• Hash Array Mapped Trie (HAMT) - For hash map/set and vector
• B-trees - For sorted map/set and linked list
• Structural Sharing - Minimizes memory usage across versions
• Copy-on-Write - Ensures immutability while maintaining performance

The Clojure wrapper adds:

• Idiomatic Clojure interfaces (IAtom, IDeref)
• Automatic type conversion between Clojure and Java types
• Thread-local read connections for scalability
• Integration with Clojure's sequence abstractions

Supported Data Types

• Maps - Hash maps with efficient key-value access
• Sorted maps - On-disk B-tree maps with ordered iteration, subseq/nth/rank
• Vectors - Array lists with indexed access
• Sets - Hash sets with unique element storage
• Sorted sets - On-disk B-tree sets with ordered iteration and ranking
• Lists - B-tree-based linked array lists
• Primitives - Numbers, strings, keywords, booleans, dates.

Performance Characteristics

• Read Operations: O(log16 n) for maps and vectors due to trie structure
• Write Operations: O(log16 n) with structural sharing for efficiency
• Memory Usage: Minimal overhead with automatic deduplication of identical subtrees
• Concurrency: Thread-safe with optimized read-write locks

License

MIT