# ol.xitstash next Binary content and file catalogs in xitdb ## ol.xitstash # ol.xitstash Stores immutable payloads and a file catalog in an application-selected xitdb subtree. ## prepare ```clojure (prepare source) (prepare source options) ``` Chunks and hashes bytes, a File/Path, or an InputStream before insertion. Returns closeable prepared content with `:digest` and `:size`. Byte arrays are copied. Caller-supplied streams are not closed. Uncompressed files are scanned without a temporary copy and must match their prepared chunk hashes when consumed. Closing prepared content releases its temporary storage, not the source file. Prepared content can be consumed repeatedly before close. Options: | | | | | --- | --- | --- | | Key | Default | Meaning | | `:algorithm` | `:sha2-256` | `:sha2-256` or `:sha2-512` | | `:min-size` | 262144 | Minimum FastCDC chunk size | | `:average-size` | 1048576 | Power-of-two target, 256 through 4194304 | | `:max-chunk-size` | 4194304 | Maximum chunk size, at most 16777216 | | `:compression` | `:none` | `:none` or per-chunk `:zlib`, only if smaller | | `:memory-threshold` | 1048576 | Spool bytes before spilling, at most 16777216 | | `:max-size` | 1099511627776 | Maximum original payload bytes | | `:max-chunks` | 65536 | Maximum chunk count | | `:max-metadata-bytes` | 16777216 | Sum of EDN-encoded chunk-record lengths | | `:max-temp-bytes` | 1099511627776 | Maximum simultaneous staged payload bytes | | `:temp-dir` | JVM temporary directory | Existing File/Path or path string | Chunk sizes must be positive and ordered. Resource limits are nonnegative signed 64-bit integers. Spool buffers and chunk buffers are bounded, but total metadata memory grows with chunk count. The metadata limit measures encoded records, not JVM heap usage. Compression may temporarily require both an input spool and an encoded spool within `:max-temp-bytes`. Invalid options and exhausted limits throw typed `ex-info`. Native I/O errors propagate. Use [`cleanup-staging!`](#cleanup-staging!) for abandoned temporary storage. [source,window=_blank](https://github.com/outskirtslabs/xitstash/blob/main/src/clj/ol/xitstash.clj#L14-L48) --- ## upload ```clojure (upload) (upload options) ``` Creates a closeable OutputStream for content preparation outside a transaction. Operations on one builder are serialized. Closing an unfinished builder discards its staging data. [`finish`](#finish) seals the builder and returns separate closeable prepared content. Closing the builder after finish does not close that result. Writes and repeated finish calls after sealing throw typed `ex-info`. Prefer bulk writes over individual bytes. Options: | | | | --- | --- | | Key | Meaning | | All [`prepare`](#prepare) options | Same defaults and resource limits | Chunking and hashing run during [`finish`](#finish), before a write transaction. [source,window=_blank](https://github.com/outskirtslabs/xitstash/blob/main/src/clj/ol/xitstash.clj#L50-L67) --- ## finish ```clojure (finish builder) ``` Seals `builder` and returns prepared content without publishing it. Call once on a builder from [`upload`](#upload). Hashing and chunking complete before return. A failed finish discards staging data. Close the returned prepared content separately from the builder. [source,window=_blank](https://github.com/outskirtslabs/xitstash/blob/main/src/clj/ol/xitstash.clj#L69-L76) --- ## cleanup-staging! ```clojure (cleanup-staging!) (cleanup-staging! options) ``` Deletes inactive, marked staging directories and returns their paths. File locks protect active preparations, including those in other processes. A directory with an invalid marker or unexpected files is not removed. No background cleanup starts automatically. Prefer an application-specific temporary directory. Unmarked directories from an interrupted initial directory creation are not removed. Options: | | | | | --- | --- | --- | | Key | Default | Meaning | | `:temp-dir` | JVM temporary directory | Existing directory to inspect | | `:min-age-ms` | 86400000 | Minimum directory age, nonnegative milliseconds | Unknown options and invalid ages throw typed `ex-info`. Native I/O errors propagate. Cleanup cannot remove published database content. [source,window=_blank](https://github.com/outskirtslabs/xitstash/blob/main/src/clj/ol/xitstash.clj#L78-L97) --- ## initialize! ```clojure (initialize! cursor) ``` Initializes an empty subtree or validates an existing xitstash store. `cursor` must be a native write cursor inside an application transaction. Returns `{:format string :initialized? boolean}`. An empty plain map is accepted. Unrelated nonempty data and unsupported formats are rejected without mutation. Repeated initialization does not write data. [source,window=_blank](https://github.com/outskirtslabs/xitstash/blob/main/src/clj/ol/xitstash.clj#L99-L107) --- ## put! ```clojure (put! cursor prepared) ``` Publishes prepared content inside the application’s write transaction. `cursor` selects an initialized store. `prepared` comes from [`prepare`](#prepare) or [`finish`](#finish) and must be open. Returns `:digest`, `:size`, and `:deduplicated?`. Existing payloads are not rewritten. Prepared bytes are checked again, including on duplicate insertion. Shared chunks are stored once. Results are provisional until the application commits. Let any exception escape the transaction callback to abort. Close prepared content separately. [source,window=_blank](https://github.com/outskirtslabs/xitstash/blob/main/src/clj/ol/xitstash.clj#L109-L120) --- ## payload ```clojure (payload cursor digest) ``` Returns realized `{:digest id :size bytes}` metadata without reading content. `cursor` selects a store. Missing content throws `:xitstash/not-found`. [source,window=_blank](https://github.com/outskirtslabs/xitstash/blob/main/src/clj/ol/xitstash.clj#L122-L127) --- ## manifest ```clojure (manifest cursor digest) (manifest cursor digest options) ``` Returns realized payload metadata with ordered `:chunks` digest/size maps. Validates native references and lengths without decoding chunk bytes. Missing content, corrupt metadata, and exceeded limits throw typed `ex-info`. The metadata limit counts encoded records, not JVM heap usage. | | | | | --- | --- | --- | | Option | Default | Meaning | | `:max-chunks` | 65536 | Maximum manifest entries | | `:max-metadata-bytes` | 16777216 | Sum of EDN-encoded chunk-record lengths | [source,window=_blank](https://github.com/outskirtslabs/xitstash/blob/main/src/clj/ol/xitstash.clj#L129-L141) --- ## open ```clojure (open cursor digest) (open cursor digest options) ``` Opens a closeable InputStream over a payload or byte range. Each decoded chunk is checked before its bytes are exposed. One decoded chunk is cached, plus checked metadata and temporary decoding buffers. Closing the stream does not close the database. Close streams before their supplied cursors become invalid, or before database compaction/replacement. Library native reads use the database monitor. Coordinate other native access, including cursor acquisition and transactions, with that monitor or prevent overlap through application-level synchronization. | | | | | --- | --- | --- | | Option | Default | Meaning | | `:start` | 0 | Inclusive payload byte offset | | `:end` | Payload size | Exclusive payload byte offset | | `:max-chunks` | 65536 | Maximum manifest entries | | `:max-metadata-bytes` | 16777216 | Sum of EDN-encoded chunk-record lengths | Requires `0 <= start <= end <= size`. Missing content, corrupt chunks, invalid ranges, and exceeded limits throw typed `ex-info`. Native I/O errors propagate. Use [`seek!`](#seek!) to move within the selected range. [source,window=_blank](https://github.com/outskirtslabs/xitstash/blob/main/src/clj/ol/xitstash.clj#L143-L166) --- ## seek! ```clojure (seek! stream offset origin) ``` Moves a stream from [`open`](#open) and returns its range-relative byte position. `origin` is `:start`/0, `:current`/1, or `:end`/2. `offset` is a signed 64-bit integer. Negative results and overflow throw typed `ex-info` without changing position. Positions after the range end are allowed and read EOF. Seeking a closed stream throws `:xitstash/closed`. [source,window=_blank](https://github.com/outskirtslabs/xitstash/blob/main/src/clj/ol/xitstash.clj#L168-L176) --- ## create! ```clojure (create! cursor requests) ``` Creates 1–4096 catalog nodes inside the application’s write transaction. Returns a vector of records in request order. Each record has `:id`, `:parent`, `:name`, `:kind` (`:directory` or `:file`), and `:revision`. Files also have `:digest` and `:size`. All records in one batch use the same revision token. | | | | --- | --- | | Request key | Meaning | | `:name` | Required name, 1–4096 valid UTF-8 bytes, no slash/NUL or dot/double dot | | `:parent` | Existing directory ID, default root `0` | | `:under` | One-based position of an earlier directory request, instead of `:parent` | | `:digest` | Published payload ID for a file, absent for a directory | | `:id` | Optional positive signed 64-bit ID, otherwise allocated automatically | Requests are processed in order. Explicit IDs advance the automatic counter. Existing IDs and sibling names conflict. Automatic IDs are not recycled, but independent branches can allocate the same ID. Applications must handle that when merging. Unknown request keys are invalid. Results are provisional until commit. Let any exception escape the transaction callback to abort all changes. Files can reference payloads published earlier in the same transaction with [`put!`](#put!). [source,window=_blank](https://github.com/outskirtslabs/xitstash/blob/main/src/clj/ol/xitstash.clj#L178-L202) --- ## node ```clojure (node cursor id) ``` Returns a realized catalog record for a positive `id`. Root `0` has no record. Missing nodes throw `:xitstash/not-found`. See [`create!`](#create!) for record fields. [source,window=_blank](https://github.com/outskirtslabs/xitstash/blob/main/src/clj/ol/xitstash.clj#L204-L210) --- ## lookup ```clojure (lookup cursor parent name) ``` Returns the child record at `parent` and `name`. `parent` is a directory ID or root `0`. Invalid names and non-directory parents throw `:xitstash/invalid-argument`. Missing parents or children throw `:xitstash/not-found`. [source,window=_blank](https://github.com/outskirtslabs/xitstash/blob/main/src/clj/ol/xitstash.clj#L212-L219) --- ## children ```clojure (children cursor parent) ``` Returns a realized vector of child records in UTF-8 byte name order. `parent` is a directory ID or root `0`. Empty directories return `[]`. Missing parents throw `:xitstash/not-found`, files are invalid parents. Result memory grows with the directory’s child count. [source,window=_blank](https://github.com/outskirtslabs/xitstash/blob/main/src/clj/ol/xitstash.clj#L221-L228) --- ## path ```clojure (path cursor id) ``` Returns a realized vector of names from root to `id`. Root `0` returns `[]`. Missing nodes or ancestors throw `:xitstash/not-found`. Corrupt parent chains throw `:xitstash/corrupt-data`. [source,window=_blank](https://github.com/outskirtslabs/xitstash/blob/main/src/clj/ol/xitstash.clj#L230-L236) --- ## files ```clojure (files cursor digest) ``` Returns a realized vector of ascending file IDs that reference `digest`. No references return `[]`, including for an unpublished valid payload ID. Invalid identifiers throw typed `ex-info`. Result memory grows with the number of referencing files. [source,window=_blank](https://github.com/outskirtslabs/xitstash/blob/main/src/clj/ol/xitstash.clj#L238-L245) --- ## update! ```clojure (update! cursor id digest expected-revision) ``` Changes a file’s payload inside the application’s write transaction. `digest` must identify a published payload. Returns the updated record. Directories and root `0` cannot be updated. An unchanged digest performs no writes and preserves the record’s revision. `expected-revision` is a nonnegative signed 64-bit integer. Zero skips the check. A stale positive value throws `:xitstash/conflict`, even for a no-op. Negative values are invalid. Changed records get a new per-store revision. Results are provisional, as described by [`create!`](#create!). [source,window=_blank](https://github.com/outskirtslabs/xitstash/blob/main/src/clj/ol/xitstash.clj#L247-L259) --- ## move! ```clojure (move! cursor id destination expected-revision) ``` Renames or reparents a catalog node inside the application’s transaction. Returns the source record with its unchanged ID and new revision. Moving to its current parent/name performs no writes and preserves its revision. Root `0` cannot move. A directory cannot move into itself or its descendants. Revision checks and provisional results follow [`update!`](#update!). | | | | | --- | --- | --- | | Destination key | Default | Meaning | | `:parent` | Current parent | Directory ID or root `0` | | `:name` | Current name | New name, subject to [`create!`](#create!) name rules | | `:replace?` | `false` | Remove an occupied destination of the same kind | Replacement directories must be empty. Replacement removes the destination record, not its payload. Without replacement, occupied destinations conflict. Unknown keys are invalid. Let any exception escape to abort the transaction. [source,window=_blank](https://github.com/outskirtslabs/xitstash/blob/main/src/clj/ol/xitstash.clj#L261-L279) --- ## remove! ```clojure (remove! cursor id expected-revision) ``` Removes a file or empty directory inside the application’s transaction. Returns the removed record, with its revision from before removal. Root `0` cannot be removed. Nonempty directories throw `:xitstash/conflict`. Removal updates catalog indexes but never deletes payload content. Automatic IDs are not recycled. Revision checks and provisional results follow [`update!`](#update!). [source,window=_blank](https://github.com/outskirtslabs/xitstash/blob/main/src/clj/ol/xitstash.clj#L281-L289) --- ## delete-payload! ```clojure (delete-payload! cursor digest) (delete-payload! cursor digest options) ``` Deletes a published payload inside the application’s write transaction. Current catalog file references cause `:xitstash/conflict`. The application must check references outside the catalog. Missing payloads throw `:xitstash/not-found`. Shared chunks stay available to other payloads. Returns `{:digest id :removed-chunks n :shared-chunks n}`. Counts refer to distinct chunk IDs, not repeated manifest occurrences. Metadata and reference counts are checked without reading chunk content. Results are provisional until commit. Let any exception escape the transaction callback to abort. | | | | | --- | --- | --- | | Option | Default | Meaning | | `:max-chunks` | 65536 | Maximum manifest entries | | `:max-metadata-bytes` | 16777216 | Sum of EDN-encoded chunk-record lengths | This is logical deletion from the selected current store. It does not compact or replace database files, revoke open streams, or erase snapshots, backups, or replicas. Physical reclamation needs application-controlled compaction with no reachable old native references. No secure-overwrite guarantee is made. [source,window=_blank](https://github.com/outskirtslabs/xitstash/blob/main/src/clj/ol/xitstash.clj#L291-L313) ## How to organise files # How to organise files Use catalog records to give stored payloads names and directories. Run changes inside the application’s write transaction against the selected store cursor. The examples call that cursor `store` and assume `objects` aliases `ol.xitstash`. ## Create a directory and file together Publish the payload first. Pass its digest as `digest`: ```clojure (objects/create! store [{:name "invoices"} {:under 1 :name "november.pdf" :digest digest}]) ``` The result is a vector containing the directory and file records. Save their `:id` and `:revision` values after the transaction commits. `:under 1` selects the first request in this batch. To put a file in an existing directory, use `:parent` with that directory’s ID. Omit both keys to create a node below root `0`. If a sibling already has the requested name, creation fails. Let the exception abort the transaction, including any payload publication earlier in that callback. ## Find a file Use a read cursor to the same store: ```clojure (let [directory (objects/lookup store 0 "invoices")] (objects/lookup store (:id directory) "november.pdf")) ``` `children` returns a directory’s records in UTF-8 byte name order. `path` returns a vector of names from the root. Neither result needs an open cursor after the call completes. ## Rename without overwriting a concurrent change Save the file record when the user starts editing. Pass that record’s revision to the later write transaction: ```clojure (objects/move! store (:id file) {:name "paid.pdf"} (:revision file)) ``` A stale revision throws `:xitstash/conflict`, even if the requested change would now be a no-op. Read the current file and resolve the conflict before retrying. Do not silently substitute its new revision. To move between directories, add `:parent` to the destination. To replace an occupied destination, add `:replace? true`. Replacement requires the same node kind and an empty destination directory. It preserves the source ID and removes the destination record. ## Replace a file's content Prepare the new bytes outside the transaction. Publish them, then update the existing file within the same callback: ```clojure (objects/put! store prepared) (objects/update! store (:id file) (:digest prepared) (:revision file)) ``` The file keeps its ID, name and parent. Its previous payload stays available until explicitly deleted. ## Remove the name ```clojure (objects/remove! store (:id file) (:revision file)) ``` This removes the catalog record, not the payload. For content removal, follow [How to delete content and reclaim space](deletion.adoc). See [`create!`](api/ol-xitstash.adoc#create-BANG-) for name and ID rules. See [Branches need an explicit merge policy](content-and-history.adoc#branches-need-an-explicit-merge-policy) before combining catalogs from separate branches. ## Content and history # Content and history A file name answers “where is it?”. A content ID answers “which bytes?”. Keeping these questions separate lets two files share a payload and lets a file change without changing its ID. ## Three levels of identity A catalog node has an integer ID, a parent and a name. Renaming or moving it keeps its ID. Updating a file changes the payload it refers to. A payload has a digest of all its original bytes. The same bytes produce the same digest regardless of their file name, chunk sizes or compression. Changing the bytes creates a different payload. A payload consists of chunks. FastCDC chooses chunk cuts from the content rather than fixed byte positions. An edit can therefore reuse chunks from the previous version instead of storing the whole file again. The amount of reuse depends on the content and the chunk settings. Compression acts on individual chunks after hashing. Two payloads can share a chunk even if they were prepared with different compression settings. The stored copy need not use the codec requested by the later preparation. ## Why preparation happens before the transaction Reading an upload, hashing it and trying compression can take much longer than updating a record. Preparation separates that work from the database write transaction. Publication checks the prepared bytes again, then stores new chunks and references existing ones. This also puts resource decisions in the application. An application can reject a large upload before starting a transaction, bound temporary storage and close prepared content after publication. It must still allow for metadata memory and concurrent requests. ## Two meanings of history An old record can store only a digest. After deletion, that record still describes the old document but resolving its digest through the current store fails. This permits a history of names and events without keeping the binary content. A native snapshot instead points to an earlier store state. That state still reaches its manifests and chunk data. Deleting from the current state leaves the snapshot readable, including after compaction. The Yggdrasil xitdb adapter saves native application states. Its historical commits therefore preserve content, not just names and digest strings. ## Deletion is not erasure Removing a catalog file leaves its payload available. Deleting a payload removes it from the selected store and drops chunks that no other payload there uses. Neither operation shrinks the database file. Compaction copies reachable native data. Old snapshots can make deleted content reachable, so compaction is not a command to erase all past copies. The original database file, backups, replicas and filesystem snapshots can also contain the bytes. This distinction matters when history and deletion have competing requirements. An audit trail of digest strings can survive loss of content. A readable historical snapshot cannot. See [How to delete content and reclaim space](deletion.adoc) for the application steps. ## Branches need an explicit merge policy Two branches can allocate the same automatic file ID for different files. The catalog also maintains child indexes, reverse payload indexes and revisions. Merging its internal maps independently can break those relationships. Applications that merge branches need a policy for file identity and catalog conflicts. xitstash does not supply that merger. Application-selected IDs address ID collisions, not the rest of the merge. ## How to delete content and reclaim space # How to delete content and reclaim space Remove catalog references and delete the payload in an application transaction. Compact separately if the database file must stop carrying unreachable data. Before deleting, check references in your application’s records outside the catalog. xitstash cannot check them for you. ## Delete the file and payload together Inside the write transaction, use the store’s write cursor as `store`. Use the file record and revision from the deletion request as `file`: ```clojure (objects/remove! store (:id file) (:revision file)) (objects/delete-payload! store (:digest file)) ``` Let errors escape the callback. If another catalog file references the payload, `delete-payload!` throws `:xitstash/conflict`. Aborting the transaction restores the file removed by the preceding call too. If there is no catalog file, call `delete-payload!` directly with the digest. A successful call returns the digest and distinct chunk counts: ```clojure {:digest "f1220..." :removed-chunks 3 :shared-chunks 2} ``` The abbreviated digest illustrates the result shape. Shared chunks continue to serve other payloads. After commit, `payload` for the deleted digest must throw `:xitstash/not-found`. Deletion does not revoke an already-open stream. ## Reclaim unreachable bytes 1. Stop access that could race with compaction or database replacement. Close affected streams. 2. Remove old native snapshots if their bytes must no longer be copied. If those snapshots must stay readable, expect their content in the compacted database. 3. Use xitdb’s `Database.compact` to write a separate target. With an adapter, use its compaction API rather than bypassing its database coordination. 4. Reopen the target and verify the application records and payloads that should survive. 5. Replace the active database using your application’s file-replacement policy. Dispose of the original only after validation. The original file can still contain deleted bytes. Backups, replicas and filesystem snapshots need separate handling. Neither deletion nor compaction promises secure overwrite. For xitdb-clj, `xdb/compact` copies the latest value with at most one history entry. The Yggdrasil adapter instead preserves reachable commits. See [Content and history](content-and-history.adoc) before choosing which history to discard. For unusually large manifests, pass the same `:max-chunks` and `:max-metadata-bytes` budgets used to read them. The defaults are listed in [`delete-payload!`](api/ol-xitstash.adoc#delete-payload-BANG-). ## How to build and test # How to build and test Use an authorised xitstash checkout with Java 21 or later, Clojure CLI and Babashka. Install `cljfmt` and `clj-kondo` for formatting and lint. The library runs on Java 17, but the test-only Closure Compiler requires Java 21. ## Run the normal checks ```shell bb ci ``` This compiles Java with warnings treated as errors, runs the unit tests and checks Clojure formatting and lint. Use `bb qa` to apply formatting before lint and tests. To run one test namespace: ```shell bb test --focus ol.xitstash.multiformats-interop-test ``` ## Run adapter tests Supply local xitdb-clj and Yggdrasil checkouts. The tested revisions are: | Project | Revision | | --- | --- | | xitdb-clj | `b9ba2598f6b3a33198b870921382c8b59ff825c6` | | Yggdrasil | `0177619468885c3521521c8557f54f0ad6b127f0` | | xitdb-java | Maven version `0.48.0` | The Yggdrasil revision contains an unpublished xitdb adapter. A released Yggdrasil dependency does not supply it. The checkout must provide `yggdrasil.adapters.xitdb` with `create`, `read!`, `transact!` and `compact!`. Replace both paths and run: ```shell bb javac clojure -Sdeps '{:deps {codeboost/xitdb-clj {:local/root "/path/to/xitdb-clj"} org.replikativ/yggdrasil {:local/root "/path/to/yggdrasil"}}}' \ -M:kaocha:integration --config-file integration/tests.edn ``` This runs unit and integration suites. Append `:integration` to select only the integration suite. The adapter tests cover mixed application and binary writes, rollback, branch isolation, historical reads and compaction/reopen. They do not cover ClojureScript, generic catalog merging or selective erasure across adapter history. Default CI and Nix package builds do not run these optional tests. ## Run measurements ```shell bb bench ``` For a fresh-JVM memory check on Linux: ```shell bb javac clojure -J-Xmx128m -M:bench --memory small clojure -J-Xmx128m -M:bench --memory large ``` See [Performance measurements](measurements.adoc) for conditions and interpretation. ## Generate documentation ```shell bb gen-docs ``` This copies `README.adoc` into the component index, generates the public API reference from `ol.xitstash` and writes `doc/manifest.edn`. Edit the README or source docstrings rather than their generated pages. Edit other reader pages under `doc/modules/ROOT/pages/`. Keep `AGENTS.md` and `docs/agents/` in Markdown. ## Build a development jar ```shell bb jar ``` The jar under `target/` contains Clojure source, Java classes and attribution resources. Do not publish it while the project license is unspecified. ## Save your first file # Save your first file Save a three-byte attachment in an in-memory database, give it a name and read it back. This tutorial assumes you can use a Clojure REPL. It does not create files on disk. ## Start the REPL From an authorised xitstash checkout, with Java 17 or later, Clojure CLI and Babashka installed, run: ```shell bb javac clojure -Sdeps '{:deps {io.github.radarroark/xitdb {:mvn/version "0.48.0"}}}' -M ``` Wait for the `user=>` prompt. ## Load the library ```clojure (require '[ol.xitstash :as objects]) (import '[io.github.radarroark.xitdb CoreMemory Database Database$ContextFunction Hasher RandomAccessMemory ReadCountedHashMap WriteCountedHashMap] '[java.security MessageDigest]) ``` The `require` returns `nil`. The `import` returns `java.security.MessageDigest`. ## Save and read the attachment Evaluate the following expression. It prepares the content, commits a file named `note.bin` and returns the saved path and bytes: ```clojure (with-open [core (CoreMemory. (RandomAccessMemory.)) prepared (objects/prepare (byte-array [1 2 3]))] (let [db (Database. core (Hasher. (MessageDigest/getInstance "SHA-256")))] (locking db (.runContext db (reify Database$ContextFunction (run [_ root] (let [store (.putCursor (WriteCountedHashMap. root) "objects")] (objects/initialize! store) (objects/put! store prepared) (objects/create! store [{:name "note.bin" :digest (:digest prepared)}]))))) (let [store (.getCursor (ReadCountedHashMap. (.rootCursor db)) "objects") file (objects/lookup store 0 "note.bin")] (with-open [input (objects/open store (:digest file))] {:path (objects/path store (:id file)) :bytes (vec (.readAllBytes input))}))))) ``` Result: ```clojure {:path ["note.bin"] :bytes [1 2 3]} ``` The file lives below catalog root `0`. Its bytes were prepared before the transaction and read after the commit. The enclosing `with-open` forms close the stream, prepared content and memory database. Run the expression again. It creates a fresh database and returns the same result. ## Next Use [xitdb-clj transactions](xitdb-clj.adoc) to save attachments alongside Clojure application records. Read [Content and history](content-and-history.adoc) for the distinction between a file name and a payload. ## Content identifiers # Content identifiers Payload and chunk IDs are full SHA2 digests encoded as multihash bytes inside lowercase base16 multibase text. They are not CIDs. | Algorithm | Digest length | Text prefix | | --- | --- | --- | | SHA2-256 | 32 bytes | `f1220` | | SHA2-512 | 64 bytes | `f1340` | The binary form is an unsigned-varint algorithm code, an unsigned-varint digest length and the digest bytes. The supported algorithm codes are `0x12` and `0x13`. Both header fields occupy one byte in this subset. The default algorithm is `:sha2-256`. Payload hashes cover all original bytes. Chunk hashes cover each chunk’s original bytes before compression. The database’s key hasher does not change these IDs. Unsupported forms include other bases, other algorithms, truncated digests, overlong varints and noncanonical spellings. ## clj-multiformats With `mvxcvi/multiformats` 1.0.125: ```clojure (require '[multiformats.base :as mbase] '[multiformats.hash :as mhash]) (mhash/decode (mbase/parse id)) (mbase/format :base16 (mhash/encode h)) ``` Here `id` is a xitstash identifier and `h` is a supported multihash value. The two expressions decode and encode, respectively. `multiformats.hash/parse` accepts bare hexadecimal text, not the full multibase-prefixed identifier. That library’s `hash:sha2-256:…` display form is not the xitstash wire format. clj-multiformats is a test dependency, not a runtime requirement. ## ol.xitstash # ol.xitstash > Binary content and file catalogs in [xitdb](https://github.com/xit-vcs/xitdb-java). ![doc](https://img.shields.io/badge/doc-outskirtslabs-orange.svg) ![status: experimental](https://img.shields.io/badge/status-experimental-orange.svg) `ol.xitstash` stores binary content alongside your application’s records in xitdb. Payloads share identical chunks, and a file catalog gives them names and directories. Use it when a record and its attachment need to commit in the same database transaction. The library provides content hashing, optional compression and seekable streams that check bytes before returning them. Your application opens the database, starts transactions and decides when to compact it. Requires Java 17 or later and application-supplied xitdb-java. The tested xitdb version is 0.48.0. Project status: **[Experimental](https://docs.outskirtslabs.com/open-source-vital-signs#experimental)**. The API is provisional. There is no published artifact, and source access is restricted. ## Installation With an authorised source checkout, follow the [installation instructions](https://docs.outskirtslabs.com/ol.xitstash/next/installation). There is no runtime dependency on xitdb-clj or Yggdrasil. ## Quick start Prepare bytes before starting a write transaction: ```clojure (require '[ol.xitstash :as objects]) (with-open [prepared (objects/prepare (byte-array [1 2 3]))] {:digest (:digest prepared) :size (:size prepared)}) ``` Result: ```clojure {:digest "f1220039058c6f2c0cb492c533b0a4d14ef77cc0f78abccced5287d84a1a2011cfb81" :size 3} ``` Preparation computes the content ID but does not publish the bytes. [Save your first file](https://docs.outskirtslabs.com/ol.xitstash/next/first-file) takes this through a transaction and reads it back. ## Documentation * [Docs](https://docs.outskirtslabs.com/ol.xitstash/next/) * [API Reference](https://docs.outskirtslabs.com/ol.xitstash/next/api) * [Support via GitHub Issues](https://github.com/outskirtslabs/xitstash/issues) The issue tracker requires repository access. ## Licensing The project license has not yet been specified. Do not redistribute a release artifact. The FastCDC implementation includes code adapted from xit under the MIT license. Its notice is in `resources/META-INF/xit-LICENSE.txt` and is included in development jars. ## How to install from source # How to install from source Use an authorised xitstash checkout. The repository is private and no artifact is published. ## Prepare the checkout Install Java 17 or later, Clojure CLI and Babashka. In the xitstash checkout, compile the Java classes: ```shell bb javac ``` The command writes classes under `target/classes`. ## Add the application dependency Add the checkout and xitdb-java to your application’s `deps.edn`. Replace the local path: ```clojure {:deps {com.outskirtslabs/xitstash {:local/root "/path/to/xitstash"} io.github.radarroark/xitdb {:mvn/version "0.48.0"}}} ``` Start a REPL from the application: ```shell clojure -M ``` Load the public namespace: ```clojure (require '[ol.xitstash :as objects]) ``` A successful load returns `nil`. After changing xitstash’s Java source, rerun `bb javac` and restart the application JVM. See [Save your first file](first-file.adoc) for a complete in-memory example. ## Performance measurements # Performance measurements These development measurements describe one workload on one machine. They are not throughput guarantees or comparisons with another object store. ## Conditions * AMD Ryzen 9 7950X3D, Linux x86-64, OpenJDK 27. * A 32 MiB byte array from `java.util.Random`, seed 42. * Five warmups and seven timed samples per operation. Reported times are medians. * SHA2-256, with FastCDC minimum 256 KiB, target 1 MiB and maximum 4 MiB. * Files under `/tmp` on ZFS, with warmed caches. * Native-storage measurements at commit `b55b618`. Catalog operations were not timed. Input creation is outside timing. Publication timings exclude preparation. Fresh-file publication includes database creation and commit syncs. Preparation timings include temporary staging and cleanup but do not force durable writes. ## Preparation | Operation | MiB/s | Milliseconds | | --- | --- | --- | | Read cached file | 9,514 | 3.363 | | Read cached file and hash twice | 1,057 | 30.287 | | Prepare cached file | 631 | 50.697 | | Prepare byte array with spill | 536 | 59.703 | | Try zlib on random file content | 58 | 556.209 | The read controls use a reusable 64 KiB buffer. The two-hash control updates two SHA2-256 states but does not find chunk cuts. Random content compresses poorly, so the zlib case pays for compression attempts without a useful reduction in stored size. ## Publication and reads | Operation | MiB/s | Milliseconds | | --- | --- | --- | | Publish to a fresh memory database | 791 | 40.480 | | Publish to a fresh file database | 380 | 84.179 | | Publish an existing payload again | 1,298 | 24.659 | | Read a cached payload with verification | 1,589 | 20.140 | | Open and read a 32 KiB range | Not reported | 2.197 | | Publish after changing one byte | 845 | 37.887 | Duplicate publication still checks the prepared source. Opening a stream checks the complete manifest. A range read verifies every covering chunk. The one-byte edit reused 24 of 25 chunks, covering 32,898,057 bytes. It appended 672,605 bytes including metadata. Its throughput counts the full logical payload, not just the new bytes. ## Small payloads Each sample uses 2,000 distinct 256-byte payloads in a memory database. Publication uses one transaction for the whole batch. | Operation | Median batch time | Objects/s | | --- | --- | --- | | Publish | 43.549 ms | 45,900 | | Open and read | 19.107 ms | 104,700 | Median-sample thread allocation was 166,706,416 bytes for publication and 80,516,520 bytes for reads. These are cumulative allocations, not peak live heap. Read allocation includes the per-stream buffer used by `InputStream.transferTo`. The results do not measure one durable transaction per small object. ## Memory Fresh JVM runs prepared, published and verified a 256 MiB file-backed payload with a 128 MiB heap limit. The source generator used a reusable 64 KiB buffer. | Payload | Peak process RSS in three runs, KiB | | --- | --- | | 256 bytes | 227,744 / 218,632 / 200,828 | | 256 MiB | 220,676 / 226,344 / 244,792 | RSS includes the JVM, compiled code and native allocations. It is neither peak live heap nor filesystem cache usage. Startup variation prevents a precise estimate of per-object memory from these runs. The benchmark and large-payload check also completed on OpenJDK 17.0.20.1. The tables above report OpenJDK 27. ## Limits of the evidence The throughput runs use no independent JVM forks, CPU pinning or confidence intervals. They do not establish uncached device bandwidth. Concurrent-reader throughput and durable per-object latency have not been measured. See [Run measurements](development.adoc#run-measurements) for reproduction commands. ## How to accept an upload # How to accept an upload Prepare an upload before entering the database transaction. Set limits for each request, then publish the prepared content and its file record together. This guide assumes an open xitdb `Database` called `db`. The application root must be a counted map or empty. The example uses its `"objects"` entry for xitstash. ## Publish from an input stream Load these names: ```clojure (require '[ol.xitstash :as objects]) (import '[io.github.radarroark.xitdb Database Database$ContextFunction WriteCountedHashMap] '[java.io InputStream]) ``` Define the upload operation: ```clojure (defn save-upload! [^Database db ^InputStream source] (with-open [prepared (objects/prepare source {:max-size 1073741824 :max-temp-bytes 1073741824})] (locking db (.runContext db (reify Database$ContextFunction (run [_ root] (let [store (.putCursor (WriteCountedHashMap. root) "objects")] (objects/initialize! store) (objects/put! store prepared) (objects/create! store [{:name "upload.bin" :digest (:digest prepared)}])))))) (:digest prepared))) ``` The function returns the committed digest. The caller must close `source`. The `with-open` closes prepared content even if publication fails. Let exceptions escape the transaction callback. If `upload.bin` already exists, this operation fails and the transaction aborts. For replacement, use `update!` with the existing file’s ID and revision instead of `create!`. See [How to organise files](catalog.adoc). Verify a successful upload with `lookup`, then open the file’s digest with `open`. Read large payloads through a bounded buffer rather than `readAllBytes`. Close readers before the database closes or compaction starts. ## Accept a producer that writes to an output stream If a producer needs an `OutputStream`, give it an upload builder. Call `finish` once after the producer stops writing: ```clojure (with-open [builder (objects/upload {:max-size 1073741824})] (.write builder (byte-array [1 2 3])) (with-open [prepared (objects/finish builder)] {:digest (:digest prepared) :size (:size prepared)})) ``` This example prepares content only. In your application, publish `prepared` inside the inner `with-open` using the transaction pattern above. Keep both `upload` and `finish` outside the transaction. Close an unfinished builder to discard an interrupted upload. Close a finished result separately from its builder. ## Bound temporary storage Set `:temp-dir` to an existing application-specific directory. Choose `:max-size`, `:max-temp-bytes`, `:max-chunks` and `:max-metadata-bytes` for your workload. The limits apply to one operation, not to all concurrent uploads. If you use `:compression :zlib`, allow temporary space for both input and encoded content. The library uses the encoded chunk only when it is smaller. After a crashed process, call `cleanup-staging!` on that directory. It skips active preparations and directories with invalid markers or unexpected files. It cannot remove an unmarked directory left by a failure during initial directory creation. See [`prepare`](api/ol-xitstash.adoc#prepare) for option defaults and [`cleanup-staging!`](api/ol-xitstash.adoc#cleanup-staging-BANG-) for cleanup options. ## How to combine records and attachments # How to combine records and attachments Use one xitdb-clj `swap!` to publish an attachment and update its application record. This guide assumes an application with xitdb-clj and xitstash on the classpath. See [the tested adapter revisions](development.adoc#run-adapter-tests). ## Select a separate map for content Load the library and adapter: ```clojure (require '[ol.xitstash :as objects] '[xitdb.common :as xc] '[xitdb.db :as xdb]) (import '[io.github.radarroark.xitdb ReadHashMap] '[java.io Closeable InputStream]) ``` Create an `:objects` map beside the application’s `:status`. Inside `swap!`, unwrap that map and take its native cursor: ```clojure (with-open [^Closeable db (xdb/xit-db :memory) ^Closeable prepared (objects/prepare (byte-array [1 2 3]))] (reset! db {:objects {} :status :draft}) (swap! db (fn [state] (let [store (.cursor ^ReadHashMap (xc/unwrap (:objects state)))] (objects/initialize! store) (objects/put! store prepared) (objects/create! store [{:name "attachment.bin" :digest (:digest prepared)}])) (assoc state :status :published))) (let [store (.cursor ^ReadHashMap (xc/unwrap (:objects @db)))] (with-open [^InputStream input (objects/open store (:digest prepared))] {:status (:status @db) :path (objects/path store 1) :bytes (vec (.readAllBytes input))}))) ``` Result: ```clojure {:status :published :path ["attachment.bin"] :bytes [1 2 3]} ``` If a write fails, let the exception leave `swap!` so both the application change and the binary change abort. Do not catch the error inside the callback and commit a partial result. In xitdb-clj `:compat` mode, use string keys for your application maps. ## Keep the object store separate from collection operations Use `xc/unwrap` and `.cursor` to reach the native subtree. `xdb-cursor` is a path view, not the native cursor required by xitstash. Do not call `xdb/materialize` on the object-store subtree or edit its internal keys with `assoc`. Use ordinary collection operations on application data beside it. Coordinate concurrent access, close and compaction. For direct native access, use the database monitor or an application lock that prevents overlap. The example is single-threaded. ## Read history and compact Use `xdb/deref-at` to get an older application value, then select its object-store subtree. Keep its streams within the database lifetime. Close affected readers before calling `xdb/compact` outside `swap!`. It creates a new database containing the latest value and at most one history entry. It does not replace the original file. Validate and replace that file using the application’s lifecycle policy. See [How to delete content and reclaim space](deletion.adoc). ## How to use snapshots and branches # How to use snapshots and branches Use the Yggdrasil xitdb adapter’s callbacks to change or read an object-store subtree. This guide is for applications already using the **unpublished** xitdb adapter. A released Yggdrasil dependency is not sufficient. See [the required checkout and test command](development.adoc#run-adapter-tests). ## Change a branch without changing an earlier snapshot Load these names: ```clojure (require '[ol.xitstash :as objects] '[yggdrasil.adapters.xitdb :as x] '[yggdrasil.protocols :as p]) (import '[io.github.radarroark.xitdb ReadHashMap WriteHashMap] '[java.io Closeable]) ``` Create a file, open a historical view and rename the file on a separate branch: ```clojure (with-open [^Closeable system (x/create) ^Closeable prepared (objects/prepare (byte-array [1 2 3]))] (x/transact! system (fn [root] (let [store (.putCursor (WriteHashMap. root) "objects")] (objects/initialize! store) (objects/put! store prepared) (objects/create! store [{:name "original.bin" :digest (:digest prepared)}])))) (with-open [^Closeable past (p/as-of system (p/snapshot-id system))] (p/branch! system :experiment) (let [experiment (p/checkout system :experiment)] (x/transact! experiment (fn [root] (objects/move! (.putCursor (WriteHashMap. root) "objects") 1 {:name "branch.bin"} 1))) (x/compact! system) (mapv (fn [view] (x/read! view (fn [root] (objects/path (.getCursor (ReadHashMap. root) "objects") 1)))) [system past experiment])))) ``` Result: ```clojure [["original.bin"] ["original.bin"] ["branch.bin"]] ``` Let write errors leave `x/transact!` to abort. Do not start a nested transaction or access the adapter’s physical catalog from the callback. ## Read payloads within the callback Open, read and close a payload stream inside `x/read!`. Return copied bytes or realised metadata. Do not return a borrowed cursor or stream. Close historical views when finished. Do not close checkout handles separately, because they share the system’s lifetime. ## Preserve catalog consistency Do not pass xitstash’s internal maps through the adapter’s generic structural merger. If branches must merge, supply a custom merger that handles IDs, revisions and all catalog indexes together. No such merger is included with xitstash. Deleting a payload on the current branch does not erase it from old commits. Compaction preserves content reachable from those commits. See [Content and history](content-and-history.adoc).