# ol.xitstash

Stores immutable payloads and a file catalog in an application-selected xitdb subtree.

## prepare

```clojure
(prepare source)
(prepare source options)
```

Chunks and hashes bytes, a File/Path, or an InputStream before insertion.

Returns closeable prepared content with `:digest` and `:size`. Byte arrays
are copied. Caller-supplied streams are not closed. Uncompressed files are
scanned without a temporary copy and must match their prepared chunk hashes
when consumed. Closing prepared content releases its temporary storage, not
the source file. Prepared content can be consumed repeatedly before close.

Options:

|     |     |     |
| --- | --- | --- |
| Key | Default | Meaning |
| `:algorithm` | `:sha2-256` | `:sha2-256` or `:sha2-512` |
| `:min-size` | 262144 | Minimum FastCDC chunk size |
| `:average-size` | 1048576 | Power-of-two target, 256 through 4194304 |
| `:max-chunk-size` | 4194304 | Maximum chunk size, at most 16777216 |
| `:compression` | `:none` | `:none` or per-chunk `:zlib`, only if smaller |
| `:memory-threshold` | 1048576 | Spool bytes before spilling, at most 16777216 |
| `:max-size` | 1099511627776 | Maximum original payload bytes |
| `:max-chunks` | 65536 | Maximum chunk count |
| `:max-metadata-bytes` | 16777216 | Sum of EDN-encoded chunk-record lengths |
| `:max-temp-bytes` | 1099511627776 | Maximum simultaneous staged payload bytes |
| `:temp-dir` | JVM temporary directory | Existing File/Path or path string |

Chunk sizes must be positive and ordered. Resource limits are nonnegative
signed 64-bit integers. Spool buffers and chunk buffers are bounded, but
total metadata memory grows with chunk count. The metadata limit measures
encoded records, not JVM heap usage. Compression may temporarily require
both an input spool and an encoded spool within `:max-temp-bytes`.

Invalid options and exhausted limits throw typed `ex-info`. Native I/O
errors propagate. Use [`cleanup-staging!`](#cleanup-staging!) for abandoned temporary storage.

[source,window=_blank](https://github.com/outskirtslabs/xitstash/blob/main/src/clj/ol/xitstash.clj#L14-L48)

---

## upload

```clojure
(upload)
(upload options)
```

Creates a closeable OutputStream for content preparation outside a transaction.

Operations on one builder are serialized. Closing an unfinished builder
discards its staging data. [`finish`](#finish) seals the builder and returns separate
closeable prepared content. Closing the builder after finish does not close
that result. Writes and repeated finish calls after sealing throw typed
`ex-info`. Prefer bulk writes over individual bytes.

Options:

|     |     |
| --- | --- |
| Key | Meaning |
| All [`prepare`](#prepare) options | Same defaults and resource limits |

Chunking and hashing run during [`finish`](#finish), before a write transaction.

[source,window=_blank](https://github.com/outskirtslabs/xitstash/blob/main/src/clj/ol/xitstash.clj#L50-L67)

---

## finish

```clojure
(finish builder)
```

Seals `builder` and returns prepared content without publishing it.

Call once on a builder from [`upload`](#upload). Hashing and chunking complete before
return. A failed finish discards staging data. Close the returned prepared
content separately from the builder.

[source,window=_blank](https://github.com/outskirtslabs/xitstash/blob/main/src/clj/ol/xitstash.clj#L69-L76)

---

## cleanup-staging!

```clojure
(cleanup-staging!)
(cleanup-staging! options)
```

Deletes inactive, marked staging directories and returns their paths.

File locks protect active preparations, including those in other processes.
A directory with an invalid marker or unexpected files is not removed.
No background cleanup starts automatically. Prefer an application-specific
temporary directory. Unmarked directories from an interrupted initial
directory creation are not removed.

Options:

|     |     |     |
| --- | --- | --- |
| Key | Default | Meaning |
| `:temp-dir` | JVM temporary directory | Existing directory to inspect |
| `:min-age-ms` | 86400000 | Minimum directory age, nonnegative milliseconds |

Unknown options and invalid ages throw typed `ex-info`. Native I/O errors
propagate. Cleanup cannot remove published database content.

[source,window=_blank](https://github.com/outskirtslabs/xitstash/blob/main/src/clj/ol/xitstash.clj#L78-L97)

---

## initialize!

```clojure
(initialize! cursor)
```

Initializes an empty subtree or validates an existing xitstash store.

`cursor` must be a native write cursor inside an application transaction.
Returns `{:format string :initialized? boolean}`. An empty plain map is
accepted. Unrelated nonempty data and unsupported formats are rejected
without mutation. Repeated initialization does not write data.

[source,window=_blank](https://github.com/outskirtslabs/xitstash/blob/main/src/clj/ol/xitstash.clj#L99-L107)

---

## put!

```clojure
(put! cursor prepared)
```

Publishes prepared content inside the application’s write transaction.

`cursor` selects an initialized store. `prepared` comes from [`prepare`](#prepare) or
[`finish`](#finish) and must be open. Returns `:digest`, `:size`, and `:deduplicated?`.
Existing payloads are not rewritten. Prepared bytes are checked again,
including on duplicate insertion. Shared chunks are stored once.

Results are provisional until the application commits. Let any exception
escape the transaction callback to abort. Close prepared content separately.

[source,window=_blank](https://github.com/outskirtslabs/xitstash/blob/main/src/clj/ol/xitstash.clj#L109-L120)

---

## payload

```clojure
(payload cursor digest)
```

Returns realized `{:digest id :size bytes}` metadata without reading content.

`cursor` selects a store. Missing content throws `:xitstash/not-found`.

[source,window=_blank](https://github.com/outskirtslabs/xitstash/blob/main/src/clj/ol/xitstash.clj#L122-L127)

---

## manifest

```clojure
(manifest cursor digest)
(manifest cursor digest options)
```

Returns realized payload metadata with ordered `:chunks` digest/size maps.

Validates native references and lengths without decoding chunk bytes.
Missing content, corrupt metadata, and exceeded limits throw typed `ex-info`.
The metadata limit counts encoded records, not JVM heap usage.

|     |     |     |
| --- | --- | --- |
| Option | Default | Meaning |
| `:max-chunks` | 65536 | Maximum manifest entries |
| `:max-metadata-bytes` | 16777216 | Sum of EDN-encoded chunk-record lengths |

[source,window=_blank](https://github.com/outskirtslabs/xitstash/blob/main/src/clj/ol/xitstash.clj#L129-L141)

---

## open

```clojure
(open cursor digest)
(open cursor digest options)
```

Opens a closeable InputStream over a payload or byte range.

Each decoded chunk is checked before its bytes are exposed. One decoded
chunk is cached, plus checked metadata and temporary decoding buffers.
Closing the stream does not close the database. Close streams before their
supplied cursors become invalid, or before database compaction/replacement.

Library native reads use the database monitor. Coordinate other native
access, including cursor acquisition and transactions, with that monitor
or prevent overlap through application-level synchronization.

|     |     |     |
| --- | --- | --- |
| Option | Default | Meaning |
| `:start` | 0 | Inclusive payload byte offset |
| `:end` | Payload size | Exclusive payload byte offset |
| `:max-chunks` | 65536 | Maximum manifest entries |
| `:max-metadata-bytes` | 16777216 | Sum of EDN-encoded chunk-record lengths |

Requires `0 <= start <= end <= size`. Missing content, corrupt chunks,
invalid ranges, and exceeded limits throw typed `ex-info`. Native I/O
errors propagate. Use [`seek!`](#seek!) to move within the selected range.

[source,window=_blank](https://github.com/outskirtslabs/xitstash/blob/main/src/clj/ol/xitstash.clj#L143-L166)

---

## seek!

```clojure
(seek! stream offset origin)
```

Moves a stream from [`open`](#open) and returns its range-relative byte position.

`origin` is `:start`/0, `:current`/1, or `:end`/2. `offset` is a signed
64-bit integer. Negative results and overflow throw typed `ex-info` without
changing position. Positions after the range end are allowed and read EOF.
Seeking a closed stream throws `:xitstash/closed`.

[source,window=_blank](https://github.com/outskirtslabs/xitstash/blob/main/src/clj/ol/xitstash.clj#L168-L176)

---

## create!

```clojure
(create! cursor requests)
```

Creates 1–4096 catalog nodes inside the application’s write transaction.

Returns a vector of records in request order. Each record has `:id`, `:parent`,
`:name`, `:kind` (`:directory` or `:file`), and `:revision`. Files also have
`:digest` and `:size`. All records in one batch use the same revision token.

|     |     |
| --- | --- |
| Request key | Meaning |
| `:name` | Required name, 1–4096 valid UTF-8 bytes, no slash/NUL or dot/double dot |
| `:parent` | Existing directory ID, default root `0` |
| `:under` | One-based position of an earlier directory request, instead of `:parent` |
| `:digest` | Published payload ID for a file, absent for a directory |
| `:id` | Optional positive signed 64-bit ID, otherwise allocated automatically |

Requests are processed in order. Explicit IDs advance the automatic counter.
Existing IDs and sibling names conflict. Automatic IDs are not recycled, but
independent branches can allocate the same ID. Applications must handle that
when merging. Unknown request keys are invalid.

Results are provisional until commit. Let any exception escape the transaction
callback to abort all changes. Files can reference payloads published earlier
in the same transaction with [`put!`](#put!).

[source,window=_blank](https://github.com/outskirtslabs/xitstash/blob/main/src/clj/ol/xitstash.clj#L178-L202)

---

## node

```clojure
(node cursor id)
```

Returns a realized catalog record for a positive `id`.

Root `0` has no record. Missing nodes throw `:xitstash/not-found`.
See [`create!`](#create!) for record fields.

[source,window=_blank](https://github.com/outskirtslabs/xitstash/blob/main/src/clj/ol/xitstash.clj#L204-L210)

---

## lookup

```clojure
(lookup cursor parent name)
```

Returns the child record at `parent` and `name`.

`parent` is a directory ID or root `0`. Invalid names and non-directory parents
throw `:xitstash/invalid-argument`. Missing parents or children throw
`:xitstash/not-found`.

[source,window=_blank](https://github.com/outskirtslabs/xitstash/blob/main/src/clj/ol/xitstash.clj#L212-L219)

---

## children

```clojure
(children cursor parent)
```

Returns a realized vector of child records in UTF-8 byte name order.

`parent` is a directory ID or root `0`. Empty directories return `[]`.
Missing parents throw `:xitstash/not-found`, files are invalid parents.
Result memory grows with the directory’s child count.

[source,window=_blank](https://github.com/outskirtslabs/xitstash/blob/main/src/clj/ol/xitstash.clj#L221-L228)

---

## path

```clojure
(path cursor id)
```

Returns a realized vector of names from root to `id`.

Root `0` returns `[]`. Missing nodes or ancestors throw `:xitstash/not-found`.
Corrupt parent chains throw `:xitstash/corrupt-data`.

[source,window=_blank](https://github.com/outskirtslabs/xitstash/blob/main/src/clj/ol/xitstash.clj#L230-L236)

---

## files

```clojure
(files cursor digest)
```

Returns a realized vector of ascending file IDs that reference `digest`.

No references return `[]`, including for an unpublished valid payload ID.
Invalid identifiers throw typed `ex-info`. Result memory grows with the
number of referencing files.

[source,window=_blank](https://github.com/outskirtslabs/xitstash/blob/main/src/clj/ol/xitstash.clj#L238-L245)

---

## update!

```clojure
(update! cursor id digest expected-revision)
```

Changes a file’s payload inside the application’s write transaction.

`digest` must identify a published payload. Returns the updated record.
Directories and root `0` cannot be updated. An unchanged digest performs no
writes and preserves the record’s revision.

`expected-revision` is a nonnegative signed 64-bit integer. Zero skips the
check. A stale positive value throws `:xitstash/conflict`, even for a no-op.
Negative values are invalid. Changed records get a new per-store revision.
Results are provisional, as described by [`create!`](#create!).

[source,window=_blank](https://github.com/outskirtslabs/xitstash/blob/main/src/clj/ol/xitstash.clj#L247-L259)

---

## move!

```clojure
(move! cursor id destination expected-revision)
```

Renames or reparents a catalog node inside the application’s transaction.

Returns the source record with its unchanged ID and new revision. Moving
to its current parent/name performs no writes and preserves its revision.
Root `0` cannot move. A directory cannot move into itself or its descendants.
Revision checks and provisional results follow [`update!`](#update!).

|     |     |     |
| --- | --- | --- |
| Destination key | Default | Meaning |
| `:parent` | Current parent | Directory ID or root `0` |
| `:name` | Current name | New name, subject to [`create!`](#create!) name rules |
| `:replace?` | `false` | Remove an occupied destination of the same kind |

Replacement directories must be empty. Replacement removes the destination
record, not its payload. Without replacement, occupied destinations conflict.
Unknown keys are invalid. Let any exception escape to abort the transaction.

[source,window=_blank](https://github.com/outskirtslabs/xitstash/blob/main/src/clj/ol/xitstash.clj#L261-L279)

---

## remove!

```clojure
(remove! cursor id expected-revision)
```

Removes a file or empty directory inside the application’s transaction.

Returns the removed record, with its revision from before removal. Root `0`
cannot be removed. Nonempty directories throw `:xitstash/conflict`. Removal
updates catalog indexes but never deletes payload content. Automatic IDs
are not recycled. Revision checks and provisional results follow [`update!`](#update!).

[source,window=_blank](https://github.com/outskirtslabs/xitstash/blob/main/src/clj/ol/xitstash.clj#L281-L289)

---

## delete-payload!

```clojure
(delete-payload! cursor digest)
(delete-payload! cursor digest options)
```

Deletes a published payload inside the application’s write transaction.

Current catalog file references cause `:xitstash/conflict`. The application
must check references outside the catalog. Missing payloads throw
`:xitstash/not-found`. Shared chunks stay available to other payloads.

Returns `{:digest id :removed-chunks n :shared-chunks n}`. Counts refer to
distinct chunk IDs, not repeated manifest occurrences. Metadata and reference
counts are checked without reading chunk content. Results are provisional
until commit. Let any exception escape the transaction callback to abort.

|     |     |     |
| --- | --- | --- |
| Option | Default | Meaning |
| `:max-chunks` | 65536 | Maximum manifest entries |
| `:max-metadata-bytes` | 16777216 | Sum of EDN-encoded chunk-record lengths |

This is logical deletion from the selected current store. It does not compact
or replace database files, revoke open streams, or erase snapshots, backups,
or replicas. Physical reclamation needs application-controlled compaction
with no reachable old native references. No secure-overwrite guarantee is made.

[source,window=_blank](https://github.com/outskirtslabs/xitstash/blob/main/src/clj/ol/xitstash.clj#L291-L313)