How to accept an upload

Prepare an upload before entering the database transaction. Set limits for each request, then publish the prepared content and its file record together.

This guide assumes an open xitdb Database called db. The application root must be a counted map or empty. The example uses its "objects" entry for xitstash.

Publish from an input stream

Load these names:

(require '[ol.xitstash :as objects])
(import '[io.github.radarroark.xitdb
          Database Database$ContextFunction WriteCountedHashMap]
        '[java.io InputStream])

Define the upload operation:

(defn save-upload! [^Database db ^InputStream source]
  (with-open [prepared (objects/prepare source {:max-size 1073741824
                                               :max-temp-bytes 1073741824})]
    (locking db
      (.runContext db
        (reify Database$ContextFunction
          (run [_ root]
            (let [store (.putCursor (WriteCountedHashMap. root) "objects")]
              (objects/initialize! store)
              (objects/put! store prepared)
              (objects/create! store [{:name "upload.bin"
                                       :digest (:digest prepared)}]))))))
    (:digest prepared)))

The function returns the committed digest. The caller must close source. The with-open closes prepared content even if publication fails.

Let exceptions escape the transaction callback. If upload.bin already exists, this operation fails and the transaction aborts. For replacement, use update! with the existing file’s ID and revision instead of create!. See How to organise files.

Verify a successful upload with lookup, then open the file’s digest with open. Read large payloads through a bounded buffer rather than readAllBytes. Close readers before the database closes or compaction starts.

Accept a producer that writes to an output stream

If a producer needs an OutputStream, give it an upload builder. Call finish once after the producer stops writing:

(with-open [builder (objects/upload {:max-size 1073741824})]
  (.write builder (byte-array [1 2 3]))
  (with-open [prepared (objects/finish builder)]
    {:digest (:digest prepared) :size (:size prepared)}))

This example prepares content only. In your application, publish prepared inside the inner with-open using the transaction pattern above. Keep both upload and finish outside the transaction.

Close an unfinished builder to discard an interrupted upload. Close a finished result separately from its builder.

Bound temporary storage

Set :temp-dir to an existing application-specific directory. Choose :max-size, :max-temp-bytes, :max-chunks and :max-metadata-bytes for your workload. The limits apply to one operation, not to all concurrent uploads.

If you use :compression :zlib, allow temporary space for both input and encoded content. The library uses the encoded chunk only when it is smaller.

After a crashed process, call cleanup-staging! on that directory. It skips active preparations and directories with invalid markers or unexpected files. It cannot remove an unmarked directory left by a failure during initial directory creation.

See prepare for option defaults and cleanup-staging! for cleanup options.