DataLab workspace capsule profile 0.1

• Identifier: https://datalab-platform.com/capsule/0.1
• Version: 0.1
• Status: experimental; may change before version 1.0
• Terms: https://datalab-platform.com/capsule/terms# (JSON-LD prefix dlc)
• Extends: RO-Crate 1.3 and Process Run Crate 0.6
• Reference implementation: DataLab-Capsule

Archive

A capsule is a ZIP archive whose entries are stored without compression: ro-crate-metadata.json (the manifest) and, unless the workspace is referenced (see below), workspace.h5 (a DataLab HDF5 workspace, provenance block /DataLab_Provenance included), and the declared attachments, if any, under attachments/ (see below). Readers must refuse absolute paths, .., drive letters, backslashes, duplicate or unexpected entries, links, encrypted or compressed entries, and must check the SHA-256 and size of the workspace against the manifest. As in RO-Crate, sha256 holds the lowercase hexadecimal digest without prefix and contentSize the size in bytes as a string.

Manifest

The manifest is an RO-Crate 1.3 metadata document whose @context is ["https://w3id.org/ro/crate/1.3/context", {"dlc": "https://datalab-platform.com/capsule/terms#"}].

• The root Dataset (./) has a name, a description, datePublished and a licence (an entity or a text). Its conformsTo lists Process Run Crate 0.6 and this profile. It contains the workspace file (hasPart), mentions every action and carries dlc:manifestFingerprint and dlc:workspaceId.
• The workspace file (workspace.h5, or an absolute IRI when referenced) is a File with encodingFormat application/x-hdf5, contentSize and sha256.
• Each attachment is a File listed in the root's hasPart after the workspace file, with dlc:attachment true (see below).
• Each activity of the workspace ledger is a CreateAction, or an UpdateAction for an in-place recomputation. Its instrument lists a SoftwareApplication for the operation contract (dlc:operationId, dlc:contractVersion) when the call names an operation (a replayable activity, whether or not the reader's computation library qualifies that contract version for replay), or for the informative implementation (no operation identifier) when it is opaque, plus one for the DataLab edition that ran it. object and result list its input and output states; because JSON-LD arrays are unordered, the roles are also given as dlc:RoleBinding entities with an explicit position (dlc:inputs, dlc:outputs). Parameters are PropertyValue entities (dlc:parameters) with a lossless dlc:jsonValue (see Parameters). An activity whose optional output had no result (e.g. a detection that found nothing) has no result.
• Each state is a CreativeWork with its object identifier, its fingerprint and, when present in workspace.h5, its HDF5 path (dlc:locator).
• Each execution environment is a dlc:Environment whose softwareRequirements list the package versions.

Manifest fingerprint

dlc:manifestFingerprint is sha256: followed by the lowercase hexadecimal SHA-256 of the RFC 8785 (JCS) canonical JSON of the manifest, computed without the dlc:manifestFingerprint property itself.

State fingerprint datalab-signal-v1

Used for a signal whose rows share one dtype (e.g. float64 signals). The payload is the UTF-8 JCS form of {"scheme": "datalab-signal-v1", "dtype": "<f8", "length": N, "rows": ["x", "y"], "units": {"x": "s", "y": ""}} (little-endian NumPy dtype string, missing units as ""), one 0x00 byte, then each row (x, y, then dx and dy when present and not all NaN) as contiguous little-endian bytes. The value is sha256: followed by the lowercase hexadecimal digest of the payload.

State fingerprint datalab-signal-v2

Used for a signal whose rows do not share one dtype, e.g. float32 or complex values with a float64 X row. The payload is the UTF-8 JCS form of {"scheme": "datalab-signal-v2", "length": N, "rows": [{"name": "x", "dtype": "<f8"}, {"name": "y", "dtype": "<f4"}], "units": {"x": "s", "y": ""}}, one 0x00 byte, then each row of the descriptor, in that order, as contiguous little-endian bytes of its own dtype. The scheme is chosen from the content alone, so that an object and its saved rows get the same record; a signal fingerprinted with datalab-signal-v1 keeps that scheme.

State fingerprint datalab-image-v1

The payload is the UTF-8 JCS form of {"scheme": "datalab-image-v1", "dtype": "<u2", "shape": [ny, nx], "coords": {...}, "units": {"x": "mm", "y": "mm", "z": ""}}, where coords is {"uniform": {"x0": ..., "y0": ..., "dx": ..., "dy": ...}} or {"nonuniform": {"x": nx, "y": ny}}, one 0x00 byte, the data in C order as little-endian bytes (any numeric dtype, complex included), then, for non-uniform coordinates, the X and Y coordinates as little-endian float64. The value is sha256: followed by the lowercase hexadecimal digest of the payload.

Regions of interest

A state whose object has a region of interest records its definition (masks derive from it) and the sha256: digest of its RFC 8785 canonical JSON next to the fingerprint, which covers the data, coordinates and units only; the manifest shows the definition as dlc:roi. A state is the same only when both its fingerprint and its region of interest (or the absence of one) match: a saved object whose data are unchanged but whose region of interest differs is reported as altered.

Analysis results

An analysis result is an artifact stored under a key of its object. The artifact records the sha256: digest of the RFC 8785 canonical JSON of the stored value (arrays as nested lists, non-finite numbers as {"$float": "NaN" | "Infinity" | "-Infinity"}), shown as dlc:resultDigest. When a later analysis stores a different value under the same key, the earlier artifact is reported as replaced, never as the later value.

Parameters

The parameters of a call are plain JSON values written by the parameter codec of the computation library (Sigima codec version 2): a parameter set is an object mapping item names to values (choices by their stable key); values without a lossless JSON form use single-key typed objects: {"$float": "NaN" | "Infinity" | "-Infinity"}, {"$complex": [re, im]}, {"$tuple": [...]}, {"$dict": {...}}, {"$date": "YYYY-MM-DD"}, {"$datetime": "..."} (ISO 8601) and {"$ndarray": {"dtype": "<f8", "shape": [...], "base64": "..."}} (raw little-endian C-order bytes). An operation taking several parameter arguments gets one entry per argument name, whose value is a parameter set, a list of parameter sets (e.g. several regions of interest) or null. A reader decodes them against the parameter classes of its own operation contract; no class, expression or pickle is ever read from a document.

Execution context

The context of an activity records what its result depends on besides inputs and parameters: the X-alignment policy applied to signals on different grids (identifier, version and outcome), the global options read by the computation (e.g. auto_normalize_kernel) and, since state fingerprints do not cover metadata, the sha256: digest of each input metadata value the computation reads (role, input position, key; e.g. the fit parameters read by evaluate_fit). A replay applies the recorded context, never the reader's current settings; it refuses an unknown policy, a missing option or unrecorded metadata, and an input whose metadata read changed since the activity (input_changed).

Verification and comparison rules

An edition verifies a replayable activity by recomputing it, on copies of its inputs, as a separate candidate (never inserted in the workspace, the ledger unchanged), then compares every output of the operation with the recorded one, by role: signals and images on their rows, regions of interest and units; analysis results on their values and their structure (kinds, column names, ROI indices); an output recorded without result must have no result again. The comparison rule is the one declared by the qualification of the operation: exact by default (same rows, dtype and shape, equal values, NaN at the same positions, both parts of complex values), or a tolerance (atol, rtol per output, with its justification) for computations whose last bits depend on the numerical libraries, which gives within_tolerance with the observed errors and may require identical results within an identical environment. A missing, altered or replaced reference gives not_verified. The report lists each output (outputs); its verdict is the most severe of theirs (exact, within_tolerance, not_verified, different).

Legacy activities

A workspace saved without provenance may hold, in each processed object's metadata, a record of the processing that produced it. Such a record becomes an action with dlc:origin legacy_metadata, and nothing missing is invented: its inputs bind objects (dlc:objectUuid) whose state at execution time is unknown, it has no parameters, no environment and no edition, it is never replayable, and dlc:legacy keeps the record as stored with the issues found at import (missing ancestry, unavailable inputs, unsupported operations, version warnings, unreadable parameters, unidentified analysis results). It claims no result (result and dlc:outputs are empty): the state of the holding object observed when the workspace was opened is named by dlc:observedState, a state that no action produced, since the data may have changed after the processing without the record changing.

Revisions

A capsule may name one parent revision with dlc:revisionOf, the manifest fingerprint of the parent. The property is part of the child's own fingerprint and must differ from it. Revisions form a chain of parents; a capsule never changes once published, a new revision is a new capsule. Comparing two revisions sorts the differences into scientific changes (activities, parameters, states, fingerprints, regions of interest), presentation changes (name, description, licence, date), storage changes (SHA-256 and size of the workspace file, whether it is included or referenced, where states are stored), environment changes (environments, package versions) and attachment changes (see below). Different workspace bytes do not show a scientific change: saving the same content again gives another file. When nothing differs in the recorded activities and states but the files differ, scientific equality is not concluded, since content outside the recorded provenance is not compared.

Referenced workspaces

Instead of including workspace.h5, a capsule may reference its workspace: the root's hasPart then names a File whose @id is an absolute IRI, with dlc:referenced true, sha256 and contentSize, and the archive holds the manifest only. Readers must not fetch the IRI on their own: the workspace is obtained by an explicit choice of the user or the application, and its bytes must match the SHA-256 and size of the manifest before use. An archive holding workspace.h5 for a referenced workspace is invalid.

Attachments

A capsule may carry a few light files chosen by its author (a protocol, notes, a small table). Each one is stored, uncompressed, as attachments/<name>, where the name has 1 to 100 letters, digits, ., _ or - and starts with a letter or a digit. It is declared in the manifest as a File whose @id is that path, with name, encodingFormat (a media type), contentSize, sha256, an optional description and dlc:attachment true. Attachments are part of the manifest fingerprint. Readers must refuse an undeclared, missing or altered attachment. They must hand attachments over as data and write one to disk only on an explicit request of the user; they must never open, interpret or execute one. No activity uses an attachment, but its content may matter scientifically (a protocol, a calibration certificate): comparing revisions lists attachment changes in their own group, without interpreting them. The reference library accepts at most 8 attachments of 16 MiB each.

Reviews

A review is a separate JSON document (review schema) about one revision: its target is the reviewed manifest fingerprint; it names the reviewer (and optionally a persistent identifier), a date, a statement, an optional verdict (endorsed, disputed or comment) and the verification reports it relies on (activity and verdict). Its review_fingerprint is the sha256: digest of its RFC 8785 canonical JSON without that property. A review never modifies the capsule; it does not authenticate the reviewer. Verification reports follow the report schema.

Mapping to W3C PROV

The manifest maps to PROV-O as follows; the mapping is informative.

Identity, compatibility and restoration

• A state is the same content as another when their fingerprints and regions of interest match; it is stored in the workspace when it has a dlc:locator and the object found there still matches. A capsule revision is identified by its manifest fingerprint, a review by its review fingerprint, an environment by the digest of its record.
• Readers refuse versions they do not know (profile, ledger, report, review, fingerprint scheme). A fingerprint scheme name is never reused for another payload.
• A call that names an operation contract keeps it, whether or not the contract is qualified for replay: the activity is replayable, and a reader recomputes it only when its own computation library qualifies that contract version (otherwise it is refused, with the reason). Opaque activities (no operation identity) and legacy activities are never recomputed. Verifying a capsule checks its integrity and computes nothing.

The full rules are in doc/rules.md of the reference implementation.

Schemas and contexts

Published with this version of the profile (SHA-256 of the files as published):

A fixture corpus of valid and invalid capsules, with the expected results of the reference command line, accompanies the reference implementation.