Store formats

What geoML writes to disk, for programs that read it without geoML: a container (to_zarr, read back by open, which takes mode="r" to refuse every write into the store) and a mesh set (MeshSet.to_zarr, read back by MeshSet.open, always read-only). Both are Zarr version 3 groups written by zarr-python 3. Since 0.8.7 every numeric array wider than a byte is written bytes, then numcodecs.shuffle at the element’s width, then zstd; one-byte arrays, and every array written earlier, bytes then zstd. Read each array’s dtype, chunk shape and codecs from its own metadata rather than assuming them.

Attributes and versions

geoML writes its root attributes under keys starting with geoml: geoml on a container, geoml_meshset on a mesh set and geoml_model on a saved model. Every other root attribute belongs to whoever wrote it. Rewriting a store keeps those and replaces the arrays and groups.

Each store records a format number. It changes when a reader of the old layout would misread the new one: a group moved or renamed, a field removed, or a field given another meaning. An added field keeps the number, so a reader ignores the fields it does not know. JSON has no NaN; a missing number is written as null.

A mesh

Every mesh is a container of its own, which Solid3D.open reads alone.

Array	Content
`_coordinates`	`float64`, `(n, 3)`: the vertices
`_triangles`	integer, `(m, 3)`: each triangle’s vertex indices
`_normals`	`float64`, `(n, 3)`: one normal per vertex

A Solid3D’s triangles face outwards: counter-clockwise seen from outside.

The root attribute geoml holds geoml_format (2), container, and metadata and variables, both empty for a mesh. container["class"] is Solid3D, Surface3D, DTM3D or Mesh3D. container["provenance"], when present, records how the mesh was made. In a mesh set it holds:

source, the contoured column’s path;
value, the level contoured, and key, the set’s key;
close, supersample and simplify, as below;
limits and exclude, the cuts applied;
nudge, how far off its level the contour was retried, zero when it closed at the level itself;
realization, null for the prediction’s body.

A mesh set

Format 2: geoml_meshset["format"]. Format 1, written before geoML 0.7.0, is read too and was always a finished set.

<store>/
    prediction/<group>/          the prediction's body for each key
    simulations/<n>/<group>/     realization n's body for each key
    limits/<name>/               each limit, as given
    excluded/<name>/             each exclusion, as given

Every leaf group is a mesh. <group> is a key’s group name, listed in groups in the order of keys. <n> is a realization number from numbers, written in decimal. A realization missing from numbers was not stored; a body that could not be made is listed in failures.

The root attribute geoml_meshset holds:

Field	Meaning
`format`	The layout’s number, 2
`complete`	False while a build is still writing realizations, or where one stopped part way; absent at format 1, which means true
`kind`	`"cutoff"` for grade shells, `"category"` for one body per category
`path`	The contoured column’s path in its container
`name`, `unit`	The variable’s name, and its unit or null
`keys`	The cut-offs, as numbers, or the category names, in order
`groups`	Each key’s group name, in the order of `keys`
`levels`	The level each key was contoured at
`close`	The side of its level a body keeps: `"above"` or `"below"`
`supersample`, `simplify`	The contour’s extra refinement levels, and its error budget or null
`rule`	How a categorical realization picks its winner; null for grades
`limits`, `excluded`	The names of the limits and exclusions
`provenance`	The set’s own provenance
`realization`	Null for a whole set; a number for one realization written alone
`numbers`	The realizations stored
`summary`	The prediction’s bodies, one value per key: `volume`, `raw` (the volume before any cut), `pieces`, `largest` (the largest piece’s share of the volume) and `triangles`; plus `taken`, the volume each cut removed, one list per cut
`measures`	The same for every realization: one table per name among `volume`, `raw`, `pieces`, `largest`, `triangles`, `gained`, `lost` and `nudge`, a row per entry of `numbers` and a column per key; `gained` and `lost` are the volume gained and lost against the prediction’s body
`taken`	The volume each cut removed from each realization’s bodies, indexed `[cut][realization][key]`, the cuts being the limits and then the exclusions
`nudge`	Per key, how far off its level the prediction’s body was retried
`failures`	`{"realization", "key", "error"}` for each body that could not be made
`repairs`	`{"keys", "values"}`, the volume `repair` removed per key, or null
`corners`, `shift`	The model’s box and the frame the cuts were computed in; a reader needs neither

The description is written before the first realization and again after each one, so a set read while its build is still running – or after one was cancelled or killed – opens and describes what it holds. numbers, measures and taken then cover the realizations actually stored, and complete is false. Before 0.7.0 the description was written once, at the end, and until then there was no geoml_meshset attribute to read at all.

Stores written before geoML 0.6.13 lack groups. Their group names spell each cut-off as Python’s str(float(key)) does, as in 30.0 or 1e-05, and each category by its name, unless the name reads as a number.

A container

Format 2: geoml["geoml_format"]. The root attribute geoml holds geoml_format, container, metadata and variables. container names the class and what rebuilds its geometry: a point cloud’s coordinates are the array _coordinates, and a grid’s are generated from its start, n and step. Each metadata column and each variable’s column gives the array holding it under key, such as _metadata/HOLEID or zn/prediction, and a column of text is stored as integer codes into its labels. Follow the keys rather than building the paths. Stores at format 1 are not read. What each variable’s columns hold – a value or an uncertainty, and on what scale – is in the catalogue’s variable_types. Every column of a categorical variable codes its classes in the variable’s order of labels since 0.8.0, a class measured outside them appended after.

A container a model predicted into, where it holds measurements, carries two more families of metadata since 0.8.0: pit_<variable> (or pit_<variable>_<component> for a part of a vector variable), where each measurement falls in the predictive distribution of a measurement, from 0 to 1; and warped_<variable>_<i>, the measurements through the likelihood’s warping, one column per column the warping gives out. Both are missing where there is no measurement.

A variable’s realizations are one (n_locations, n_sim) array, and since 0.7.0 a wide one is chunked on both axes: the location axis as always, and the realization axis in groups of ten, so that reading one realization reads about a tenth of the array rather than all of it. Read the chunk shape from the array’s own metadata – a store written earlier keeps the chunks it was written with, and a narrow array is still one chunk across. Realizations are written as float32 since 0.8.7 (float64 under geoml.set_realization_dtype("float64"), and in every store written earlier); the prediction, its variances and every other column stay float64.