# Store formats What geoML writes to disk, for programs that read it without geoML: a container (`to_zarr`, read back by `open`, which takes `mode="r"` to refuse every write into the store) and a mesh set (`MeshSet.to_zarr`, read back by `MeshSet.open`, always read-only). Both are Zarr version 3 groups written by zarr-python 3. Since 0.8.7 every numeric array wider than a byte is written `bytes`, then `numcodecs.shuffle` at the element's width, then `zstd`; one-byte arrays, and every array written earlier, `bytes` then `zstd`. Read each array's dtype, chunk shape and codecs from its own metadata rather than assuming them. ## Attributes and versions geoML writes its root attributes under keys starting with `geoml`: `geoml` on a container, `geoml_meshset` on a mesh set and `geoml_model` on a saved model. Every other root attribute belongs to whoever wrote it. Rewriting a store keeps those and replaces the arrays and groups. Each store records a format number. It changes when a reader of the old layout would misread the new one: a group moved or renamed, a field removed, or a field given another meaning. An added field keeps the number, so a reader ignores the fields it does not know. JSON has no NaN; a missing number is written as `null`. ## A mesh Every mesh is a container of its own, which `Solid3D.open` reads alone. | Array | Content | |---|---| | `_coordinates` | `float64`, `(n, 3)`: the vertices | | `_triangles` | integer, `(m, 3)`: each triangle's vertex indices | | `_normals` | `float64`, `(n, 3)`: one normal per vertex | A `Solid3D`'s triangles face outwards: counter-clockwise seen from outside. The root attribute `geoml` holds `geoml_format` (2), `container`, and `metadata` and `variables`, both empty for a mesh. `container["class"]` is `Solid3D`, `Surface3D`, `DTM3D` or `Mesh3D`. `container["provenance"]`, when present, records how the mesh was made. In a mesh set it holds: - `source`, the contoured column's path; - `value`, the level contoured, and `key`, the set's key; - `close`, `supersample` and `simplify`, as below; - `limits` and `exclude`, the cuts applied; - `nudge`, how far off its level the contour was retried, zero when it closed at the level itself; - `realization`, null for the prediction's body. ## A mesh set Format 2: `geoml_meshset["format"]`. Format 1, written before geoML 0.7.0, is read too and was always a finished set. ``` / prediction// the prediction's body for each key simulations/// realization n's body for each key limits// each limit, as given excluded// each exclusion, as given ``` Every leaf group is a mesh. `` is a key's group name, listed in `groups` in the order of `keys`. `` is a realization number from `numbers`, written in decimal. A realization missing from `numbers` was not stored; a body that could not be made is listed in `failures`. The root attribute `geoml_meshset` holds: | Field | Meaning | |---|---| | `format` | The layout's number, 2 | | `complete` | False while a build is still writing realizations, or where one stopped part way; absent at format 1, which means true | | `kind` | `"cutoff"` for grade shells, `"category"` for one body per category | | `path` | The contoured column's path in its container | | `name`, `unit` | The variable's name, and its unit or null | | `keys` | The cut-offs, as numbers, or the category names, in order | | `groups` | Each key's group name, in the order of `keys` | | `levels` | The level each key was contoured at | | `close` | The side of its level a body keeps: `"above"` or `"below"` | | `supersample`, `simplify` | The contour's extra refinement levels, and its error budget or null | | `rule` | How a categorical realization picks its winner; null for grades | | `limits`, `excluded` | The names of the limits and exclusions | | `provenance` | The set's own provenance | | `realization` | Null for a whole set; a number for one realization written alone | | `numbers` | The realizations stored | | `summary` | The prediction's bodies, one value per key: `volume`, `raw` (the volume before any cut), `pieces`, `largest` (the largest piece's share of the volume) and `triangles`; plus `taken`, the volume each cut removed, one list per cut | | `measures` | The same for every realization: one table per name among `volume`, `raw`, `pieces`, `largest`, `triangles`, `gained`, `lost` and `nudge`, a row per entry of `numbers` and a column per key; `gained` and `lost` are the volume gained and lost against the prediction's body | | `taken` | The volume each cut removed from each realization's bodies, indexed `[cut][realization][key]`, the cuts being the limits and then the exclusions | | `nudge` | Per key, how far off its level the prediction's body was retried | | `failures` | `{"realization", "key", "error"}` for each body that could not be made | | `repairs` | `{"keys", "values"}`, the volume `repair` removed per key, or null | | `corners`, `shift` | The model's box and the frame the cuts were computed in; a reader needs neither | The description is written before the first realization and again after each one, so a set read while its build is still running -- or after one was cancelled or killed -- opens and describes what it holds. `numbers`, `measures` and `taken` then cover the realizations actually stored, and `complete` is false. Before 0.7.0 the description was written once, at the end, and until then there was no `geoml_meshset` attribute to read at all. Stores written before geoML 0.6.13 lack `groups`. Their group names spell each cut-off as Python's `str(float(key))` does, as in `30.0` or `1e-05`, and each category by its name, unless the name reads as a number. ## A container Format 2: `geoml["geoml_format"]`. The root attribute `geoml` holds `geoml_format`, `container`, `metadata` and `variables`. `container` names the class and what rebuilds its geometry: a point cloud's coordinates are the array `_coordinates`, and a grid's are generated from its `start`, `n` and `step`. Each metadata column and each variable's column gives the array holding it under `key`, such as `_metadata/HOLEID` or `zn/prediction`, and a column of text is stored as integer codes into its `labels`. Follow the keys rather than building the paths. Stores at format 1 are not read. What each variable's columns hold -- a value or an uncertainty, and on what scale -- is in the catalogue's `variable_types`. Every column of a categorical variable codes its classes in the variable's order of `labels` since 0.8.0, a class measured outside them appended after. A container a model predicted into, where it holds measurements, carries two more families of metadata since 0.8.0: `pit_` (or `pit__` for a part of a vector variable), where each measurement falls in the predictive distribution of a measurement, from 0 to 1; and `warped__`, the measurements through the likelihood's warping, one column per column the warping gives out. Both are missing where there is no measurement. A variable's realizations are one `(n_locations, n_sim)` array, and since 0.7.0 a wide one is chunked on both axes: the location axis as always, and the realization axis in groups of ten, so that reading one realization reads about a tenth of the array rather than all of it. Read the chunk shape from the array's own metadata -- a store written earlier keeps the chunks it was written with, and a narrow array is still one chunk across. Realizations are written as float32 since 0.8.7 (float64 under `geoml.set_realization_dtype("float64")`, and in every store written earlier); the prediction, its variances and every other column stay float64.