Progress and cancelling

What a long call is doing, and how to stop it. geoml.progress(callback) is a context manager: every long call made inside it reports what it has finished, and the callback raising is how the caller cancels.

import geoml

def watch(event):
    print(event.task, event.done, "of", event.total)
    if stop_file.exists():
        raise geoml.Cancelled()

with geoml.progress(watch):
    model.train_full(max_iter=2000)
    model.predict(blocks, n_sim=50)

The callback travels on a context variable rather than an argument, because the long calls nest: a refinement predicts, a cross-validation trains and predicts. An event names the enclosing tasks in within, so a refinement’s predictions can be told from a bare one.

done counts units finished, never units attempted, which is what makes the number worth acting on: it is what a cancel at that moment would leave behind.

Resuming a cancelled prediction

A location’s values do not depend on what else is in its batch, so the batches a cancelled prediction finished are exactly as they would have been. unpredicted() names the rest, on any container:

model.predict(blocks, n_sim=50)                 # cancelled part way
model.predict(blocks, n_sim=50,
              where=blocks.unpredicted())       # finishes the rest

The answer is read off the missing values rather than remembered from a call, so it stays true however the container was arrived at – reopened from a store, subsetted, carried into. Each kind of variable declares the column that marks it: a grade has a prediction, a rock type an entropy, a vector variable an uncertainty.

A mesh set cancelled part way leaves a store that opens and says it is incomplete, holding the realizations it finished.

geoml.progress(callback)

Report what long calls are doing, and cancel them by raising.

Every call made inside the block reports through callback, which receives one Progress per unit finished. Raising from the callback cancels: the exception travels out of the geoML call untouched, and what was finished stays finished.

Parameters:

callback (Callable[[Progress], None] | None) – Called with one Progress argument. None disables reporting for the block, which is how a caller silences an inner call inside an outer reporting one.

Notes

What a cancelled call leaves behind, per task:

• train: the model at the last completed iteration or batch, its parameters refreshed and its training_log appended to, saveable and trainable again from there.

• predict: the batches that finished, written into the target; unpredicted() names the locations left, and predicting only those gives what predicting the lot would have.

• refine: nothing. Each pass builds a new block model and only the return hands it over, so a cancel loses what the passes made. Write the loop out instead – predict, needs_splitting, split – which is the documented way to stop part way and keep the model.

• cross_validate: nothing – the scores are of the whole, so a cancelled run has none to give.

• mesh_set: the bodies and realizations already contoured, recorded in the store, which opens and says it is incomplete. A realization still being contoured when the cancel comes is discarded with its worker; only what was recorded survives.

Reporting is per finished unit, so done is what would survive a cancel at that moment rather than what is being attempted.

Examples

def watch(event):
    print(event.task, event.done, "of", event.total)
    if stop_file.exists():
        raise geoml.Cancelled()

with geoml.progress(watch):
    model.train_full(max_iter=2000)
    model.predict(blocks, n_sim=50)

class geoml.Progress(task, done, total=None, unit='step', bound=None, within=())

Bases: object

One report from a call running inside geoml.progress().

task

What is running: “train”, “predict”, “refine”, “cross_validate” or “mesh_set”.

Type:

str

done

Units finished, counted in whatever unit names. Never decreases within one task.

Type:

int

total

Units expected, or None where the count is not known in advance. A cap rather than a promise wherever training may stop early.

Type:

int | None

unit

What done counts: “iteration”, “epoch”, “batch”, “pass”, “fold”, “body” or “realization”.

Type:

str

bound

The evidence lower bound, on the tasks that have one, else None.

Type:

float | None

within

The enclosing tasks, outermost first – (“refine”,) for the predictions a refinement makes, () at the top.

Type:

tuple[str, …]

task: str

done: int

total: int | None = None

unit: str = 'step'

bound: float | None = None

within: tuple[str, ...] = ()

exception geoml.Cancelled

Bases: Exception

Raised by a progress callback to stop the call reporting to it.

geoML never raises this itself, and never catches it: it is a name for the caller to raise so that the cancel reads as one at the other end. Any exception cancels – this one only says why.