satpy.writers.core.compute module

Utilities for writers.

satpy.writers.core.compute._flatten(results)[source]
satpy.writers.core.compute.compute_writer_results(results)[source]

Compute all the given dask graphs results so that the files are saved.

Parameters:

results (Iterable[list[Array | Delayed] | tuple[list[Array], list[Any]]]) – Iterable of dask collections resulting from calls to scn.save_datasets(..., compute=False). The iterable passed should always contain the results of the writer/save_datasets call, not be the results themselves. Each collection of results from a writer should be either a list of dask containers to be computed (ex. Array or Delayed) or a 2-element tuple where the first element is a list of dask Array objects and the second element is a list of target file-like objects supporting __setitem__ syntax. In the case of the 2-element tuple, these lists will be passed to dask.array.store() to write the Arrays to the targets.

Return type:

list[ndarray | str | PathLike | None]

Returns:

A list of the computed results. These are typically string or Path filenames that were created or None if the store function (see above) was called. If the dask collection to be computed is an array then the computed array will be in the returned list.

satpy.writers.core.compute.group_results_by_output_file(sources, targets)[source]

Group results by output file.

For writers that return sources and targets for compute=False, split the results by output file.

When not only the data but also GeoTIFF tags are dask arrays, then save_datasets(..., compute=False)` returns a tuple of flat lists, where the second list consists of a mixture of RIOTag and RIODataset objects (from trollimage). In some cases, we may want to get a seperate delayed object for each file; for example, if we want to add a wrapper to do something with the file as soon as it’s finished. This function unflattens the flat lists into a list of (src, target) tuples.

For example, to close files as soon as computation is completed:

>>> @dask.delayed
>>> def closer(obj, targs):
...     for targ in targs:
...         targ.close()
...     return obj
>>> (srcs, targs) = sc.save_datasets(writer="ninjogeotiff", compute=False, **ninjo_tags)
>>> for (src, targ) in group_results_by_output_file(srcs, targs):
...     delayed_store = da.store(src, targ, compute=False)
...     wrapped_store = closer(delayed_store, targ)
...     wrapped.append(wrapped_store)
>>> compute_writer_results(wrapped)

In the wrapper you can do other useful tasks, such as writing a log message or moving files to a different directory.

Warning

Adding a callback may impact runtime and RAM. The pattern or cause is unclear. Tests with FCI data show that for resampling with high RAM use (from around 15 GB), runtime increases when a callback is added. Tests with ABI or low RAM consumption rather show a decrease in runtime. More information, see these GitHub comments Users who find out more are encouraged to contact the Satpy developers with clues.

Parameters:
Returns:

List of Tuple(List[sources], List[targets]) with a length equal to the number of output files planned to be written by satpy.scene.Scene.save_datasets().

satpy.writers.core.compute.split_results(results)[source]

Split results.

Get sources, targets, and objects to be computed into separate lists from a list of results collected from one or more writers. This function will treat dask Arrays and Delayed objects with no targets as dask collections to be computed. Otherwise dask Arrays can be supplied with an associated target array-like file object and will be passed to dask.array.store().

We assume that the provided input is a list containing any combination of:

  1. List of dask Arrays (to be computed, not stored).

  2. List of Delayed objects.

  3. A two-element tuple where the first element is a collection of dask Arrays and the second is a collection of array-like file objects of the same size.

  4. A list made up of the above.

Return type:

tuple[list[Array], list[Any], list[Array | Delayed]]