satpy.aux_download module

Functions and utilities for downloading ancillary data.

class satpy.aux_download.DataDownloadMixin

Bases: object

Mixin class for Satpy components to download files.

This class simplifies the logic needed to download and cache data files needed for operations in a Satpy component (readers, writers, etc). It does this in a two step process where files that might be downloaded are “registered” and then “retrieved” when they need to be used.

To use this class include it as one of the subclasses of your Satpy component. Then in the __init__ method, call the register_data_files function during initialization.

Note

This class is already included in the FileYAMLReader and Writer base classes. There is no need to define a custom class.

The below code is shown as an example:

from satpy.readers.yaml_reader import AbstractYAMLReader
from satpy.aux_download import DataDownloadMixin

class MyReader(AbstractYAMLReader, DataDownloadMixin):
    def __init__(self, *args, **kwargs):
        super().__init__(*args, **kwargs)
        self.register_data_files()

This class expects data files to be configured in either a self.info['data_files'] (standard for readers/writers) or self.config['data_files'] list. The data_files item itself is a list of dictionaries. This information can also be passed directly to register_data_files for more complex cases. In YAML, for a reader, this might look like this:

reader:
    name: abi_l1b
    short_name: ABI L1b
    long_name: GOES-R ABI Level 1b
    ... other metadata ...
    data_files:
      - url: "https://example.com/my_data_file.dat"
      - url: "https://raw.githubusercontent.com/pytroll/satpy/main/README.rst"
        known_hash: "sha256:5891286b63e7745de08c4b0ac204ad44cfdb9ab770309debaba90308305fa759"
      - url: "https://raw.githubusercontent.com/pytroll/satpy/main/RELEASING.md"
        filename: "satpy_releasing.md"

In this example we register two files that might be downloaded. If known_hash is not provided or None (null in YAML) then the data file will not be checked for validity when downloaded. See register_file() for more information. You can optionally specify filename to define the in-cache name when this file is downloaded. This can be useful in cases when the filename can not be easily determined from the URL.

When it comes time to needing the file, you can retrieve the local path by calling ~satpy.aux_download.retrieve(cache_key) with the “cache key” generated during registration. These keys will be in the format: <component_type>/<filename>. For a reader this would be readers/satpy_release.md.

This Mixin is not the only way to register and download files for a Satpy component, but is the most generic and flexible. Feel free to use the register_file() and retrieve() functions directly. However, find_registerable_files() must also be updated to support your component (if files are not register during initialization).

DATA_FILE_COMPONENTS = {'composit': 'composites', 'corr': 'modifiers', 'modifi': 'modifiers', 'reader': 'readers', 'writer': 'writers'}

property _data_file_component_type

static _register_data_file(data_file_entry, comp_type)

register_data_files(data_files=None)

Register a series of files that may be downloaded later.

See DataDownloadMixin for more information on the assumptions and structure of the data file configuration dictionary.

satpy.aux_download._find_registerable_files_compositors(sensors=None)

Load all compositor configs so that files are registered.

Compositor objects should register files when they are initialized.

satpy.aux_download._find_registerable_files_readers(readers=None)

Load all readers so that files are registered.

satpy.aux_download._find_registerable_files_writers(writers=None)

Load all writers so that files are registered.

satpy.aux_download._generate_filename(filename, component_type)

satpy.aux_download._register_modifier_files(modifiers)

satpy.aux_download._retrieve_all_with_pooch(pooch_kwargs)

satpy.aux_download._retrieve_offline(data_dir, cache_key)

satpy.aux_download._should_download(cache_key)

Check if we’re running tests and can download this file.

satpy.aux_download.find_registerable_files(readers=None, writers=None, composite_sensors=None)

Load all Satpy components so they can be downloaded.

Parameters:

• readers (list or None) – Limit searching to these readers. If not specified or None then all readers are searched. If an empty list then no readers are searched.

• writers (list or None) – Limit searching to these writers. If not specified or None then all writers are searched. If an empty list then no writers are searched.

• composite_sensors (list or None) – Limit searching to composite configuration files for these sensors. If None then all sensor configs will be searched. If an empty list then no composites will be searched.

satpy.aux_download.register_file(url, filename, component_type=None, known_hash=None)

Register file for future retrieval.

This function only prepares Satpy to be able to download and cache the provided file. It will not download the file. See satpy.aux_download.retrieve() for more information.

Parameters:

• url (str) – URL where remote file can be downloaded.

• filename (str) – Filename used to identify and store the downloaded file as.

• component_type (str or None) – Name of the type of Satpy component that will use this file. Typically “readers”, “composites”, “writers”, or “enhancements” for consistency. This will be prepended to the filename when storing the data in the cache.

• known_hash (str) – Hash used to verify the file is downloaded correctly. See https://www.fatiando.org/pooch/v1.3.0/beginner.html#hashes for more information. If not provided then the file is not checked.

Returns:

Cache key that can be used to retrieve the file later. The cache key consists of the component_type and provided filename. This should be passed to satpy.aux_download_retrieve() when the file will be used.

satpy.aux_download.retrieve(cache_key, pooch_kwargs=None)

Download and cache the file associated with the provided cache_key.

Cache location is controlled by the config data_dir key. See Data Directory for more information.

Parameters:

• cache_key (str) – Cache key returned by register_file().

• pooch_kwargs (dict or None) – Extra keyword arguments to pass to pooch.Pooch.fetch().

Returns:

Local path of the cached file.

satpy.aux_download.retrieve_all(readers=None, writers=None, composite_sensors=None, pooch_kwargs=None)

Find cache-able data files for Satpy and download them.

The typical use case for this function is to download all ancillary files before going to an environment/system that does not have internet access.

Parameters:

• readers (list or None) – Limit searching to these readers. If not specified or None then all readers are searched. If an empty list then no readers are searched.

• writers (list or None) – Limit searching to these writers. If not specified or None then all writers are searched. If an empty list then no writers are searched.

• composite_sensors (list or None) – Limit searching to composite configuration files for these sensors. If None then all sensor configs will be searched. If an empty list then no composites will be searched.

• pooch_kwargs (dict) – Additional keyword arguments to pass to pooch fetch.

satpy.aux_download.retrieve_all_cmd(argv=None)

Call ‘retrieve_all’ function from console script ‘satpy_retrieve_all’.