satpy.readers package

Subpackages

Submodules

Module contents

Shared objects of the various reader classes.

class satpy.readers.FSFile(file, fs=None)[source]

Bases: PathLike

Implementation of a PathLike file object, that can be opened.

Giving the filenames to Scene with valid transfer protocols will automatically use this class so manual usage of this class is needed mainly for fine-grained control.

This class is made to be used in conjuction with fsspec or s3fs. For example:

from satpy import Scene

import fsspec
filename = 'noaa-goes16/ABI-L1b-RadC/2019/001/17/*_G16_s20190011702186*'

the_files = fsspec.open_files("simplecache::s3://" + filename, s3={'anon': True})

from satpy.readers import FSFile
fs_files = [FSFile(open_file) for open_file in the_files]

scn = Scene(filenames=fs_files, reader='abi_l1b')
scn.load(['true_color_raw'])

Initialise the FSFile instance.

Parameters:

  • file (str, Pathlike, or OpenFile) – String, object implementing the os.PathLike protocol, or an fsspec.OpenFile instance. If passed an instance of fsspec.OpenFile, the following argument fs has no effect.

  • fs (fsspec filesystem, optional) – Object implementing the fsspec filesystem protocol.

_abc_impl = <_abc._abc_data object>
_update_with_fs_open_kwargs(user_kwargs)[source]

Complement keyword arguments for opening a file via file system.

property fs

Return the underlying private filesystem attribute.

open(*args, **kwargs)[source]

Open the file.

This is read-only.

satpy.readers._assign_files_to_readers(files_to_sort, reader_names, reader_kwargs)[source]

Assign files to readers.

Given a list of file names (paths), match those to reader instances.

Internal helper for group_files.

Parameters:

  • files_to_sort (Collection[str]) – Files to assign to readers.

  • reader_names (Collection[str]) – Readers to consider

  • reader_kwargs (Mapping)

Returns:

Mapping[str, Tuple[reader, Set[str]]] Mapping where the keys are reader names and the values are tuples of (reader_configs, filenames).

satpy.readers._check_reader_instances(reader_instances)[source]
satpy.readers._check_remaining_files(remaining_filenames)[source]
satpy.readers._early_exit(filenames, reader)[source]
satpy.readers._filter_groups(groups, missing='pass')[source]

Filter multi-reader group-files behavior.

Helper for group_files. When group_files is called with multiple readers, make sure that the desired behaviour for missing files is enforced: if missing is "raise", raise an exception if at least one group has at least one reader without files; if it is "skip", remove those. If it is "pass", do nothing. Yields groups to be kept.

Parameters:

  • groups (List[Mapping[str, List[str]]]) – groups as found by group_files.

  • missing (str) – String controlling behaviour, see documentation above.

Yields:

Mapping[str:, List[str]] – groups to be retained

satpy.readers._get_compression(file)[source]
satpy.readers._get_file_keys_for_reader_files(reader_files, group_keys=None)[source]

From a mapping from _assign_files_to_readers, get file keys.

Given a mapping where each key is a reader name and each value is a tuple of reader instance (typically FileYAMLReader) and a collection of files, return a mapping with the same keys, but where the values are lists of tuples of (keys, filename), where keys are extracted from the filenames according to group_keys and filenames are the names those keys were extracted from.

Internal helper for group_files.

Returns:

Mapping[str, List[Tuple[Tuple, str]]], as described.

satpy.readers._get_fs_open_kwargs(file)[source]

Get keyword arguments for opening a file via file system.

For example compression.

satpy.readers._get_keys_with_empty_values(grp)[source]

Find mapping keys where values have length zero.

Helper for _filter_groups, which is in turn a helper for group_files. Given a mapping key -> Collection[Any], return the keys where the length of the collection is zero.

Parameters:

grp (Mapping[Any, Collection[Any]]) – dictionary to check

Returns:

set of keys

satpy.readers._get_loadables_for_reader_config(base_dir, reader, sensor, reader_configs, reader_kwargs, fs)[source]

Get loadables for reader configs.

Helper for find_files_and_readers.

Parameters:

  • base_dir – as for find_files_and_readers

  • reader – as for find_files_and_readers

  • sensor – as for find_files_and_readers

  • reader_configs – reader metadata such as returned by configs_for_reader.

  • reader_kwargs – Keyword arguments to be passed to reader.

  • fs (FileSystem) – as for find_files_and_readers

satpy.readers._get_reader_and_filenames(reader, filenames)[source]
satpy.readers._get_reader_kwargs(reader, reader_kwargs)[source]

Help load_readers to form reader_kwargs.

Helper for load_readers to get reader_kwargs and reader_kwargs_without_filter in the desirable form.

satpy.readers._get_sorted_file_groups(all_file_keys, time_threshold)[source]

Get sorted file groups.

Get a list of dictionaries, where each list item consists of a dictionary mapping a tuple of keys to a mapping of reader names to files. The files listed in each list item are considered to be grouped within the same time.

Parameters:

  • all_file_keys

  • _get_file_keys_for_reader_files (as returned by)

  • time_threshold – temporal threshold

Returns:

List[Mapping[Tuple, Mapping[str, List[str]]]], as described

Internal helper for group_files.

satpy.readers.available_readers(as_dict=False, yaml_loader=<class 'yaml.loader.UnsafeLoader'>)[source]

Available readers based on current configuration.

Parameters:

  • as_dict (bool) – Optionally return reader information as a dictionary. Default: False.

  • yaml_loader (Optional[Union[yaml.BaseLoader, yaml.FullLoader, yaml.UnsafeLoader]]) – The yaml loader type. Default: yaml.UnsafeLoader.

Returns:

List of available reader names. If as_dict is True then a list of dictionaries including additionally reader information is returned.

Return type:

Union[list[str], list[dict]]

satpy.readers.configs_for_reader(reader=None)[source]

Generate reader configuration files for one or more readers.

Parameters:

reader (Optional[str]) – Yield configs only for this reader

Returns: Generator of lists of configuration files

satpy.readers.find_files_and_readers(start_time=None, end_time=None, base_dir=None, reader=None, sensor=None, filter_parameters=None, reader_kwargs=None, missing_ok=False, fs=None)[source]

Find files matching the provided parameters.

Use start_time and/or end_time to limit found filenames by the times in the filenames (not the internal file metadata). Files are matched if they fall anywhere within the range specified by these parameters.

Searching is NOT recursive.

Files may be either on-disk or on a remote file system. By default, files are searched for locally. Users can search on remote filesystems by passing an instance of an implementation of fsspec.spec.AbstractFileSystem (strictly speaking, any object of a class implementing a glob method works).

If locating files on a local file system, the returned dictionary can be passed directly to the Scene object through the filenames keyword argument. If it points to a remote file system, it is the responsibility of the user to download the files first (directly reading from cloud storage is not currently available in Satpy).

The behaviour of time-based filtering depends on whether or not the filename contains information about the end time of the data or not:

  • if the end time is not present in the filename, the start time of the filename is used and has to fall between (inclusive) the requested start and end times

  • otherwise, the timespan of the filename has to overlap the requested timespan

Example usage for querying a s3 filesystem using the s3fs module:

>>> import s3fs, satpy.readers, datetime
>>> satpy.readers.find_files_and_readers(
...     base_dir="s3://noaa-goes16/ABI-L1b-RadF/2019/321/14/",
...     fs=s3fs.S3FileSystem(anon=True),
...     reader="abi_l1b",
...     start_time=datetime.datetime(2019, 11, 17, 14, 40))
{'abi_l1b': [...]}
Parameters:

  • start_time (datetime) – Limit used files by starting time.

  • end_time (datetime) – Limit used files by ending time.

  • base_dir (str) – The directory to search for files containing the data to load. Defaults to the current directory.

  • reader (str or list) – The name of the reader to use for loading the data or a list of names.

  • sensor (str or list) – Limit used files by provided sensors.

  • filter_parameters (dict) – Filename pattern metadata to filter on. start_time and end_time are automatically added to this dictionary. Shortcut for reader_kwargs[‘filter_parameters’].

  • reader_kwargs (dict) – Keyword arguments to pass to specific reader instances to further configure file searching.

  • missing_ok (bool) – If False (default), raise ValueError if no files are found. If True, return empty dictionary if no files are found.

  • fs (fsspec.spec.AbstractFileSystem) – Optional, instance of implementation of fsspec.spec.AbstractFileSystem (strictly speaking, any object of a class implementing .glob is enough). Defaults to searching the local filesystem.

Returns:

Dictionary mapping reader name string to list of filenames

Return type:

dict

satpy.readers.get_valid_reader_names(reader)[source]

Check for old reader names or readers pending deprecation.

satpy.readers.group_files(files_to_sort, reader=None, time_threshold=10, group_keys=None, reader_kwargs=None, missing='pass')[source]

Group series of files by file pattern information.

By default this will group files by their filename start_time assuming it exists in the pattern. By passing the individual dictionaries returned by this function to the Scene classes’ filenames, a series Scene objects can be easily created.

Parameters:

  • files_to_sort (iterable) – File paths to sort in to group

  • reader (str or Collection[str]) – Reader or readers whose file patterns should be used to sort files. If not given, try all readers (slow, adding a list of readers is strongly recommended).

  • time_threshold (int) – Number of seconds used to consider time elements in a group as being equal. For example, if the ‘start_time’ item is used to group files then any time within time_threshold seconds of the first file’s ‘start_time’ will be seen as occurring at the same time.

  • group_keys (list or tuple) – File pattern information to use to group files. Keys are sorted in order and only the first key is used when comparing datetime elements with time_threshold (see above). This means it is recommended that datetime values should only come from the first key in group_keys. Otherwise, there is a good chance that files will not be grouped properly (datetimes being barely unequal). Defaults to a reader’s group_keys configuration (set in YAML), otherwise ('start_time',). When passing multiple readers, passing group_keys is strongly recommended as the behaviour without doing so is undefined.

  • reader_kwargs (dict) – Additional keyword arguments to pass to reader creation.

  • missing (str) – Parameter to control the behavior in the scenario where multiple readers were passed, but at least one group does not have files associated with every reader. Valid values are "pass" (the default), "skip", and "raise". If set to "pass", groups are passed as-is. Some groups may have zero files for some readers. If set to "skip", groups for which one or more readers have zero files are skipped (meaning that some files may not be associated to any group). If set to "raise", raise a FileNotFoundError in case there are any groups for which one or more readers have no files associated.

Returns:

List of dictionaries mapping ‘reader’ to a list of filenames. Each of these dictionaries can be passed as filenames to a Scene object.

satpy.readers.load_reader(reader_configs, **reader_kwargs)[source]

Import and setup the reader from reader_info.

satpy.readers.load_readers(filenames=None, reader=None, reader_kwargs=None)[source]

Create specified readers and assign files to them.

Parameters:

  • filenames (iterable or dict) – A sequence of files that will be used to load data from. A dict object should map reader names to a list of filenames for that reader.

  • reader (str or list) – The name of the reader to use for loading the data or a list of names.

  • reader_kwargs (dict) – Keyword arguments to pass to specific reader instances. This can either be a single dictionary that will be passed to all reader instances, or a mapping of reader names to dictionaries. If the keys of reader_kwargs match exactly the list of strings in reader or the keys of filenames, each reader instance will get its own keyword arguments accordingly.

Returns: Dictionary mapping reader name to reader instance

satpy.readers.open_file_or_filename(unknown_file_thing, mode=None)[source]

Try to open the provided file “thing” if needed, otherwise return the filename or Path.

This wraps the logic of getting something like an fsspec OpenFile object that is not directly supported by most reading libraries and making it usable. If a pathlib.Path object or something that is not open-able is provided then that object is passed along. In the case of fsspec OpenFiles their .open() method is called and the result returned.

satpy.readers.read_reader_config(config_files, loader=<class 'yaml.loader.UnsafeLoader'>)[source]

Read the reader config_files and return the extracted reader metadata.