satpy.readers.yaml_reader module

Base classes and utilities for all readers configured by YAML files.

class satpy.readers.yaml_reader.AbstractYAMLReader(config_dict)

Bases: object

Base class for all readers that use YAML configuration files.

This class should only be used in rare cases. Its child class FileYAMLReader should be used in most cases.

Load information from YAML configuration file about how to read data files.

_abc_impl = <_abc._abc_data object>

_build_id_permutations(dataset, id_keys)

Build each permutation/product of the dataset.

property all_dataset_ids

Get DataIDs of all datasets known to this reader.

property all_dataset_names

Get names of all datasets known to this reader.

property available_dataset_ids

Get DataIDs that are loadable by this reader.

property available_dataset_names

Get names of datasets that are loadable by this reader.

abstract property end_time

End time of the reader.

abstract filter_selected_filenames(filenames)

Filter provided filenames by parameters in reader configuration.

Returns: iterable of usable files

classmethod from_config_files(*config_files, **reader_kwargs)

Create a reader instance from one or more YAML configuration files.

get_dataset_key(key, **kwargs)

Get the fully qualified DataID matching key.

See satpy.readers.get_key for more information about kwargs.

abstract load(dataset_keys)

Load dataset_keys.

load_ds_ids_from_config()

Get the dataset ids from the config.

select_files_from_directory(directory=None, fs=None)

Find files for this reader in directory.

If directory is None or ‘’, look in the current directory.

Searches the local file system by default. Can search on a remote filesystem by passing an instance of a suitable implementation of fsspec.spec.AbstractFileSystem.

Parameters:

• directory (Optional[str]) – Path to search.

• fs (Optional[FileSystem]) – fsspec FileSystem implementation to use. Defaults to None, using local file system.

Returns:

list of strings describing matching files

select_files_from_pathnames(filenames)

Select the files from filenames this reader can handle.

property sensor_names

Names of sensors whose data is being loaded by this reader.

abstract property start_time

Start time of the reader.

supports_sensor(sensor)

Check if sensor is supported.

Returns True is sensor is None.

class satpy.readers.yaml_reader.FileYAMLReader(config_dict, filter_parameters=None, filter_filenames=True, **kwargs)

Bases: AbstractYAMLReader, DataDownloadMixin

Primary reader base class that is configured by a YAML file.

This class uses the idea of per-file “file handler” objects to read file contents and determine what is available in the file. This differs from the base AbstractYAMLReader which does not depend on individual file handler objects. In almost all cases this class should be used over its base class and can be used as a reader by itself and requires no subclassing.

Set up initial internal storage for loading file data.

_abc_impl = <_abc._abc_data object>

static _assign_coords_from_dataarray(coords, ds)

Assign coords from the ds dataarray if needed.

_coords_cache: WeakValueDictionary = <WeakValueDictionary>

_file_handlers_available_datasets()

Generate a series of available dataset information.

This is done by chaining file handler’s satpy.readers.file_handlers.BaseFileHandler.available_datasets() together. See that method’s documentation for more information.

Returns:

Generator of (bool, dict) where the boolean tells whether the current dataset is available from any of the file handlers. The boolean can also be None in the case where no loaded file handler is configured to load the dataset. The dictionary is the metadata provided either by the YAML configuration files or by the file handler itself if it is a new dataset. The file handler may have also supplemented or modified the information.

_gather_ancillary_variables_ids(datasets)

Gather ancillary variables’ ids.

This adds/modifies the dataset’s ancillary_variables attr.

_get_coordinates_for_dataset_key(dsid)

Get the coordinate dataset keys for dsid.

_get_coordinates_for_dataset_keys(dsids)

Get all coordinates.

_get_file_handlers(dsid)

Get the file handler to load this dataset.

_get_lons_lats_from_coords(coords)

Get lons and lats from the coords list.

_load_ancillary_variables(datasets, **kwargs)

Load the ancillary variables of datasets.

_load_area_def(dsid, file_handlers, **kwargs)

Load the area definition of dsid.

static _load_dataset(dsid, ds_info, file_handlers, dim='y', **kwargs)

Load only a piece of the dataset.

_load_dataset_area(dsid, file_handlers, coords, **kwargs)

Get the area for dsid.

_load_dataset_data(file_handlers, dsid, **kwargs)

_load_dataset_with_area(dsid, coords, **kwargs)

Load dsid and its area if available.

_make_area_from_coords(coords)

Create an appropriate area with the given coords.

_make_swath_definition_from_lons_lats(lons, lats)

Make a swath definition instance from lons and lats.

_new_filehandler_instances(filetype_info, filename_items, fh_kwargs=None)

Generate new filehandler instances.

_new_filehandlers_for_filetype(filetype_info, filenames, fh_kwargs=None)

Create filehandlers for a given filetype.

_preferred_filetype(filetypes)

Get the preferred filetype out of the filetypes list.

At the moment, it just returns the first filetype that has been loaded.

property available_dataset_ids

Get DataIDs that are loadable by this reader.

static check_file_covers_area(file_handler, check_area)

Check if the file covers the current area.

If the file doesn’t provide any bounding box information or ‘area’ was not provided in filter_parameters, the check returns True.

create_filehandlers(filenames, fh_kwargs=None)

Organize the filenames into file types and create file handlers.

property end_time

End time of the latest file used by this reader.

static filename_items_for_filetype(filenames, filetype_info)

Iterate over the filenames matching filetype_info.

filter_fh_by_metadata(filehandlers)

Filter out filehandlers using provide filter parameters.

filter_filenames_by_info(filename_items)

Filter out file using metadata from the filenames.

Currently only uses start and end time. If only start time is available from the filename, keep all the filename that have a start time before the requested end time.

filter_selected_filenames(filenames)

Filter provided files based on metadata in the filename.

find_required_filehandlers(requirements, filename_info)

Find the necessary file handlers for the given requirements.

We assume here requirements are available.

Raises:

• KeyError, if no handler for the given requirements is available. –

• RuntimeError, if there is a handler for the given requirements, –

• but it doesn't match the filename info. –

get_dataset_key(key, available_only=False, **kwargs)

Get the fully qualified DataID matching key.

This will first search through available DataIDs, datasets that should be possible to load, and fallback to “known” datasets, those that are configured but aren’t loadable from the provided files. Providing available_only=True will stop this fallback behavior and raise a KeyError exception if no available dataset is found.

Parameters:

• key (str, float, DataID, DataQuery) – Key to search for in this reader.

• available_only (bool) – Search only loadable datasets for the provided key. Loadable datasets are always searched first, but if available_only=False (default) then all known datasets will be searched.

• kwargs – See satpy.readers.get_key() for more information about kwargs.

Returns:

Best matching DataID to the provided key.

Raises:

KeyError – if no key match is found.

load(dataset_keys, previous_datasets=None, **kwargs)

Load dataset_keys.

If previous_datasets is provided, do not reload those.

metadata_matches(sample_dict, file_handler=None)

Check that file metadata matches filter_parameters of this reader.

property sensor_names

Names of sensors whose data is being loaded by this reader.

sorted_filetype_items()

Sort the instance’s filetypes in using order.

property start_time

Start time of the earlier file used by this reader.

time_matches(fstart, fend)

Check that a file’s start and end time mtach filter_parameters of this reader.

update_ds_ids_from_file_handlers()

Add or modify available dataset information.

Each file handler is consulted on whether or not it can load the dataset with the provided information dictionary. See satpy.readers.file_handlers.BaseFileHandler.available_datasets() for more information.

class satpy.readers.yaml_reader.GEOFlippableFileYAMLReader(config_dict, filter_parameters=None, filter_filenames=True, **kwargs)

Bases: FileYAMLReader

Reader for flippable geostationary data.

Set up initial internal storage for loading file data.

_abc_impl = <_abc._abc_data object>

_load_dataset_with_area(dsid, coords, upper_right_corner='native', **kwargs)

Load dsid and its area if available.

class satpy.readers.yaml_reader.GEOSegmentYAMLReader(config_dict, filter_parameters=None, filter_filenames=True, **kwargs)

Bases: GEOFlippableFileYAMLReader

Reader for segmented geostationary data.

This reader pads the data to full geostationary disk if necessary.

This reader uses an optional pad_data keyword argument that can be passed to Scene.load() to control if padding is done (True by default). Passing pad_data=False will return data unpadded.

When using this class in a reader’s YAML configuration, segmented file types (files that may have multiple segments) should specify an extra expected_segments piece of file_type metadata. This tells this reader how many total segments it should expect when padding data. Alternatively, the file patterns for a file type can include a total_segments field which will be used if expected_segments is not defined. This will default to 1 segment.

Set up initial internal storage for loading file data.

_abc_impl = <_abc._abc_data object>

_get_empty_segment(**kwargs)

_get_new_areadef_for_padded_segment(area, filetype, seg_size, segment, padding_type)

_get_new_areadef_heights(previous_area, previous_seg_size, **kwargs)

_get_segments_areadef_with_later_padded(file_handlers, filetype, dsid, available_segments, expected_segments)

_get_y_area_extents_for_padded_segment(area, filetype, padding_type, seg_size, segment)

_load_area_def(dsid, file_handlers, pad_data=True)

Load the area definition of dsid with padding.

_load_area_def_with_padding(dsid, file_handlers)

Load the area definition of dsid with padding.

_load_dataset(dsid, ds_info, file_handlers, dim='y', pad_data=True)

Load only a piece of the dataset.

_pad_earlier_segments_area(file_handlers, dsid, area_defs)

Pad area definitions for missing segments that are earlier in sequence than the first available.

_pad_later_segments_area(file_handlers, dsid)

Pad area definitions for missing segments that are later in sequence than the first available.

_sort_segment_filehandler_by_segment_number()

create_filehandlers(filenames, fh_kwargs=None)

Create file handler objects and determine expected segments for each.

Additionally, sort the filehandlers by segment number to avoid issues with filenames where start_time or alphabetic sorting does not produce the correct order.

class satpy.readers.yaml_reader.GEOVariableSegmentYAMLReader(config_dict, filter_parameters=None, filter_filenames=True, **kwargs)

Bases: GEOSegmentYAMLReader

GEOVariableSegmentYAMLReader for handling segmented GEO products with segments of variable height.

This YAMLReader overrides parts of the GEOSegmentYAMLReader to account for formats where the segments can have variable heights. It computes the sizes of the padded segments using the information available in the file(handlers), so that gaps of any size can be filled as needed.

This implementation was motivated by the FCI L1c format, where the segments (called chunks in the FCI world) can have variable heights. It is however generic, so that any future reader can use it. The requirement for the reader is to have a method called get_segment_position_info, returning a dictionary containing the positioning info for each segment (see example in satpy.readers.fci_l1c_nc.FCIL1cNCFileHandler.get_segment_position_info()).

For more information on please see the documentation of satpy.readers.yaml_reader.GEOSegmentYAMLReader().

Initialise the GEOVariableSegmentYAMLReader object.

_abc_impl = <_abc._abc_data object>

_collect_segment_position_infos(filetype)

_extract_segment_location_dicts(filetype)

_get_empty_segment(dim=None, idx=None, filetype=None)

_get_new_areadef_heights(previous_area, previous_seg_size, segment_n=None, filetype=None)

_initialise_segment_infos(filetype)

_segment_heights(filetype, grid_width)

Compute optimal padded segment heights (in number of pixels) based on the location of available segments.

satpy.readers.yaml_reader._compute_optimal_missing_segment_heights(seg_infos, grid_type, expected_vertical_size)

satpy.readers.yaml_reader._compute_positioning_data_for_missing_group(segment_start_rows, segment_end_rows, segment_heights, group)

satpy.readers.yaml_reader._compute_proposed_sizes_of_missing_segments_in_group(group, segment_end_rows, segment_start_rows)

satpy.readers.yaml_reader._find_missing_segments(file_handlers, ds_info, dsid)

Find missing segments.

satpy.readers.yaml_reader._flip_dataset_data_and_area_extents(dataset, area_extents_to_update, flip_direction)

Flip the data and area extents array for a dataset.

satpy.readers.yaml_reader._get_current_scene_orientation(area_extents_to_update)

Get the current scene orientation from the area_extents.

satpy.readers.yaml_reader._get_dataset_area_extents_array(dataset_area_attr)

Get dataset area extents in a numpy array for further flipping.

satpy.readers.yaml_reader._get_empty_segment_with_height(empty_segment, new_height, dim)

Get a new empty segment with the specified height.

satpy.readers.yaml_reader._get_filebase(path, pattern)

Get the end of path of same length as pattern.

satpy.readers.yaml_reader._get_grid_width_to_grid_type(seg_info)

satpy.readers.yaml_reader._get_new_flipped_area_definition(dataset_area_attr, area_extents_to_update, flip_areadef_stacking)

Get a new area definition with updated area_extents for flipped geostationary datasets.

satpy.readers.yaml_reader._get_projection_type(dataset_area_attr)

Get the projection type from the crs coordinate operation method name.

satpy.readers.yaml_reader._get_target_scene_orientation(upper_right_corner)

Get the target scene orientation from the target upper_right_corner.

‘NE’ corresponds to target_eastright and target_northup being True.

satpy.readers.yaml_reader._init_positioning_arrays_for_variable_padding(chk_infos, grid_type, exp_segment_nr)

satpy.readers.yaml_reader._load_area_def(dsid, file_handlers)

Load the area definition of dsid.

satpy.readers.yaml_reader._match_filenames(filenames, pattern)

Get the filenames matching pattern.

satpy.readers.yaml_reader._populate_group_end_row_using_later_segment(group, segment_end_rows, segment_start_rows)

satpy.readers.yaml_reader._populate_group_start_end_row_using_neighbour_segments(group, segment_end_rows, segment_start_rows)

satpy.readers.yaml_reader._populate_group_start_row_using_previous_segment(group, segment_end_rows, segment_start_rows)

satpy.readers.yaml_reader._populate_positioning_arrays_with_available_segment_info(chk_infos, grid_type, segment_start_rows, segment_end_rows, segment_heights)

satpy.readers.yaml_reader._populate_start_end_rows_of_missing_segments_with_proposed_sizes(group, proposed_sizes_missing_segments, segment_start_rows, segment_end_rows, segment_heights)

satpy.readers.yaml_reader._set_orientation(dataset, upper_right_corner)

Set the orientation of geostationary datasets.

Allows to flip geostationary imagery when loading the datasets. Example call: scn.load([‘VIS008’], upper_right_corner=’NE’)

Parameters:

• dataset – Dataset to be flipped.

• upper_right_corner (str) – Direction of the upper right corner of the image after flipping. Possible options are ‘NW’, ‘NE’, ‘SW’, ‘SE’, or ‘native’. The common upright image orientation corresponds to ‘NE’. Defaults to ‘native’ (no flipping is applied).

satpy.readers.yaml_reader._stack_area_defs(area_def_dict)

Stack given dict of area definitions and return a StackedAreaDefinition.

satpy.readers.yaml_reader._verify_reader_info_assign_config_files(config, config_files)

satpy.readers.yaml_reader.listify_string(something)

Take something and make it a list.

something is either a list of strings or a string, in which case the function returns a list containing the string. If something is None, an empty list is returned.

satpy.readers.yaml_reader.load_yaml_configs(*config_files, loader=<class 'yaml.cyaml.CLoader'>)

Merge a series of YAML reader configuration files.

Parameters:

• *config_files (str) – One or more pathnames to YAML-based reader configuration files that will be merged to create a single configuration.

• loader – Yaml loader object to load the YAML with. Defaults to CLoader if libyaml is available, Loader otherwise.

Returns: dict

Dictionary representing the entire YAML configuration with the addition of config[‘reader’][‘config_files’] (the list of YAML pathnames that were merged).

satpy.readers.yaml_reader.split_integer_in_most_equal_parts(x, n)

Split an integer number x in n parts that are as equally-sizes as possible.