satpy.readers.li_base_nc module

Base class used for the MTG Lighting Imager netCDF4 readers.

The base LI reader class supports generating the available datasets programmatically: to achieve this, each LI product type should provide a "file description" which is itself retrieved directly from the YAML configuration file for the reader of interest, as a custom file_desc entry inside the 'file_type' section corresponding to that product type.

Each of the file_desc entry describes what are the variables that are available into that product that should be used to register the available satpy datasets.

Each of those description entries may contain the following elements:

• product_type [required]:

  Indicate the processing_level / product_type name to use internally for that type of product file. This should correspond to the {processing_level}-{product_type} part of the full file_pattern.

• search_paths [optional]:

  A list of the possible paths that should be prefixed to a given variable name when searching for that variable in the NetCDF file to register a dataset on it. The list is given in priority order. If no search path is provided (or an empty array is provided) then the variables will only be searched directly in the root group of the NetCDF structure.

• swath_coordinates [required]:

  The LI reader will use a SwathDefinition object to define the area/coordinates of each of the provided datasets depending on the content of this entry. The user can either:

  • Specify a swath_coordinates entry directly with latitude and longitude entries, in which case, the datasets that will match one of the 'variable_patterns' provided will use those lat/lon variables as coordinate providers.

  • Specify a swath_coordinates entry directly with projection, azimuth and elevation entries instead, in which case, the reader will first use the variables pointed by those 3 entries compute the corresponding latitude/longitude data from the scan angles contained in the product file. And then, continue with assigned those lat/lon datasets as coordinates for datasets that will match one of the variable_patterns provided.

  Note: It is acceptable to specify an empty array for the list of variable_patterns, in this case, the swath coordinates will not be assigned to any dataset.

• sectors [optional]:

  The custom dataset description mechanism makes a distinction between "ordinary" variables which should be used to create a "single dataset" and "sectored variables" which will be found per sector and will thus be used to generate as many datasets as there are sectors (see below). So this entry is used to specify the list of sector names there should be available in the NetCDF structure.

• sector_variables [optional]:

  This entry is used to provide a list of the variables that are available per sector in the NetCDF file. Thus, assuming the sectors entry is set to the standard list ['north', 'east', 'south', 'west'], 4 separated datasets will be registered for each variable listed here (using the conventional suffix "{sector_name}_sector")

• variables [optional]:

  This entry is used to provide a list of "ordinary variables" (ie. variables that are not available per sector). Each of those variables will be used to register one dataset.

  Note: A single product may provide both the "variables" and the "sector_variables" at the same time (as this is the case for LI LEF for instance)

• variable_transforms [optional]:

  This entry is may be used to provide specific additional entries per variable name (ie. will apply to both in sector or out of sector variables) that should be added to the dataset infos when registering a dataset with that variable. While any kind of info could be added this way to the final dataset infos, we are currently using the entry mainly to provide our LI reader with the following traits which will then be used to "transform" the data of the dataset as requested on loading:

  • broadcast_to: if this extra info is found in a dataset_info on dataset loading, then the initial data array will be broadcast to the shape of the variable found under the variable path specified as value for that entry. Note that, if the pattern {sector_name} if found in this entry value, then the reader will assume that we are writing a dataset from an in sector variable, and use the current sector name to find the appropriate alternate variable that will be used as reference to broadcast the current variable data.

  • seconds_to_datetime: This transformation is used to internally convert variables provided as float values to the np.datetime64 data type. The value specified for this entry should be the reference epoch time used as offsets for the elapsed seconds when converting the data.

  • seconds_to_timedelta: This transformation is used to internally convert variables (assumed to use a "second" unit) provided as float values to the np.timedelta64 data type. This entry should be set to true to activate this transform. During the conversion, we internally use a nanosecond resolution on the input floating point second values.

  • milliseconds_to_timedelta: Same kind of transformation as seconds_to_timedelta except that the source data is assumed to contain millisecond float values.

  • accumulate_index_offset: if this extra info is found in a dataset_info on dataset loading, then we will consider that the dataset currently being generated is an array of indices inside the variable pointed by the path provided as value for that entry. Note that the same usage of the pattern {sector_name} mentioned for the entry "broadcast_to" will also apply here. This behavior is useful when multiple input files are loaded together in a single satpy scene, in which case, the variables from each files will be concatenated to produce a single dataset for each variable, and thus the need to correct the reported indices accordingly.

    An example of usage of this entry is as follows:

    variable_transforms:
      integration_frame_index:
        accumulate_index_offset: "{sector_name}/exposure_time"
    

    In the example above the integration_frame_index from each sector (i.e. optical channel) provides a list of indices in the corresponding exposure_time array from that same sector. The final indices will thus correctly take into account that the final exposure_time array contains all the values concatenated from all the input files in the scene.

  • use_rescaling: By default, we currently apply variable rescaling as soon as we find one (or more) of the attributes named 'scale_factor', 'scaling_factor' or 'add_offset' in the source netcdf variable. This automatic transformation can be disabled for a given variable specifying a value of false for this extra info element, for instance:

    variable_transforms:
      latitude:
        use_rescaling: false
    

    Note: We are currently not disabling rescaling for any dataset, so that entry is not used in the current version of the YAML config files for the LI readers.

class satpy.readers.li_base_nc.LINCFileHandler(filename, filename_info, filetype_info, cache_handle=True)

Bases: NetCDF4FsspecFileHandler

Base class used as parent for the concrete LI reader classes.

Initialize LINCFileHandler.

add_provided_dataset(ds_infos)

Add a provided dataset to our internal list.

apply_accumulate_index_offset(data_array, ds_info)

Apply the accumulate_index_offset transform on a given array.

apply_broadcast_to(data_array, ds_info)

Apply the broadcast_to transform on a given array.

apply_fill_value(arr, fill_value)

Apply fill values, unless it is None.

apply_milliseconds_to_timedelta(data_array, _ds_info)

Apply the milliseconds_to_timedelta transform on a given array.

apply_seconds_to_datetime(data_array, ds_info)

Apply the seconds_to_datetime transform on a given array.

apply_seconds_to_timedelta(data_array, _ds_info)

Apply the seconds_to_timedelta transform on a given array.

apply_transforms(data_array, ds_info)

Apply all transformations requested in the ds_info on the provided data array.

apply_use_rescaling(data_array, ds_info=None)

Apply the use_rescaling transform on a given array.

available_datasets(configured_datasets=None)

Determine automatically the datasets provided by this file.

Uses a per product type dataset registration mechanism using the dataset descriptions declared in the reader construction above.

check_variable_extra_info(ds_infos, vname)

Check if we have extra infos for that variable.

combine_info(all_infos)

Re-implement combine_info.

This is to be able to reset our __index_offset attribute in the shared ds_info currently being updated.

property end_time

Get the end time.

generate_coords_from_scan_angles()

Generate the latitude/longitude coordinates from the scan azimuth and elevation angles.

get_coordinate_names(ds_infos)

Get the target coordinate names, applying the sector name as needed.

get_daskified_lon_lat(proj_dict)

Get daskified lon and lat array using map_blocks.

get_dataset(dataset_id, ds_info=None)

Get a dataset.

get_dataset_infos(dname)

Retrieve the dataset infos corresponding to one of the registered datasets.

get_first_valid_variable(var_paths)

Select the first valid path for a variable from the given input list and returns the data.

get_latlon_names()

Retrieve the user specified names for latitude/longitude coordinates.

Use default ‘latitude’ / ‘longitude’ if not specified.

get_measured_variable(var_paths, fill_value=nan)

Retrieve a measured variable path taking into account the potential old data formatting schema.

And also replace the missing values with the provided fill_value (except if this is explicitly set to None). Also, if a slice index is provided, only that slice of the array (on the axis=0) is retrieved (before filling the missing values).

get_projection_config()

Retrieve the projection configuration details.

get_transform_reference(transform_name, ds_info)

Retrieve a variable that should be used as reference during a transform.

get_transformed_dataset(ds_info)

Retrieve a dataset with all transformations applied on it.

get_variable_search_paths(var_paths)

Get the search paths from the dataset descriptions.

inverse_projection(azimuth, elevation, proj_dict)

Compute inverse projection.

is_prod_in_accumulation_grid()

Check if the current product is an accumulated product in geos grid.

register_available_datasets()

Register all the available dataset that should be made available from this file handler.

register_coords_from_scan_angles()

Register lat lon datasets in this reader.

register_dataset(var_name, oc_name=None)

Register a simple dataset given name elements.

register_sector_datasets()

Register all the available sector datasets.

register_variable_datasets()

Register all the available raw (i.e. not in sectors).

property sensor_names

List of sensors represented in this file.

property start_time

Get the start time.

update_array_attributes(data_array, ds_info)

Inject the attributes from the ds_info structure into the final data array, ignoring the internal entries.

validate_array_dimensions(data_array, ds_info=None)

Ensure that the dimensions of the provided data_array are valid.

variable_path_exists(var_path)

Check if a given variable path is available in the underlying netCDF file.

All we really need to do here is to access the file_content dictionary and check if we have a variable under that var_path key.