satpy.writers.cf_writer module
Writer for netCDF4/CF.
Example usage
The CF writer saves datasets in a Scene as CF-compliant netCDF file. Here is an example with MSG SEVIRI data in HRIT format:
>>> from satpy import Scene
>>> import glob
>>> filenames = glob.glob('data/H*201903011200*')
>>> scn = Scene(filenames=filenames, reader='seviri_l1b_hrit')
>>> scn.load(['VIS006', 'IR_108'])
>>> scn.save_datasets(writer='cf', datasets=['VIS006', 'IR_108'], filename='seviri_test.nc',
exclude_attrs=['raw_metadata'])
You can select the netCDF backend using the
engine
keyword argument. If None if followsto_netcdf()
engine choices with a preference for ‘netcdf4’.For datasets with area definition you can exclude lat/lon coordinates by setting
include_lonlats=False
. If the area has a projected CRS, units are assumed to be in metre. If the area has a geographic CRS, units are assumed to be in degrees. The writer does not verify that the CRS is supported by the CF conventions. One commonly used projected CRS not supported by the CF conventions is the equirectangular projection, such as EPSG 4087.By default non-dimensional coordinates (such as scanline timestamps) are prefixed with the corresponding dataset name. This is because they are likely to be different for each dataset. If a non-dimensional coordinate is identical for all datasets, the prefix can be removed by setting
pretty=True
.Some dataset names start with a digit, like AVHRR channels 1, 2, 3a, 3b, 4 and 5. This doesn’t comply with CF https://cfconventions.org/Data/cf-conventions/cf-conventions-1.7/build/ch02s03.html. These channels are prefixed with CHANNEL_ by default. This can be controlled with the variable numeric_name_prefix to save_datasets. Setting it to None or ‘’ will skip the prefixing.
Grouping
All datasets to be saved must have the same projection coordinates x
and y
. If a scene holds datasets with
different grids, the CF compliant workaround is to save the datasets to separate files. Alternatively, you can save
datasets with common grids in separate netCDF groups as follows:
>>> scn.load(['VIS006', 'IR_108', 'HRV'])
>>> scn.save_datasets(writer='cf', datasets=['VIS006', 'IR_108', 'HRV'],
filename='seviri_test.nc', exclude_attrs=['raw_metadata'],
groups={'visir': ['VIS006', 'IR_108'], 'hrv': ['HRV']})
Note that the resulting file will not be fully CF compliant.
Dataset Encoding
Dataset encoding can be specified in two ways:
Via the
encoding
keyword argument ofsave_datasets
:>>> my_encoding = { ... 'my_dataset_1': { ... 'compression': 'zlib', ... 'complevel': 9, ... 'scale_factor': 0.01, ... 'add_offset': 100, ... 'dtype': np.int16 ... }, ... 'my_dataset_2': { ... 'compression': None, ... 'dtype': np.float64 ... } ... } >>> scn.save_datasets(writer='cf', filename='encoding_test.nc', encoding=my_encoding)
Via the
encoding
attribute of the datasets in a scene. For example>>> scn['my_dataset'].encoding = {'compression': 'zlib'} >>> scn.save_datasets(writer='cf', filename='encoding_test.nc')
See the xarray encoding documentation for all encoding options.
Note
Chunk-based compression can be specified with the compression
keyword
since
netCDF4-1.6.0 libnetcdf-4.9.0 xarray-2022.12.0
The zlib
keyword is deprecated. Make sure that the versions of
these modules are all above or all below that reference. Otherwise,
compression might fail or be ignored silently.
Attribute Encoding
In the above examples, raw metadata from the HRIT files have been excluded. If you want all attributes to be included,
just remove the exclude_attrs
keyword argument. By default, dict-type dataset attributes, such as the raw metadata,
are encoded as a string using json. Thus, you can use json to decode them afterwards:
>>> import xarray as xr
>>> import json
>>> # Save scene to nc-file
>>> scn.save_datasets(writer='cf', datasets=['VIS006', 'IR_108'], filename='seviri_test.nc')
>>> # Now read data from the nc-file
>>> ds = xr.open_dataset('seviri_test.nc')
>>> raw_mda = json.loads(ds['IR_108'].attrs['raw_metadata'])
>>> print(raw_mda['RadiometricProcessing']['Level15ImageCalibration']['CalSlope'])
[0.020865 0.0278287 0.0232411 0.00365867 0.00831811 0.03862197
0.12674432 0.10396091 0.20503568 0.22231115 0.1576069 0.0352385]
Alternatively it is possible to flatten dict-type attributes by setting flatten_attrs=True
. This is more human
readable as it will create a separate nc-attribute for each item in every dictionary. Keys are concatenated with
underscore separators. The CalSlope attribute can then be accessed as follows:
>>> scn.save_datasets(writer='cf', datasets=['VIS006', 'IR_108'], filename='seviri_test.nc',
flatten_attrs=True)
>>> ds = xr.open_dataset('seviri_test.nc')
>>> print(ds['IR_108'].attrs['raw_metadata_RadiometricProcessing_Level15ImageCalibration_CalSlope'])
[0.020865 0.0278287 0.0232411 0.00365867 0.00831811 0.03862197
0.12674432 0.10396091 0.20503568 0.22231115 0.1576069 0.0352385]
This is what the corresponding ncdump
output would look like in this case:
$ ncdump -h test_seviri.nc
...
IR_108:raw_metadata_RadiometricProcessing_Level15ImageCalibration_CalOffset = -1.064, ...;
IR_108:raw_metadata_RadiometricProcessing_Level15ImageCalibration_CalSlope = 0.021, ...;
IR_108:raw_metadata_RadiometricProcessing_MPEFCalFeedback_AbsCalCoeff = 0.021, ...;
...
- class satpy.writers.cf_writer.AttributeEncoder(*, skipkeys=False, ensure_ascii=True, check_circular=True, allow_nan=True, sort_keys=False, indent=None, separators=None, default=None)[source]
Bases:
JSONEncoder
JSON encoder for dataset attributes.
Constructor for JSONEncoder, with sensible defaults.
If skipkeys is false, then it is a TypeError to attempt encoding of keys that are not str, int, float or None. If skipkeys is True, such items are simply skipped.
If ensure_ascii is true, the output is guaranteed to be str objects with all incoming non-ASCII characters escaped. If ensure_ascii is false, the output can contain non-ASCII characters.
If check_circular is true, then lists, dicts, and custom encoded objects will be checked for circular references during encoding to prevent an infinite recursion (which would cause an RecursionError). Otherwise, no such check takes place.
If allow_nan is true, then NaN, Infinity, and -Infinity will be encoded as such. This behavior is not JSON specification compliant, but is consistent with most JavaScript based encoders and decoders. Otherwise, it will be a ValueError to encode such floats.
If sort_keys is true, then the output of dictionaries will be sorted by key; this is useful for regression tests to ensure that JSON serializations can be compared on a day-to-day basis.
If indent is a non-negative integer, then JSON array elements and object members will be pretty-printed with that indent level. An indent level of 0 will only insert newlines. None is the most compact representation.
If specified, separators should be an (item_separator, key_separator) tuple. The default is (’, ‘, ‘: ‘) if indent is
None
and (‘,’, ‘: ‘) otherwise. To get the most compact JSON representation, you should specify (‘,’, ‘:’) to eliminate whitespace.If specified, default is a function that gets called for objects that can’t otherwise be serialized. It should return a JSON encodable version of the object or raise a
TypeError
.
- class satpy.writers.cf_writer.CFWriter(name=None, filename=None, base_dir=None, **kwargs)[source]
Bases:
Writer
Writer producing NetCDF/CF compatible datasets.
Initialize the writer object.
- Parameters:
name (str) – A name for this writer for log and error messages. If this writer is configured in a YAML file its name should match the name of the YAML file. Writer names may also appear in output file attributes.
filename (str) –
Filename to save data to. This filename can and should specify certain python string formatting fields to differentiate between data written to the files. Any attributes provided by the
.attrs
of a DataArray object may be included. Format and conversion specifiers provided by thetrollsift
package may also be used. Any directories in the provided pattern will be created if they do not exist. Example:{platform_name}_{sensor}_{name}_{start_time:%Y%m%d_%H%M%S}.tif
base_dir (str) – Base destination directories for all created files.
kwargs (dict) – Additional keyword arguments to pass to the
Plugin
class.
- _collect_datasets(datasets, epoch='seconds since 1970-01-01 00:00:00', flatten_attrs=False, exclude_attrs=None, include_lonlats=True, pretty=False, include_orig_name=True, numeric_name_prefix='CHANNEL_')[source]
Collect and prepare datasets to be written.
- static da2cf(dataarray, epoch='seconds since 1970-01-01 00:00:00', flatten_attrs=False, exclude_attrs=None, include_orig_name=True, numeric_name_prefix='CHANNEL_')[source]
Convert the dataarray to something cf-compatible.
- Parameters:
dataarray (xr.DataArray) – The data array to be converted
epoch (str) – Reference time for encoding of time coordinates
flatten_attrs (bool) – If True, flatten dict-type attributes
exclude_attrs (list) – List of dataset attributes to be excluded
include_orig_name (bool) – Include the original dataset name in the netcdf variable attributes
numeric_name_prefix (str) – Prepend dataset name with this if starting with a digit
- save_dataset(dataset, filename=None, fill_value=None, **kwargs)[source]
Save the dataset to a given filename.
- save_datasets(datasets, filename=None, groups=None, header_attrs=None, engine=None, epoch='seconds since 1970-01-01 00:00:00', flatten_attrs=False, exclude_attrs=None, include_lonlats=True, pretty=False, include_orig_name=True, numeric_name_prefix='CHANNEL_', **to_netcdf_kwargs)[source]
Save the given datasets in one netCDF file.
Note that all datasets (if grouping: in one group) must have the same projection coordinates.
- Parameters:
datasets (list) – Datasets to be saved
filename (str) – Output file
groups (dict) – Group datasets according to the given assignment: {‘group_name’: [‘dataset1’, ‘dataset2’, …]}. Group name None corresponds to the root of the file, i.e. no group will be created. Warning: The results will not be fully CF compliant!
header_attrs – Global attributes to be included
engine (str) – Module to be used for writing netCDF files. Follows xarray’s
to_netcdf()
engine choices with a preference for ‘netcdf4’.epoch (str) – Reference time for encoding of time coordinates
flatten_attrs (bool) – If True, flatten dict-type attributes
exclude_attrs (list) – List of dataset attributes to be excluded
include_lonlats (bool) – Always include latitude and longitude coordinates, even for datasets with area definition
pretty (bool) – Don’t modify coordinate names, if possible. Makes the file prettier, but possibly less consistent.
include_orig_name (bool) – Include the original dataset name as an varaibel attribute in the final netcdf
numeric_name_prefix (str) – Prefix to add the each variable with name starting with a digit. Use ‘’ or None to leave this out.
- satpy.writers.cf_writer._check_backend_versions()[source]
Issue warning if backend versions do not match.
- satpy.writers.cf_writer._encode_nc(obj)[source]
Try to encode obj as a netcdf compatible datatype which most closely resembles the object’s nature.
- Raises:
ValueError if no such datatype could be found –
- satpy.writers.cf_writer._encode_python_objects(obj)[source]
Try to find the datatype which most closely resembles the object’s nature.
If on failure, encode as a string. Plain lists are encoded recursively.
- satpy.writers.cf_writer._set_default_chunks(encoding, dataset)[source]
Update encoding to preserve current dask chunks.
Existing user-defined chunks take precedence.
- satpy.writers.cf_writer._set_default_fill_value(encoding, dataset)[source]
Set default fill values.
Avoid _FillValue attribute being added to coordinate variables (https://github.com/pydata/xarray/issues/1865).
- satpy.writers.cf_writer._set_default_time_encoding(encoding, dataset)[source]
Set default time encoding.
Make sure time coordinates and bounds have the same units. Default is xarray’s CF datetime encoding, which can be overridden by user-defined encoding.
- satpy.writers.cf_writer._set_encoding_dataset_names(encoding, dataset, numeric_name_prefix)[source]
Set Netcdf variable names encoding according to numeric_name_prefix.
A lot of channel names in satpy starts with a digit. When writing data with the satpy_cf_nc these channels are prepended with numeric_name_prefix. This ensures this is also done with any matching variables in encoding.
- satpy.writers.cf_writer.area2cf(dataarray, strict=False, got_lonlats=False)[source]
Convert an area to at CF grid mapping or lon and lats.
- satpy.writers.cf_writer.area2lonlat(dataarray)[source]
Convert an area to longitudes and latitudes.
- satpy.writers.cf_writer.assert_xy_unique(datas)[source]
Check that all datasets share the same projection coordinates x/y.
- satpy.writers.cf_writer.create_grid_mapping(area)[source]
Create the grid mapping instance for area.
- satpy.writers.cf_writer.dataset_is_projection_coords(dataset)[source]
Check if dataset is a projection coords.
- satpy.writers.cf_writer.encode_attrs_nc(attrs)[source]
Encode dataset attributes in a netcdf compatible datatype.
- satpy.writers.cf_writer.encode_nc(obj)[source]
Encode the given object as a netcdf compatible datatype.
- satpy.writers.cf_writer.get_extra_ds(dataset, keys=None)[source]
Get the extra datasets associated to dataset.
- satpy.writers.cf_writer.has_projection_coords(ds_collection)[source]
Check if collection has a projection coords among data arrays.
- satpy.writers.cf_writer.link_coords(datas)[source]
Link dataarrays and coordinates.
If the coordinates attribute of a data array links to other dataarrays in the scene, for example coordinates=’lon lat’, add them as coordinates to the data array and drop that attribute. In the final call to xr.Dataset.to_netcdf() all coordinate relations will be resolved and the coordinates attributes be set automatically.
- satpy.writers.cf_writer.make_alt_coords_unique(datas, pretty=False)[source]
Make non-dimensional coordinates unique among all datasets.
Non-dimensional (or alternative) coordinates, such as scanline timestamps, may occur in multiple datasets with the same name and dimension but different values. In order to avoid conflicts, prepend the dataset name to the coordinate name. If a non-dimensional coordinate is unique among all datasets and
pretty=True
, its name will not be modified.Since all datasets must have the same projection coordinates, this is not applied to latitude and longitude.