veranda.raster.mosaic package

Submodules

veranda.raster.mosaic.base module

Collection of base classes for managing multi-dimensional I/O of raster data.

class veranda.raster.mosaic.base.RasterAccess(src_raster_geom, dst_raster_geom, src_root_raster_geom=None)

Bases: object

Helper class to build the link between indexes of the source array (access) and the target array (assignment).

property dst_col_slice: slice

Indices for the cols of the data to assign.

property dst_row_slice: slice

Indices for the rows of the data to assign.

property src_col_slice: slice

Indices for the columns of the data to access.

property src_row_slice: slice

Indices for the rows of the data to access.

class veranda.raster.mosaic.base.RasterData(file_register, mosaic, data=None, stack_dimension='layer_id', stack_coords=None, tile_dimension='tile_id', **kwargs)

Bases: object

Combines spatial information represented as a mosaic with raster mosaic given as 3D array data in memory or as geospatial files on disk.

apply_nan(nodatavals=None)

Converts no data values given as an attribute ‘_FillValue’ or keyword nodatavals to np.nan.

Parameters:

nodatavals (dict) – Data variable name to no data value map.

Notes

This replacement implicitly converts the data format to float.

clear_ram()

Releases memory allocated by the internal data object.

close(layer_ids=None)

Closes open file handles and optionally data stored in RAM.

Parameters:

layer_ids (list, optional) – Layer IDs indicating the file handles which should be closed. Defaults to None, i.e. all file handles are closed.

property data_geom: RasterGeometry

Raster/tile geometry of the raster mosaic files.

property data_view: Dataset

View on internal raster data.

property file_register: DataFrame

File register of the raster data object.

property filepaths: List[str]

Unique list of file paths stored in the file register.

property layer_ids: list

Sorted layers.

Type:

List

abstract load(*args, **kwargs)

An abstract method for loading data either from disk or RAM.

property mosaic: MosaicGeometry

Mosaic geometry of the raster mosaic files.

property n_layers: int

Maximum number of layers.

property n_tiles: int

Number of tiles.

static raster_geom_from_data(data, sref=None, **kwargs) → RasterGeometry

Creates a raster geometry from an xarray dataset.

Parameters:

• data (xr.Dataset) – Raster data.

• sref (geospade.crs.SpatialRef, optional) – CRS of the mosaic if not given under the ‘spatial_ref’ variable.

• kwargs (dict) – Key-word arguments for the constructor of RasterGeometry.

Returns:

raster_geom – Raster geometry representing the spatial extent of the xarray dataset.

Return type:

geospade.raster.RasterGeometry

select(cmds, inplace=False) → RasterData

Executes several select operations from a dict/JSON compatible set of commands.

Parameters:

• cmds (list of 3-tuple) – List of tuples containing the select operator to execute, its positional arguments, and its key-word arguments.

• inplace (bool, optional) – If True, the current raster data object is modified. If False, a new raster data instance will be returned (default).

Returns:

Raster data object with a mosaic and a file register in compliance with the provided select operations.

Return type:

RasterData

select_bbox(bbox, sref=None, inplace=False) → RasterData

Selects tile and pixel coordinates according to the given bounding box.

Parameters:

• bbox (list of 2 2-tuple) – Bounding box to select, i.e. [(x_min, y_min), (x_max, y_max)]

• sref (geospade.crs.SpatialRef, optional) – CRS of the given bounding box coordinates. Defaults to the CRS of the mosaic.

• inplace (bool, optional) – If True, the current raster data object is modified. If False, a new raster data instance will be returned (default).

Returns:

Raster data object with a file register and a mosaic only consisting of the intersected tiles.

Return type:

RasterData

select_layers(layer_ids, inplace=False) → RasterData

Selects layers according to the given layer IDs.

Parameters:

• layer_ids (list) – Layer IDs to select.

• inplace (bool, optional) – If True, the current raster data object is modified. If False, a new raster data instance will be returned (default).

Returns:

Raster data object with a file register only consisting of the given layer IDs.

Return type:

RasterData

select_polygon(polygon, sref=None, apply_mask=True, inplace=False) → RasterData

Selects tile and pixel coordinates according to the given polygon.

Parameters:

• polygon (ogr.Geometry) – Polygon specifying the pixels to collect.

• sref (geospade.crs.SpatialRef, optional) – CRS of the given bounding box coordinates. Defaults to the CRS of the mosaic.

• apply_mask (bool, optional) – True if pixels outside the polygon should be set to a no data value (default). False if every pixel withing the bounding box of the polygon should be included.

• inplace (bool, optional) – If True, the current raster data object is modified. If False, a new raster data instance will be returned (default).

Returns:

Raster data object with a file register and a mosaic only consisting of the intersected tiles.

Return type:

RasterData

select_px_window(row, col, height=1, width=1, inplace=False) → RasterData

Selects the pixel coordinates according to the given pixel window.

Parameters:

• row (int) – Top-left row number of the pixel window anchor.

• col (int) – Top-left column number of the pixel window anchor.

• height (int, optional) – Number of rows/height of the pixel window. Defaults to 1.

• width (int, optional) – Number of columns/width of the pixel window. Defaults to 1.

• inplace (bool, optional) – If True, the current raster data object is modified. If False, a new raster data instance will be returned (default).

Returns:

Raster data object with a data and a mosaic geometry only consisting of the intersected tile with the pixel window.

Return type:

RasterData

Notes

The mosaic will be only sliced if it consists of one tile to prevent ambiguities in terms of the definition of the pixel window.

select_tiles(tile_names, inplace=False) → RasterData

Selects certain tiles from a raster data object.

Parameters:

• tile_names (list of str) – Tile names/IDs.

• inplace (bool, optional) – If True, the current raster data object is modified. If False, a new raster data instance will be returned (default).

Returns:

Raster data object with a mosaic and a file register only consisting of the given tiles.

Return type:

RasterData

select_xy(x, y, sref=None, inplace=False) → RasterData

Selects a pixel according to the given coordinate tuple.

Parameters:

• x (number) – Coordinate in X direction.

• y (number) – Coordinate in Y direction.

• sref (geospade.crs.SpatialRef, optional) – CRS of the given coordinate tuple. Defaults to the CRS of the mosaic.

• inplace (bool, optional) – If True, the current raster data object is modified. If False, a new raster data instance will be returned (default).

Returns:

Raster data object with a file register and a mosaic only consisting of the intersected tile containing information on the location of the time series.

Return type:

RasterData

class veranda.raster.mosaic.base.RasterDataReader(file_register, mosaic, stack_dimension='layer_id', stack_coords=None, tile_dimension='tile_id')

Bases: RasterData

Allows to read and manage a stack of raster data.

load(*args, **kwargs)

Loads mosaic either from disk or RAM.

Parameters:

• args – Positional arguments for the read method.

• kwargs – Key-word arguments for the read method.

abstract read(*args, auto_decode=False, decoder=None, decoder_kwargs=None, **kwargs)

Read data from disk.

Parameters:

• auto_decode (bool, optional) – True if data should be decoded according to the information available in its metadata. Defaults to False.

• decoder (callable, optional) – Function allowing to decode data read from disk.

• decoder_kwargs (dict, optional) – Keyword arguments for the decoder.

class veranda.raster.mosaic.base.RasterDataWriter(mosaic, file_register=None, data=None, stack_dimension='layer_id', stack_coords=None, tile_dimension='tile_id', dirpath=None, fn_pattern='{layer_id}.xyz', fn_formatter=None)

Bases: RasterData

Allows to write and manage a stack of raster data.

abstract export(apply_tiling=False, encoder=None, encoder_kwargs=None, overwrite=False, **kwargs)

Writes all the internally stored data to disk.

Parameters:

• apply_tiling (bool, optional) – True if the internal data should be tiled according to the mosaic. False if the internal data composes a new tile and should not be tiled (default).

• encoder (callable, optional) – Function allowing to encode data before writing it to disk.

• encoder_kwargs (dict, optional) – Keyword arguments for the encoder.

• overwrite (bool, optional) – True if data should be overwritten, False if not (default).

classmethod from_xarray(data, file_register, mosaic=None, **kwargs) → RasterDataWriter

Converts an xarray dataset and a file register to a RasterDataWriter instance.

Parameters:

• data (xr.Dataset) – Raster data.

• file_register (pd.Dataframe, optional) –

  Data frame managing a stack/list of files containing the following columns:

  • ’filepath’str

    Full file path to a geospatial file.

  • stack_dimensionobject

    Specifies an ID to which layer a file belongs to, e.g. a layer counter or a timestamp. Must correspond to stack_dimension.

  • tile_dimensionstr

    Tile name or ID to which tile a file belongs to.

• mosaic (geospade.raster.MosaicGeometry, optional) – Mosaic representing the spatial allocation of the given files. The tiles of the mosaic have to match the ID’s/names of the tile_dimension column. If it is None, a one-tile mosaic will be created from the given data.

• kwargs – Key-word arguments for the RasterDataWriter constructor.

Return type:

RasterDataWriter

load(*args, **kwargs)

Loads data from RAM.

abstract write(data, encoder=None, encoder_kwargs=None, overwrite=False, **kwargs)

Writes a certain chunk of data to disk.

Parameters:

• data (xr.Dataset) – Data chunk to be written to disk or being appended to existing data.

• encoder (callable, optional) – Function allowing to encode data before writing it to disk.

• encoder_kwargs (dict, optional) – Keyword arguments for the encoder.

• overwrite (bool, optional) – True if data should be overwritten, False if not (default).

veranda.raster.mosaic.geotiff module

Raster data class managing I/O for multiple GeoTIFF files.

class veranda.raster.mosaic.geotiff.GeoTiffAccess(src_raster_geom, dst_raster_geom, src_root_raster_geom=None)

Bases: RasterAccess

Helper class to build the link between indexes of the source array (access) and the target array (assignment). The base class RasterAccess is extended with some properties needed for accessing GeoTIFF files.

property gdal_args: Tuple[int, int, int, int]

Prepares the needed positional arguments for GDAL’s ReadAsArray() function.

property read_args: Tuple[int, int, int, int]

Prepares the needed positional arguments for the read() function of the internal GeoTIFF native.

class veranda.raster.mosaic.geotiff.GeoTiffReader(file_register, mosaic, stack_dimension='layer_id', stack_coords=None, tile_dimension='tile_id', space_dims=None, file_class=<class 'veranda.raster.native.geotiff.GeoTiffFile'>, file_class_kwargs=None)

Bases: RasterDataReader

Allows to read and manage a stack of GeoTIFF data.

apply_nan()

Converts no data values given as an attribute ‘_FillValue’ or keyword nodatavals to np.nan.

classmethod from_filepaths(filepaths, mosaic_class=<class 'geospade.raster.MosaicGeometry'>, mosaic_kwargs=None, tile_kwargs=None, stack_dimension='layer_id', tile_dimension='tile_id', **kwargs) → GeoTiffReader

Creates a GeoTiffReader instance as one stack of GeoTIFF files.

Parameters:

• filepaths (list of str) – List of full system paths to a GeoTIFF file.

• mosaic_class (geospade.raster.MosaicGeometry, optional) – Mosaic class used to manage the spatial properties of the file stack. If None, the most generic mosaic will be used by default. The initialised mosaic will only contain one tile.

• mosaic_kwargs (dict, optional) – Additional arguments for initialising mosaic_class.

• tile_kwargs (dict, optional) – Additional arguments for initialising a tile class associated with mosaic_class.

• stack_dimension (str, optional) – Dimension/column name of the dimension, where to stack the files along (first axis), e.g. time, bands etc. Defaults to ‘layer_id’, i.e. the layer ID’s are used as the main coordinates to stack the files.

• tile_dimension (str, optional) – Dimension/column name of the dimension containing tile ID’s in correspondence with the tiles in mosaic. Defaults to ‘tile_id’.

• kwargs (dict, optional) – Key-word arguments for the GeoTiffReader constructor.

Return type:

GeoTiffReader

classmethod from_mosaic_filepaths(filepaths, mosaic_class=<class 'geospade.raster.MosaicGeometry'>, mosaic_kwargs=None, stack_dimension='layer_id', tile_dimension='tile_id', **kwargs) → GeoTiffReader

Creates a GeoTiffDataReader instance as multiple stacks of GeoTIFF files.

Parameters:

• filepaths (list of str) – List of full system paths to a GeoTIFF file.

• mosaic_class (geospade.raster.MosaicGeometry, optional) – Mosaic class used to manage the spatial properties of the file stacks. If None, the most generic mosaic will be used by default.

• mosaic_kwargs (dict, optional) – Additional arguments for initialising mosaic_class.

• stack_dimension (str, optional) – Dimension/column name of the dimension, where to stack the files along (first axis), e.g. time, bands etc. Defaults to ‘layer_id’, i.e. the layer ID’s are used as the main coordinates to stack the files.

• tile_dimension (str, optional) – Dimension/column name of the dimension containing tile ID’s in correspondence with the tiles in mosaic. Defaults to ‘tile_id’.

• kwargs (dict, optional) – Key-word arguments for the GeoTiffReader constructor.

Return type:

GeoTiffReader

read(bands=1, band_names=None, engine='vrt', n_cores=1, auto_decode=False, decoder=None, decoder_kwargs=None) → GeoTiffReader

Reads data from disk.

Parameters:

• bands (tuple of int or int, optional) – The GeoTIFF bands of interest. Defaults to the first band, i.e. 1.

• band_names (tuple of str or str, optional) – Names associated with the respective bands of the GeoTIFF files. Defaults to None, i.e. the band numbers will be used as a name.

• engine (str, optional) –

  Engine used in the background to read data. The following options are available:

  • ’vrt’Uses GDAL’s VRT format to stack the GeoTIFF files per tile and load them as once. This option

    yields good performance if the mosaic is stored locally on one drive. Parallelisation is applied across tiles.

  • ’parallel’Reads file by file, but in a parallelised manner. This option yields good performance if

    the mosaic is stored on a distributed file system.

• n_cores (int, optional) – Number of cores used to read data in a parallelised manner (defaults to 1).

• auto_decode (bool, optional) – True if data should be decoded according to the information available in its metadata. Defaults to False.

• decoder (callable, optional) – Function allowing to decode data read from disk.

• decoder_kwargs (dict, optional) – Keyword arguments for the decoder.

class veranda.raster.mosaic.geotiff.GeoTiffWriter(mosaic, file_register=None, data=None, stack_dimension='layer_id', stack_coords=None, tile_dimension='tile_id', dirpath=None, fn_pattern='{layer_id}.tif', fn_formatter=None)

Bases: RasterDataWriter

Allows to write and manage a stack of GeoTIFF files.

export(use_mosaic=False, data_variables=None, encoder=None, encoder_kwargs=None, overwrite=False, **kwargs)

Writes all internally stored data to disk.

Parameters:

• use_mosaic (bool, optional) – True if data should be written according to the mosaic. False if data composes a new tile and should not be tiled (default).

• data_variables (list of str, optional) – Data variables to write. Defaults to None, i.e. all data variables are written.

• encoder (callable, optional) – Function allowing to encode data before writing it to disk.

• encoder_kwargs (dict, optional) – Keyword arguments for the encoder.

• overwrite (bool, optional) – True if data should be overwritten, False if not (default).

write(data, use_mosaic=False, data_variables=None, encoder=None, encoder_kwargs=None, overwrite=False, **kwargs)

Writes a certain chunk of data to disk.

Parameters:

• data (xr.Dataset) – Data chunk to be written to disk or being appended to existing data.

• use_mosaic (bool, optional) – True if data should be written according to the mosaic. False if data composes a new tile and should not be tiled (default).

• data_variables (list of str, optional) – Data variables to write. Defaults to None, i.e. all data variables are written.

• encoder (callable, optional) – Function allowing to encode data before writing it to disk.

• encoder_kwargs (dict, optional) – Keyword arguments for the encoder.

• overwrite (bool, optional) – True if data should be overwritten, False if not (default).

veranda.raster.mosaic.geotiff.read_init(fr, am, sm, sd, td, si, ad, dc, dk)

Helper method for setting the entries of global variable PROC_OBJS to be available during multiprocessing.

veranda.raster.mosaic.geotiff.read_single_files(file_idx)

Function being responsible to read data from a single GeoTIFF file and assign it to a shared memory array. This function is meant to be executed in parallel on different cores.

Parameters:

file_idx (any) – Index value to access a specific row of the file register. The actual value should come from the Pool’s mapping function.

veranda.raster.mosaic.geotiff.read_vrt_stack(tile_id)

Function being responsible to create a new VRT file from a stack of GeoTIFF files, read data from this files, and assign it to a shared memory array. This function is meant to be executed in parallel on different cores.

Parameters:

tile_id (str) – Tile/geometry ID coming from the Pool’s mapping function.

veranda.raster.mosaic.netcdf module

Raster data class managing I/O for multiple NetCDF files.

class veranda.raster.mosaic.netcdf.NetCdfReader(file_register, mosaic, stack_dimension='layer_id', stack_coords=None, tile_dimension='tile_id', file_class=<class 'veranda.raster.native.netcdf.NetCdf4File'>, file_class_kwargs=None)

Bases: RasterDataReader

Allows to read and manage a stack of NetCDF files.

apply_nan()

Converts no data values given as an attribute ‘_FillValue’ or keyword nodatavals to np.nan.

classmethod from_filepaths(filepaths, mosaic_class=<class 'geospade.raster.MosaicGeometry'>, mosaic_kwargs=None, tile_kwargs=None, stack_dimension='layer_id', tile_dimension='tile_id', file_class=<class 'veranda.raster.native.netcdf.NetCdf4File'>, file_class_kwargs=None, **kwargs) → NetCdfReader

Creates a NetCdfReader instance as one stack of NetCDF files.

Parameters:

• filepaths (list of str) – List of full system paths to a NetCDF file.

• mosaic_class (geospade.raster.MosaicGeometry, optional) – Mosaic class used to manage the spatial properties of the file stack. If None, the most generic mosaic will be used by default. The initialised mosaic will only contain one tile.

• mosaic_kwargs (dict, optional) – Additional arguments for initialising mosaic_class.

• tile_kwargs (dict, optional) – Additional arguments for initialising a tile class associated with mosaic_class.

• stack_dimension (str, optional) – Dimension/column name of the dimension, where to stack the files along (first axis), e.g. time, bands etc. Defaults to ‘layer_id’, i.e. the layer ID’s are used as the main coordinates to stack the files.

• tile_dimension (str, optional) – Dimension/column name of the dimension containing tile ID’s in correspondence with the tiles in mosaic. Defaults to ‘tile_id’.

• file_class (class, optional) – Class to use for opening files. Defaults to NetCdf4File.

• file_class_kwargs (dict, optional) – Keyword arguments for calling file_class. Defaults to none.

• kwargs (dict, optional) – Key-word arguments for the NetCdfReader constructor.

Return type:

NetCdfReader

classmethod from_mosaic_filepaths(filepaths, mosaic_class=<class 'geospade.raster.MosaicGeometry'>, mosaic_kwargs=None, stack_dimension='layer_id', tile_dimension='tile_id', file_class=<class 'veranda.raster.native.netcdf.NetCdf4File'>, file_class_kwargs=None, **kwargs) → NetCdfReader

Creates a NetCdfReader instance as multiple stacks of NetCDF files.

Parameters:

• filepaths (list of str) – List of full system paths to a NetCDF file.

• mosaic_class (geospade.raster.MosaicGeometry, optional) – Mosaic class used to manage the spatial properties of the file stacks. If None, the most generic mosaic will be used by default.

• mosaic_kwargs (dict, optional) – Additional arguments for initialising mosaic_class.

• stack_dimension (str, optional) – Dimension/column name of the dimension, where to stack the files along (first axis), e.g. time, bands etc. Defaults to ‘layer_id’, i.e. the layer ID’s are used as the main coordinates to stack the files.

• tile_dimension (str, optional) – Dimension/column name of the dimension containing tile ID’s in correspondence with the tiles in mosaic. Defaults to ‘tile_id’.

• file_class (class, optional) – Class to use for opening files. Defaults to NetCdf4File.

• file_class_kwargs (dict, optional) – Keyword arguments for calling file_class. Defaults to none.

• kwargs (dict, optional) – Key-word arguments for the NetCdfReader constructor.

Return type:

NetCdfReader

read(data_variables=None, engine='netcdf4', agg_dim='time', parallel=True, compute=True, auto_decode=False, decoder=None, decoder_kwargs=None, **kwargs) → NetCdfReader

Reads NetCdf data from disk and assigns it to the class.

Parameters:

• data_variables (list, optional) – Data variables to read. Default is to read all available data variables.

• engine (str, optional) –

  Engine used in the background to read NetCDF data. The following options are available:

  • ’netcdf4’ : Uses the netCDF4 library to create an MFDataset object.

  • ’xarray’ : Uses xarray’s open_mfdataset function.

• agg_dim (str, optional) – Dimension to aggregate on (defaults to ‘layer_id’).

• parallel (bool, optional) – Flag to activate parallelisation or not when using ‘xarray’ as an engine. Defaults to True.

• compute (bool, optional) – True if values from a dask array should be loaded into RAM (default).

• auto_decode (bool, optional) – True if NetCDF data should be decoded according to the information available in its metadata. Defaults to False.

• decoder (callable, optional) – Function allowing to decode NetCDF data read from disk.

• decoder_kwargs (dict, optional) – Keyword arguments for the decoder.

class veranda.raster.mosaic.netcdf.NetCdfWriter(mosaic, file_register=None, data=None, stack_dimension='layer_id', stack_coords=None, tile_dimension='tile_id', dirpath=None, fn_pattern='{layer_id}.tif', fn_formatter=None)

Bases: RasterDataWriter

Allows to write and manage a stack of NetCDF files.

export(use_mosaic=False, data_variables=None, encoder=None, encoder_kwargs=None, overwrite=False, unlimited_dims=None, **kwargs)

Writes all internally stored data to disk.

Parameters:

• use_mosaic (bool, optional) – True if data should be written according to the mosaic. False if data composes a new tile and should not be tiled (default).

• data_variables (list of str, optional) – Data variables to write. Defaults to None, i.e. all data variables are written.

• encoder (callable, optional) – Function allowing to encode an xarray dataset before writing it to disk.

• encoder_kwargs (dict, optional) – Keyword arguments for the encoder.

• overwrite (bool, optional) – True if the NetCDF file(s) should be overwritten, False if not (default).

• unlimited_dims (list of str or str, optional) – List of dimension names specifying the dimensions to be stored as unlimited.

• kwargs (dict, optional) – Key-word arguments for creating a NetCdf4File instance.

classmethod from_data(data, filepath, mosaic=None, tile_dimension='tile_id', **kwargs) → NetCdfWriter

Creates NetCdfWriter instance from an xarray.Dataset instance and a target file path, i.e. this function should help to write/export the whole image stack to one file.

Parameters:

• data (xr.Dataset) – Dataset to write to disk.

• filepath (str) – Full system path to NetCDF file to write to.

• mosaic (geospade.raster.MosaicGeometry, optional) – Mosaic representing the spatial allocation of the given file. The tiles of the mosaic have to match the ID’s/names of the ‘tile_id’ column. If it is None, a one-tile mosaic will be created from the given mosaic.

• tile_dimension (str, optional) – Dimension/column name of the dimension containing tile ID’s in correspondence with the tiles in mosaic. Defaults to ‘tile_id’.

• kwargs (dict, optional) – Key-word arguments for initialising the NetCdfWriter class.

Return type:

NetCdfWriter

write(data, use_mosaic=False, data_variables=None, encoder=None, encoder_kwargs=None, overwrite=False, unlimited_dims=None, **kwargs)

Writes a certain chunk of NetCDF data to disk.

Parameters:

• data (xr.Dataset) – Data chunk to be written to disk or being appended to existing mosaic.

• use_mosaic (bool, optional) – True if data should be written according to the mosaic. False if data composes a new tile and should not be tiled (default).

• data_variables (list of str, optional) – Data variables to write. Defaults to None, i.e. all data variables are written.

• encoder (callable, optional) – Function allowing to encode a xarray dataset before writing it to disk.

• encoder_kwargs (dict, optional) – Keyword arguments for the encoder.

• overwrite (bool, optional) – True if the NetCDF file(s) should be overwritten, False if not (default).

• unlimited_dims (list of str or str, optional) – List of dimension names specifying the dimensions to be stored as unlimited.

• kwargs (dict, optional) – Key-word arguments for creating a NetCdf4File instance.

Module contents