4.6 pde.storage package

Module defining classes for storing simulation data.

`StorageBase`	Base class for storing time series of discretized fields.
`FileStorage`	Store discretized fields in a hdf5 file.
`get_memory_storage`	A context manager that can be used to create a MemoryStorage.
`MemoryStorage`	Store discretized fields in memory.
`ModelrunnerStorage`	Store discretized fields in a `modelrunner` storage.
`MovieStorage`	Store discretized fields in a movie file.

class FileStorage(filename, *, info=None, write_mode='truncate_once', max_length=None, compression=True, keep_opened=True, check_mpi=True)[source]

Bases: StorageBase

Store discretized fields in a hdf5 file.

Parameters:

filename (str) – The path to the hdf5-file where the data is stored
info (dict) – Supplies extra information that is stored in the storage
write_mode (str) – Determines how new data is added to already existing data. Possible values are: ‘append’ (data is always appended), ‘truncate’ (data is cleared every time this storage is used for writing), or ‘truncate_once’ (data is cleared for the first writing, but appended subsequently). Alternatively, specifying ‘readonly’ will disable writing completely.
max_length (int, optional) – Maximal number of entries that will be stored in the file. This can be used to preallocate data, which can lead to smaller files, but is also less flexible. Giving max_length = None, allows for arbitrarily large data, which might lead to larger files.
compression (bool) – Whether to store the data in compressed form. Automatically enabled chunked storage.
keep_opened (bool) – Flag indicating whether the file should be kept opened after each writing. If False, the file will be closed after writing a dataset. This keeps the file in a consistent state, but also requires more work before data can be written.
check_mpi (bool) – If True, files will only be opened in the main node for an parallel simulation using MPI. This flag has no effect in serial code.

clear(clear_data_shape=False)[source]

Truncate the storage by removing all stored data.

Parameters:: clear_data_shape (bool) – Flag determining whether the data shape is also deleted.

close()[source]

Close the currently opened file.

Return type:: None

property data

The actual data for all time.

Type:: ndarray

end_writing()[source]

Finalize the storage after writing.

This makes sure the data is actually written to a file when self.keep_opened == False

Return type:: None

start_writing(field, info=None)[source]

Initialize the storage for writing data.

Parameters:

field (FieldBase) – An example of the data that will be written to extract the grid and the data_shape
info (dict) – Supplies extra information that is stored in the storage

Return type:

None

property times

The times at which data is available.

Type:: ndarray

class MemoryStorage(times=None, data=None, *, info=None, field_obj=None, write_mode='truncate_once')[source]

Bases: StorageBase

Store discretized fields in memory.

Parameters:

times (ndarray) – Sequence of times for which data is known
data (list of ndarray) – The field data at the given times
field_obj (FieldBase) – An instance of the field class store data for a single time point.
info (dict) – Supplies extra information that is stored in the storage
write_mode (str) – Determines how new data is added to already existing data. Possible values are: ‘append’ (data is always appended), ‘truncate’ (data is cleared every time this storage is used for writing), or ‘truncate_once’ (data is cleared for the first writing, but appended subsequently). Alternatively, specifying ‘readonly’ will disable writing completely.

clear(clear_data_shape=False)[source]

Truncate the storage by removing all stored data.

Parameters:: clear_data_shape (bool) – Flag determining whether the data shape is also deleted.
Return type:: None

classmethod from_collection(storages, label=None, *, rtol=1e-05, atol=1e-08)[source]

Combine multiple memory storages into one.

This method can be used to combine multiple time series of different fields into a single representation. This requires that all time series contain data at the same time points.

Parameters:

storages (list) – A collection of instances of StorageBase whose data will be concatenated into a single MemoryStorage
label (str, optional) – The label of the instances of FieldCollection that represent the concatenated data
rtol (float) – Relative tolerance used when checking times for merging
atol (float) – Absolute tolerance used when checking times for merging

Returns:

Storage containing all the data.

Return type:

MemoryStorage

classmethod from_fields(times=None, fields=None, info=None, write_mode='truncate_once')[source]

Create MemoryStorage from a list of fields.

Parameters:

times (ndarray) – Sequence of times for which data is known
fields (list of FieldBase) – The fields at all given time points
info (dict) – Supplies extra information that is stored in the storage
write_mode (str) – Determines how new data is added to already existing data. Possible values are: ‘append’ (data is always appended), ‘truncate’ (data is cleared every time this storage is used for writing), or ‘truncate_once’ (data is cleared for the first writing, but appended subsequently). Alternatively, specifying ‘readonly’ will disable writing completely.

Return type:

MemoryStorage

start_writing(field, info=None)[source]

Initialize the storage for writing data.

Parameters:

field (FieldBase) – An instance of the field class store data for a single time point.
info (dict) – Supplies extra information that is stored in the storage

Return type:

None

class ModelrunnerStorage(storage, *, loc='trajectory', info=None, write_mode='truncate_once')[source]

Bases: StorageBase

Store discretized fields in a modelrunner storage.

This storage class acts as a wrapper for the trajectory module, which allows handling time-dependent data in modelrunner storages. In principle, all backends are supported, but it is advisable to use binary formats like HDFStorage or ZarrStorage to write large amounts of data.

from modelrunner import Result

r = Result.from_file("data.hdf5")
r.result.plot()  # plots the final state
r.storage["trajectory"]  # allows accessing the stored trajectory

Parameters:

storage (StorageGroup) – Modelrunner storage used for storing the trajectory
loc (str or list of str) – The location in the storage where the trajectory data is written.
info (dict) – Supplies extra information that is stored in the storage
write_mode (str) – Determines how new data is added to already existing data. Possible values are: ‘append’ (data is always appended), ‘truncate’ (data is cleared every time this storage is used for writing), or ‘truncate_once’ (data is cleared for the first writing, but appended subsequently). Alternatively, specifying ‘readonly’ will disable writing completely.

clear(clear_data_shape=False)[source]

Truncate the storage by removing all stored data.

Parameters:: clear_data_shape (bool) – Flag determining whether the data shape is also deleted.

close()[source]

Close the currently opened trajectory writer.

Return type:: None

property data

The actual data for all time.

Type:: ndarray

end_writing()[source]

Finalize the storage after writing.

This makes sure the data is actually written to a file when self.keep_opened == False

Return type:: None

start_writing(field, info=None)[source]

Initialize the storage for writing data.

Parameters:

field (FieldBase) – An example of the data that will be written to extract the grid and the data_shape
info (dict) – Supplies extra information that is stored in the storage

Return type:

None

property times

The times at which data is available.

Type:: ndarray

class MovieStorage(filename, *, vmin=0, vmax=1, bits_per_channel=16, video_format='auto', bitrate=-1, info=None, write_mode='truncate_once', write_times=False, loglevel='warning')[source]

Bases: StorageBase

Store discretized fields in a movie file.

This storage only works when the ffmpeg program and ffmpeg is installed. The default codec is FFV1, which supports lossless compression for various configurations. Not all video players support this codec, but VLC usually works quite well.

Note that important meta information is stored as a comment in the movie, so this data must not be deleted or altered if the video should be read again.

Warning

This storage potentially compresses data and can thus lead to loss of some information. The data quality depends on many parameters, but most important are the bits per channel of the video format and the range that is encoded (determined by vmin and vmax).

Note also that selecting individual time points might be quite slow since the video needs to be read from the beginning each time. Instead, it is much more efficient to process entire videos (by iterating over them or using items()).

Parameters:

filename (str) – The path where the movie is stored. The file extension determines the container format of the movie. The standard codec FFV1 plays well with the “.avi”, “.mkv”, and “.mov” container format.
vmin (float or array) – Lowest values that are encoded (per field). Smaller values are clipped to this value.
vmax (float or array) – Highest values that are encoded (per field). Larger values are clipped to this value.
bits_per_channel (int) – The number of bits used per color channel. Typical values are 8 and 16. The relative accuracy of stored values is 0.01 and 0.0001, respectively.
video_format (str) – Identifier for a video format from formats, which determines the number of channels, the bit depth of individual colors, and the codec. The special value auto tries to find a suitable format automatically, taking bits_per_channel into account.
bitrate (float) – The bitrate of the movie (in kilobits per second). The default value of -1 let’s the encoder choose an appropriate bit rate.
info (dict) – Supplies extra information that is stored in the storage alongside additional information necessary to reconstruct fields and grids.
write_mode (str) – Determines how new data is added to already existing data. Possible values are: ‘append’ (data is always appended), ‘truncate’ (data is cleared every time this storage is used for writing), or ‘truncate_once’ (data is cleared for the first writing, but appended subsequently). Alternatively, specifying ‘readonly’ will disable writing completely.
write_times (bool) – Flag determining whether timestamps are written to a file. If True, a separate file with name filename + ".times" is created where the times are written as plain text. Without these timestamps, the time information might be inaccurate.
loglevel (str) – FFmpeg log level determining the amount of data sent to stdout. The default only emits warnings and errors, but setting this to “info” can be useful to get additional information about the encoding.

clear()[source]: Truncate the storage by removing all stored data.

close()[source]

Close the currently opened file.

Return type:: None

property data

The actual data for all times.

Type:: ndarray

end_writing()[source]

Finalize the storage after writing.

Return type:: None

items()[source]

Iterate over all times and stored fields, returning pairs.

Return type:: Iterator[tuple[float, FieldBase]]

start_writing(field, info=None)[source]

Initialize the storage for writing data.

Parameters:

field (FieldBase) – An example of the data that will be written to extract the grid and the data_shape
info (dict) – Supplies extra information that is stored in the storage

Return type:

None

times

The times at which data is available.

Type:: ndarray

tracker(interrupts=1, *, transformation=None)[source]

Create object that can be used as a tracker to fill this storage.

Parameters:

interrupts (InterruptData) – Determines when the tracker interrupts the simulation. A single numbers determines an interval (measured in the simulation time unit) of regular interruption. A string is interpreted as a duration in real time assuming the format ‘hh:mm:ss’. A list of values is taken as explicit simulation time points. More fine- grained control is possible by passing an instance of classes defined in interrupts.
transformation (callable, optional) – A function that transforms the current state into a new field or field collection, which is then stored. This allows to store derived quantities of the field during calculations. The argument needs to be a callable function taking 1 or 2 arguments. The first argument always is the current field, while the optional second argument is the associated time.

Returns:

The tracker that fills the current storage

Return type:

StorageTracker

Example

The transformation argument allows storing additional fields:

def add_to_state(state):
    transformed_field = state.smooth(1)
    return field.append(transformed_field)


storage = pde.MemoryStorage()
tracker = storage.tracker(1, transformation=add_to_state)
eq.solve(..., tracker=tracker)

In this example, storage will contain a trajectory of the fields of the simulation as well as the smoothed fields. Other transformations are possible by defining appropriate add_to_state()

get_memory_storage(field, info=None)[source]

A context manager that can be used to create a MemoryStorage.

Example

This can be used to quickly store data:

with get_memory_storage(field_class) as storage:
    storage.append(numpy_array0, 0)
    storage.append(numpy_array1, 1)

# use storage thereafter

Parameters:

field (FieldBase) – An instance of the field class store data for a single time point.
info (dict) – Supplies extra information that is stored in the storage

Yields:

MemoryStorage