DownloadDataFile

class openmsistream.data_file_io.DownloadDataFile(filepath, *args, **kwargs)

Bases: DataFile, ABC

Class to represent a data file that will be read as messages from a topic

Parameters:

filepath (pathlib.Path) – Path to the file

static get_full_filepath(dfc)

Return the full filepath of a downloaded file given one of its DataFileChunks

Parameters:

dfc (DataFileChunk) – One of the DataFileChunk objects contributing to the file

Returns:

the full path to the file

Return type:

pathlib.Path

abstract property bytestring

A bytestring of the data contained in the file. Not implemented in the base class.

abstract property check_file_hash

The hash of the data in the file after it was read. Not implemented in the base class.

property relative_filepath

The path to the file, relative to its root directory

add_chunk(dfc, thread_lock=<contextlib.nullcontext object>)

Process a chunk that’s been read from a topic. Returns a number of codes based on what effect adding the chunk had.

This function calls _on_add_chunk(), with the DataFileChunk as the argument.

Parameters:

• dfc (DataFileChunk) – the DataFileChunk object whose data should be added

• thread_lock (threading.Lock, optional) – the lock object to acquire/release so that race conditions don’t affect reconstruction of the files (only needed if running this function asynchronously)

Returns:

an internal code indicating whether the chunk: was successfully added to a file in progress, was already received, was the last chunk needed and the file is successfully reconstructed according to its hash or the post-reconstruction hash is mismatched to the hash of the file contents originally read from disk.

Return type:

int

abstract _on_add_chunk(dfc)

A function to actually process a new chunk being added to the file. This function is executed while a thread lock is acquired so it will never run asynchronously. Also any DataFileChunks passed to this function are guaranteed to have unique offsets.

Not implemented in the base class.

Parameters:

dfc (DataFileChunk) – the DataFileChunk object whose data should be added