spinn_storage_handlers package

Submodules

spinn_storage_handlers.buffered_bytearray_data_storage module

class spinn_storage_handlers.buffered_bytearray_data_storage.BufferedBytearrayDataStorage

Bases: spinn_storage_handlers.abstract_classes.abstract_buffered_data_storage.AbstractBufferedDataStorage, spinn_storage_handlers.abstract_classes.abstract_context_manager.AbstractContextManager

Data storage based on a bytearray buffer with two pointers, one for reading and one for writing.

close()

Closes the data storage.

Return type:None
Raises:spinn_storage_handlers.exceptions.DataReadException – If the data storage cannot be closed
eof()

Check if the read pointer is a the end of the data storage.

Returns:Whether the read pointer is at the end of the data storage
Return type:bool
read(data_size)

Read data from the data storage from the position indicated by the read pointer index.

Parameters:data_size (int) – number of bytes to be read
Returns:a bytearray containing the data read
Return type:bytearray
read_all()

Read all the data contained in the data storage starting from position 0 to the end.

Returns:a bytearray containing the data read
Return type:bytearray
readinto(data)

Read some bytes of data from the underlying storage into a predefined array. Will block until some bytes are available, but may not fill the array completely.

Parameters:data (bytearray) – The place where the data is to be stored
Returns:The number of bytes stored in data
Return type:int
Raises:IOError – If an error occurs reading from the underlying storage
seek_read(offset, whence=0)

Set the data storage’s current read position to the offset.

Parameters:
  • offset (int) – Position of the read pointer within the buffer
  • whence – One of: * os.SEEK_SET which means absolute buffer positioning (default) * os.SEEK_CUR which means seek relative to the current read position * os.SEEK_END which means seek relative to the buffer’s end
Return type:

None
seek_write(offset, whence=0)

Set the data storage’s current write position to the offset.

Parameters:
  • offset (int) – Position of the write pointer within the buffer
  • whence – One of: * os.SEEK_SET which means absolute buffer positioning (default) * os.SEEK_CUR which means seek relative to the current write position * os.SEEK_END which means seek relative to the buffer’s end
Return type:

None
tell_read()

The current position of the read pointer.

Returns:The current position of the read pointer
Return type:int
tell_write()

The current position of the write pointer.

Returns:The current position of the write pointer
Return type:int
write(data)

Store data in the data storage in the position indicated by the write pointer index.

Parameters:data (bytearray) – the data to be stored
Return type:None

spinn_storage_handlers.buffered_file_data_storage module

class spinn_storage_handlers.buffered_file_data_storage.BufferedFileDataStorage(filename, mode)

Bases: spinn_storage_handlers.abstract_classes.abstract_buffered_data_storage.AbstractBufferedDataStorage, spinn_storage_handlers.abstract_classes.abstract_context_manager.AbstractContextManager

Data storage based on a temporary file with two pointers, one for reading and one for writing.

close()

Closes the data storage.

Return type:None
Raises:spinn_storage_handlers.exceptions.DataReadException – If the data storage cannot be closed
eof()

Check if the read pointer is a the end of the data storage.

Returns:Whether the read pointer is at the end of the data storage
Return type:bool
filename

The name of the file.

read(data_size)

Read data from the data storage from the position indicated by the read pointer index.

Parameters:data_size (int) – number of bytes to be read
Returns:a bytearray containing the data read
Return type:bytearray
read_all()

Read all the data contained in the data storage starting from position 0 to the end.

Returns:a bytearray containing the data read
Return type:bytearray
readinto(data)

Read some bytes of data from the underlying storage into a predefined array. Will block until some bytes are available, but may not fill the array completely.

Parameters:data (bytearray) – The place where the data is to be stored
Returns:The number of bytes stored in data
Return type:int
Raises:IOError – If an error occurs reading from the underlying storage
seek_read(offset, whence=0)

Set the data storage’s current read position to the offset.

Parameters:
  • offset (int) – Position of the read pointer within the buffer
  • whence – One of: * os.SEEK_SET which means absolute buffer positioning (default) * os.SEEK_CUR which means seek relative to the current read position * os.SEEK_END which means seek relative to the buffer’s end
Return type:

None
seek_write(offset, whence=0)

Set the data storage’s current write position to the offset.

Parameters:
  • offset (int) – Position of the write pointer within the buffer
  • whence – One of: * os.SEEK_SET which means absolute buffer positioning (default) * os.SEEK_CUR which means seek relative to the current write position * os.SEEK_END which means seek relative to the buffer’s end
Return type:

None
tell_read()

The current position of the read pointer.

Returns:The current position of the read pointer
Return type:int
tell_write()

The current position of the write pointer.

Returns:The current position of the write pointer
Return type:int
write(data)

Store data in the data storage in the position indicated by the write pointer index.

Parameters:data (bytearray) – the data to be stored
Return type:None

spinn_storage_handlers.buffered_tempfile_data_storage module

class spinn_storage_handlers.buffered_tempfile_data_storage.BufferedTempfileDataStorage

Bases: spinn_storage_handlers.abstract_classes.abstract_buffered_data_storage.AbstractBufferedDataStorage, spinn_storage_handlers.abstract_classes.abstract_context_manager.AbstractContextManager

Data storage based on a temporary file with two pointers, one for reading and one for writing. Under the covers, it actually opens and closes the temporary file as it chooses in order to limit the number of file descriptors in use.

close()

Closes the data storage.

Return type:None
Raises:spinn_storage_handlers.exceptions.DataReadException – If the data storage cannot be closed
eof()

Check if the read pointer is a the end of the data storage.

Returns:Whether the read pointer is at the end of the data storage
Return type:bool
classmethod initialise(lru_max)
read(data_size)

Read data from the data storage from the position indicated by the read pointer index.

Parameters:data_size (int) – number of bytes to be read
Returns:a bytearray containing the data read
Return type:bytearray
read_all()

Read all the data contained in the data storage starting from position 0 to the end.

Returns:a bytearray containing the data read
Return type:bytearray
readinto(data)

Read some bytes of data from the underlying storage into a predefined array. Will block until some bytes are available, but may not fill the array completely.

Parameters:data (bytearray) – The place where the data is to be stored
Returns:The number of bytes stored in data
Return type:int
Raises:IOError – If an error occurs reading from the underlying storage
seek_read(offset, whence=0)

Set the data storage’s current read position to the offset.

Parameters:
  • offset (int) – Position of the read pointer within the buffer
  • whence – One of: * os.SEEK_SET which means absolute buffer positioning (default) * os.SEEK_CUR which means seek relative to the current read position * os.SEEK_END which means seek relative to the buffer’s end
Return type:

None
seek_write(offset, whence=0)

Set the data storage’s current write position to the offset.

Parameters:
  • offset (int) – Position of the write pointer within the buffer
  • whence – One of: * os.SEEK_SET which means absolute buffer positioning (default) * os.SEEK_CUR which means seek relative to the current write position * os.SEEK_END which means seek relative to the buffer’s end
Return type:

None
tell_read()

The current position of the read pointer.

Returns:The current position of the read pointer
Return type:int
tell_write()

The current position of the write pointer.

Returns:The current position of the write pointer
Return type:int
write(data)

Store data in the data storage in the position indicated by the write pointer index.

Parameters:data (bytearray) – the data to be stored
Return type:None

spinn_storage_handlers.exceptions module

exception spinn_storage_handlers.exceptions.BufferedBytearrayOperationNotImplemented

Bases: exceptions.NotImplementedError

An exception that denotes that the operation required is unavailable for a byteArray buffer

exception spinn_storage_handlers.exceptions.DataReadException

Bases: exceptions.Exception

An exception that indicates that there was an error reading from the underlying medium

exception spinn_storage_handlers.exceptions.DataWriteException

Bases: exceptions.Exception

An exception that indicates that there was an error writing to the underlying medium

spinn_storage_handlers.file_data_reader module

class spinn_storage_handlers.file_data_reader.FileDataReader(filename)

Bases: spinn_storage_handlers.abstract_classes.abstract_data_reader.AbstractDataReader, spinn_storage_handlers.abstract_classes.abstract_context_manager.AbstractContextManager

A reader that can read data from a file.

Parameters:filename (str) – The file to read
Raises:spinn_storage_handlers.exceptions.DataReadException – If the file cannot found or opened for reading
close()

Closes the file.

Return type:None
Raises:spinn_storage_handlers.exceptions.DataReadException – If the file cannot be closed
read(n_bytes)

Read some bytes of data from the underlying storage. Will block until some bytes are available, but might not return the full n_bytes. The size of the returned array indicates how many bytes were read.

Parameters:n_bytes (int) – The number of bytes to read
Returns:An array of bytes
Return type:bytearray
Raises:IOError – If an error occurs reading from the underlying storage
readall()

Read the rest of the bytes from the underlying stream.

Returns:The bytes read
Return type:bytearray
Raises:IOError – If there is an error obtaining the bytes
readinto(data)

Read some bytes of data from the underlying storage into a predefined array. Will block until some bytes are available, but may not fill the array completely.

Parameters:data (bytearray) – The place where the data is to be stored
Returns:The number of bytes stored in data
Return type:int
Raises:IOError – If an error occurs reading from the underlying storage
tell()

Returns the position of the file cursor.

Returns:Position of the file cursor
Return type:int

spinn_storage_handlers.file_data_writer module

class spinn_storage_handlers.file_data_writer.FileDataWriter(filename)

Bases: spinn_storage_handlers.abstract_classes.abstract_data_writer.AbstractDataWriter, spinn_storage_handlers.abstract_classes.abstract_context_manager.AbstractContextManager

Parameters:filename (str) – The file to write to
Raises:spinn_storage_handlers.exceptions.DataWriteException – If the file cannot found or opened for writing
close()

Closes the file.

Return type:None
Raises:spinn_storage_handlers.exceptions.DataWriteException – If the file cannot be closed
filename

The name of the file that is being written to.

tell()

Returns the position of the file cursor.

Returns:Position of the file cursor
Return type:int
write(data)

Write some bytes of data to the underlying storage. Does not return until all the bytes have been written.

Parameters:data (bytearray or bytes) – The data to write
Returns:Nothing is returned
Return type:None
Raises:IOError – If an error occurs writing to the underlying storage

spinn_storage_handlers.utils module

spinn_storage_handlers.utils.file_length(f)

The size of an open file.

Parameters:f (file) – The file to get the size of
Returns:The size of the file
Return type:int

Module contents

class spinn_storage_handlers.BufferedBytearrayDataStorage

Bases: spinn_storage_handlers.abstract_classes.abstract_buffered_data_storage.AbstractBufferedDataStorage, spinn_storage_handlers.abstract_classes.abstract_context_manager.AbstractContextManager

Data storage based on a bytearray buffer with two pointers, one for reading and one for writing.

close()

Closes the data storage.

Return type:None
Raises:spinn_storage_handlers.exceptions.DataReadException – If the data storage cannot be closed
eof()

Check if the read pointer is a the end of the data storage.

Returns:Whether the read pointer is at the end of the data storage
Return type:bool
read(data_size)

Read data from the data storage from the position indicated by the read pointer index.

Parameters:data_size (int) – number of bytes to be read
Returns:a bytearray containing the data read
Return type:bytearray
read_all()

Read all the data contained in the data storage starting from position 0 to the end.

Returns:a bytearray containing the data read
Return type:bytearray
readinto(data)

Read some bytes of data from the underlying storage into a predefined array. Will block until some bytes are available, but may not fill the array completely.

Parameters:data (bytearray) – The place where the data is to be stored
Returns:The number of bytes stored in data
Return type:int
Raises:IOError – If an error occurs reading from the underlying storage
seek_read(offset, whence=0)

Set the data storage’s current read position to the offset.

Parameters:
  • offset (int) – Position of the read pointer within the buffer
  • whence – One of: * os.SEEK_SET which means absolute buffer positioning (default) * os.SEEK_CUR which means seek relative to the current read position * os.SEEK_END which means seek relative to the buffer’s end
Return type:

None
seek_write(offset, whence=0)

Set the data storage’s current write position to the offset.

Parameters:
  • offset (int) – Position of the write pointer within the buffer
  • whence – One of: * os.SEEK_SET which means absolute buffer positioning (default) * os.SEEK_CUR which means seek relative to the current write position * os.SEEK_END which means seek relative to the buffer’s end
Return type:

None
tell_read()

The current position of the read pointer.

Returns:The current position of the read pointer
Return type:int
tell_write()

The current position of the write pointer.

Returns:The current position of the write pointer
Return type:int
write(data)

Store data in the data storage in the position indicated by the write pointer index.

Parameters:data (bytearray) – the data to be stored
Return type:None
class spinn_storage_handlers.BufferedFileDataStorage(filename, mode)

Bases: spinn_storage_handlers.abstract_classes.abstract_buffered_data_storage.AbstractBufferedDataStorage, spinn_storage_handlers.abstract_classes.abstract_context_manager.AbstractContextManager

Data storage based on a temporary file with two pointers, one for reading and one for writing.

close()

Closes the data storage.

Return type:None
Raises:spinn_storage_handlers.exceptions.DataReadException – If the data storage cannot be closed
eof()

Check if the read pointer is a the end of the data storage.

Returns:Whether the read pointer is at the end of the data storage
Return type:bool
filename

The name of the file.

read(data_size)

Read data from the data storage from the position indicated by the read pointer index.

Parameters:data_size (int) – number of bytes to be read
Returns:a bytearray containing the data read
Return type:bytearray
read_all()

Read all the data contained in the data storage starting from position 0 to the end.

Returns:a bytearray containing the data read
Return type:bytearray
readinto(data)

Read some bytes of data from the underlying storage into a predefined array. Will block until some bytes are available, but may not fill the array completely.

Parameters:data (bytearray) – The place where the data is to be stored
Returns:The number of bytes stored in data
Return type:int
Raises:IOError – If an error occurs reading from the underlying storage
seek_read(offset, whence=0)

Set the data storage’s current read position to the offset.

Parameters:
  • offset (int) – Position of the read pointer within the buffer
  • whence – One of: * os.SEEK_SET which means absolute buffer positioning (default) * os.SEEK_CUR which means seek relative to the current read position * os.SEEK_END which means seek relative to the buffer’s end
Return type:

None
seek_write(offset, whence=0)

Set the data storage’s current write position to the offset.

Parameters:
  • offset (int) – Position of the write pointer within the buffer
  • whence – One of: * os.SEEK_SET which means absolute buffer positioning (default) * os.SEEK_CUR which means seek relative to the current write position * os.SEEK_END which means seek relative to the buffer’s end
Return type:

None
tell_read()

The current position of the read pointer.

Returns:The current position of the read pointer
Return type:int
tell_write()

The current position of the write pointer.

Returns:The current position of the write pointer
Return type:int
write(data)

Store data in the data storage in the position indicated by the write pointer index.

Parameters:data (bytearray) – the data to be stored
Return type:None
class spinn_storage_handlers.BufferedTempfileDataStorage

Bases: spinn_storage_handlers.abstract_classes.abstract_buffered_data_storage.AbstractBufferedDataStorage, spinn_storage_handlers.abstract_classes.abstract_context_manager.AbstractContextManager

Data storage based on a temporary file with two pointers, one for reading and one for writing. Under the covers, it actually opens and closes the temporary file as it chooses in order to limit the number of file descriptors in use.

close()

Closes the data storage.

Return type:None
Raises:spinn_storage_handlers.exceptions.DataReadException – If the data storage cannot be closed
eof()

Check if the read pointer is a the end of the data storage.

Returns:Whether the read pointer is at the end of the data storage
Return type:bool
classmethod initialise(lru_max)
read(data_size)

Read data from the data storage from the position indicated by the read pointer index.

Parameters:data_size (int) – number of bytes to be read
Returns:a bytearray containing the data read
Return type:bytearray
read_all()

Read all the data contained in the data storage starting from position 0 to the end.

Returns:a bytearray containing the data read
Return type:bytearray
readinto(data)

Read some bytes of data from the underlying storage into a predefined array. Will block until some bytes are available, but may not fill the array completely.

Parameters:data (bytearray) – The place where the data is to be stored
Returns:The number of bytes stored in data
Return type:int
Raises:IOError – If an error occurs reading from the underlying storage
seek_read(offset, whence=0)

Set the data storage’s current read position to the offset.

Parameters:
  • offset (int) – Position of the read pointer within the buffer
  • whence – One of: * os.SEEK_SET which means absolute buffer positioning (default) * os.SEEK_CUR which means seek relative to the current read position * os.SEEK_END which means seek relative to the buffer’s end
Return type:

None
seek_write(offset, whence=0)

Set the data storage’s current write position to the offset.

Parameters:
  • offset (int) – Position of the write pointer within the buffer
  • whence – One of: * os.SEEK_SET which means absolute buffer positioning (default) * os.SEEK_CUR which means seek relative to the current write position * os.SEEK_END which means seek relative to the buffer’s end
Return type:

None
tell_read()

The current position of the read pointer.

Returns:The current position of the read pointer
Return type:int
tell_write()

The current position of the write pointer.

Returns:The current position of the write pointer
Return type:int
write(data)

Store data in the data storage in the position indicated by the write pointer index.

Parameters:data (bytearray) – the data to be stored
Return type:None
class spinn_storage_handlers.FileDataReader(filename)

Bases: spinn_storage_handlers.abstract_classes.abstract_data_reader.AbstractDataReader, spinn_storage_handlers.abstract_classes.abstract_context_manager.AbstractContextManager

A reader that can read data from a file.

Parameters:filename (str) – The file to read
Raises:spinn_storage_handlers.exceptions.DataReadException – If the file cannot found or opened for reading
close()

Closes the file.

Return type:None
Raises:spinn_storage_handlers.exceptions.DataReadException – If the file cannot be closed
read(n_bytes)

Read some bytes of data from the underlying storage. Will block until some bytes are available, but might not return the full n_bytes. The size of the returned array indicates how many bytes were read.

Parameters:n_bytes (int) – The number of bytes to read
Returns:An array of bytes
Return type:bytearray
Raises:IOError – If an error occurs reading from the underlying storage
readall()

Read the rest of the bytes from the underlying stream.

Returns:The bytes read
Return type:bytearray
Raises:IOError – If there is an error obtaining the bytes
readinto(data)

Read some bytes of data from the underlying storage into a predefined array. Will block until some bytes are available, but may not fill the array completely.

Parameters:data (bytearray) – The place where the data is to be stored
Returns:The number of bytes stored in data
Return type:int
Raises:IOError – If an error occurs reading from the underlying storage
tell()

Returns the position of the file cursor.

Returns:Position of the file cursor
Return type:int
class spinn_storage_handlers.FileDataWriter(filename)

Bases: spinn_storage_handlers.abstract_classes.abstract_data_writer.AbstractDataWriter, spinn_storage_handlers.abstract_classes.abstract_context_manager.AbstractContextManager

Parameters:filename (str) – The file to write to
Raises:spinn_storage_handlers.exceptions.DataWriteException – If the file cannot found or opened for writing
close()

Closes the file.

Return type:None
Raises:spinn_storage_handlers.exceptions.DataWriteException – If the file cannot be closed
filename

The name of the file that is being written to.

tell()

Returns the position of the file cursor.

Returns:Position of the file cursor
Return type:int
write(data)

Write some bytes of data to the underlying storage. Does not return until all the bytes have been written.

Parameters:data (bytearray or bytes) – The data to write
Returns:Nothing is returned
Return type:None
Raises:IOError – If an error occurs writing to the underlying storage