Utils

Utilities for key-value stores.

File-like Interface Utilities

These utilities help with the management of file-like objects. In particular buffer_iterator() is of particular use, as it produces an iterator which generates chunks of bytes in the file-like object which permits memory-efficient streaming of the data. This is preferred over reading in all the data and then processing it if the data is even moderately big.

The BufferIteratorIO class is a class whick provides a file-like API around a buffer iterator. This is particularly useful for Stores which wrap another store and implementing streaming filters on the data.

class encore.storage.utils.BufferIteratorIO(iterator)

A file-like object based on an iterable of buffers

This takes an iterator of bytes objects, such as produced by the buffer_iterator function, and wraps it in a file-like interface which is usable with the store API.

This uses less memory than a StringIO, at the cost of some flexibility.

Parameters:

iterator (iterator of bytes objects) – An iterator that produces a bytes object on each iteration.

read(buffer_size=1048576)

Read at most buffer_size bytes, returned as a string.

encore.storage.utils.buffer_iterator(filelike, buffer_size=1048576, progress=None, max_bytes=None)

Return an iterator of byte buffers

The buffers of bytes default to the provided buffer_size. This is a useful method when copying one data stream to another.

Parameters:

• filelike (a file-like object) – An object which implements the read(buffer_size)() method.

• buffer_size (int) – The number of bytes to read at a time.

• progress (callable) – A callback for progress indication. A StoreProgressManager instance inside a with block would be appropriate, but anthing that takes a step parameter which is the total number of bytes read so far will work.

• max_bytes (int) – The maximum number of bytes to return.

encore.storage.utils.tee(filelike, n=2, buffer_size=1048576)

Clone a filelike stream into n parallel streams

This uses itertools.tee and buffer iterators, with the corresponding cautions about memory usage. In general it should be more memory efficient than pulling everything into memory.

Parameters:

• filelike (a file-like object) – An object which implements the read(buffer_size)() method.

• n (int) – The number of filelike streams to produce.

• buffer_size (int) – The number of bytes to read at a time.

Transaction Support

These are two simple context managers for transactions. The DummyTransactionContext should be used by Store implementations which have no notion of a transaction. The SimpleTransactionContext is a complete transaction manager for implementations with begin/commit/rollback semantics.

class encore.storage.utils.DummyTransactionContext(store)

A dummy class that can be returned by stores which don’t support transactions

This class guarantees that there is only one transaction object for each store instance.

Parameters:

store (key-value store instance) – The store that this transaction context is associated with.

class encore.storage.utils.SimpleTransactionContext(store)

A simple class that adds support for simple transactions

This is a base class that ensures transactions are appropriately handled in terms of nesting and event generation. Subclasses should override the start, commit and rollback methods to perform appropriate implementation-specific actions.

This class correctly handles nested transactions by ensuring that each store has precisely one active transaction context and by tracking the number of times the context has been entered and exited. The transaction is only committed once the top-level context has exited.

Parameters:

store (key-value store instance) – The store that this transaction context is associated with.

begin()

Begin a transaction

By default, this calls the store’s _begin_transaction method. Override in subclasses if you need different behaviour.

commit()

Commit a transaction

By default, this calls the store’s _commit_transaction method. Override in subclasses if you need different behaviour.

rollback()

Roll back a transaction

By default, this calls the store’s _rollback_transaction method. Override in subclasses if you need different behaviour.

Event Support

class encore.storage.utils.StoreProgressManager(event_manager=None, source=None, operation_id=None, message='Performing operation', steps=-1, **kwargs)

encore.events.progress_events.ProgressManager subclass that generates encore.storage.events.StoreProgressEvent instances

EndEventType

alias of StoreProgressEndEvent

StartEventType

alias of StoreProgressStartEvent

StepEventType

alias of StoreProgressStepEvent