Dynamic URL Store

This module contains the DynamicURLStore store that communicates with a remote HTTP server which provides the actual data storage. This is a store which implements the basic operations via HTTP GET, POST, PUT and DELETE commands as described in the class documentation.

The implementation relies on the third-party requests library to handle the HTTP operations.

class encore.storage.dynamic_url_store.DynamicURLStore(base_url, query_url, url_format='{base}/{key}/{part}', url_format_no_part='{base}/{key}', parts={'data': 'data', 'metadata': 'metadata', 'permissions': 'auth'})[source]

Store implementation which gets and sets from a web server

This store expects a server which exposes URLs for each key. By default these URLs are of the form:

<base>/<key>/<part>

Where <base> is a common prefix, <key> is the key of interest, and <part> is one of “data”, “metadata” or “auth”. If the store does not follow this format, you can provide a differnt url_format argument and a different mapping of <part> to aspects of the key.

The server is expected to respond to queries against these URLS in the following ways:

GET <base>/<key>/data

return the bytes in the body of the response

PUT <base>/<key>/data

accept the data bytes from the body of the request

GET <base>/<key>/metadata

return metadata as JSON

PUT <base>/<key>/metadata

set the metadata based on JSON contained in the body of the request

POST <base>/<key>/metadata

update the metadata based on JSON contained in the body of the request (as dict.update())

GET <base>/<key>/auth

return permissions information as JSON

PUT <base>/<key>/auth

set the permissions based on JSON contained in the body of the request

POST <base>/<key>/metadata

update the permissions based on JSON contained in the body of the request

In addition, a DELETE request to a URL of the form <base>/<key> should remove the key from the remote store. This pattern is configurable via the url_format_no_part argument to the constructor.

In addition, the server should have a query URL which accepts GET reuqests containing a JSON data structure of metadata key, value pairs to filter with, and should return a list of macthing keys, one per line.

connect(credentials=None)[source]

Connect to a DynamicURLStore

Parameters:

credentials ((user_tag, requests.Session)) – The credentials are a tuple containing ther user’s permission tag and a requests Session initialized with appropriate authentication.

delete(key)[source]

Delete a key from the repsository.

This may be left unimplemented by subclasses that represent a read-only key-value store.

Parameters:

  • key (string) – The key for the resource in the key-value store. They key is a unique identifier for the resource within the key-value store.

  • Events –

  • ------ –

  • StoreDeleteEvent – On successful completion of a transaction, a StoreDeleteEvent should be emitted with the key.

disconnect()[source]

Disconnect from the key-value store

This method disposes or disconnects to any long-lived resources that the store requires.

get(key)[source]

Retrieve a stream of data and metdata from a given key in the key-value store.

Parameters:

key (string) – The key for the resource in the key-value store. They key is a unique identifier for the resource within the key-value store.

Returns:

value – An instance of a Value subclass which holds references to the data, metadata and other information about the key.

Return type:

instance of Value

Raises:

KeyError : – If the key is not found in the store, a KeyError is raised.

get_data(key)[source]

Retrieve a stream from a given key in the key-value store.

Parameters:

key (string) – The key for the resource in the key-value store. They key is a unique identifier for the resource within the key-value store.

Returns:

data – A readable file-like object the that provides stream of data from the key-value store.

Return type:

file-like

Raises:

KeyError : – This will raise a key error if the key is not present in the store.

get_metadata(key, select=None)[source]

Retrieve the metadata for a given key in the key-value store.

Parameters:

  • key (string) – The key for the resource in the key-value store. They key is a unique identifier for the resource within the key-value store.

  • select (iterable of strings or None) – Which metadata keys to populate in the result. If unspecified, then return the entire metadata dictionary.

Returns:

metadata – A dictionary of metadata associated with the key. The dictionary has keys as specified by the select argument. If a key specified in select is not present in the metadata, then it will not be present in the returned value.

Return type:

dict

Raises:

KeyError : – This will raise a key error if the key is not present in the store.

get_permissions(key)[source]

Return the set of permissions the user has

Parameters:

key (str) – The key for the resource which you want to know the permissions.

Returns:

permissions – A dictionary whose keys are the permissions and values are sets of tags which have that permission.

Return type:

dict of str: set of str

Raises:

  • KeyError : – This error will be raised if the key does not exist or the user is not authorized to see it.

  • AuthorizationError : – This error will be raised if user is authorized to see the key, but is not an owner.

info()[source]

Get information about the key-value store

Returns:

metadata – A dictionary of metadata giving information about the key-value store.

Return type:

dict

is_connected()[source]

Whether or not the store is currently connected

Returns:

connected – Whether or not the store is currently connected.

Return type:

bool

query(select=None, **kwargs)[source]

Query for keys and metadata matching metadata provided as keyword arguments

This provides a very simple querying interface that returns precise matches with the metadata. If no arguments are supplied, the query will return the complete set of metadata for the key-value store.

Parameters:

  • select (iterable of strings or None) – An optional list of metadata keys to return. If this is not None, then the metadata dictionaries will only have values for the specified keys populated.

  • kwargs – Arguments where the keywords are metadata keys, and values are possible values for that metadata item.

Returns:

result – An iterable of (key, metadata) tuples where metadata matches all the specified values for the specified metadata keywords. If a key specified in select is not present in the metadata of a particular key, then it will not be present in the returned value.

Return type:

iterable

query_keys(**kwargs)[source]

Query for keys matching metadata provided as keyword arguments

This provides a very simple querying interface that returns precise matches with the metadata. If no arguments are supplied, the query will return the complete set of keys for the key-value store.

This is equivalent to self.query(**kwargs).keys(), but potentially more efficiently implemented.

Parameters:

kwargs – Arguments where the keywords are metadata keys, and values are possible values for that metadata item.

Returns:

result – An iterable of key-value store keys whose metadata matches all the specified values for the specified metadata keywords.

Return type:

iterable

set(key, value, buffer_size=1048576)[source]

Store a stream of data into a given key in the key-value store.

This may be left unimplemented by subclasses that represent a read-only key-value store.

Parameters:

  • key (string) – The key for the resource in the key-value store. They key is a unique identifier for the resource within the key-value store.

  • value (instance of Value) – An instance of a Value subclass.

  • buffer_size (int) – An optional indicator of the number of bytes to read at a time. Implementations are free to ignore this hint or use a different default if they need to. The default is 1048576 bytes (1 MiB).

  • Events –

  • ------ –

  • StoreProgressStartEvent – For buffering implementations, this event should be emitted prior to writing any data to the underlying store.

  • StoreProgressStepEvent – For buffering implementations, this event should be emitted periodically as data is written to the underlying store.

  • StoreProgressEndEvent – For buffering implementations, this event should be emitted after finishing writing to the underlying store.

  • StoreSetEvent – On successful completion of a transaction, a StoreSetEvent should be emitted with the key & metadata

set_data(key, data, buffer_size=1048576)[source]

Replace the data for a given key in the key-value store.

Parameters:

  • key (string) – The key for the resource in the key-value store. They key is a unique identifier for the resource within the key-value store.

  • data (file-like) – A readable file-like object the that provides stream of data from the key-value store.

  • buffer_size (int) – An optional indicator of the number of bytes to read at a time. Implementations are free to ignore this hint or use a different default if they need to. The default is 1048576 bytes (1 MiB).

  • Events –

  • ------ –

  • StoreProgressStartEvent – For buffering implementations, this event should be emitted prior to writing any data to the underlying store.

  • StoreProgressStepEvent – For buffering implementations, this event should be emitted periodically as data is written to the underlying store.

  • StoreProgressEndEvent – For buffering implementations, this event should be emitted after finishing writing to the underlying store.

  • StoreSetEvent – On successful completion of a transaction, a StoreSetEvent should be emitted with the key & metadata

set_metadata(key, metadata)[source]

Set new metadata for a given key in the key-value store.

This replaces the existing metadata set for the key with a new set of metadata.

Parameters:

  • key (string) – The key for the resource in the key-value store. They key is a unique identifier for the resource within the key-value store.

  • metadata (dict) – A dictionary of metadata to associate with the key. The dictionary keys should be strings which are valid Python identifiers.

  • Events –

  • ------ –

  • StoreSetEvent – On successful completion of a transaction, a StoreSetEvent should be emitted with the key & metadata

set_permissions(key, permissions)[source]

Set the permissions on a key the user owns

Parameters:

  • key (str) – The key for the resource which you want to know the permissions.

  • permissions (dict of str: set of str) – A dictionary whose keys are the permissions and values are sets of tags which have that permission. There must be an ‘owned’ permission with at least one tag.

Raises:

  • KeyError : – This error will be raised if the key does not exist or the user is not authorized to see it.

  • AuthorizationError : – This error will be raised if user is authorized to see the key, but is not an owner.

transaction(notes)[source]

Provide a transaction context manager

This class does not support transactions, so it returns a dummy object.

update_metadata(key, metadata)[source]

Update the metadata for a given key in the key-value store.

This performs a dictionary update on the existing metadata with the provided metadata keys and values

Parameters:

  • key (string) – The key for the resource in the key-value store. They key is a unique identifier for the resource within the key-value store.

  • metadata (dict) – A dictionary of metadata to associate with the key. The dictionary keys should be strings which are valid Python identifiers.

  • Events –

  • ------ –

  • StoreSetEvent – On successful completion of a transaction, a StoreSetEvent should be emitted with the key & metadata

update_permissions(key, permissions)[source]

Add permissions on a key the user owns

The tags provided in the permissions dictionary will be added to the existing set of tags for each permission.

Parameters:

  • key (str) – The key for the resource which you want to know the permissions.

  • permissions (dict of str: set of str) – A dictionary whose keys are the permissions and values are sets of tags which have that permission.

Raises:

  • KeyError : – This error will be raised if the key does not exist or the user is not authorized to see it.

  • AuthorizationError : – This error will be raised if user is authorized to see the key, but is not an owner.

user_tag()[source]

A tag that represents the user