API

Dogpile

class dogpile.core.Lock(mutex, creator, value_and_created_fn, expiretime, async_creator=None)

Dogpile lock class.

Provides an interface around an arbitrary mutex that allows one thread/process to be elected as the creator of a new value, while other threads/processes continue to return the previous version of that value.

New in version 0.4.0: The Lock class was added as a single-use object representing the dogpile API without dependence on any shared state between multiple instances.

Parameters:

• mutex – A mutex object that provides acquire() and release() methods.
• creator – Callable which returns a tuple of the form (new_value, creation_time). “new_value” should be a newly generated value representing completed state. “creation_time” should be a floating point time value which is relative to Python’s time.time() call, representing the time at which the value was created. This time value should be associated with the created value.
• value_and_created_fn – Callable which returns a tuple of the form (existing_value, creation_time). This basically should return what the last local call to the creator() callable has returned, i.e. the value and the creation time, which would be assumed here to be from a cache. If the value is not available, the NeedRegenerationException exception should be thrown.
• expiretime – Expiration time in seconds. Set to None for never expires. This timestamp is compared to the creation_time result and time.time() to determine if the value returned by value_and_created_fn is “expired”.
• async_creator –

  A callable. If specified, this callable will be passed the mutex as an argument and is responsible for releasing the mutex after it finishes some asynchronous value creation. The intent is for this to be used to defer invocation of the creator callable until some later time.

  New in version 0.4.1: added the async_creator argument.

class dogpile.core.NeedRegenerationException

An exception that when raised in the ‘with’ block, forces the ‘has_value’ flag to False and incurs a regeneration of the value.

Utilities

class dogpile.core.ReadWriteMutex

A mutex which allows multiple readers, single writer.

ReadWriteMutex uses a Python threading.Condition to provide this functionality across threads within a process.

The Beaker package also contained a file-lock based version of this concept, so that readers/writers could be synchronized across processes with a common filesystem. A future Dogpile release may include this additional class at some point.

acquire_read_lock(wait=True)

Acquire the ‘read’ lock.

acquire_write_lock(wait=True)

Acquire the ‘write’ lock.

release_read_lock()

Release the ‘read’ lock.

release_write_lock()

Release the ‘write’ lock.

class dogpile.core.NameRegistry(creator)

Generates and return an object, keeping it as a singleton for a certain identifier for as long as its strongly referenced.

e.g.:

class MyFoo(object):
    "some important object."
    def __init__(self, identifier):
        self.identifier = identifier

registry = NameRegistry(MyFoo)

# thread 1:
my_foo = registry.get("foo1")

# thread 2
my_foo = registry.get("foo1")

Above, my_foo in both thread #1 and #2 will be the same object. The constructor for MyFoo will be called once, passing the identifier foo1 as the argument.

When thread 1 and thread 2 both complete or otherwise delete references to my_foo, the object is removed from the NameRegistry as a result of Python garbage collection.

Parameters:creator – A function that will create a new value, given the identifier passed to the NameRegistry.get() method.

get(identifier, *args, **kw)

Get and possibly create the value.

Parameters:

• identifier – Hash key for the value. If the creation function is called, this identifier will also be passed to the creation function.
• **kw (*args,) – Additional arguments which will also be passed to the creation function if it is called.

Legacy API

class dogpile.core.Dogpile(expiretime, init=False, lock=None)

Dogpile lock class.

Deprecated since version 0.4.0: The Lock object specifies the full API of the Dogpile object in a single way, rather than providing multiple modes of usage which don’t necessarily work in the majority of cases. Dogpile is now a wrapper around the Lock object which provides dogpile.core’s original usage pattern. This usage pattern began as something simple, but was not of general use in real-world caching environments without several extra complicating factors; the Lock object presents the “real-world” API more succinctly, and also fixes a cross-process concurrency issue.

Parameters:

• expiretime – Expiration time in seconds. Set to None for never expires.
• init – if True, set the ‘createdtime’ to the current time.
• lock – a mutex object that provides acquire() and release() methods.

acquire(creator, value_fn=None, value_and_created_fn=None)

Acquire the lock, returning a context manager.

Parameters:

• creator – Creation function, used if this thread is chosen to create a new value.
• value_fn – Optional function that returns the value from some datasource. Will be returned if regeneration is not needed.
• value_and_created_fn – Like value_fn, but returns a tuple of (value, createdtime). The returned createdtime will replace the “createdtime” value on this dogpile lock. This option removes the need for the dogpile lock itself to remain persistent across usages; another dogpile can come along later and pick up where the previous one left off.

createdtime = -1

The last known ‘creation time’ of the value, stored as an epoch (i.e. from time.time()).

If the value here is -1, it is assumed the value should recreate immediately.

has_value

Return true if the creation function has proceeded at least once.

is_expired

Return true if the expiration time is reached, or no value is available.

class dogpile.core.SyncReaderDogpile(*args, **kw)

Provide a read-write lock function on top of the Dogpile class.

Deprecated since version 0.4.0: The ReadWriteMutex object can be used directly.

acquire_write_lock(*args, **kwds)

Return the “write” lock context manager.

This will provide a section that is mutexed against all readers/writers for the dogpile-maintained value.