keystone.common.cache.backends package

Submodules

keystone.common.cache.backends.memcache_pool module

dogpile.cache backend that uses Memcached connection pool

class keystone.common.cache.backends.memcache_pool.ClientProxy(client_pool)[source]

Bases: object

class keystone.common.cache.backends.memcache_pool.PooledMemcachedBackend(arguments)[source]

Bases: dogpile.cache.backends.memcached.MemcachedBackend

client[source]

keystone.common.cache.backends.mongo module

class keystone.common.cache.backends.mongo.AbstractManipulator[source]

Bases: object

Abstract class with methods which need to be implemented for custom manipulation.

Adding this as a base class for BaseTransform instead of adding import dependency of pymongo specific class i.e. pymongo.son_manipulator.SONManipulator and using that as base class. This is done to avoid pymongo dependency if MongoDB backend is not used.

transform_incoming(son, collection)[source]

Used while saving data to MongoDB.

Parameters:
  • son – the SON object to be inserted into the database
  • collection – the collection the object is being inserted into
Returns:

transformed SON object

transform_outgoing(son, collection)[source]

Used while reading data from MongoDB.

Parameters:
  • son – the SON object being retrieved from the database
  • collection – the collection this object was stored in
Returns:

transformed SON object

will_copy()[source]

Will this SON manipulator make a copy of the incoming document?

Derived classes that do need to make a copy should override this method, returning True instead of False.

Returns:boolean
class keystone.common.cache.backends.mongo.BaseTransform[source]

Bases: keystone.common.cache.backends.mongo.AbstractManipulator

Base transformation class to store and read dogpile cached data from MongoDB.

This is needed as dogpile internally stores data as a custom class i.e. dogpile.cache.api.CachedValue

Note: Custom manipulator needs to always override transform_incoming and transform_outgoing methods. MongoDB manipulator logic specifically checks that overridden method in instance and its super are different.

transform_incoming(son, collection)[source]

Used while saving data to MongoDB.

transform_outgoing(son, collection)[source]

Used while reading data from MongoDB.

class keystone.common.cache.backends.mongo.MongoApi(arguments)[source]

Bases: object

Class handling MongoDB specific functionality.

This class uses PyMongo APIs internally to create database connection with configured pool size, ensures unique index on key, does database authentication and ensure TTL collection index if configured so. This class also serves as handle to cache collection for dogpile cache APIs.

In a single deployment, multiple cache configuration can be defined. In that case of multiple cache collections usage, db client connection pool is shared when cache collections are within same database.

delete(key)[source]
delete_multi(keys)[source]
get(key)[source]
get_cache_collection()[source]
get_multi(keys)[source]
set(key, value)[source]
set_multi(mapping)[source]

Insert multiple documents specified as key, value pairs.

In this case, multiple documents can be added via insert provided they do not exist. Update of multiple existing documents is done one by one

class keystone.common.cache.backends.mongo.MongoCacheBackend(arguments)[source]

Bases: dogpile.cache.api.CacheBackend

A MongoDB based caching backend implementing dogpile backend APIs.

Arguments accepted in the arguments dictionary:

Parameters:
  • db_hosts – string (required), hostname or IP address of the MongoDB server instance. This can be a single MongoDB connection URI, or a list of MongoDB connection URIs.
  • db_name – string (required), the name of the database to be used.
  • cache_collection – string (required), the name of collection to store cached data. Note: Different collection name can be provided if there is need to create separate container (i.e. collection) for cache data. So region configuration is done per collection.

Following are optional parameters for MongoDB backend configuration,

Parameters:
  • username – string, the name of the user to authenticate.
  • password – string, the password of the user to authenticate.
  • max_pool_size – integer, the maximum number of connections that the pool will open simultaneously. By default the pool size is 10.
  • w

    integer, write acknowledgement for MongoDB client

    If not provided, then no default is set on MongoDB and then write acknowledgement behavior occurs as per MongoDB default. This parameter name is same as what is used in MongoDB docs. This value is specified at collection level so its applicable to cache_collection db write operations.

    If this is a replica set, write operations will block until they have been replicated to the specified number or tagged set of servers. Setting w=0 disables write acknowledgement and all other write concern options.

  • read_preference – string, the read preference mode for MongoDB client Expected value is primary, primaryPreferred, secondary, secondaryPreferred, or nearest. This read_preference is specified at collection level so its applicable to cache_collection db read operations.
  • use_replica – boolean, flag to indicate if replica client to be used. Default is False. replicaset_name value is required if True.
  • replicaset_name – string, name of replica set. Becomes required if use_replica is True
  • son_manipulator

    string, name of class with module name which implements MongoDB SONManipulator. Default manipulator used is BaseTransform.

    This manipulator is added per database. In multiple cache configurations, the manipulator name should be same if same database name db_name is used in those configurations.

    SONManipulator is used to manipulate custom data types as they are saved or retrieved from MongoDB. Custom impl is only needed if cached data is custom class and needs transformations when saving or reading from db. If dogpile cached value contains built-in data types, then BaseTransform class is sufficient as it already handles dogpile CachedValue class transformation.

  • mongo_ttl_seconds

    integer, interval in seconds to indicate maximum time-to-live value. If value is greater than 0, then its assumed that cache_collection needs to be TTL type (has index at ‘doc_date’ field). By default, the value is -1 and its disabled. Reference: <http://docs.mongodb.org/manual/tutorial/expire-data/>

    Note

    This parameter is different from Dogpile own expiration_time, which is the number of seconds after which Dogpile will consider the value to be expired. When Dogpile considers a value to be expired, it continues to use the value until generation of a new value is complete, when using CacheRegion.get_or_create(). Therefore, if you are setting mongo_ttl_seconds, you will want to make sure it is greater than expiration_time by at least enough seconds for new values to be generated, else the value would not be available during a regeneration, forcing all threads to wait for a regeneration each time a value expires.

  • ssl – boolean, If True, create the connection to the server using SSL. Default is False. Client SSL connection parameters depends on server side SSL setup. For further reference on SSL configuration: <http://docs.mongodb.org/manual/tutorial/configure-ssl/>
  • ssl_keyfile – string, the private keyfile used to identify the local connection against mongod. If included with the certfile then only the ssl_certfile is needed. Used only when ssl is True.
  • ssl_certfile – string, the certificate file used to identify the local connection against mongod. Used only when ssl is True.
  • ssl_ca_certs – string, the ca_certs file contains a set of concatenated ‘certification authority’ certificates, which are used to validate certificates passed from the other end of the connection. Used only when ssl is True.
  • ssl_cert_reqs – string, the parameter cert_reqs specifies whether a certificate is required from the other side of the connection, and whether it will be validated if provided. It must be one of the three values ssl.CERT_NONE (certificates ignored), ssl.CERT_OPTIONAL (not required, but validated if provided), or ssl.CERT_REQUIRED (required and validated). If the value of this parameter is not ssl.CERT_NONE, then the ssl_ca_certs parameter must point to a file of CA certificates. Used only when ssl is True.

Rest of arguments are passed to mongo calls for read, write and remove. So related options can be specified to pass to these operations.

Further details of various supported arguments can be referred from <http://api.mongodb.org/python/current/api/pymongo/>

client[source]

Initializes MongoDB connection and collection defaults.

This initialization is done only once and performed as part of lazy inclusion of MongoDB dependency i.e. add imports only if related backend is used.

Returns:MongoApi instance
delete(key)[source]
delete_multi(keys)[source]
get(key)[source]
get_multi(keys)[source]
set(key, value)[source]
set_multi(mapping)[source]

keystone.common.cache.backends.noop module

class keystone.common.cache.backends.noop.NoopCacheBackend(*args)[source]

Bases: dogpile.cache.api.CacheBackend

A no op backend as a default caching backend.

The no op backend is provided as the default caching backend for keystone to ensure that dogpile.cache.memory is not used in any real-world circumstances unintentionally. dogpile.cache.memory does not have a mechanism to cleanup it’s internal dict and therefore could cause run-away memory utilization.

delete(key)[source]
delete_multi(keys)[source]
get(key)[source]
get_multi(keys)[source]
set(key, value)[source]
set_multi(mapping)[source]

Module contents