This is a companion repo for the blog post, "Preventing Cache Stampedes Using python-redis-lock" on the EFL Technology Blog.
It contains three example scripts that show how to use locks to prevent "cache stampedes" when cached values become stale:
cache_stampede/example_no_protection.py
shows what happens during a cache stampede.cache_stampede/example_threading.py
proves the concept by usingthreading.Lock
to control access to the cache among multiple threads in a single process.cache_stampede/example_redis.py
takes it a step further by moving the lock into Redis itself, allowing access to be controlled across multiple processes, even multiple servers.
This project is compatible with Python 3.6 and 2.7.
Set up the Django project (only has to be done once):
- Create a virtualenv, if desired.
pip install -r requirements.txt
- Ensure Redis is running, and configure
cache_stampede/settings.py:CACHES
if necessary (by default it is configured to connect tolocalhost:6379
). cd
into thecache_stampede
directory of the project.
To run the example scripts, execute the following commands (tested in bash and zsh):
This is the "control group", showing what happens during a cache stampede.
redis-cli -n 1 FLUSHDB
DJANGO_SETTINGS_MODULE="settings" python example_no_protection.py
This example uses threading.Lock
to prevent a cache stampede.
redis-cli -n 1 FLUSHDB
DJANGO_SETTINGS_MODULE="settings" python example_threading.py
This example uses python-redis-lock
to prevent a cache stampede.
redis-cli -n 1 FLUSHDB
for _ in {1..5}; do DJANGO_SETTINGS_MODULE="settings" python example_redis.py & done
You may see a single UNLOCK_SCRIPT not cached.
warning in the output.
Releasing a lock requires several commands that must run atomically (this
requires a LUA script). For better
performance, python-redis-lock
uses the
EVALSHA command so that the LUA script gets
cached after the first time it is eval'd.
UNLOCK_SCRIPT not cached.
simply means that the LUA script hasn't been cached
by Redis yet. It is safe to ignore this warning, as long as it only appears
once.