Quickstart Guide¶

This section aims to get you started within a few couple of minutes, while still explaining what is going on, so that someone with only limited experience with Python can follow along.

Setup TL;DR

docker run --name redis_gears --rm -d -p 127.0.0.1:6379:6379 redislabs/redisgears:1.0.6

virtualenv -p python3.7 .venv
source .venv/bin/activate

pip install redgrease[all]

docker run --name redis_gears --rm -d -p 127.0.0.1:6379:6379 redislabs/redisgears:1.0.6

virtualenv -p python3.7 .venv
\venv\Scripts\activate.bat

pip install redgrease[all]

Note

This is not tested. If anyone is using this OS, please let me know if this works or not. :)

If this was obvious to you, you can jump straight to the first code examples.

Running Redis Gears¶

The easiest way to run a Redis Engine with Redis Gears is by running one of the official Docker images, so firstly make sure that you have Docker engine installed.

(Eh? I’ve been living under a rock. What the hedge is Docker?)

With Docker is installed, open a terminal or command prompt and enter:

docker run --name redis_gears --rm -d -p 127.0.0.1:6379:6379 redislabs/redisgears:1.0.6

This will run a single Redis engine, with the Redis Gears module loaded, inside a Docker container. The first time you issue the command it may take a little time to launch, as Docker needs to fetch the container image from Docker Hub.

Lets break the command down:

docker run is the base command telling Docker that we want to run a new containerized process.
--name redis_gears gives the name “redis_gears” to the container for easier identification. You can change it for whatever you like or omit it to assign a randomized name.
--rm instructs Docker that we want the container to be removed, in case it stops. This is optional but makes starting and stopping easer, although the state (and stored data) will be lost between restarts.
-d instructs Docker to run the container in the background as a ‘daemon’. You can omit this too, but your terminal / command prompt will be hijacked by the output / logs from the container. Which could be interesting enough.
-p 127.0.0.1:6379:6379 instructs Docker that we want your host computer to locally (127.0.0.1) expose port 6379 and route it to 6379 in the container. This is the default port for Redis communication and this argument is necessary for application on your computer to be able to talk to the Redis engine inside the Docker container.
redislabs/redisgears:1.0.6 is the name and version of the Docker image that we want to run. This specific image is prepared by RedisLabs and has the Gears module pre-installed and configured.

Note

If its your first time trying Redis Gears, stick to the command above, but if you want to try running a cluster image instead, you can issue the following command:

docker run --name redis_gears_cluster --rm -d -p 127.0.0.1:30001:30001 -p 127.0.0.1:30002:30002 -p 127.0.0.1:30003:30003 redislabs/rgcluster:1.0.6

This will run a 3-shard cluster exposed locally on ports 30001-30003.

Refer to the official documentation for more information and details on how how to install Redis Gears.

Checking Logs:¶

You can confirm that the container is running by inspecting the logs / output of the Redis Gears container by issuing the command:

docker logs redis_gears

You can optionally add the argument --follow to continuously follow the log output.
You can optionally add the argument --tail 100 to start showing the logs from the 100 most recent entries only.

If you just started the single instance engine, the logs should hold 40 odd lines starting and ending something like this:

1:C 03 Apr 2021 07:41:37.250 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
1:C 03 Apr 2021 07:41:37.251 # Redis version=6.0.1, bits=64, commit=00000000, modified=0, pid=1, just started
...
...
...
1:M 03 Apr 2021 07:41:37.309 * Module 'rg' loaded from /var/opt/redislabs/lib/modules/redisgears.so
1:M 03 Apr 2021 07:41:37.309 * Ready to accept connections

Stopping¶

You can stop the container by issuing:

docker stop redis_gears

If successful, it should simply output the name of the stopped container: redis_gears

RedGrease Installation¶

For the client application environment, it is strongly recommended that you set up a virtual Python environment, with Python 3.7 specifically.

Note

The Redis Gears containers above use Python 3.7 for executing Gear functions, and using the same version the client application will enable more features.

Warning

The RedGrease client package works with any Python version from 3.6 and later, but execution of dynamically created GearFunction objects is only possible when the client Python version match the Python version on the Redis Gears server runtime.

If the versions mismatch, Gear function execution is limited to execution by string or execution of script files

With Python 3.7, and virtualenv installed on your system:

Create a virtual python3.7 environment
virtualenv -p python3.7 .venv
Python packages, including RedGrease that you install within this virtual environment will not interfere with the rest of your system.
Activate the environment
source .venv/bin/activate
.venv\Scripts\activate.bat
Install redgrease
pip install redgrease[all]
Note

The [all] portion is important, as it will include all the Redgrease extras, and include the dependencies for the RedisGears client module as well as the RedGrease Command Line Interface (CLI).

See here for more details on the various extras options.

Basic Commands¶

In this section we’ll walk through some of the basic commands and interactions with the RedGrease Gears client, including executing some very basic Gear functions.

The next chapter “RedGrease Client”, goes into all commands in more details, but for now we’ll just look at the most important things.

You can take a sneak-peek at the full code that we will walk through in this section, by expanding the block below (click “▶ Show”).

If you find this rather self-explanatory, then you can probably jump directly to the next section where we do some RedGrease Gear Function Comparisons with “vanilla” RedisGears functions.

Otherwise just continue reading and we’ll, go through it step-by-step.

From examples/basics.py on the official GitHub repo:¶

from operator import add

import redgrease

# Create connection / client for single instance Redis
r = redgrease.RedisGears()

# # Normal Redis Commands
# Clearing the database
r.flushall()

# String operations
r.set("Foo-fighter", 2021)
r.set("Bar-fighter", 63)
r.set("Baz-fighter", -747)

# Hash Opertaions
r.hset("noodle", mapping={"spam": "eggs", "meaning": 8})
r.hincrby("noodle", "meaning", 34)

# Stream operations
r.xadd("transactions:0", {"msg": "First", "from": 0, "to": 2, "amount": 1000})


# # Redis Gears Commands
# Get Statistics on the Redis Gears Python runtime
gears_runtime_python_stats = r.gears.pystats()
print(f"Gears Python runtime stats: {gears_runtime_python_stats}")

# Get info on any registered Gear functions, if any.
registered_gear_functions = r.gears.dumpregistrations()
print(f"Registered Gear functions: {registered_gear_functions}")

# Execute nothing as a Gear function
empty_function_result = r.gears.pyexecute()
print(f"Result of nothing: {empty_function_result}")

# Execute a Gear function string that just iterates through and returns the key-space.
all_records_gear = r.gears.pyexecute("GearsBuilder('KeysReader').run()")
print("All-records gear results: [")
for result in all_records_gear:
    print(f"  {result}")
print("]")

# Gear function string to count all the keys
key_count_gearfun_str = "GearsBuilder('KeysReader').count().run()"
key_count_result = r.gears.pyexecute(key_count_gearfun_str)
print(f"Total number of keys: {int(key_count_result)}")


# # GearFunctions
# GearFunction object to count all keys
key_count_gearfun = redgrease.KeysReader().count().run()
key_count_result = r.gears.pyexecute(key_count_gearfun)
print(f"Total number of keys: {key_count_result}")


# Simple Aggregation
add_gear = redgrease.KeysReader("*-fighter").values().map(int).aggregate(0, add)
simple_sum = add_gear.run(on=r)
print(f"Multiplication of '-fighter'-keys values: {simple_sum}")

Let’s look at some code examples of how to use RedGrease, warming up with the basics.

Instantiation¶

Naturally, the first thing is to import of the RedGrease package and instantiate Redis Gears client / connection object:

Package import and client / connection instantiation:¶

from operator import add

import redgrease

# Create connection / client for single instance Redis
r = redgrease.RedisGears()

This will attempt to connect to a Redis server using the default port (6379) on “localhost”, which, if you followed the instructions above should be exactly what you set up and have running. There are of course arguments to set other targets, but more on that later.

The imported add function from the operator module is not part of the RedgGrease package, but we will use it later in one of the examples.

Note

If you created a Redis cluster above then you have to specify the initial master nodes you want to connect to:

import redgrease

r = redgrease.RedisGears(port=30001)

Redis Commands¶

The instantiated client / connection, r, accepts all the normal Redis commands, exactly as expected. The subsequent lines populate the Redis instance it with some data.

Some normal Redis commands:¶

# Clearing the database
r.flushall()

# String operations
r.set("Foo-fighter", 2021)
r.set("Bar-fighter", 63)
r.set("Baz-fighter", -747)

# Hash Opertaions
r.hset("noodle", mapping={"spam": "eggs", "meaning": 8})
r.hincrby("noodle", "meaning", 34)

# Stream operations
r.xadd("transactions:0", {"msg": "First", "from": 0, "to": 2, "amount": 1000})

Gears Commands¶

The client / connection also has a gears attribute that gives access to RedisGears Commands.

Some Redis Gears commands:¶

# Get Statistics on the Redis Gears Python runtime
gears_runtime_python_stats = r.gears.pystats()
print(f"Gears Python runtime stats: {gears_runtime_python_stats}")

# Get info on any registered Gear functions, if any.
registered_gear_functions = r.gears.dumpregistrations()
print(f"Registered Gear functions: {registered_gear_functions}")

# Execute nothing as a Gear function
empty_function_result = r.gears.pyexecute()
print(f"Result of nothing: {empty_function_result}")

# Execute a Gear function string that just iterates through and returns the key-space.
all_records_gear = r.gears.pyexecute("GearsBuilder('KeysReader').run()")
print("All-records gear results: [")
for result in all_records_gear:
    print(f"  {result}")
print("]")

# Gear function string to count all the keys
key_count_gearfun_str = "GearsBuilder('KeysReader').count().run()"
key_count_result = r.gears.pyexecute(key_count_gearfun_str)
print(f"Total number of keys: {int(key_count_result)}")

The highlighted lines show the commands Gears.pystats(), Gears.dumpregistrations() and Gears.pyexecute() respectively and the output should look something like this:

Gears Python runtime stats: PyStats(TotalAllocated=41275404, PeakAllocated=11867779, CurrAllocated=11786368)
Registered Gear functions: []
Result of nothing: True
All-records gear results: [
    b"{'event': None, 'key': 'Baz-fighter', 'type': 'string', 'value': '-747'}"
    b"{'event': None, 'key': 'Bar-fighter', 'type': 'string', 'value': '63'}"
    b"{'event': None, 'key': 'transactions:0', 'type': 'unknown', 'value': None}"
    b"{'event': None, 'key': 'Foo-fighter', 'type': 'string', 'value': '2021'}"
    b"{'event': None, 'key': 'noodle', 'type': 'hash', 'value': {'meaning': '42', 'spam': 'eggs'}}"
]
Total number of keys: 5

The command Gears.pystats() gets some memory usage statistics about the Redis Gears Python runtime environment on the server.

The command Gears.dumpregistrations() gets information about any registered Gears functions, in this cas none.

And finally, the command Gears.pyexecute() is the most important command, which sends a Gears function to the server for execution or registration. In the above example, we are invoking it three times:

Firstly (line 35) - We pass nothing, i.e. no function at all, which naturally doesn’t do anything, but is perfectly valid, and the call thus just returns True.
Secondly (line 39) - We execute a Raw Function String, that reads through the Redis keys (indicated by the 'KeysReader') and just returns the result by running the function as a batch job (indicated by the run() operation). The result is consequently a list of dicts, representing the keys and their respective values and types in the Redis keyspace, I.e. the keys we added just before.
Thirdly (lines 46-47) - We pass a very similar function, but with an additional count() operation, which is a Gear operation that simply aggregates and counts the incoming records, in this case all key-space records on the server. The result is simply the number or keys in the database: 5.

There are other Gears commands too, and the next chapter, “Redgrease Client”, will run through all of them.

GearFunctions¶

Composing Gear functions by using strings is not at all very practical, so RedGrease provides a more convenient way of constructing Gear functions programmatically, using various GearFunction objects.

GearFunction objects instead of strings:¶

# GearFunction object to count all keys
key_count_gearfun = redgrease.KeysReader().count().run()
key_count_result = r.gears.pyexecute(key_count_gearfun)
print(f"Total number of keys: {key_count_result}")

This Gear function does the same thing as the last function of the previous example, but instead of being composed by a string, it is composed programmatically using RedGrease’s GearFunction objects, in this case using the KeysReader class.

The output is, just as expected:

Total number of keys: 5

Warning

Note that execution of GearFunction objects only work if your local Python environment version matches the version on the Redis Gear server, i.e. Python 3.7.

If the versions mismatch, Gear function execution is limited to execution by string or execution of script files

The final basic example shows a GearFunction that has a couple of operations stringed together.

Simple aggregation - Add keyspace values:¶

add_gear = redgrease.KeysReader("*-fighter").values().map(int).aggregate(0, add)
simple_sum = add_gear.run(on=r)
print(f"Multiplication of '-fighter'-keys values: {simple_sum}")

This Gear function adds the values of all the simple keys, with names ending in “-fighter”, which were the first three keys created in the example.

And indeed, the result is:

Sum of '-fighter'-keys values: 1337

Here is a quick run down of how it works:

Firstly, the KeysReader is parameterized with a key pattern *-fighter meaning it will only read the matching keys.
Secondly, the map() operation uses a simple lambda function, to lift out the value and ensure it is an integer, on each of the keys.
Thirdly, the aggregate() operation is used to add the values together, using the imported add function, starting from the value 0.
Lastly, the run() operation is used to specify that the function should run as a batch job. The on argument states that we want to run it immediately on our client / connection, r.

The chapter “Readers” will go through the various types of readers, and the chapter operations will go through the various types of operations, and how to use them.

RedGrease Gear Function Comparisons¶

Now let’s move on to some more examples of smaller Gear functions, before we move on to some more elaborate examples.

The examples in this section are basically comparisons of how the examples in the official Gears documentation, could be simplified by using RedGrease.

Note

RedGrease is backwards compatible with the “vanilla” syntax and structure, and all versions below are still perfectly valid Gear functions when executing using RedGrease.

Word Count¶

Counting of words.

Assumptions¶

All keys store Redis String values. Each value is a sentence.

Vanilla Version¶

This is the the ‘Word Count’ example from the official RedisGears documentation.

Vanilla - Word Count¶

gb = GearsBuilder()
gb.map(lambda x: x["value"])  # map records to "sentence" values
gb.flatmap(lambda x: x.split())  # split sentences to words
gb.countby()  # count each word's occurances
gb.run()

RedGrease Version¶

This is an example of how the same Gear function could be rewritten using RedGrease.

RedGrease - Word Count¶

from redgrease import KeysReader

KeysReader().values().flatmap(str.split).countby().run()

Delete by Key Prefix¶

Deletes all keys whose name begins with a specified prefix and return their count.

Assumptions¶

There may be keys in the database. Some of these may have names beginning with the “delete_me:” prefix.

Vanilla Version¶

This is the the ‘Delete by Key Prefix’ example from the official RedisGears documentation.

Vanilla - Delete by Key Prefix¶

gb = GearsBuilder()
gb.map(lambda x: x["key"])  # map the records to key names
gb.foreach(lambda x: execute("DEL", x))  # delete each key
gb.count()  # count the records
gb.run("delete_me:*")

RedGrease Version¶

This is an example of how the same Gear function could be rewritten using RedGrease.

RedGrease - Delete by Key Prefix¶

from redgrease import KeysReader, cmd

delete_fun = KeysReader().keys().foreach(cmd.delete).count()
delete_fun.run("delete_me:*")

Basic Redis Stream Processing¶

Copy every new message from a Redis Stream to a Redis Hash key.

Assumptions¶

An input Redis Stream is stored under the “mystream” key.

Vanilla Version¶

This is the the ‘Basic Redis Stream Processing’ example from the official RedisGears documentation.

Vanilla - Basic Redis Stream Processing¶

gb = GearsBuilder("StreamReader")
gb.foreach(
    lambda x: execute("HMSET", x["id"], *sum([[k, v] for k, v in x.items()], []))
)  # write to Redis Hash
gb.register("mystream")

RedGrease Version¶

This is an example of how the same Gear function could be rewritten using RedGrease.

RedGrease - Basic Redis Stream Processing¶

from redgrease import StreamReader, cmd

StreamReader().foreach(lambda x: cmd.hmset(x["id"], x)).register(  # write to Redis Hash
    "mystream"
)

Automatic Expiry¶

Sets the time to live (TTL) for every updated key to one hour.

Assumptions¶

None.

Vanilla Version¶

This is the the ‘Automatic Expiry’ example from the official RedisGears documentation.

Vanilla - Automatic Expiry¶

gb = GB()
gb.foreach(lambda x: execute("EXPIRE", x["key"], 3600))
gb.register("*", mode="sync", readValue=False)

RedGrease Version¶

This is an example of how the same Gear function could be rewritten using RedGrease.

RedGrease - Automatic Expiry¶

from redgrease import KeysReader, cmd

expire = KeysReader().keys().foreach(lambda x: cmd.expire(x, 3600))
expire.register("*", mode="sync", readValue=False)

Keyspace Notification Processing¶

This example demonstrates a two-step process that:

Synchronously captures distributed keyspace events
Asynchronously processes the events’ stream

Assumptions¶

The example assumes there is a process function defined, that does the actual processing of the deleted records. For the purpose of the example we can assume that it just outputs the name of the expired keys to the Redis logs, as follows:

def process(x):
    """
    Processes a message from the local expiration stream
    Note: in this example we simply print to the log, but feel free to replace
    this logic with your own, e.g. an HTTP request to a REST API or a call to an
    external data store.
    """
    log(f"Key '{x['value']['key']}' expired at {x['id'].split('-')[0]}")

Vanilla Version¶

This is the the ‘Keyspace Notification Processing’ example from the official RedisGears documentation.

Vanilla - Keyspace Notification Processing¶

# Capture an expiration event and adds it to the shard's local 'expired' stream
cap = GB("KeysReader")
cap.foreach(lambda x: execute("XADD", f"expired:{hashtag()}", "*", "key", x["key"]))
cap.register(prefix="*", mode="sync", eventTypes=["expired"], readValue=False)

# Consume new messages from expiration streams and process them somehow
proc = GB("StreamReader")
proc.foreach(process)
proc.register(prefix="expired:*", batch=100, duration=1)

RedGrease Version¶

This is an example of how the same Gear function could be rewritten using RedGrease.

RedGrease - Keyspace Notification Processing¶

from redgrease import KeysReader, StreamReader, cmd, hashtag, log

# Capture an expiration event and adds it to the shard's local 'expired' stream
KeysReader().keys().foreach(
    lambda key: cmd.xadd(f"expired:{hashtag()}", {"key": key})
).register(prefix="*", mode="sync", eventTypes=["expired"], readValue=False)

# Consume new messages from expiration streams and process them somehow
StreamReader().foreach(process).register(prefix="expired:*", batch=100, duration=1)

Reliable Keyspace Notification¶

Capture each keyspace event and store to a Stream.

Assumptions¶

…

Vanilla Version¶

This is the the ‘Reliable Keyspace Notification’ example from the official RedisGears documentation.

Vanilla - Reliable Keyspace Notification¶

GearsBuilder().foreach(
    lambda x: execute(
        "XADD", "notifications-stream", "*", *sum([[k, v] for k, v in x.items()], [])
    )
).register(prefix="person:*", eventTypes=["hset", "hmset"], mode="sync")

RedGrease Version¶

This is an example of how the same Gear function could be rewritten using RedGrease.

RedGrease - Reliable Keyspace Notification¶

from redgrease import KeysReader, cmd

KeysReader().foreach(lambda x: cmd.xadd("notifications-stream", x)).register(
    prefix="person:*", eventTypes=["hset", "hmset"], mode="sync"
)

Cache Get Command¶

As a final example of this quickstart tutorial, let’s look at how we can build caching into Redis as a new command, with the help of Redis Gears and RedGrease.

It may look a bit intimidating at first, but theres actually not not that much to it. Most of it is just comments, logging or testing code.

Simple Caching command:¶

import timeit

import redgrease

# Bind / register the function on some Redis instance.
r = redgrease.RedisGears()


# CommandReader Decorator
# The `command` decorator tunrs the function to a CommandReader,
# registerered on the Redis Gears sever if using the `on` argument
@redgrease.command(on=r, requirements=["requests"], replace=False)
def cache_get(url):
    import requests

    # Check if the url is already in the cache,
    # And if so, simply return the cached result.
    if redgrease.cmd.exists(url):
        return bytes(redgrease.cmd.get(url))

    # Otherwise fetch the url.
    response = requests.get(url)

    # Return nothing if request fails
    if response.status_code != 200:
        return bytes()

    # If ok, set the cache data and return.
    response_data = bytes(response.content)
    redgrease.cmd.set(url, response_data)

    return response_data


# Test caching on some images
some_image_urls = [
    "http://images.cocodataset.org/train2017/000000483381.jpg",
    "http://images.cocodataset.org/train2017/000000237137.jpg",
    "http://images.cocodataset.org/train2017/000000017267.jpg",
    "http://images.cocodataset.org/train2017/000000197756.jpg",
    "http://images.cocodataset.org/train2017/000000193332.jpg",
    "http://images.cocodataset.org/train2017/000000475564.jpg",
    "http://images.cocodataset.org/train2017/000000247368.jpg",
    "http://images.cocodataset.org/train2017/000000416337.jpg",
]


# Get all the images and write them to disk
def get_em_all():

    for image_url in some_image_urls:

        # This will invoke the cache_get function **on the Redis server**
        image_data = cache_get(image_url)

        # Quick and dirty way of getting the image file name.
        image_name = image_url.split("/")[-1]

        # Write to file
        with open(image_name, "wb") as img_file:
            img_file.write(image_data.value)


# Test it
# Time how long it takes to get images when the cache is empty.
t1 = timeit.timeit(get_em_all, number=1)
print(f"Cache-miss time: {t1:.3f} seconds")

# Time how long it takes to get the images when they are all in the cache.
t2 = timeit.timeit(get_em_all, number=1)
print(f"Cache-hit time: {t2:.3f} seconds")
print(f"That is {t1/t2:.1f} times faster!")


# Clean the database
def cleanup(r: redgrease.RedisGears):

    # Unregister all registrations
    for reg in r.gears.dumpregistrations():
        r.gears.unregister(reg.id)

    # Remove all executions
    for exe in r.gears.dumpexecutions():
        r.gears.dropexecution(str(exe.executionId))

    # Clear all keys
    r.flushall()

    # Check that there are no keys
    return len(r.keys()) == 0


# print(cleanup(r))

Let’s go through the code, step by step, and it will hopefully make some sense.

Instantiation as usual:¶

import timeit

import redgrease

# Bind / register the function on some Redis instance.
r = redgrease.RedisGears()

The instantiation of the client / connection is business as usual.

Cache-Get function¶

Lets now go for the core of the solution; The code that we want to run on Redis for each resource request.

Cache handling function:¶

    import requests

    # Check if the url is already in the cache,
    # And if so, simply return the cached result.
    if redgrease.cmd.exists(url):
        return bytes(redgrease.cmd.get(url))

    # Otherwise fetch the url.
    response = requests.get(url)

    # Return nothing if request fails
    if response.status_code != 200:
        return bytes()

    # If ok, set the cache data and return.
    response_data = bytes(response.content)
    redgrease.cmd.set(url, response_data)

    return response_data

Look at the highlighted lines, and notice:

The logic of handling requests with caching is simply put in a normal function, much like we would if the caching logic was handled by each client.
The argument of the function is what we could expect, the url to the resource to get.
The function return value is either:
- The contents of the response to requests to the URL (line 32), or
- A cached value (line 19)

Which is exactly what you would expect from a cached fetching function.

The really interesting part, however, is this little line, on top of the function.

CommandReader function decorator:¶

# The `command` decorator tunrs the function to a CommandReader,
# registerered on the Redis Gears sever if using the `on` argument
@redgrease.command(on=r, requirements=["requests"], replace=False)
def cache_get(url):
    import requests

    # Check if the url is already in the cache,
    # And if so, simply return the cached result.
    if redgrease.cmd.exists(url):
        return bytes(redgrease.cmd.get(url))

All the Redis Gears magic is hidden in this function decorator, and it does a couple of important things:

It embeds the function in a CommandReader Gear function.
It ensures that the function is registered on our Redis server(s).
It captures the relevant requirements, for the function to work.
It ensures that we only register this function once.
It creates a new function, with the same name, which when called, triggers the corresponding registered Gear function, and returns the result from the server.

This means that you can now call the decorated function, just as if it was a local function:

result = cache_get("http://images.cocodataset.org/train2017/000000169188.jpg")

This may look like it is actually executing the function locally, but the cache_get function is actually executed on the server.

This means that the registered cache_get Gear function can not only be triggered by the client that defined the decorated function, but can be triggered by any client by invoking the Redis Gear RG.TRIGGER command with the the functions’ trigger name and arguments.

In our case, using redis-cli as an example:

> RG.TRIGGER cache_get http://images.cocodataset.org/train2017/000000169188.jpg

The arguments for the @command decorator, are the same as to the OpenGearFunction.register() method, inherited by the CommandReader class.

Note

This simplistic cache function is only for demonstrating the command function decorator. The design choices of this particular caching implementation is far from ideal for all use-cases.

For example:

Only the response content data is returned, not response status or headers.
Cache is never expiring.
If multiple requests for the same resource is made in close successions, there may be duplicate external requests.
The entire response contents is copied into memory before writing to cache.
… etc …

Naturally, the solution could easily be modified to accommodate other behaviors.

Testing the Cache¶

To test the caching, we create a very simple function that iterates through some URLs and tries to get them from the cache and saving the contents to local files.

Test function:¶

some_image_urls = [
    "http://images.cocodataset.org/train2017/000000483381.jpg",
    "http://images.cocodataset.org/train2017/000000237137.jpg",
    "http://images.cocodataset.org/train2017/000000017267.jpg",
    "http://images.cocodataset.org/train2017/000000197756.jpg",
    "http://images.cocodataset.org/train2017/000000193332.jpg",
    "http://images.cocodataset.org/train2017/000000475564.jpg",
    "http://images.cocodataset.org/train2017/000000247368.jpg",
    "http://images.cocodataset.org/train2017/000000416337.jpg",
]


# Get all the images and write them to disk
def get_em_all():

    for image_url in some_image_urls:

        # This will invoke the cache_get function **on the Redis server**
        image_data = cache_get(image_url)

        # Quick and dirty way of getting the image file name.
        image_name = image_url.split("/")[-1]

        # Write to file
        with open(image_name, "wb") as img_file:
            img_file.write(image_data.value)


# Test it
# Time how long it takes to get images when the cache is empty.
t1 = timeit.timeit(get_em_all, number=1)
print(f"Cache-miss time: {t1:.3f} seconds")

# Time how long it takes to get the images when they are all in the cache.
t2 = timeit.timeit(get_em_all, number=1)
print(f"Cache-hit time: {t2:.3f} seconds")
print(f"That is {t1/t2:.1f} times faster!")

Calling the this function twice reveals that the caching does indeed seem to work.

Cache-miss time: 10.954 seconds
Cache-hit time: 0.013 seconds
That is 818.6 times faster!

We can also inspect the logs of the Redis node to confirm that the cache function was indeed executed on the server.

docker logs --tail 100

And you should indeed see that the expected log messages appear:

M 06 Apr 2021 08:58:06.314 * <module> GEARS: Cache request #1 for resource 'http://images.cocodataset.org/train2017/000000416337.jpg'
M 06 Apr 2021 08:58:06.314 * <module> GEARS: Cache miss #1 - Downloading resource 'http://images.cocodataset.org/train2017/000000416337.jpg'.
M 06 Apr 2021 08:58:07.855 * <module> GEARS: Cache update #1 - Request status for resource 'http://images.cocodataset.org/train2017/000000416337.jpg': 200

...

M 06 Apr 2021 08:58:07.860 * <module> GEARS: Cache request #2 for resource 'http://images.cocodataset.org/train2017/000000416337.jpg'

The last piece of code is jut to clean up the database by un-registering the cache_get Gear function, cancel and drop any ongoing Gear function executions and flush the key-space.

Clean up the database:¶

def cleanup(r: redgrease.RedisGears):

    # Unregister all registrations
    for reg in r.gears.dumpregistrations():
        r.gears.unregister(reg.id)

    # Remove all executions
    for exe in r.gears.dumpexecutions():
        r.gears.dropexecution(str(exe.executionId))

    # Clear all keys
    r.flushall()

    # Check that there are no keys
    return len(r.keys()) == 0


# print(cleanup(r))

That wraps up the Quickstart Guide! Good luck building Gears!

Courtesy of : Pte. Ltd.