Comprehensive guide to Redis. Part 4.

vladimir

November 19, 2015

Comprehensive guide to Redis. Part 4.

In this tutorial, we are going to overview other Redis commands and features.

Pub/Sub

Publish-Subscribe is a pattern where publishers send messages to channels, and subscribers receive these messages if they are listening to a given channel.

Pub/Sub use cases:
– chat apps
– push notifications
– command dashboards
– remote code execution

PUBLISH: sends a message to the Redis channel. Returns the number of clients that received that message.
SUBSCRIBE: subscribes a client to one or many channels
UNSUBSCRIBE: unsubscribes a client from one or many channels
PSUBSCRIBE: subscribes a client to one or many channels. Accepts glob-style patterns as channel names.
PUNSUBSCRIBE: unsubscribes a client from one or many channels. Accepts glob-style patterns as channel names.
PUBSUB: checks the state of the Redis Pub/Sub system. Accepts 3 subcommands: CHANNELS, NUMSUB, NUMPAT.
PUBSUB CHANNELS [pattern]: returns all channels with at least 1 subscriber. Accepts an optional glob-style pattern.
PUBSUB NUMSUB [channel1 … channelN]: returns the number of clients connected to channels via the SUBSCRIBE command. Accepts channel names as arguments.
PUBSUB NUMPAT: returns the number of clients connected to channels via the PSUBSCRIBE command.

Notice that when a Redis client executes the SUBSCRIBE or PSUBSCRIBE command, it stops accepting commands, except for SUBSCRIBE, PSUBSCRIBE, UNSUBSCRIBE, PUNSUBSCRIBE.

We are going to create a remote command execution system. In this system, a command is sent to a channel and the server that is subscribed to that channel executes the command.

Create a file called publisher.js with the following code:

var redis = require("redis");
var client = redis.createClient();

//assign the 3rd argument from the command line to the variable channel
var channel = process.argv[2];
//assign the 4th argumemt to the var command
var command = process.argv[3];

//the PUBLISH command
client.publish(channel, command);

client.quit();

var redis = require("redis");

var client = redis.createClient();

//assign the 3rd argument from the command line to the variable channel

var channel = process.argv[2];

//assign the 4th argumemt to the var command

var command = process.argv[3];

//the PUBLISH command

client.publish(channel, command);

client.quit();

Create a file called subscriber.js with the following code:

//require the Node.js module os
var os = require("os");
var redis = require("redis");
var client = redis.createClient();

//command namespace
var COMMANDS = {};

//displays the current date
COMMANDS.DATE = function(){
    var now = new Date();
    console.log("DATE " + now.toISOString());
};

//displays PONG
COMMANDS.PING = function(){
    console.log("PONG");
};

//displays the server hostname
COMMANDS.HOSTNAME = function(){
    console.log("HOSTNAME " + os.hostname());
};

//channel listener
//executes commands based on the channel messages
client.on("message", function(channel, commandName){
    //if the command exists
    if(COMMANDS.hasOwnProperty(commandName)){
        var commandFunction = COMMANDS[commandName];
        commandFunction();
    } else {
        console.log("Unknown command: " + commandName);
    }
});

//the SUBSCRIBE command
//passing 2 variables
//global is the channel that all clients subscribe to
//the second argument is a channel from the command line
client.subscribe("global", process.argv[2]);

//require the Node.js module os

var os = require("os");

var redis = require("redis");

var client = redis.createClient();

//command namespace

var COMMANDS = {};

//displays the current date

COMMANDS.DATE = function(){

var now = new Date();

console.log("DATE " + now.toISOString());

};

//displays PONG

COMMANDS.PING = function(){

console.log("PONG");

};

//displays the server hostname

COMMANDS.HOSTNAME = function(){

console.log("HOSTNAME " + os.hostname());

};

//channel listener

//executes commands based on the channel messages

client.on("message", function(channel, commandName){

//if the command exists

if(COMMANDS.hasOwnProperty(commandName)){

var commandFunction = COMMANDS[commandName];

commandFunction();

} else {

console.log("Unknown command: " + commandName);

}

});

//the SUBSCRIBE command

//passing 2 variables

//global is the channel that all clients subscribe to

//the second argument is a channel from the command line

client.subscribe("global", process.argv[2]);

Now open 3 terminal windows and run the previous files. You should see the following:

Transactions

A Redis transaction is a sequence of commands executed in order and atomically. The MULTI command marks the beginning of the transaction. The EXEC command marks the end. Any commands between the MULTI and EXEC commands are executed as an atomic operator. To prevent a transaction from being executed use the DISCARD command instead of EXEC.

Notice that transactions in Redis are not rolled back. If one of the commands fail, Redis proceeds to the next command.

The following example simulates a bank transfer. Money is transferred from the source account to a destination account.

Create a file called bank-transaction.js with the following code:

var redis = require("redis");
var client = redis.createClient();

/**
 * Transfers money
 *
 * @param from account ID from which to withdraw money
 * @param to account ID to receive money
 * @param value money
 * @param callback function to call after the transfer
 */
function transfer(from, to, value, callback){
    //retrieve the current balance
    client.get(from, function(err, balance){
        //start transaction
        var multi = client.multi();
        multi.decrby(from, value);
        multi.incrby(to, value);
        //if enough money
        if(balance >= value){
            multi.exec(function(err, reply){
                callback(null, reply[0]);
            });
        } else {
            multi.discard();
            callback(new Eror("Insufficient funds"), null);
        }
    });
}

//set the initial balance of each account to $100
client.mset("max:checkings", 100, "hugo:checkings", 100, function(err, reply){
    console.log("Max checkings: 100");
    console.log("Hugo checkings: 100");
    transfer("max:checkings", "hugo:checkings", 40, function(err, balance){
        if(err){
            console.log(err);
        } else {
            console.log("Transferred 40 from Max to Hugo");
            console.log("Max balance:", balance);
        }
        client.quit();
    })
});

var redis = require("redis");

var client = redis.createClient();

/**

* Transfers money

* @param from account ID from which to withdraw money

* @param to account ID to receive money

* @param value money

* @param callback function to call after the transfer

function transfer(from, to, value, callback){

//retrieve the current balance

client.get(from, function(err, balance){

//start transaction

var multi = client.multi();

multi.decrby(from, value);

multi.incrby(to, value);

//if enough money

if(balance >= value){

multi.exec(function(err, reply){

callback(null, reply[0]);

});

} else {

multi.discard();

callback(new Eror("Insufficient funds"), null);

}

});

}

//set the initial balance of each account to $100

client.mset("max:checkings", 100, "hugo:checkings", 100, function(err, reply){

console.log("Max checkings: 100");

console.log("Hugo checkings: 100");

transfer("max:checkings", "hugo:checkings", 40, function(err, balance){

if(err){

console.log(err);

} else {

console.log("Transferred 40 from Max to Hugo");

console.log("Max balance:", balance);

}

client.quit();

})

});

Now execute the file. You should see the following output:

WATCH: implements an optimistic lock on a group of keys. Marks keys as being watched so that the EXEC command executes the transaction only if the keys were not changed. Otherwise, returns null and the operation needs to be repeated.
UNWATCH: removes keys from a watch list.

The following example implements a zpop function, which removes the first element of a Sorted Set and passes it to a callback function, using a transaction with WATCH.

Create a file called watch-transaction.js with the following code:

var redis = require("redis");
var client = redis.createClient();

function zpop(key, callback){
    //execute the WATCH command on the key passed as an argument
    client.watch(key, function(watchErr, watchReply){
        //retrieve the 1st element from a Sorted Set
        client.zrange(key, 0, 0, function(zrangeErr, zrangeReply){
            //start transaction
            var multi = client.multi();
            multi.zrem(key, zrangeReply);
            multi.exec(function(transactionErr, transactionReply){
                //execute the callback function if the key being watched has not been changed
                if(transactionReply){
                    console.log("reply");
                    callback(zrangeReply[0]);
                } else {
                    console.log("no reply");
                    zpop(key, callback);
                }
            });
        });
    });
}
client.zadd("coaches", 2010, "Muslin");
client.zadd("coaches", 2011, "Kononov");
client.zadd("coaches", 2012, "Bruce-Lee");

zpop("coaches", function(member){
    console.log("The first coach in the group is: ", member);
    client.quit();
});

var redis = require("redis");

var client = redis.createClient();

function zpop(key, callback){

//execute the WATCH command on the key passed as an argument

client.watch(key, function(watchErr, watchReply){

//retrieve the 1st element from a Sorted Set

client.zrange(key, 0, 0, function(zrangeErr, zrangeReply){

//start transaction

var multi = client.multi();

multi.zrem(key, zrangeReply);

multi.exec(function(transactionErr, transactionReply){

//execute the callback function if the key being watched has not been changed

if(transactionReply){

console.log("reply");

callback(zrangeReply[0]);

} else {

console.log("no reply");

zpop(key, callback);

}

});

}

client.zadd("coaches", 2010, "Muslin");

client.zadd("coaches", 2011, "Kononov");

client.zadd("coaches", 2012, "Bruce-Lee");

zpop("coaches", function(member){

console.log("The first coach in the group is: ", member);

client.quit();

});

Now execute the file. You should see the following output:

Pipelines

A pipeline is a way to send multiple commands together to the Redis server without waiting for replies. The replies are read all at once by a client. The time taken for a Redis client to send a command and receive a response from the Redis server is called RTT (Round Trip Time).

Redis without pipelines:

Redis commands run sequentially in the server, but they are neither transactional nor atomic. By default, node_redis, the Node.js library, sends commands in pipelines. However, other Redis clients may not use pipelines by default.

Redis with pipelines:

Lua scripting

Redis 2.6 introduced Lua scripting feature. Lua scripts are atomic, which means that the Redis server is blocked during script execution. Redis has a default timeout of 5 seconds to run any script. This value can ba changed through the configuration lua-time-limit.

When Lua script times out Redis will not automatically terminate it. The Redis server will start to reply with a BUSY message to every command. In this case, you should abort script execution with the command SCRIPT KILL or SHUTDOWN NOSAVE.

A Redis client must send Lua scripts as strings to the Redis server. There are 2 functions that execute Redis commands: redis.call and redis.pcall.

redis.call: requires the command name and all it parameters. Returns the result of the executed command. If there are errors, aborts the script.
redis.pcall: similar to redis.call, but when it is an error, this function returns the error as a Lua table and continues the script execution.

It is possible to pass Redis key names and parameters to a Lua script. They will be available through the KEYS and ARGV variables.

There are 2 commands to run Lua scripts: EVAL and EVALSHA.

Syntax:
EVAL script numkeys key [key …] arg [arg …]
script – the Lua script itself
numkeys – the number of Redis keys being passed
key – the key name that will be available through the KEYS variable inside the script
arg – an additional argument. It will be available through the ARGV variable.

The following example uses Lua to run the GET command and retrieve a key value. Create a file called luaget.js with the following code:

var redis = require("redis");
var client = redis.createClient();

//create a key called "testkey"
client.set("testkey", "testvalue");

//create a variable and assign Lua code to it
//This Lua code uses the redis.call function to run the GET command
//KEYS is an array with all key names passed to the script
var luaScript = 'return redis.call("GET", KEYS[1])';
//execute the script
client.eval(luaScript, 1, "mykey", function(err, reply){
    //display the return of the Lua script
    console.log(reply);
    client.quit();
});

var redis = require("redis");

var client = redis.createClient();

//create a key called "testkey"

client.set("testkey", "testvalue");

//create a variable and assign Lua code to it

//This Lua code uses the redis.call function to run the GET command

//KEYS is an array with all key names passed to the script

var luaScript = 'return redis.call("GET", KEYS[1])';

//execute the script

client.eval(luaScript, 1, "mykey", function(err, reply){

//display the return of the Lua script

console.log(reply);

client.quit();

});

Then execute it. You should see the following:

The next example will be the implementation of the zpop function as a Lua script. It will be atomic as Redis will always guarantee that there are no parallel changes to the Sorted Set during script execution.

Create a file called zpop-lua.js with the following code:

var redis = require("redis");
var client = redis.createClient();

client.zadd("visits", 2013, 100000);
client.zadd("visits", 2014, 200000);
client.zadd("visits", 2015, 300000);

//Lua code
//uses the redis.call function to execute the Redis command
//ZRANGE to retrieve an array with only the first element in
//the Sorted Set. Then it executes the ZREM to remove the first element
//of the Sorted Set
var luaScript = [
    'local elements = redis.call("ZRANGE", KEYS[1], 0, 0)',
    'redis.call("ZREM", KEYS[1], elements[1])',
    'return elements[1]'
].join('\n');

//execute the Lua script
client.eval(luaScript, 1, "visits", function(err, reply){
    console.log("The first value in the group is: ", reply);
    client.quit();
});

var redis = require("redis");

var client = redis.createClient();

client.zadd("visits", 2013, 100000);

client.zadd("visits", 2014, 200000);

client.zadd("visits", 2015, 300000);

//Lua code

//uses the redis.call function to execute the Redis command

//ZRANGE to retrieve an array with only the first element in

//the Sorted Set. Then it executes the ZREM to remove the first element

//of the Sorted Set

var luaScript = [

'local elements = redis.call("ZRANGE", KEYS[1], 0, 0)',

'redis.call("ZREM", KEYS[1], elements[1])',

'return elements[1]'

].join('\n');

//execute the Lua script

client.eval(luaScript, 1, "visits", function(err, reply){

console.log("The first value in the group is: ", reply);

client.quit();

});

Run the above code. You should see the following console output:

When executing the same script multiple times, you can save network bandwidth usage by using the commands SCRIPT LOAD and EVALSHA instead of EVAL. The SCRIPT LOAD command caches a Lua script and returns an identifier. The EVALSHA command executes a Lua script based on that identifier. With EVALSHA, over a small identifier is transferred over the network.

Create a file called evalsha-example.js with the following code:

var redis = require("redis");
var client = redis.createClient();

var luaScript = 'return "Lua script using EVALSHA"';

client.script("load", luaScript, function(err, reply){
    var scriptId = reply;

    client.evalsha(scriptId, 0, function(err, reply){
        console.log(reply);
        client.quit();
    });
});

var redis = require("redis");

var client = redis.createClient();

var luaScript = 'return "Lua script using EVALSHA"';

client.script("load", luaScript, function(err, reply){

var scriptId = reply;

client.evalsha(scriptId, 0, function(err, reply){

console.log(reply);

client.quit();

});

Then execute the script. You should see the following output:

Misc commands

INFO: returns all Redis server statistics, including the Redis version, OS, connected clients, memory usage, persistence, keyspace, and replication. By default, shows all available sections: memory, persistence, CPU, command, cluster, replication, and clients.

DBSIZE: returns the number of existing keys in a Redis server.
DEBUG SEGFAULT: crashes the Redis server process by performing an invalid memory access.
MONITOR: shows all the commands processed by the Redis server in real time.

CLIENT LIST: returns a list of all clients connected to the server.
CLIENT SETNAME: changes a client name.
CLIENT KILL: terminates a client connection. It is possible to terminate by IP, port, ID, or type.

FLUSHALL: deletes all keys from Redis.
RANDOMKEY: returns a random existing key name.
EXPIRE: sets a timeout in seconds for a given key. The key will be deleted after the specified amount of seconds. A negative timeout will delete the key instantaneously.
EXPIREAT: sets a timeout for a given key based on a Unix timestamp.
TTL: returns the remaining time to live(in seconds) of a key that has an associated timeout. Returns -1 if the key does not have an associated timeout. Returns -2 if the key does not exist.
PTTL: the same as TTL, but the return value is in milliseconds.
SET: set a value to a given key.
Syntax:
SET key value [EX seconds|PX milliseconds] [NX|XX]
EX – set an expiration time in seconds
PX – set an expiration time in milliseconds
NX – only set the key if it does not exist
XX – only set the key if it already exists

PERSIST: removes the existing timeout of a given key. Returns 1 if the timeout is removed of 0 if the key does not have an associated timeout.
SETEX: sets a value to a given key and an expiration.
DEL: removes one or many keys from Redis. Returns the number of removed keys.
EXISTS: returns 1 if a certain key exists and 0 if it does not.
PING: returns “PONG”. Useful for testing server/client connection.
MIGRATE: moves a given key to a destination Redis server. This command is atomic, and both Redis servers will be blocked during the key migration.
Syntax:
MIGRATE host port key destination-db timeout [COPY] [REPLACE]
COPY – keep the key in the local Redis server and create a copy in the destination server.
REPLACE – replace the existing key in the destination server.
SELECT: changes the current database that the client is connected to. Redis has 16 databases by default.
AUTH: is used to authorize a client to connect to Redis.
SCRIPT KILL: terminates the running Lua script if no write operations have been performed by the script. If the script has performed any write operations, the SHUTDOWN NOSAVE command must be executed. Returns OK, NOTBUSY, and UNKILLABLE.
SHUTDOWN: stops all client, causes data to persist if enabled, and shuts down the Redis server. Accepts optional parameters:
SAVE – forces Redis to save all of the data to a file called dump.rdb.
NOSAVE – prevents Redis from persisting data to the disk.
OBJECT ENCODING: returns the encoding used by a given key.

Optimizations

All data types in Redis can use different encodings to improve performance or save memory. A String that has only digits (1234) uses less memory that a string of letters because they use different encodings. Data types use different encodings based on thresholds defined in the Redis configuration file (redis.conf).

Start a Redis server with low values for all configurations.

String

Available String encodings:
int: is used when the string is represented by a 64-bit signed integer
embstr: is used for strings fewer that 40 bytes
raw: is used for strings more than 40 bytes

List

Available encodings for Lists:
ziplist: is used when the List size has fewer elements than the configuration list-max-ziplist-entries and each List element has fewer bytes than the configuration list-max-ziplist-value
linkedlist: us used when the previous limits are exceeded

Set

Available encodings for Sets:
intset: is used when all elements of a Set are integers and the Set cardinality is smaller than set-max-intset-entries
hashtable: is used when any element of a Set is not an integer or the Set cardinality exceeds set-max-intset-entries

Hash

Available encodings for Hashes:
ziplist: is used when the number of fields in the Hash does to exceed the hash-max-ziplist-entries and each field name and value of the Hash is less(in bytes) that the hash-max-ziplist-value
hashtable: is used when a Hash size or any of its values exceed the hash-max-ziplist-entries and hash-max-ziplist-value

Sorted Set

Available encodings:
ziplist: is used when a Sorted Set has fewer entries than the set-max-ziplist-entries and each of its values are smaller(in bytes) than zset-max-ziplist-value
skiplist: is used when the Sorted Set number of entries or size of any of its values exceeds the set-max-ziplist-entries and zset-max-ziplist-value

To sum up, if you have a large dataset and need to optimize for memory, you can tweak these configurations until you find a good trade-off between memory and performance. That’s all for today 🙂

Comprehensive guide to Redis. Part 4.

Pub/Sub

Transactions

Pipelines

Lua scripting

Misc commands

Optimizations

String

List

Set

Hash

Sorted Set

Leave a Reply Cancel reply