Encrypted Redis Replication with PeerVPN Tunnel

Redis is a NoSQL database used by many modern web applications to store ephemeral data. It stores the data set completely in-memory, making it lightning fast compared to file based caches. Application caching and message queues make excellent use cases for Redis, as these types of data typically have a short lifespan.

Redis databases are intended to be puny, compared to RDBMS such as MySQL or Postgres, or even other NoSQL databases such as MongoDB. The Redis data architecture represented by key-value pairs, is very simple by comparison to relational databases with tables, keys, and constraints. If your application needs to store schemable data, yet you want the speed and agility of a NoSQL database, MongoDB is a more suitable choice than Redis.

Another caveat of Redis’ in-memory but persistent on disk nature is the trade-off of data consistency vs speed. Redis regularly flushes data to disk (e.g. every hour) to persist data between reboots, but cannot recover from a crash as effectively as ACID compliant databases. If a single Redis master is unexpectedly shut down, the data written since the last snapshot will be forgotten. For this reason, Redis is usually used to store non-transactional data that is relatively inconsequential if lost.

Setting up Redis master/slave replication is a good strategy for migrating a Redis database to a new server, maintaining a backup Redis to failover to, or scaling Redis with up to 5 read replicas. In this tutorial, we’ll mostly focus on the former two cases for setting up Redis replication. A cluster mode also exists for sharding data, but it is outside the scope of this article.

Redis replication is ridiculously simple to set up, with far fewer parameters than setting up MySQL master/slave replication for instance. Some features which you might expect, such as SSL encryption, are also notably missing from Redis. According to Redis Labs, the developers of Redis, this is by design — they recommend setting up a tunnel between the Redis servers if replicating over an untrusted network.

PeerVPN is an open source peer-to-peer VPN application that can be used to secure Redis replication. The first node must have an address or hostname accessible by all the peers on the underlay network, as it serves the role of initializing the other peers.

Step 1 – Install PeerVPN on the Redis Master

First, install the PeerVPN binaries on the Redis Master. Download the latest from https://peervpn.net/, untar the archive, and move the peervpn executable to /usr/sbin. Then create a config file at /etc/peervpn.conf.

port 7000
networkname RedisReplication
psk mysecretpassword
enabletunneling yes
interface peervpn0
ifconfig4 10.8.0.1/24

The networkname can be modified to anything you prefer. The psk should certainly be changed to a long and difficult-to-guess key, as anybody possessing this pre-shared key can join their node to the network. The ifconfig4 specifies the IP address this node will be assigned within the VPN. It should be incremented by one, within the same subnet for each subsequent node (10.8.0.2, 10.8.0.3…)

Use the following openssl command to generate a random 512-bit key for psk. All nodes connected to the peer-to-peer VPN must use the identical networkname and psk.

openssl rand -base64 382 | tr -d '\n' && echo

Then create a systemd service at /usr/lib/systemd/system/peervpn.service so the VPN can be enabled on boot.

[Unit]
Description=PeerVPN network service
Wants=network-online.target
After=network-online.target
[Service]
ExecStart=/usr/sbin/peervpn /etc/peervpn/peervpn.conf
[Install]
WantedBy=multi-user.target

Enable and start the service.

systemctl enable peervpn.service

systemctl start peervpn.service

Create a firewall rule to ensure port 7000 is open to traffic from the other VPN peers.

Step 2 – Install PeerVPN on the Redis Slave

Do the same as above on the Redis Slave, but with this config file instead at /etc/peervpn.conf.

port 7000
networkname RedisReplication
psk mysecretpassword
enabletunneling yes
interface peervpn0
ifconfig4 10.8.0.2/24
initpeers 111.111.111.111 7000

initpeers is the IP address or hostname, and port, of the first peer on the underlay network (not the address within the VPN). In this case, it is the public IP address of the Redis master, which also serves as the initialization peer for PeerVPN.

As mentioned above, you can join additional Redis slaves to the VPN by assigning them an ifconfig4 address incremented by one in the peervpn.conf file.

Remember to enable and start the PeerVPN service on each node.

Try pinging the other VPN peer with ping 10.8.0.1 (from the Redis slave) and vice versa, ping 10.8.0.2 (from the Redis master), to check that you are able to communicate through the peervpn0 network interface. If not, review your firewall settings to ensure you have opened port 7000 on both hosts.

Step 3 – Secure Redis Master For Replication

By default, Redis listens only on localhost or 127.0.0.1, which is a sane setting as Redis does not require password authentication unless you set it up. For replication through the PeerVPN tunnel however, the Redis Master must bind itself to the IP on the peervpn0 interface, 10.8.0.1.

Redis’ config file, redis.conf, is located at /etc/redis.conf for most distributions.

In Redis 2.8 and above, Redis can be bound to multiple IPs. If possible, we recommend binding Redis only to the networks which should have access to the Redis service. For example, if you wanted Redis to be accessible from localhost and the other VPN peers, specify the following.

bind 127.0.0.1 10.8.0.1

If your Redis instance is currently in production, and your developers haven’t configured a password for Redis, use caution if you decide to add one now. Since Redis doesn’t have multiple user authentication like other DBMS, setting a Redis password will break access for all your applications which are reliant on Redis. You must provide the password through each application’s config file, or a supported environment variable.

To set a password on the master, add the following line to redis.conf.

requirepass your_redis_master_password

Earlier versions of Redis can only be bound to a single IP or all IPs (0.0.0.0). If replicating from older versions of Redis below version 2.8, you must comment out the line in redis.conf with the bind directive to have Redis listen on all IPs (so existing applications can still access Redis). If you bound Redis to the PeerVPN address alone, Redis would become unavailable other applications aside from replication.

# bind 127.0.0.1

To reduce the likelihood of excessive timeouts during Redis replication, set the TCP keepalive timer to 300 seconds.

tcp-keepalive 300

After making changes to redis.conf, restart the redis service.

sudo systemctl restart redis.service

Fair word of warning: If you bind Redis to all IPs and there is no password authentication, your Redis database will be accessible to the world. Use a software firewall such as iptables or ufw to limit access to only the hosts which should have access, such as your application server.

For users of CentOS 7, firewalld zones are an excellent way to easily control this.

You could specify peervpn0 as a network interface in the redis zone, and allow access to the Redis port 6379 only for hosts included in the redis zone. Remember to create the firewalld rules using the firewall-cmd utility with the –permanent flag to persist them on reboot.

sudo firewall-cmd --new-zone=redis
sudo firewall-cmd --zone=redis --permanent --add-port=6379/tcp
sudo firewall-cmd --zone=redis --permanent --add-interface=peervpn0
sudo firewall-cmd --zone=redis --permanent --add-interface=lo
sudo firewall-cmd --zone=redis --list-all
sudo firewall-cmd reload

If you wanted to limit access to Redis even more precisely based on the IPs of the allowed hosts, instead of adding the interface peervpn0 to the zone, you would add each IP address separately, like so.

sudo firewall-cmd --new-zone=redis
sudo firewall-cmd --zone=internal --permanent --add-port=6379/tcp
sudo firewall-cmd --zone=internal --permanent --add-source=10.8.0.2/32
sudo firewall-cmd --zone=redis --permanent --add-interface=lo
sudo firewall-cmd --zone=redis --list-all
sudo firewall-cmd reload

You could include both the interface and a list of source IPs in the redis zone, so hosts matching either would have access to Redis. Other Linux distributions such as Ubuntu or Debian have a similar mechanism such as ufw to control iptables rules and limit access to Redis port 6379.

Use a port scan utility such as nmap to check Redis port 6379 is open and closed to the appropriate hosts. The syntax for using nmap is nmap -p <port> <hostname or IP> when scanning from the local to the remote machine.

Step 4 – Configure Redis Slave For Replication

Once the Redis slave can visibly see the master over the network, initiating replication is a trivial process. First ensure you can access the Redis CLI using the slave machine as a client, and the master machine as the server.

redis-cli -h 10.8.0.1 -p 6379

If successful, modify the following line(s) in redis.conf as follows:

slaveof 10.8.0.1 6379

Should the master Redis server require a password, specify it in this line.

masterauth your_redis_master_password

After making changes to redis.conf, restart the redis service.

sudo systemctl restart redis.service

Drop into the Redis CLI by typing redis-cli at a terminal. Then from the Redis prompt, run:

redis-cli > replication info

The replication info command will show whether the slave is connected to a master server, and

master_sync_in_progress:0 indicates whether the slave is currently copying data from the master. A value of 0 means the slave and master are currently in sync, and a value of 1 means the slave is synchronizing with the master.

# Replication
role:slave
master_host:111.111.111.111
master_port:6379
master_link_status:up
master_last_io_seconds_ago:3
master_sync_in_progress:0
slave_repl_offset:1401
slave_priority:100
slave_read_only:1
connected_slaves:0
master_repl_offset:0
repl_backlog_active:0
repl_backlog_size:1048576
repl_backlog_first_byte_offset:0
repl_backlog_histlen:0

Step 5 – Promote Redis Slave to Master

If using Redis Master/Slave replication to migrate Redis data from one server to another, or in a disaster recovery scenario failing over to the Redis slave, you will want to promote the Redis slave to the master.

For a Redis data migration, be sure that replication info currently shows that the Redis slave is synchronized with the master. To ensure data consistency, stop all applications that make writes to the Redis database, and remap their Redis credentials to the new Redis server.

On the slave (destined to become the master), comment out the slaveof line in redis.conf.

# slaveof 10.8.0.1 6379

After making changes to redis.conf, restart the redis service.

sudo systemctl restart redis.service

Drop into the Redis CLI by typing redis-cli at a terminal. Then from the Redis prompt, run:

redis-cli > slaveof no one

After stopping replication, the Redis slave will no longer be in read-only mode and it will be writeable by any applications that can connect to it. The promoted slave, which is now a Redis master server, can have other slaves connect to it for master/slave replication.