Redis官方文档笔记

An introduction to Redis data type

Redis keys:
- Very long keys are not a good idea. For instance a key of 1024 bytes is a bad idea not only memory-wise, but also because the lookup of the key in the dataset may require several costly key-comparisons. Even when the task at hand is to match thte existence of a large value, hashing it is a better idea, espically from the prespective of memory and bandwidth.
- Very short keys are often not a good idea. There is little point in writing “u1000flw” as a key if you can instead write “user:1000:followers”. The latter is more readable and the added space is minor compared to the space used by the key object itself and the value object. While short keys will obviously consume a bit less memory, your job is to find the right balance.
- Try to stick with a schema. For instance “object-type:id” is a good idea, as in “user:1000”. Dots or dashes are often used for multi-word fields, as in “comment:1234:reply.to” or “comment:1234:reply-to”
- The maximum allowed key size is 512MB。
Redis string Values can be strings(including binary data) of every kind, for instance you can store a jpeg image inside a value. A value can’t be bigger than 512MB.
Redis List Redis List is implemented via LinkedLists. So random access is a bad idea.

When fast access to the middle of a large collection of elements is important, there is a different data structure that can be used, called sorted sets.

Lists are useful for a number of tasks, two very representative use cases are the following:

Remember the latest updates posted by users into a social network
Communication between processes, using a consumer-producer pattern where producer pushes items into a list, and a consumer consumes those items and executed actions.

Redis allows us to use lists as a capped collection, only remembering the latest N items and discarding all the oldest items using CTRIM command.

Blocking operations on lists: Redis implements commands called BRPOP and BLPOP which are versions of RPOP and LPOP able to block if the list is empty: they’ll return to the caller only when a new element is added to the list, or when a user-specified timeout is reached.

It is possible to build safer queues or rotating queues using RPOPLPUSH

Redis Hashes

The number of fields you can put inside a hash has no practical limits(other than available memory)

Redis Sets

The elements in set are not sorted, since there is no contract with the user about element ordering.

Redis Sorted Sets

Every element in a sorted set is associated with a floating point value, called the score.

Elements in a sorted sets are taken in order. They are ordered according to the following value:

If A and B are two elements with a different score, then A > B if A.score is larger than B.score
If A and B have exactly the same score, then A > B if the A string is lexicographically greater than the B string. A and B strings can’t be equal since sorted sets only have unique elements.

Sorted sets are implemented via a dual-ported data structure containing both a skip list and a hash table, so every time we add an element Redis performing an O(logN) operation.

Sorted sets’ scores can be updated at any time. Just calling ZADD against an element already included in the sorted set will update its score with 0(logN) time complexity. As such, sorted sets are suitable when there are tons of updates.

Bitmaps

Bitmaps are not an actual data type, but a set of bit-oriented operations defined on the string type. Since strings are binary safe blobs and their maximum length is 512MB, they are suitable to set up to 2^32 different bits.

HyperLogLogs

A HyperLogLog is a probabilistic data structure used in order to count unique things. Usually counting unique items requires using an amount of memory proportional to the number of items you want to count, because you need to member the elements you have already seen in the past in order to avoid counting them multiple times. Howerver, there is a set of algorithm that trade memory for precesion: you end with an estimated measure with a standard error, which in the case of the Redis implementation is less than 1%. The magic of this algorithm is that you no longer need to use an amount of memory proportional to the number of items counted, and instead can use a constant amount of memory. 12k bytes in the worst case, or a lot less if your HyperLogLog has seen very few elements.

Redis Pipelining

A Request/Response server can be implemented so that it is able to process new requests even if the client didn’t already read the old response. This way it is possible to send multiple commands to the server without waiting for this replies at all, and finally read the replies in a single step.
While the client sends commands using pipelining, the server will be forced to queue the replies, using memory. So if you need to send a lot of commands with pipelining, it is better to send them as batches having a reasonable number, for instance 10k commands, read the replies, and then send another 10k commands again, and so forth. The speed will be nearly the same, but the additional memory used will be at max the amount need to queue the replies for this 10k commands.
Without using pipelining, serving each command is very cheap from the point of view of accessing the data structures and producing the reply, but it is very costly from the point of view of doing the socket I/O. This involuves calling the read() and write() syscall, that means going from user land to kernal land. This context switch is a huge speed penalty. When pipelining is used, many commands are usually read with a single read() system call, and multiple replies are delivered with a single write() system call. Because of this, the number of total queries performed per second initially increases almost linearly with longer pipelines, and eventually reaches 10 times the baseline obtained not using pipelining.

Pub/Sub

The format of pushed mesages is a Array reply with three elements. The first element is the kind of message:
- Subscribe: means that we successfully subscribed to the channel given as the second element in the reply. The third argument represents the number of channels we are currently subscribed to.
- Unsubscribe: means that we successfully unsubscribed from the channel given as second element in the reply. The third argument represents the number of channels we are currently subscribed to. When the last argument is zero, we are no longer subscribed to any channel, and the client can issue any kinds of Redis command as we are outside the Pub/Sub state.
- message: it is a message received as result of a PUBLISH command issued by another client. The second element is the name of the orginating channel, and the third argument is the actual message payload.
Clients may subscribe to glob-style patterns in order to receive all the messages sent to channel names matching a given pattern by PSUBSCRIBE command.
- The type of the message is pmessage: it is a message received as result of a PUBLISH command issued by another client, matching a pattern-matching subscription. The second element is the original pattern matched, the third element is the name of the originating channel, and the last element the actual message payload.
A client may receive a single message multiple times of it’s subscribed to multiple patterns matching a published message, or if it is subscribed to both patterns and channels matching the message.

Memory Optimization

Hases, Lists, Sets composed of just integers, and Sorted Sets, when smaller than a given number of elements, and up tp a maximum element size, are encoded in a very memory efficient way that uses up to 10 times less memory.
Redis compiled with 32 bit target uses a lot less memory per key, since pointers are small, but such an instance will be limited to 4GB of maximum memory usage. To compile Redis as 32 bit binary use make 32bit. RDB and AOF files are compatible between 32bit and 64bit instances so you can switch from 32 to 64bit, or the contrary, without problems.
Redis support bit or byte operations to reduce the memory usage.
Small hashes are encoded in a very small space, so you should try representing your data using hashes every time it is possible. For instance if you have objects representing users in a web application, instead of using different keys for name, username, email, password, use a single hash with all the required fields.
To store user keys, Redis allocates at most as much memory as the maxmemory setting enables. There are a few things that should be noted about how Redis manages memory.
- Redis will not always free up memory to the OS when keys are removed. This is not something special about Redis, but it is how most malloc() implementations work. For example if you fill an instance with 5GB worth of data, and then remove the equivalent of 2GB of data, the Resident Set Size (also known as RDD, which is the number of memory pages consumed by the process) will probably still be 5GB, even if Redis will claim that the user memory is around 3GB. This happens because the underlying allocator can’t easily release the memory. For example often most of the removed keys were allocated in the same pages as the other keys that still exist.
- The previous point means that you need to provison memory based on your peak memory usage. If your workload from time to time requires 10GB, even if most of the times 5GB could do, you need to prevision for 10GB.
- However allocators are smart and are able to reuse free chunks of memory, so after you freed 2GB of your 5GB data set, when you start adding more keys again, you’ll see the RSS to stay steady and don’t grow more, as you add up to 2GB of additional keys. The allocator is basically trying to reuse the 2GB of memory previously freed.
- Because of all this, the fragmentation ratio is not reliable when you had a memory usage that at peak is much larger than the currently used meomry. The fragmentation is calculated as the amount of memory currently in use divided by the physical memory actually used. Because the RSS reflects the peak memory, when the used memory is low since a lot of keys/values were freed, but the RSS is high, the ratio mem_used/RSS will be very high. Ifmaxmemory is not set Redis will keep allocating memory as it finds fit and thus it can eat up all your free memory. Therefore it is generally advisable to configure some limit. You may also want to set maxmemory-policy to noeviction. It makes Redis return an out of memory error for write commands if and when it reaches the limit - which in turn may result in errors in the application but will not render the whole machine dead because of memory starvation.

Redis Expire

How Redis expires keys? A key is passively expired simply when some client tries to access it , and the key is found to be timed out. Redis also tests a few keys at random among keys with an expire set periodically. All the keys that are already expired are deleted from the key space. Specially this is what Redis does 10 times per second:
- Test 20 random keys from the set of keys with an associated expire
- Delete all the keys found expired
- If more than 25% of keys were expired, start again from step1 This is a trivial probabilistic algorithm, basically the assumption is that our sample is representatiive of the whole key space, and we continue to expire until the percentage of keys that are likely to be expired is under 25%. This means that at any given moment the maximum amount of keys already expired that are using memory is at max equal to max amount of write operations per second divide by 4.
In order to obtain a correct behaviour without sacrificing consistency, when a key expires, a DEL operation is synthesized in both the AOF file and gains all the attached slaves. This way the expiration process is centralized in the master instance, and there is no change of consistency errors.

Redis CRU

Eviction policies The exact behaviour Redis follows when the max memory limit is reached is configured using the maxmemory-policy configuration directive. The following policies are available:
- noeviction: return errors when the memory limit was reached and the client is trying to execute commands that could result in more memory to be used.
- allkeys-lru: evict keys by trying to remove the less recently used(LRU) keys first, in order to make space for the new data added.
- volatile-lru: evict keys by trying to remove the less recently used keys first, but only among keys that have an expire set, in order to make space for the new data added.
- allkeys-random: evict keys randomly in order to make space for the new data added.
- volatile-random: evict keys randomly in order to make space for the new data, but only evict keys with an expire set.
- volatile-ttl: evict keys with an expire set, and try to evict keys with a shorter time to live(TTL) first, in order to make space for the new data added. The policies volatile-lru, volatile-random and volatile-ttl behave like noeviction if there are no keys to evit matching the prerequisites. Ingeneral as a rule of thumb:
- Use the allkey-lru policy when you expect a power-law distribution in the popularity of your requests, that is, you expect that a subset of elements will be accessed far more often than the rest. This is a good pick if you are unsure.
- Use the allkeys-random if you have a cyclic access where all the keys are scanned continuously, or when you expect the distribution to be uniform.
- Use the volatile-ttl and volatile-random policies are mainly useful when you want to use a single instance for both caching and to have a set of persistent keys. However it is usually a better idea to run two Redis instances to solve such a problem. It is also worth to note that setting an expire to a key costs memory, so using a policy like allkeys-lru is more memory efficient since there is no need to set an expire for the key to be evicted under memory pressure.
Redis LRU algorithm is not an exact implementation. This means that Redis is not able to pick the best candidate for eviction, that is, the access that was accessed the most in the past. Instead it will try to run an approximation of the LRU algorithm, by sampling a small number of keys, and evicting the one that is the best(with the oldest access time) among the sampled keys. However since Redis 3.0 the algorithm was improved to also take a pool of good candidates for eviction. This improved the performance of the algorithm, making it able to approximate more closely the behavior of a read LRU algorithm. What is important about the Redis LRU algorithm is that you are able to tunce the precesion of the algorithm by changing the number of samples to check for every eviction. This parameter is controleed by the following configuration directive: maxmemory-samples 5 The reason why Redis does not use a true LRU implementation is because it costs more memory.
Starting with Redis 4.0, a new Least Frequently Used eviction mode is available. THis mode may work better in certain cases, since using LFU Redis wil try to track the frequency of access of items, so that the ones used rarely are evicted while the one used often have on higher chance of remaining in memory. If you think at LRU, an item that was recently accessed but is actually almost never requested, will not get expired, so the risk is to evict a key that has an higher change to be requested in the future. LRU does not have this problem, and in general should adapt better to different access patterns. LFU is approximated like LRU it uses a probablistic counter, called Morris counter in order to estimate the object access frequentcy using just a few bits per object, combined with a delay period so that the counter is reduced over time: at some point we no longer want to consider keys are frequenctly accessed, even if they were in the past, so that the algorithm can adapt a shift in the access pattern. LFU is approximated like LRU: it uses a probablistic counter, called Morris counter in order to estimate the object access frequency using just a few bits per object, combined with a delay period so that the counter is reduced over time: at some point we no longer want to consider keys as frequently accessed, even if they were in the past, so that the algorithm can adapt a shift in the access pattern. However unlike LRU, LFU has certain tunable parameters: for instance, how fast should a frequent item lower in rank if it gets no longer accessed? It is also possible to tune the morris counters range in order to better adapt the algorithm to specific use cases. By default Redis 4.0 is configured to:
- Satuarate the counter at, around, one million requests
- Decay the counter every one minute

Redis Transaction

WHen using the append-only file Redis makes sure to use a single write(2) syscall to write the transaction on disk. However if the Redis server crashes or is billed by the system administrator in some hard way it is possible that only a partial number of operations are registered. Redis will detect this condition at restart, and will exit with an error. Using the redis-check-aof tool it is possible to fix the append only file that will remove the partial transaction so that the server can start again.
A Redis transaction is entered using the MULTI command. The command always replies with “OK”. At this point the user can issue multiple commands. Instead of executing these commands, Redis will queue them. All the commands are executed once “Exec” is called. Calling “DISCARD” instead will flush the trasaction queue and will exit the transaction.
Even when a command fails, all the other commands in the queue are processed. Redis will not stop the processing of commands. Because all the commands in queue are syntax-correct.
During a transaction it is possible to encounter two kinds of command errors:
- A command may fail to be queued, so there may be an error before “EXEC” is called. For instance the command may be syntatically wrong, or there may be some critical condition like an out of memroy condition
- A command may fail after EXEC is called, for instance since we perform an operation against a key with the wrong value.
Redis doesn’t support roll back because the syntax errors will be found out when queue the command.
WATCH is used to provide a check-and-set behavior to Redis tranactions. WATCHed keys are monitored in order to detect changes against them. If at least one watched key is monified before the EXEC command, the whole command aborts, and EXEC returns a Null reply to notify the transaction failed. If you WATCH a volatile key and Redis expires the key after you WATCHed it, EXEC will still work.
When EXEC is called, all keys are UNWATCHED, regardless of whether the transaction was aborted or not. Also when a client connection is closed, everything gets UNWATCHed.

Redis Mass Insertion

Using a normal Redis client to perform mass insertion is not a good idea for a few reasons: the naive approach of sending one command after the other is slow because you have to pay for the round trip time for every command. It is possible to use pipelining, but for mass insertion of many records you need to write new commands while you read replies at the same time to make sure you are inserting as fast as possible.
By generating Redis protocol files.
How the pipe mode works under the hoods?
- redis-cli–pipe tries to send data as fast as possible to the server
- At the same time it reads data when available, trying to parse it
- Once there is no more data to read from stdin, it sends a special ECHO command with a random 20 bytes string: we are sure this is the latest command sent, and we can match the reply checking if we receive the same 20bytes as a bulk reply
- Once a special final command is sent, the code receiving replies start to match replies with this 20 bytes. When the matching reply is reached it can exit with success.

Redis partition

Different implementations of partitioning:
- Client side partitioning
- Proxy assisted partitioning
- Query routing: You can send your query to a random instance, and the instance will make sure to forward your query to the right node.
Disadvantages of partitioning:
- Operations involving multiple keys are usually not supported. For instance you can’t perform the intersection between two sets if they are stored in keys that are mapped to different Redis instances.
- Redis transactions involving multiple keys can not be used
- The partitioning granularity is the key, so it is not possible to shared a dataset with a single huge key like a very big sorted set.
- When partitioning is used, data handling is more complex, for instance you have to handle multiple RDB/AOF files, and to make a backup of your data you need to aggregate the persistence files from multple instances and hosts.
- Adding and removing capacity can be complex. For instance Redis Cluster supports mostly transparent rebalancing of data with the ability to add and remove nodes at runtime, but other systems like client side partitioning and proxies don’t support this feature.

Redis as Distributed Lock

For single node:

Set a key only if it doesn’t already exist(NX option), with an expire. And the value must be unique across all clients and all lock requests. For example, “timestamp + client id” Remove the key only if it exists and the value stored at that key is exactly the one that client sets
For Distributed nodes: Redlock