Code with Music Abhinav Singhs Homepage. In the last post we saw how Redis can easily be modified to persist the last published message on Pub. Sub channels. Without subscribing to the Pub. Sub channel we were able to get the last published message from Redis db. In this post, I will take that idea one step ahead and add native capabilities within Redis to persist all the unprocessed messages published on Pub. Sub channel in channel specific lists. Well also preserve our capability to send the last published message to clients upon subscription. Im learning Apigility Apigility docu REST Service Tutorial and trying to send a POST request with basic authentication via cURL curl X POST i H Content. But why are we doing this Popular open source application that provide support for Redis are based out of its list class of API. For example, lets look at Celery which is a distributed task queue written in Python. Start redis cli MONITOR on a terminal and then start celery in another window as follows pip install celery. Youll find celery polling Redis periodically as indicated by log lines like BRPOP testing testingx. Celery uses the Redis Pub. Sub mechanism that we disabled in the above command only for internal features. Sentry, a popular exception logging and aggregation library, internally depends upon Celery. There is an open pull request that claims to add Redis pubsub based support to Celery. In the world of Ruby, background processing frameworks like Requeue and Sidekiq depend upon Redis list class of APIs. However, with no native support for persistence of Pub. A short tutorial walking you through how to obtain a Facebook Access Token and use it to access and display your newsstatus feed on your own website. My mom loves me. But she also likes mea lot. And apparently, when she does so on Facebook, its hurting my chances of becoming the next viral sensation. Sub messages in Redis, its not difficult to understand why adopting to Redis Pub. Sub can be tricky for some. Currently, Redis simply drops the message if no subscribers are found. Hence, question really is whether your application is tolerant to loss of published messages for example, dropped messages while you were upgrading your application To solve persistence problem with Redis pubsub, the usual approach is to start multiple application instances. Some instances can continue to serve while others get deployed. However still, your active instances might be experiencing a network partition and unable to receive published messages. After all, primary goal is to guarantee processing of every message received by Redis irrespective of whether we are using list or pubsub based backend. A native support to solve persistence problems with Redis Pub. Sub is clearly desirable. Persisting dropped Redis Pub. Sub messages in a list. In the last post we added a single line of code to persist the last published message on channels in a separate Redis key. Well update implementation to push every received message at the end of a channel specific list. Replace set. Keyc db, c argv1, c argv2 line that we added the last time with following code Persist messages in list only if no receivers were found. REDISTAIL. Fetch list key from the database. Key. Writec db,c argv1. For every published message on the channel. Object. Encodingc argvj. Ensure we have our quicklist initialized. Quicklist. Object. Set. Optionslobj ptr, server. Addc db,c argv1,lobj. Push message at the tail of the list. Type. Pushlobj,c argvj,where. Signal key watchers and internal event subscribers. REDISHEAD lpush rpush. Modified. Keyc db,c argv1. Keyspace. EventREDISNOTIFYLIST,event,c argv1,c db id. I have added some comments in the code for clarity. This code is merely is a rip off of srctlist. Generic. Command function. We dont want to send replies to the client that are usually sent after an RPUSH command. Frankly, we can further refactor push. Generic. Command function to turn the above code into a single function call. Using. srcredis cli try. srcredis cli. Voila Since we published a few messages with no active subscriber, they all got persisted in a list named after the channel itself. Now incoming subscribers can process pending messages by fetching them from the list which otherwise would have been dropped. Delivering unprocessed messages to subscribers upon subscription. Instead of depending upon clients to poll for channel list length, server can deliver unprocessed messages to subscribers upon successful subscription. Since this can get overwhelming for subscribers if there are several pending messages waiting in the list, we may not want to do this at all and leave it up to the clients. Lets preserve our feature from the last post i. Here we dont want to remove the last published message from our persistent list. We simply wish to send it to the incoming subscribers. For example, imagine cases like user status, location and mood being published on separated channels. Here is a method that will give back the last element from the list without removing it equivalent to LRANGE key 1 1 robj llastredis. Client, robj ey. Type. Entry entry. NULL. Fetch list object from db. Key. Readc db, key. Ensure key contains a list. NULL o encoding REDISENCODINGQUICKLIST. Get list iterator for lrange key 1 1 use case. Type. Lengtho. list. Type. Iterator ter list. Type. Init. Iteratoro, llen 1, REDISTAIL. Fetch last item, prepare value. Type. Nextiter, entry. Entry e entry. String. Objectchar qe value. String. Object. From. Long. Longqe longval. Type. Release. Iteratoriter. We now need to replace our modifications within subscription handling methods from the last post. Note, callee must call decr. Ref. Count on the returned robj to free up created string object after delivery. Kindly check this github commit for all the changes since the last post. You can also checkout my Redis fork and directly play with the modified code. We saw how easy it is to modify Redis for fun and profit. By adding native persistence capabilities, we offload our task of ensuring processing of every message received by Redis cluster. After all, none of the magical client side logics will ever be as reliable as native support. By the way, Redis 3. Leave your comments and feedback.
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. Archives
November 2017
Categories |