Opensource on paper.archive

redis data types

Wed, 27 Jul 2016 14:32:00 +0000

Redis Data Type Summary

There are plenty of good resources on the internet, but I’m writing this up as study

Testing is done with redis-cli, but for real use I’d probably use an api library for the language

Overview

http://redis.io/topics/data-types-intro

Examples

http://redis.io/topics/data-types

Strings

Stores characters, numbers, etc. as values
No separate type on save (no distinction between numbers and strings)
Numbers can be stored too, and atomic counter operations like incr, incrby, decr, decrby are possible on them
incrby, decrby add or subtract a specific number, used like incrby "test_strings" 10

# redis-cli
127.0.0.1:6379> set "test_strings" 1
OK
127.0.0.1:6379> get "test_strings"
"1"
127.0.0.1:6379> incr "test_strings"
(integer) 2
127.0.0.1:6379> get "test_strings"
"2"
127.0.0.1:6379>

Lists

Stores a list as the value
lrange: reads values, where -1 means fetch everything

127.0.0.1:6379> lpush "test_lists" 1
127.0.0.1:6379> lpush test_lists 2
127.0.0.1:6379> lrange test_lists 0 -1
1) "2"
2) "1"
127.0.0.1:6379> rpush test_lists 3
(integer) 3
127.0.0.1:6379> lrange test_lists 0 -1
1) "2"
2) "1"
3) "3"

Seems like you could implement a queue with rpop

127.0.0.1:6379> rpop test_lists
"3"
127.0.0.1:6379> lrange test_lists 0 -1
1) "2"
2) "1"

With rbpop it seems you could even implement sequential distributed work. Similar to rpop but if there’s no data it waits in a block state until data comes in

Sets

Holds the value as a set
Lists allow duplicates but sets don’t

127.0.0.1:6379> sadd test_sets 1
(integer) 1
127.0.0.1:6379> smembers test_sets
1) "1"
127.0.0.1:6379> sadd test_sets 1 2 3 4
(integer) 3
127.0.0.1:6379> smembers test_sets
1) "1"
2) "2"
3) "3"
4) "4"

Hashes

Hashes hold a key/value list as the value

127.0.0.1:6379> hset htest username hi
(integer) 1
127.0.0.1:6379> hset htest userpwd asdf
(integer) 1
127.0.0.1:6379> hget htest username
"hi"
127.0.0.1:6379> hgetall htest
1) "username"
2) "hi"
3) "userpwd"
4) "asdf"

hget returns (nil) if there’s no value

1
2

127.0.0.1:6379> hget htest temp
(nil)

Changing the value for a hashkey

127.0.0.1:6379> hget htest userpwd
"asdf"
127.0.0.1:6379> hset htest userpwd 1234
(integer) 0
127.0.0.1:6379> hget htest userpwd
"1234"

Sorted sets

Holds the value as a set
Like Sets, no duplicates
Stored together with a score, and sorted by score
Seems usable like a list, with the advantage of being sorted

127.0.0.1:6379> zadd test_ssets 1 1
(integer) 1
127.0.0.1:6379> zadd test_ssets 2 2
(integer) 1
127.0.0.1:6379> zrange test_ssets 0 -1
1) "1"
2) "2"

Don’t use a string for the score

1
2

127.0.0.1:6379> zadd test_ssets "a" 2
(error) ERR value is not a valid float

Since duplicates aren’t allowed, inserting the same value overwrites the existing data’s score and the order changes

127.0.0.1:6379> zadd test_ssets "0" 2
(integer) 0
127.0.0.1:6379> zrange test_ssets 0 -1
1) "2"
2) "1"

Bitmaps

Stores bit values
Can store 2^32 (4.2 billion) bit values in 512MB
Seems good for storing boolean option values (like whether each member has viewed a notice)

127.0.0.1:6379> setbit test_bits 0 1
(integer) 0
127.0.0.1:6379> getbit test_bits 0
(integer) 1
127.0.0.1:6379> setbit test_bits 0 0
(integer) 1
127.0.0.1:6379> getbit test_bits 0
(integer) 0

Collecting logs with fluentd

Wed, 22 Jun 2016 15:47:00 +0000

I’ve been collecting error logs with flume, but I’m switching to fluentd.
~~actually the goal is monitoring, not collecting~~

Install

Just run the command shown on the site below and it handles everything including chkconfig registration. Official site for reference
http://docs.fluentd.org/articles/install-by-rpm#step-0-before-installation

The architecture seems similar to flume
Set up client -> server, and logs collected on the client get forwarded to the server.

Run this command

`1`	`curl -L https://toolbelt.treasuredata.com/sh/install-redhat-td-agent2.sh \| sh`

Start the service

`1`	`service td-agent start`

Obviously service td-agent stop stops it

Getting average response time, call count, Min, Max on a restful api server

Wed, 08 Jun 2016 15:32:00 +0000

There’s an api server that handles data traffic from an app
I computed the average response time per interface using the apache logs

The server is apache+tomcat, implemented with spring

For the hadoop, spark, and python setup see
Analyzing log files with Python and Spark (with hadoop)

1. Log the response time in the apache TransferLog

Before analyzing, make apache record the response time in its log

I edited the following at /etc/httpd/conf.d/ssl.conf.
Of course. the path and name of the config file may differ per server

Analyzing log files with Python and Spark (with hadoop)

Mon, 30 May 2016 16:35:00 +0000

File analysis with Spark

I did this without really knowing spark or hadoop, so there will be mistakes.
For reference, this was done on OSX.

Setup

1. Install and run Hadoop

2. Put the file on hdfs.

1
2

cd /logs
hdfs dfs -put test.log /input/

You can browse files as below. Check the uploaded file here! http://localhost:50070/explorer.html#/input

3. Test spark’s python command..

$SPARK_HOME/bin/pyspark With hadoop running, launching pyspark shows this

Installing redis 3.2.3

Sun, 20 Sep 2015 21:47:00 +0000

Install the compiler first. (Skip if already installed)

`1`	`yum install gcc gcc-c++ autoconf automake`

Reference) http://www.redis.io/download

Install

# 다운로드 및 설치 (컴파일)
wget http://download.redis.io/releases/redis-3.2.3.tar.gz
tar xzf redis-3.2.3.tar.gz
cd redis-3.2.3
make && make install
cd utils
./install_server.sh

Check the run log

`1`	`tail -f /var/log/redis_6379.log -n 1000`

Removing the warnings in the log

Add the following to vi /etc/sysctl.conf

1
2

vm.overcommit_memory=1
fs.file-max = 1048576

Add the following to vi /etc/rc.local

1
2

echo never > /sys/kernel/mm/transparent_hugepage/enabled
sysctl -w net.core.somaxconn=65535

Add the following to vi /etc/security/limits.conf

* soft nofile 65536
* hard nofile 65536
* soft nproc 131072
* hard nproc 131072