Collecting logs with fluentd

I’ve been collecting error logs with flume, but I’m switching to fluentd.
~~actually the goal is monitoring, not collecting~~

Install

Just run the command shown on the site below and it handles everything including chkconfig registration. Official site for reference
http://docs.fluentd.org/articles/install-by-rpm#step-0-before-installation

The architecture seems similar to flume
Set up client -> server, and logs collected on the client get forwarded to the server.

Run this command

1
curl -L https://toolbelt.treasuredata.com/sh/install-redhat-td-agent2.sh | sh

Start the service

1
service td-agent start

Obviously service td-agent stop stops it

Configuration

/etc/td-agent/td-agent.conf is the config file

1
vi /etc/td-agent/td-agent.conf

Let’s keep it simple: collect only the error.log of each service and write them to a file on the log server.

The items I need for the setup I’m going for are:
<source />: the data collection path, or receiving from a remote server <match />: the action to handle collected logs (forward to another server or save to a local file)

Check the other config values at the link below (there are a lot..) http://docs.fluentd.org/articles/config-file#2-ldquomatchrdquo-tell-fluentd-what-to-do

Log collection td-agent.conf

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
<source>
  @type tail
  path /logs/error.log
  tag xx.devwas.error

  format multiline
  format_firstline /\[[^\s]+\] .*/
  format1 /\[(?<level>[^\s]+)\] (?<message>.*)/
</source>

<match xx.*.*>
  @type forward
  send_timeout 60s
  recover_wait 10s
  heartbeat_interval 1s
  phi_threshold 16
  hard_timeout 60s
  <server>
    name logserver
    host log-server
    port 24224
    weight 60
  </server>
</match>

Log storage td-agent.conf

1
2
3
4
5
6
7
8
9
<source>
  @type forward
  port 24224
</source>

<match xx.**>
  @type file
  path /logs/td/error
</match>

If you get the error below on startup, give td-agent permission on the storage path

1
Starting td-agent: 2016-06-22 14:13:53 +0900 [error]: fluent/supervisor.rb:359:rescue in main_process: config error file="/etc/td-agent/td-agent.conf" error="out_file: `/logs/td/~~~` is not writable"

1
chown td-agent td

You can see files saved as error.timestamp in the folder below

1
2
3
4
5
6
[root@logserver td]# ls -al
합계 224
drwxr-xr-x. 2 td-agent root       4096 2016-06-22 15:02 .
drwxr-xr-x. 4 root     root       4096 2016-06-22 15:16 ..
-rw-r--r--. 1 td-agent td-agent 220529 2016-06-22 15:38 error.20160622.b535d7aec407e9135
[root@logserver td]#

But it’s being saved as json. Probably inconvenient to read with the naked eye

Or there are lots of plugins, let’s look for one.