Nginx日志分析ELK安装文档

1.准备工作

1.1安装包

1、jdk-8u60-linux-x.gz

2、elasticsearch-2.2.0.tar.gz

3、logstash-2.2.2.tar.gz

4、kibana-4.4.2-linux-x.tar.gz

5、redis-3.0.7.tar.gz

6、keepalived-1.2.19.tar.gz

1.2操作系统环境

# cat /etc/redhat-release 

CentOS release 6.4 (Final)

1.3安装流程

1.4服务器及软件清单

2.1Java环境的安装

Java环境的安装

#cd /usr/src

#tar xf jdk-8u60-linux-x.gz -C /usr/local/

#cd /usr/local

# mv jdk1.8.0_60 java-1.8.0

# ln -s java-1.8.0 java

设置java环境变量

#cat >> /etc/profile <#To start添加java的环境变量by xxx 20160216

export JAVA_HOME=/usr/local/java

export JRE_HOME=/usr/local/java/jre

export CLASSPATH=.:%JAVA_HOME%/lib/dt.jar:%JAVA_HOME%/lib/tools.jar

export PATH=$JAVA_HOME/bin:$JRE_HOME/bin:$PATH

#To end添加java的环境变量by xxx 20160216

<#使Java环境生效

#source /etc/profile

#查看java版本验证java环境是否正确

#java -version

java version "1.8.0_60"

Java(TM) SE Runtime Environment (build 1.8.0_60-b27)

Java HotSpot(TM) -Bit Server VM (build 25.60-b23, mixed mode)

至此java环境安装正确。

2.2安装logstash软件把log日志写到redis中去

2.2.1安装logstash。

#cd /usr/src

# tarxf logstash-2.2.2.tar.gz -C /usr/local/

#cd /usr/local

# ln -s logstash-2.2.2 logstash

#cd logstash

#mkdir –p etc/conf.d

#mkdirvar

#mkdir log

2.2.2设置配置文件把日志写到redis中

#cd /usr/local/logstash/etc/conf.d

#tcouch s2-android-redis.conf

#vim  s2-android-redis.conf

input {

file {

               path => "/usr/local/nginx/logs/access_android.log" #日志文件路径

start_position => "beginning"

             }

}

output{

redis {

data_type => "list"

key => "s2-android"

host => "172.11.13.30" #rediskeeplived虚拟地址

        port => "6379"          #redis端口号

db => "1"              #选择数据库“1”

        }

}

2.2.3启动logstash

# /usr/local/logstash/bin/logstash agent -f /usr/local/logstash/etc/conf.d/s2-android-redis.conf &

2.3logstash相关知识点

2.3.1 logstash插件目录：/usr/local/logstash/vendor/bundle/jruby/1.9/gems

3.搭建高可用redis server(IP 172.11.13.34;172.11.13.35)

3.1redis server1(主172.11.13.34)的搭建

3.1.1redis的安装

#cd /usr/src

#tar xf redis-3.0.7.tar.gz

# cd redis-3.0.7

#make

# make PREFIX=/usr/local/redis-3.0.7 install

# ln -s /usr/local/redis-3.0.7 /usr/local/redis

# cd /usr/local/redis

# mkdir data

# mkdiretc

# mkdir log

# mkdir –p var/6379

3.1.1.1设置配置文件

#cd /usr/local/redis/etc

#touch 6379.conf

# cat 6379.conf |grep '^[a-z]'

daemonize yes

pidfile /usr/local/redis/var/redis_6379.pid

port 6379

tcp-backlog 511

timeout 0

tcp-keepalive 0

loglevel notice

logfile /usr/local/redis/log/redis_6379.log

databases 16

save 900 1

save 300 10

save 60 10000

stop-writes-on-bgsave-error yes

rdbcompression yes

rdbchecksum yes

dbfilenamedump.rdb

dir /usr/local/redis/var/6379

slave-serve-stale-data yes

slave-read-only yes

repl-diskless-sync no

repl-diskless-sync-delay 5

repl-disable-tcp-nodelay no

slave-priority 100

maxmemory 3GB

appendonly yes

appendfilename "appendonly.aof"

appendfsynceverysec

no-appendfsync-on-rewrite no

auto-aof-rewrite-percentage 100

auto-aof-rewrite-min-size mb

aof-load-truncated yes

lua-time-limit 5000

slowlog-log-slower-than 10000

slowlog-max-len 128

latency-monitor-threshold 0

notify-keyspace-events ""

hash-max-ziplist-entries 512

hash-max-ziplist-value 

list-max-ziplist-entries 512

list-max-ziplist-value 

set-max-intset-entries 512

zset-max-ziplist-entries 128

zset-max-ziplist-value 

hll-sparse-max-bytes 3000

activerehashing yes

client-output-buffer-limit normal 0 0 0

client-output-buffer-limit slave 256mb mb 60

client-output-buffer-limitpubsub 32mb 8mb 60

hz 10

aof-rewrite-incremental-fsync yes

3.1.1.2设置开机启动

#echo 'PATH=$PATH:/usr/local/redis/bin' >>/etc/profile

#source /etc/profile

3.1.1.3启动redis

#/usr/local/redis/bin/redis-server /usr/local/redis/etc/6379.conf

3.1.2Keepalived的安装

3.1.2.1Keepalived的安装及配置

#cd /usr/src

#wgethttp://120.52.72.45/www.keepalived.org/c3pr90ntcsf0/software/keepalived-1.2.19.tar.gz

#tar xf keepalived-1.2.19.tar.gz

# cd keepalived-1.2.19

#./configure --prefix=/usr/local/keepalived-1.2.19  --datadir=/usr/local/keepalived/data --docdir=/usr/local/keepalived/doc

# make&& make install

#cd /usr/local/

#ln -s keepalived-1.2.19  keepalived

#cd /usr/local/keepalived/etc/keepalived

#vim keepalived.conf

! Configuration File for keepalived

global_defs {

router_id redis34

}

vrrp_scriptchk_redis

{

script "/etc/keepalived/scripts/redis_check.sh 127.0.0.1 6379"

interval 2

timeout 2

fall 3

}

vrrp_instanceredis {

state MASTER # master set to SLAVE also

interface eth0

virtual_router_id 50

priority  150

nopreempt # no seize,must add

advert_int 1

authentication {   #all node must same

auth_type PASS

auth_pass 1111

    }

virtual_ipaddress {

                 172.11.13.30/24      #keepalived的虚拟IP

    }

track_script {

chk_redis

    }

notify_master "/etc/keepalived/scripts/redis_master.sh 127.0.0.1 172.11.13.35 6379"

notify_backup "/etc/keepalived/scripts/redis_backup.sh 127.0.0.1 172.11.13.35 6379"

notify_fault /etc/keepalived/scripts/redis_fault.sh

notify_stop /etc/keepalived/scripts/redis_stop.sh

}

3.1.2.2Redis check脚本

#cd /etc/keepalived/scripts/

1.

#touch redis_backup.sh

#chmod +x redis_backup.sh

#vim redis_backup.sh

#!/bin/bash 

REDISCLI="/usr/local/redis/bin/redis-cli"

LOGFILE="/usr/local/redis/log/keepalived-redis-state.log"

echo "[backup]" >> $LOGFILE

date>> $LOGFILE

echo "Run SLAVEOF cmd ..." >> $LOGFILE

$REDISCLI SLAVEOF $2 $3 >> $LOGFILE 2>&1

# echo "Being slave...." >> $LOGFILE 2>&1

sleep 15 #delay 15 s wait data sync exchange role

2.

#touch redis_check.sh

#chmod +x redis_check.sh

#vim redis_check.sh

#!/bin/bash 

ALIVE=`/usr/local/redis/bin/redis-cli -h $1 -p $2 PING`

LOGFILE="/usr/local/redis/log/keepalived-redis-check.log"

echo "[CHECK]" >> $LOGFILE

date>> $LOGFILE

if [ $ALIVE == "PONG" ]; then :

echo "Success: redis-cli -h $1 -p $2 PING $ALIVE" >> $LOGFILE 2>&1

exit 0

else

echo "Failed:redis-cli -h $1 -p $2 PING $ALIVE " >> $LOGFILE 2>&1

exit 1

fi

3.

#touch redis_fault.sh

#chmod +x redis_fault.sh

#vim redis_fault.sh

#!/bin/bash 

LOGFILE=/usr/local/redis/log/keepalived-redis-state.log

echo "[fault]" >> $LOGFILE

date>> $LOGFILE

4.

#touch redis_master.sh

#chmod +x  redis_master.sh

#vim redis_master.sh

#!/bin/bash 

REDISCLI="/usr/local/redis/bin/redis-cli -h $1 -p $3"

LOGFILE="/usr/local/redis/log/keepalived-redis-state.log"

echo "[master]" >> $LOGFILE

date>> $LOGFILE

echo "Being master...." >> $LOGFILE 2>&1

echo "Run MASTER cmd ..." >> $LOGFILE 2>&1

$REDISCLI SLAVEOF $2 $3 >> $LOGFILE

sleep 10 #delay 10 s wait data async cancel sync

echo "Run SLAVEOF NO ONE cmd ..." >> $LOGFILE

$REDISCLI SLAVEOF NO ONE >> $LOGFILE 2>&1

5.

#touch redis_stop.sh

#chmod +x redis_stop.sh

#vim redis_stop.sh

#!/bin/bash 

LOGFILE=/usr/local/redis/log/keepalived-redis-state.log

echo "[stop]" >> $LOGFILE

date>> $LOGFILE

3.1.2.3设置开机启动

#chkconfig –add /etc/init.d/keepalivd

#chkconfigkeepalived on

3.1.2.4启动keepalived

#/etc/init.d/keepalived start

3.2Redis server2(备172.11.13.35)的搭建 

3.2.1Redis的安装

3.2.1.1配置文件不同如下：

#cd /usr/local/redis/etc

#cat 6379.conf |grep '^[a-z]'

daemonize yes

pidfile /var/run/redis_6379.pid

port 6379

tcp-backlog 511

timeout 0

tcp-keepalive 0

loglevel notice

logfile /usr/local/redis/log/redis_6379.log

databases 16

save 900 1

save 300 10

save 60 10000

stop-writes-on-bgsave-error yes

rdbcompression yes

rdbchecksum yes

dbfilenamedump.rdb

dir /usr/local/redis/var/6379

slaveof 172.11.13.34 6379  #redis master IP

slave-serve-stale-data yes

slave-read-only yes

repl-diskless-sync no

repl-diskless-sync-delay 5

repl-disable-tcp-nodelay no

slave-priority 100

maxmemory 3GB

appendonly no

appendfilename "appendonly.aof"

appendfsynceverysec

no-appendfsync-on-rewrite no

auto-aof-rewrite-percentage 100

auto-aof-rewrite-min-size mb

aof-load-truncated yes

lua-time-limit 5000

slowlog-log-slower-than 10000

slowlog-max-len 128

latency-monitor-threshold 0

notify-keyspace-events ""

hash-max-ziplist-entries 512

hash-max-ziplist-value 

list-max-ziplist-entries 512

list-max-ziplist-value 

set-max-intset-entries 512

zset-max-ziplist-entries 128

zset-max-ziplist-value 

hll-sparse-max-bytes 3000

activerehashing yes

client-output-buffer-limit normal 0 0 0

client-output-buffer-limit slave 256mb mb 60

client-output-buffer-limitpubsub 32mb 8mb 60

hz 10

aof-rewrite-incremental-fsync yes

keepalived的安装

keepalived的配置文件：

#cd /usr/local/keepalived/etc/keepalived

#vim keepalived.conf

! Configuration File for keepalived

global_defs {

router_id redis35

}

vrrp_scriptchk_redis

{

script "/etc/keepalived/scripts/redis_check.sh 127.0.0.1 6379"

interval 2

timeout 2

fall 3

}

vrrp_instanceredis {

state BACKUP

interface eth0

virtual_router_id 50

priority  100

advert_int 1

authentication {   #all node must same

auth_type PASS

auth_pass 1111

    }

virtual_ipaddress {

    172.11.13.30/24         #keepalved虚拟IP

    }

track_script {

chk_redis

    }

notify_master "/etc/keepalived/scripts/redis_master.sh 127.0.0.1 172.11.13.34 6379"

notify_backup "/etc/keepalived/scripts/redis_backup.sh 127.0.0.1 172.11.13.34 6379"

notify_fault /etc/keepalived/scripts/redis_fault.sh

notify_stop /etc/keepalived/scripts/redis_stop.sh

}

3.2.1.2Redis check 脚本

#cd  cd /etc/keepalived/scripts/   

1.

#touch redis_backup.sh

#chmod +x redis_backup.sh

#vim redis_backup.sh

#!/bin/bash 

REDISCLI="/usr/local/redis/bin/redis-cli"

LOGFILE="/usr/local/redis/log/keepalived-redis-state.log"

echo "[BACKUP]" >> $LOGFILE

date>> $LOGFILE

echo "Being slave...." >> $LOGFILE 2>&1

echo "Run SLAVEOF cmd ..." >> $LOGFILE 2>&1

$REDISCLI SLAVEOF $2 $3 >> $LOGFILE

sleep 100 #delay 10 s wait data async cancel sync 

exit(0)

2.

#touch redis_check.sh

#chmod +x redis_check.sh

#vim redis_check.sh

#!/bin/bash 

ALIVE=`/usr/local/redis/bin/redis-cli -h $1 -p $2 PING`

LOGFILE="/usr/local/redis/log/keepalived-redis-check.log"

echo "[CHECK]" >> $LOGFILE

date>> $LOGFILE

if [ $ALIVE == "PONG" ]; then :

echo "Success: redis-cli -h $1 -p $2 PING $ALIVE" >> $LOGFILE 2>&1

exit 0

else

echo "Failed:redis-cli -h $1 -p $2 PING $ALIVE " >> $LOGFILE 2>&1

exit 1

fi

3.

#touch redis_fault.sh

#chmod +x redis_fault.sh

#vim redis_fault.sh

#!/bin/bash 

LOGFILE=/usr/local/redis/log/keepalived-redis-state.log

echo "[fault]" >> $LOGFILE

date>> $LOGFILE

4.

#touch redis_master.sh

#chmod +x  redis_master.sh

#vim redis_master.sh

#!/bin/bash 

REDISCLI="/usr/local/redis/bin/redis-cli -h $1 -p $3"

LOGFILE="/usr/local/redis/log/keepalived-redis-state.log"

echo "[master]" >> $LOGFILE

date>> $LOGFILE

echo "Being master...." >> $LOGFILE 2>&1

echo "Run SLAVEOF cmd ... " >> $LOGFILE

$REDISCLI SLAVEOF $2 $3 >> $LOGFILE 2>&1

#echo "SLAVEOF $2 cmd can't excute... ">> $LOGFILE

sleep 10 ##delay 15 s wait data sync exchange role

echo "Run SLAVEOF NO ONE cmd ..." >> $LOGFILE

$REDISCLI SLAVEOF NO ONE >> $LOGFILE 2>&1

5.

#touch redis_stop.sh

#chmod +x redis_stop.sh

#vim redis_stop.sh

#!/bin/bash 

LOGFILE=/usr/local/redis/log/keepalived-redis-state.log

echo "[stop]" >> $LOGFILE

date>> $LOGFILE

3.2.1.3设置开机启动及启动keepalived同上

4.安装logstash server转换日志（IP 172.11.13.33）

4.1Java的安装（略）

4.2Logstash的安装

4.2.1Logstash的安装

#cd /usr/src

#tar xf logstash-2.2.2.tar.gz -C /usr/local/

#cd /usr/local

#ln -s logstash-2.2.2 logstash

#cd logstash

#mkdir -p etc/conf.d

#mkdir log

#mkdir pattern#与logstash在nginxlog server上不同

#mkdirvar

4.2.2配置logstash.conf文件

#cd /usr/local/logstash/etc/conf.d/

#touch s2.android-es.conf

#vim s2.android-es.conf

input {

redis {

        host => "172.11.13.30"     #rediskeepalived的虚拟IP

port => "6379"

db => "1"

data_type => "list"

key => "s2-android"

        }

}

filter {

grok {

match => {"message" => "%{WCC_ANDROID_NGINX}"}

patterns_dir => "/usr/local/logstash/pattern"

        }

kv {

add_field =>{"request"=>"%{interface}?%{interface_parameter}"}

source => "interface_parameter"

field_split => "&?"

value_split => "="

        }

geoip {

source =>"clientip"

        }

urldecode {

all_fields => true

        }

}

output {

elasticsearch {

hosts => "172.11.13.31"

index => "s2-android-%{+YYYY.MM.dd}"

                }

}

4.2.3配置partten （kibana需要的拆分依据）

#cd /usr/local/logstash/pattern

#touch nginx_access

#vim nginx_access

NGUSERNAME [a-zA-Z\\.\\@\\-\\+_%]+

NGUSER %{NGUSERNAME}

WCC_ANDROID_NGINX %{IPORHOST:clientip} %{NGUSER:ident} %{NGUSER:auth} \\[%{HTTPDATE:timestamp}\\] \\"%{WORD:verb} %{NOTSPACE:interface}\\?%{NOTSPACE:interface_parameter} HTTP/%{NUMBER:httpversion}" %{NUMBER:response} %{NUMBER:bytes} "-" %{QS:agent}\\s+(?:%{IP:http_forward}|-)\\s+"(?:%{WORD:grid}|-)"\\s+"(?:%{WORD:imsi}|-)"\\s+"(?:%{WORD:urid}|-)"\\s+"%{BASE10NUM:request_time}"

4.2.4启动logstash

#/usr/local/logstash/bin/logstash agent -f /usr/local/logstash/etc/conf.d/s2.android-es.conf &

5.Elasticsearchserver的安装（IP172.11.13.31；172.11.13.32）

5.1Elasticsearch server1的安装（IP 172.11.13.31）

5.1.1Java的安装（略）

5.1.2Elasticsearch的安装

#cd /usr/src

#tar xf elasticsearch-2.2.0.tar.gz -C /usr/local/

#cd /usr/local/

#ln -s elasticsearch-2.2.0 elasticsearch

#cd /usr/local/elasticsearch

#mkdir data

#mkdir logs

5.1.3配置elasticsearch.yml文件

#/usr/local/elasticsearch/config

# catelasticsearch.yml|grep '^[a-z]'

cluster.name: s2-es

node.name: s2-es-node1

path.data: /usr/local/elasticsearch/data

path.logs: /usr/local/elasticsearch/logs

bootstrap.mlockall: true

network.host: 172.11.13.31

http.port: 9200

discovery.zen.ping.unicast.hosts: ["172.11.13.31", "172.11.13.32"]

5.1.4root启动报错

#/usr/local/elasticsearch/bin/elasticsearch

Exception in thread "main" java.lang.RuntimeException: don't run elasticsearch as root.

#在之前的版本中可以使用root账户启动，但2.0版本及以后的版本估计就不行了。

解决办法，创建elasticsearch启动用户

#groupaddes

#useradd -g es -s /sbin/nologines

#chown –R es.es /usr/local/elasticsearch-2.2.0

#sues

#/usr/local/elasticsearch/bin/elasticsearch  #启动elasticsearch

5.1.5在es用户下安装elasticsearch的插件

#/usr/local/elasticsearch/bin/plugin install license

#/usr/local/elasticsearch/bin/plugin install marvel-agent

#/usr/local/elasticsearch/bin/plugin install mobz/elasticsearch-head

#/usr/local/elasticsearch/bin/plugin install lmenezes/elasticsearch-kopf

5.2Elasticsearch server2的安装（IP 172.11.13.32）

5.2.1Java的安装（略）

5.2.2Elasticsearch的安装(略)

5.2.3配置elasticsearch.yuml文件

#/usr/local/elasticsearch/config

# catelasticsearch.yml|grep '^[a-z]'

cluster.name: s2-es

node.name: s2-es-node2

path.data: /usr/local/elasticsearch/data

path.logs: /usr/local/elasticsearch/logs

bootstrap.mlockall: true

network.host: 172.11.13.32

http.port: 9200

discovery.zen.ping.unicast.hosts: ["172.11.13.31", "172.11.13.32"]

5.3elasticsearch界面截图

6.kibana server的安装（IP 172.11.13.36）

6.1java的安装（略）

6.2kibana的安装及配置

#cd /usr/src

#tar xf kibana-4.4.2-linux-x.tar.gz -C /usr/local/

#cd /usr/local

#ln -s kibana-4.4.2-linux-x kibana

#cd kibana

# cdconfig

#cat kibana.yml|grep '^[a-z]'

server.port: 5601

server.host: "172.11.13.36"

elasticsearch.url: http://172.11.13.31:9200

6.3安装kibana插件

#/usr/local/kibana/bin/kibana plugin install elasticsearch/marvel/latest

#/usr/local/kibana/bin/kibana plugin install elastic/sense

6.4启动kibana

#/usr/local/kibana/bin/kibana

6.5Kibana页面截图

Kibana界面截图展示：

Kibana插件marvel截图：

Kibana插件sense截图