Centos 7.3 部署ELk环境

2018-02-02 18:42:00
aolens
原创
722
环境:
centos 7
172.16.16.83 nginx logstash
172.16.16.84 elasticsearch(master) kibana
172.16.16.86 elasticsearch(slave)

版本:

elasticsearch-2.2.0.tar.gz
kibana-4.4.0-linux-x64.tar.gz
logstash-2.2.1.tar.gz


安装依赖
所有环境安装 jdk-1.8.0_65
elk安装在/home/elk下
1,安装logstash
mkdir /home/elk ;cd elk
tar zxf logstash-2.2.1.tar.gz
mv logstash-2.2.1 logstash
创建systemd管理脚本
vim /etc/systemd/system/logstash.service
[Unit]
Description=no description given
[Service]
Type=simple
User=root
Group=root
EnvironmentFile=-/etc/default/logstash
EnvironmentFile=-/etc/sysconfig/logstash
ExecStart=/home/elk/logstash/bin/logstash "-f" "/home/elk/logstash/conf/nginx.conf" "-l" "/home/elk/logstash/log/logstash.log"
Restart=always
WorkingDirectory=/
[Install]
WantedBy=multi-user.target
cd /home/elk/logstash/
mkdir conf log ;touch conf/nginx.conf
input {
file {
path => "/var/log/nginx/access_json.log"
type => "json"
codec => "json"
start_position => "beginning"
}
}
filter {
geoip {
source => "client"
#fields => ["city_name","country_code2","country_name","latitude","longitude"] #geoip指定输出的列
}
}
output {
stdout {
codec => rubydebug
}
elasticsearch {
hosts => ["172.16.16.84:9200"]
index => "logstash-nginx-dev-%{type}-%{+YYYY.MM.dd}"
document_type => "nginx"
workers => 1
flush_size => 5000
idle_flush_time => 100
template_overwrite => true
}
}

配置nginx 日志格式:

log_format json '{"@timestamp":"$time_iso8601",'
'"@version":"1",'
'"host":"$server_addr",'
'"client":"$remote_addr",'
'"size":$body_bytes_sent,'
'"responsetime":$request_time,'
'"domain":"$host",'
'"url":"$request",'
'"refer":"$http_referer",'
'"agent":"$http_user_agent",'
'"status":"$status"}';
access_log /var/log/nginx/access_json.log json;


logstash就配置好了
2,配置elasticsearch集群
先启动的就是master节点
先配置master 172.16.16.84
useradd es
tar zxf elasticsearch-2.2.0.tar.gz ;mv elasticsearch-2.2.0 elasticsearch ;
chown -R es.es elasticsearch ;cd elasticsearch
mkdir data log
vim config/elasticsearch.yml
cluster.name: nginx_log_maps #集群必须一样
discovery.zen.ping.multicast.enabled: false
discovery.zen.ping.unicast.hosts: ["172.16.16.84:9300", "172.16.16.86:9300"]
network.publish_host: 172.16.16.84 #elasticsearch从2.2不支持自动发现集群,需要添加上折三项单播功能
node.name: prod-nginx-02
path.data: /home/elk/elasticsearch/data
path.logs: /home/elk/elasticsearch/log
network.host: 0.0.0.0
http.port: 9200 #需要不一样,一样会有冲突
安装插件:
[root@prod-nginx-02 /home/elk/elasticsearch/bin]# ./plugin list
Installed plugins in /home/elk/elasticsearch/plugins:
- kopf
- head
- analysis-icu

安装:
plugin install analysis-icu
plugin install lmenezes/elasticsearch-kopf
plugin install mobz/elasticsearch-head
slave 172.16.16.86
useradd es
tar zxf elasticsearch-2.2.0.tar.gz ;mv elasticsearch-2.2.0 elasticsearch
chown -R es.es elasticsearch ;cd elasticsearch
mkdir data log
vim config/elasticsearch.yml
cluster.name: nginx_log_maps
node.name: prod-mq-02
path.data: /home/elk/elasticsearch/data
path.logs: /home/elk/elasticsearch/log
network.host: 0.0.0.0
http.port: 9201

创建systemd管理脚本 master slave 是一样的
vim /etc/systemd/system/elasticsearch.service
[Unit]
Description=Elasticsearch
Documentation=http://www.elastic.co
Wants=network-online.target
After=network-online.target
[Service]
Environment=ES_HOME=/home/elk/elasticsearch
Environment=CONF_DIR=/home/elk/elasticsearch/config
Environment=DATA_DIR=/home/elk/elasticsearch/data
Environment=LOG_DIR=/home/elk/elasticsearch/log
Environment=PID_DIR=/home/elk/elasticsearch/var/run
EnvironmentFile=-/etc/sysconfig/elasticsearch
WorkingDirectory=/home/elk/elasticsearch
User=es
Group=es
ExecStart=/home/elk/elasticsearch/bin/elasticsearch \
-Des.pidfile=${PID_DIR}/elasticsearch.pid \
-Des.default.path.home=${ES_HOME} \
-Des.default.path.logs=${LOG_DIR} \
-Des.default.path.data=${DATA_DIR} \
-Des.default.path.conf=${CONF_DIR}
StandardOutput=journal
StandardError=inherit
LimitNOFILE=65536
TimeoutStopSec=0
KillSignal=SIGTERM
SendSIGKILL=no
SuccessExitStatus=143
[Install]
WantedBy=multi-user.target
3,配置kibana
tar zxf kibana-4.4.0-linux-x64.tar.gz
mv kibana-4.4.0 kibana ;cd kibana/bin
安装插件:
./kibana plugin --install elastic/sense API 调试插件
配置systemd管理
vim /etc/systemd/system/kibana.service
[Unit]
Description=no description given
[Service]
Type=simple
User=root
Group=root
EnvironmentFile=-/etc/default/kibana
EnvironmentFile=-/etc/sysconfig/kibana
ExecStart=/home/elk/kibana/bin/kibana "--user" "kibana"
Restart=always
WorkingDirectory=/
[Install]
WantedBy=multi-user.target
4,启动
先启动elasticsearch master systemctl start elasticsearch.service
再启动elasticsearch slave systemctl start elasticsearch.service
启动kibana systemctl start kibana.service
最后启动logstash systemctl start logstash.service