← 返回列表

日志分析系列(二):平台实现篇

单站 bloodzer0 发布 Mon, 06 Jan 2020 08:15:00 +0800 入库 2026-11-18T06:46:07

原创 bloodzer0 2020-01-06 08:15

本系列故事纯属虚构,如有雷同纯属巧合平台实现前的说明小B在给老板汇报了"统一日志分析平台"项目后,老板拍板立

平台实现前的说明

踩坑记录,在文章末尾)
实现日志采集处理与展示

服务器日志

首先在我们的测试服务器上安装filebeat

rpm -ivh filebeat-7.4.1-x86_64.rpm

修改filebeat配置文件中的ES和Kibana地址,并初始化filebeat

vim /etc/filebeat/filebeat.yml

setup.kibana:

host: "10.10.10.9:5601"

output.elasticsearch:

hosts: ["10.10.10.9:9200"]

初始化filebeat

filebeat setup

启动filebeat模块

filebeat modules enable system

filebeat modules enable auditd

初始化filebeat模块的pipelines

filebeat setup --pipelines --modules system

filebeat setup --pipelines --modules auditd

修改system模块中日志文件路径

vim /etc/filebeat/modules.d/system.yml

syslog:

enabled: true

var.paths: ["/var/log/messages"]

auth:

enabled: true

var.paths: ["/var/log/secure"]

修改audit模块中日志文件路径

vim /etc/filebeat/modules.d/auditd.yml

log:

enabled: true

var.paths: ["/var/log/audit/audit.log"]

修改filebeat配置文件中输出 为Kafka

vim /etc/filebeat/filebeat.yml

output.elasticsearch:

hosts: ["10.10.10.9:9200"]

output.kafka:

hosts: ["10.10.10.9:9092"]

topics:

这里的含义:当存在@metadata.pipeline字段时(这个字段是使用module默认生成的字段),topic命名为%{[service.type]}-%{[fileset.name]}

在这里推荐大家,如果不想使用我这种命名方式,可以使用kafka消费者查看详细信息,从信息中获取字段来命名

when.has_fields: ["@metadata.pipeline"]

测试配置文件

filebeat test config -c /etc/filebeat/filebeat.yml

启动filebeat

filebeat -e

创建Logstash配置文件

vim /etc/logstash/conf.d/system.conf

input {

kafka {

bootstrap_servers => "10.10.10.9:9092"

topics => ["system-auth","system-syslog","auditd-log"]

codec => json

}

}

output {

if [@metadata][pipeline] and [@metadata][topic] == "auditd-log" {

elasticsearch {

hosts => ["10.10.10.9:9200"]

index => "%{[@metadata][beat]}-%{[@metadata][version]}-audit-log-%{+YYYY.MM.dd}"

pipeline => "%{[@metadata][pipeline]}"

}

}

else if [@metadata][pipeline] and [@metadata][topic] == "system-syslog" {

elasticsearch {

hosts => ["10.10.10.9:9200"]

index => "%{[@metadata][beat]}-%{[@metadata][version]}-system-syslog-%{+YYYY.MM.dd}"

pipeline => "%{[@metadata][pipeline]}"

}

}

****

}

启动Logstash

/usr/share/logstash/bin/logstash -f /etc/logstash/conf.d

查看Kibana中的索引信息

Web日志

首先修改采集端filebeat的配置

vim /etc/filebeat/filebeat.yml

filebeat.inputs:

paths:

fields:

nginx: access

json.keys_under_root: true

json.overwrite_keys: true

output.kafka:

hosts: ["10.10.10.9:9092"]

topics:

when.has_fields: ["@metadata.pipeline"]

when.has_fields: ["fields.nginx"]

修改Logstash配置

vim /etc/logstash/conf.d/nginx.conf

input {

kafka {

bootstrap_servers => "10.10.10.9:9092"

topics => ["nginx-access"]

codec => json

}

}

output {

if [fields][nginx] == "access" {

elasticsearch {

hosts => ["10.10.10.9:9200"]

index => "nginx-access-%{+YYYY.MM.dd}"

}

}

}

堡垒机日志

配置jumpserver Logstash采集

vim /etc/logstash/conf.d/jumpserver.conf

input {

jdbc {

jdbc_driver_library => "/usr/share/logstash/logstash-core/lib/jars/mysql-connector-java-5.1.36.jar"

jdbc_driver_class => "com.mysql.jdbc.Driver"

jdbc_paging_enabled => "true"

jdbc_connection_string => "jdbc:mysql://10.10.10.6:3306/jumpserver"

jdbc_user => "root"

jdbc_password => "root"

schedule => "/1 * * * "

statement => "select * from terminal_session where date_start >= SUBDATE(now(),interval 481 minute)"

type => "terminal_session"

}

jdbc {

jdbc_driver_library => "/usr/share/logstash/logstash-core/lib/jars/mysql-connector-java-5.1.36.jar"

jdbc_driver_class => "com.mysql.jdbc.Driver"

jdbc_paging_enabled => "true"

jdbc_connection_string => "jdbc:mysql://10.10.10.6:3306/jumpserver"

jdbc_user => "root"

jdbc_password => "root"

schedule => "/1 * * * "

statement => "select id,user,asset,system_user,input,session,timestamp,org_id from terminal_command where FROM_UNIXTIME(timestamp) >= SUBDATE(now(),interval 1 minute);"

type => "terminal_command"

}

}

output {

if [type] == "terminal_session" {

elasticsearch {

hosts => ["10.10.10.9:9200"]

index => "jumpserver-session-%{+YYYY.MM.dd}"

}

}

else if [type] == "terminal_command" {

elasticsearch {

hosts => ["10.10.10.9:9200"]

index => "jumpserver-command-%{+YYYY.MM.dd}"

}

}

}

入侵检测系统日志

安装依赖

yum install cmake.x86_64 gcc.x86_64 gcc-c++.x86_64 flex.x86_64 bison.x86_64 libpcap.x86_64 libpcap-devel.x86_64 openssl-devel.x86_64 python-devel.x86_64 swig.x86_64 zlib-devel.x86_64 -y

配置yum源

wget http://download.opensuse.org/repositories/network:bro/CentOS_7/network:bro.repo -O /etc/yum.repos.d/network:bro.repo

yum install bro.x86_64 -y

配置环境变量

echo 'export PATH=/opt/bro/bin:$PATH' >> /etc/profile

source /etc/profile

vim /opt/bro/share/bro/site/local.bro

在文件末尾添加

@load tuning/json-logs

redef LogAscii::json_timestamps = JSON::TS_ISO8601;

redef LogAscii::use_json = T;

启动zeek模块,执行这里时需要将output.kafka注释,还原output.elasticsearch

filebeat setup --pipelines --modules zeek

如果你已经执行初始化就不用重复执行了

filebeat -e

Filebeat的配置文件与收集系统日志一致,只需要修改logstash的配置文件的input中的topic内容

input {

kafka {

bootstrap_servers => "10.10.10.9:9092"

topics => ["system-auth","system-syslog","auditd-log","zeek-connection","zeek-dns","zeek-http","zeek-files"]

codec => json

}

}

output内容不用变换,只需要替换条件即可

filebeat.modules:

alerts:

enabled: true

archives:

enabled: false

setup.template.json.enabled: true

setup.template.json.path: '/etc/filebeat/wazuh-template.json'

setup.template.json.name: 'wazuh'

setup.template.overwrite: true

setup.ilm.enabled: false

output.kafka:

hosts: ["10.10.10.9:9092"]

topics:

when.has_fields: ["@metadata.pipeline"]

input {

kafka {

bootstrap_servers => "10.10.10.9:9092"

topics => ["wazuh-alert"]

codec => json

}

}

output {

if [@metadata][pipeline] and [@metadata][topic] == "auditd-log" {

elasticsearch {

hosts => ["10.10.10.9:9200"]

index => "wazuh-alerts-3.x-%{+YYYY.MM.dd}"

}

}

}

收集方式 --> 存储 --> 展示 。因为篇幅原因不想写的太冗长影响阅读。后续还会有一些番外传:
- 利用大数据处理日志信息;
- 基于Elastic的监控报警系统;
- 统一日志分析平台的安全性;

vim /etc/hosts10.10.10.9 big_data

vim /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-patterns-core-4.1.2/patterns/grok-patterns# 将 HOSTNAME \b(?:[0-9A-Za-z][0-9A-Za-z-]{0,62})(?:.(?:[0-9A-Za-z][0-9A-Za-z-]{0,62}))*(.?|\b) 修改为

HOSTNAME \b(?:[0-9A-Za-z][0-9A-Za-z_-]{0,62})(?:.(?:[0-9A-Za-z][0-9A-Za-z_-]{0,62}))*(.?|\b) # 注意看两者多了两个下划线

/usr/share/logstash/bin/logstash-plugin install logstash-input-jdbc

wgethttps://repo1.maven.org/maven2/mysql/mysql-connector-java/5.1.36/mysql-connector-java-5.1.36.jar -O /usr/share/logstash/logstash-core/lib/jars/mysql-connector-java-5.1.36.jar

chown logstash:logstash /usr/share/logstash/logstash-core/lib/jars/mysql-connector-java-5.1.36.jar

chmod 644 /usr/share/logstash/logstash-core/lib/jars/mysql-connector-java-5.1.36.jar

sudo -u kibana /usr/share/kibana/bin/kibana-plugin install https://packages.wazuh.com/wazuhapp/wazuhapp-3.10.2_7.4.1.zip

原文链接:https://mp.weixin.qq.com/s?__biz=MzU2NzY5MDY3MQ==&mid=2247483869&idx=1&sn=5b43cdc88622a39b9f3eddf930d653cc