Filebeat es 同步服务器日志到es

资源

ubuntu es 7.10 kibana7.10 filebeat:7.10.2 metricbeat:7.10.2对应的版本必须相同否在会有兼容问题

es kibana

内网地址
192.168.0.94:9200
127.0.0.1:9200
https://127.0.0.1:9200
账户 admin 
密码 123456
#端口
9200 es
kibana
https://127.0.0.1:5601/app/login?nextUrl=%2F
账户 admin 
密码 123456

日志es kibana服务器安装docker-compose

开放端口

5601,9200

设置系统参数（在宿主机执行）

# 1. 设置内核映射限制参数
sudo sysctl -w vm.max_map_count=262144
# 2. 永久写入配置
echo "vm.max_map_count=262144" | sudo tee -a /etc/sysctl.conf
# 3. 使配置生效
sudo sysctl -p

目录准备

# 创建基础目录
sudo mkdir -p /www/es-kibana/{metricbeat/modules.d,metricbeat/config,elasticsearch/config,elasticsearch/data,elasticsearch/logs,kibana/config,kibana/logs}
# 拷贝或新建配置文件
# （如果之前已经编辑过，直接 mv 到相应目录即可）
# Elasticsearch 配置
sudo tee /www/es-kibana/elasticsearch/config/elasticsearch.yml > /dev/null  /dev/null  /dev/null  /dev/null 在调试docker启动是否正常同步->启动镜像->启动正式容器

生产prd v99_mian配置filebeat

目录

mkdir -p /www/filebeat/
mkdir -p /www/filebeat/modules.d
/www/filebeat/
├── docker-compose.yml
├── Dockerfile
└── filebeat.docker.yml

vim filebeat.docker.yml

filebeat.config:
  modules:
    path: ${path.config}/modules.d/*.yml
    reload.enabled: false
filebeat.inputs:
  - type: log
    enabled: true
    paths:
      - /var/log/v99mian/**/*.log
      - /var/log/nginx/**/*.log
    json.keys_under_root: true
    json.add_error_key: true
    json.overwrite_keys: true
    fields:
      log_source: mian
processors:
  - decode_json_fields:
      fields: ["message"]
      target: ""
      overwrite_keys: true
  - timestamp:
      field: "@timestamp"
      layouts:
        - '2006-01-02T15:04:05.000Z07:00'
      timezone: "UTC"
  - add_host_metadata: {}
  - add_cloud_metadata: {}
  - add_docker_metadata: {}
  - add_kubernetes_metadata: {}
output.elasticsearch:
  hosts: ["127.0.0.1:9200"]
  username: "elastic"
  password: "123456"
  ssl.verification_mode: "none"
setup.template.name: "metricbeat-mian-prd"
setup.template.pattern: "metricbeat-*"
setup.template.priority: 260
setup.ilm.enabled: true
setup.ilm.rollover_alias: "metricbeat-mian-prd"
setup.ilm.pattern: "{now/d}-000001"
setup.ilm.policy_name: "metricbeat-mian-prd-policy"
setup.ilm.policy:
  policy:
    phases:
      hot:
        actions:
          rollover:
            max_age: "1d"
            max_size: "50gb"
      delete:
        min_age: "30d"
        actions:
          delete: {}
setup.template.settings:
  index.mapping.total_fields.limit: 2000
  index.mapping.ignore_malformed: true
  index.number_of_shards: 1
  index.number_of_replicas: 0

vim Dockerfile

FROM docker.elastic.co/beats/filebeat:7.10.2
# 切换到 root（确保有权限修改配置文件属主）
USER root
# 复制配置文件到镜像中
COPY filebeat.docker.yml /usr/share/filebeat/filebeat.yml
# 如果 modules.d 目录下有自定义模块，也一并复制
COPY modules.d /usr/share/filebeat/modules.d
# 确保 filebeat 用户可以读取配置
RUN chown -R root:filebeat /usr/share/filebeat/filebeat.yml \
 && chmod 0644 /usr/share/filebeat/filebeat.yml
# 切回非 root 用户
USER filebeat
# 挂载日志目录
VOLUME ["/var/log/mian"]
VOLUME ["/var/log/nginx"]
# 启动命令
CMD ["filebeat", "-e", "--strict.perms=false", "-c", "/usr/share/filebeat/filebeat.yml"]

vim docker-compose.yml

version: '3.8'
services:
  filebeat:
    build:
      context: .
      dockerfile: Dockerfile
    container_name: filebeat-mian
    restart: always
    user: root
    volumes:
      - /var/log/v99mian:/var/log/v99mian:ro
      - /var/log/nginx:/var/log/nginx:ro
      - /var/run/docker.sock:/var/run/docker.sock:ro

启动构建Docker镜像

cd /www/filebeat
docker-compose down -v
docker-compose up -d
docker-compose up --build -d #调试启动
docker ps         # 查看容器运行状态
docker logs -f filebeat-mian   # 实时查看输出日志

验证es

curl -u elastic:123456 \
  'http://127.0.0.1:9200/metricbeat-v99mian-prd-*/_search?size=5&pretty'
curl -u elastic:123456 'http://127.0.0.1:9200/_cluster/health?pretty'
curl -u elastic:123456 'http://127.0.0.1:9200/_cat/indices?v'