tonglin0325的个人主页

Filebeat采集文本文件内容发送到kafka

  1. 安装filebeat,选择tar包安装方式,下载并解压
1
2
3
wget https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-7.7.0-linux-x86_64.tar.gz
tar -zxvf filebeat-7.7.0-linux-x86_64.tar.gz

  1. 修改配置文件
1
2
3
mv filebeat.yml filebeat.yml.bak
touch filebeat.yml

配置文件内容
 

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
logging.level: info
logging.to_files: true
logging.files:
path: /data/log_path/filebeat
name: filebeat
keepfiles: 7
permissions: 0644


filebeat.inputs:
- type: log
enabled: true
paths:
- /your_file_name
scan_frequency: 1s
backoff: 1s
max_backoff: 1s
ignore_older: 24h
close_inactive: 30m
close_timeout: 24h
clean_inactive: 720h
encoding: utf-8

output.kafka:
version: "2.0.0"
enabled: true
hosts: your_kafka_ip:tour_kafka_port
topic: your_kafka_topic # topic name
username: xxxxxx # kafka username
password: xxxxxx # kafka password
required_acks: 1
compression: gzip
max_message_bytes: 1000000
codec.format:
string: '%{[message]}'

  1. 启动脚本