Elastic Stack 生产环境部署

I. 前提条件

安装 Elastic Stack 时，必须在整个堆栈中使用相同的版本。例如，如果使用 Elasticsearch 8.12.0，则需要安装 Beats 8.12.0、APM Server 8.12.0、Elasticsearch Hadoop 8.12.0、Kibana 8.12.0 和 Logstash 8.12.0。

如果要升级现有安装，请参阅升级 Elastic Stack ，了解如何确保与 8.12.0 兼容。

1.1 网络要求

要在内部安装 Elastic Stack，需要为每个组件打开以下端口。

默认端口	组件
3002	Enterprise Search
5044	Elastic Agent → Logstash Beats → Logstash
5601	Kibana Elastic Agent → Fleet Fleet Server → Fleet
8220	Elastic Agent → Fleet Server APM Server
9200-9300	Elasticsearch REST API
9300-9400	Elasticsearch node transport and communication
9600-9700	Logstash REST API

1.2 安装顺序

按以下顺序安装要使用的 Elastic Stack 产品：

Elasticsearch (install instructions )
Kibana (install )
Logstash (install )
Elastic Agent (install instructions ) or Beats (install instructions )
APM (install instructions )
Elasticsearch Hadoop (install instructions )

按此顺序安装可确保每个产品所依赖的组件都已就位。

1.3 节点规划

参数	VM1	VM2	VM3	VM4
Hostname	app1	app2	app3	app4
Services	elasticsearch, kibana	elasticsearch, logstash	elasticsearch	elasticsearch
IP Address	192.168.31.61	192.168.31.62	192.168.31.63	192.168.31.64
Node Roles	master,	master	master, data, voting_only	master, data
OS	CentOS 7.9	CentOS 7.9	CentOS 7.9	CentOS 7.9

1.4 服务器配置

RHEL 8 初始化

II. Elasticsearch 安装

2.1 登录 sudo 用户

1su user

2.2 Hosts 修改

修改各节点服务器 hosts

1sudo vim /etc/hosts

添加内容如下：

1192.168.31.61  app1
2192.168.31.62  app2
3192.168.31.63  app3
4192.168.31.64  app4

2.3 创建工作目录

1mkdir elastic-install-files
2cd elastic-install-files

2.4 导入 Elasticsearch GPG 密钥

1sudo rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch

2.5 下载并手动安装 RPM

最新的 Elasticsearch 稳定版可在 Download Elasticsearch 页面找到。其他版本可在 Past Releases page 页面找到。

安装 perl-Digest-SHA

1sudo yum install perl-Digest-SHA

安装 Elasticsearch

1wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-8.12.0-x86_64.rpm
2wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-8.12.0-x86_64.rpm.sha512
3shasum -a 512 -c elasticsearch-8.12.0-x86_64.rpm.sha512 
4sudo rpm --install elasticsearch-8.12.0-x86_64.rpm

导出安装后的输出信息。
将 elasticsearch 服务配置为使用 systemd 自动启动。

1sudo systemctl daemon-reload
2sudo systemctl enable elasticsearch

⚠️ 重要：先不要启动 Elasticsearch 服务！在重新启动之前，还有几个配置步骤要做。

2.6 创建数据目录

1sudo mkdir -p /mnt/sdc/elastic/elasticsearch/{data,log,tmp}

1cd /mnt/sdc/elastic/
2sudo chown -R elasticsearch:elasticsearch ./elasticsearch/
3sudo chmod -R 2750 ./elasticsearch/

2.7 在节点之间配置 TLS

2.7.1 创建证书存放目录

1sudo mkdir /etc/elasticsearch/certs
2sudo chmod 750 /etc/elasticsearch/certs

2.7.2 生成证书授权

启动 Elasticsearch 之前，请在任何单节点上使用 [elasticsearch-certutil](https://www.elastic.co/guide/en/elasticsearch/reference/current/certutil.html) 工具为群集生成 CA。

创建一个CA证书，并指定有效期3650天，默认1095天(三年)，密码为 capasswd （使用默认路径直接回车，生产环境请修改密码，密码不要有符号）。

1sudo /usr/share/elasticsearch/bin/elasticsearch-certutil ca --days 3650 --pass "capasswd" --silent

在任何单个节点上，为群集中的节点生成证书和私钥。其中包括上一步生成的 elastic-stack-ca.p12 输出文件（使用默认路径直接回车）。

1sudo /usr/share/elasticsearch/bin/elasticsearch-certutil cert --ca elastic-stack-ca.p12 --ca-pass "capasswd" --pass "transpasswd" --silent

在群集的每个节点上，将 elastic-certificates.p12 文件复制到 $ES_PATH_CONF 目录（如果使用软件包发行版（Debian 或 RPM），变量默认为 /etc/elasticsearch）。

1sudo scp /usr/share/elasticsearch/elastic-certificates.p12 root@appX:/etc/elasticsearch/certs/

2.7.3 将密码存储到密钥存储中

如果在创建节点证书时输入了密码，在集群中的每个节点上，运行以下命令将传输密码（transpasswd）存储到 Elasticsearch 密钥存储中：

1sudo /usr/share/elasticsearch/bin/elasticsearch-keystore add xpack.security.transport.ssl.keystore.secure_password

1sudo /usr/share/elasticsearch/bin/elasticsearch-keystore add xpack.security.transport.ssl.truststore.secure_password

为集群中的每个节点完成前面的步骤。
在集群中的每个节点上，启动 Elasticsearch。启动和停止 Elasticsearch 的方法因安装方式而异。

📢 注意：
您必须重新启动整个群集。配置为使用 TLS 传输的节点无法与使用未加密传输连接的节点通信（反之亦然）。

2.8 在 HTTP 层启用 TLS

2.8.1 生成 HTTP 证书

前提条件：完成前面 2.7 中的步骤。
在集群中的每个节点上，停止 Elasticsearch 和 Kibana（如果它们正在运行）。
运行 Elasticsearch HTTP 证书工具以生成证书签名请求 (CSR)。

1sudo /usr/share/elasticsearch/bin/elasticsearch-certutil http --silent

💡说明：
此命令生成一个 .zip 文件，其中包含用于 Elasticsearch 和 Kibana 的证书和密钥。每个文件夹都包含一个解释如何使用这些文件的 README.txt。HTTPS 私钥密码： <font style="color:#E8323C;">httppasswd</font> （生产环境按需修改，密码不要有符号）

  1[root@es1 security]# sudo /usr/share/elasticsearch/bin/elasticsearch-certutil http --silent
  2## Elasticsearch HTTP Certificate Utility
  3## Do you wish to generate a Certificate Signing Request (CSR)?
  4## 是否生成 CSR -> n(否)
  5Generate a CSR? [y/N]n
  6## Do you have an existing Certificate Authority (CA) key-pair that you wish to use to sign your certificate?
  7## 是否使用已存在的 CA -> y(是)
  8Use an existing CA? [y/N]y
  9## What is the path to your CA?
 10## 粘贴 CA 绝对路径
 11CA Path: /usr/share/elasticsearch/elastic-stack-ca.p12
 12## 输入 CA 密码
 13Password for elastic-stack-ca.p12:
 14## How long should your certificates be valid?
 15## 输入证书有效时间（这里写 10 年）
 16For how long should your certificate be valid? [5y] 10y
 17## Do you wish to generate one certificate per node?
 18## 是否为每个节点生成证书 -> y(是)
 19Generate a certificate per node? [y/N]y
 20## What is the name of node #1?
 21## 配置节点1信息
 22node #1 name: app1
 23## 哪些主机名将用于连接到 app1
 24## Which hostnames will be used to connect to app1?
 25app1
 26## 回车继续
 27You entered the following hostnames.
 28
 29 - app1
 30## 是否正确？
 31Is this correct [Y/n]y
 32## Which IP addresses will be used to connect to app1?
 33192.168.31.61
 34## 回车继续
 35You entered the following IP addresses.
 36
 37 - 192.168.31.61
 38Is this correct [Y/n]y
 39## Other certificate options
 40Key Name: app1
 41Subject DN: CN=app1
 42Key Size: 2048
 43
 44## 您想更改这些选项中的任何一个吗? -> n(否)
 45Do you wish to change any of these options? [y/N]n
 46## 生成其他证书? -> y(是)
 47Generate additional certificates? [Y/n]y
 48
 49## 配置节点2信息
 50## What is the name of node #2?
 51node #2 name: app2
 52## Which hostnames will be used to connect to app2?
 53app2
 54
 55You entered the following hostnames.
 56
 57 - app2
 58Is this correct [Y/n]y
 59## Which IP addresses will be used to connect to app2?
 60192.168.31.62
 61
 62You entered the following IP addresses.
 63
 64 - 192.168.31.62
 65Is this correct [Y/n]y
 66## Other certificate options
 67Key Name: app2
 68Subject DN: CN=app2
 69Key Size: 2048
 70
 71Do you wish to change any of these options? [y/N]n
 72Generate additional certificates? [Y/n]y
 73
 74## 配置节点3信息
 75## What is the name of node #3?
 76node #3 name: app3
 77## Which hostnames will be used to connect to app3?
 78app3
 79
 80You entered the following hostnames.
 81
 82 - app3
 83Is this correct [Y/n]y
 84## Which IP addresses will be used to connect to app3?
 85192.168.31.63
 86
 87You entered the following IP addresses.
 88
 89 - 192.168.31.63
 90Is this correct [Y/n]y
 91## Other certificate options
 92Key Name: app3
 93Subject DN: CN=app3
 94Key Size: 2048
 95
 96## 配置节点4信息
 97## What is the name of node #4?
 98node #4 name: app4
 99## Which hostnames will be used to connect to app4?
100app4
101
102You entered the following hostnames.
103
104 - app4
105Is this correct [Y/n]y
106## Which IP addresses will be used to connect to app4?
107192.168.31.64
108
109You entered the following IP addresses.
110
111 - 192.168.31.64
112Is this correct [Y/n]y
113## Other certificate options
114Key Name: app4
115Subject DN: CN=app4
116Key Size: 2048
117
118Do you wish to change any of these options? [y/N]n
119Generate additional certificates? [Y/n]y
120
121Do you wish to change any of these options? [y/N]n
122## 生成额外的证书？ -> n(否)
123Generate additional certificates? [Y/n]n
124## What password do you want for your private key(s)?
125## 输入 HTTPS 证书密码，此处使用 httppasswd 
126Provide a password for the "http.p12" file:  [<ENTER> for none]
127## 重复密码以确认
128Repeat password to confirm:
129## Where should we save the generated files?
130## 粘贴生成 HTTPS 证书的绝对路径 (使用默认路径直接回车)
131What filename should be used for the output zip file? [/usr/share/elasticsearch/elasticsearch-ssl-http.zip]

解压生成的 elasticsearch-ssl-http.zip 文件。这个压缩文件包含一个用于 Elasticsearch 和 Kibana 的目录。

1sudo unzip /usr/share/elasticsearch/elasticsearch-ssl-http.zip

1sudo tree ~/elasticsearch

1sudo tree ~/kibana

查看证书详情（验证配置密码 httppasswd）

1sudo openssl pkcs12 -in ~/elasticsearch/app1/http.p12 -info

2.8.2 拷贝证书至各节点

1sudo scp ~/elasticsearch/appX/http.p12 root@appX:/etc/elasticsearch/certs/

2.8.3 修改各节点证书权限

1sudo chmod -R 660 /etc/elasticsearch/certs/*

2.8.4 将密码存储到密钥存储中

将您的 HTTPS 私钥密码（httppasswd）添加到 Elasticsearch 的安全设置中。

1sudo /usr/share/elasticsearch/bin/elasticsearch-keystore add xpack.security.http.ssl.keystore.secure_password

为集群中的每个节点完成前面的步骤。
在集群中的每个节点上，启动 Elasticsearch。启动和停止 Elasticsearch 的方法因安装方式而异。

2.9 Elasticsearch 配置

2.9.1 备份默认置文件

1cp -avc /etc/elasticsearch/elasticsearch.yml /etc/elasticsearch/elasticsearch.yml.bak

2.9.2 节点1配置参考

1sudo vim /etc/elasticsearch/elasticsearch.yml

内容如下：

 1cluster.name: elk-cluster
 2node.name: app1
 3
 4path.data: /mnt/sdc/elastic/elasticsearch/data
 5path.logs: /mnt/sdc/elastic/elasticsearch/log
 6
 7bootstrap.memory_lock: true
 8network.host: 192.168.31.61
 9discovery.seed_hosts: ["192.168.31.61", "192.168.31.62", "192.168.31.63", "192.168.31.64"]
10cluster.initial_master_nodes: ["app1", "app2", "app3"]
11
12xpack.security.enabled: true
13xpack.security.enrollment.enabled: true
14
15xpack.security.http.ssl:
16  enabled: true
17  keystore.path: certs/http.p12
18
19xpack.security.transport.ssl:
20  enabled: true
21  verification_mode: certificate
22  client_authentication: required
23  keystore.path: certs/elastic-certificates.p12
24  truststore.path: certs/elastic-certificates.p12
25
26http.host: 0.0.0.0
27transport.host: 192.168.31.61
28# Manual additional configuration
29node.roles: [ master ]

查看配置

1sudo grep -vxE '[[:blank:]]*([#;].*)?' /etc/elasticsearch/elasticsearch.yml

或

1sudo awk '$1 ~ /^[^;#]/' /etc/elasticsearch/elasticsearch.yml

2.9.3 jvm.options 配置

1sudo vim /etc/elasticsearch/jvm.options.d/jvm-heap.options

内容如下：

 1## JVM heap size
 2-Xms2g
 3-Xmx2g
 4
 5## Expert settings
 6# specify an alternative path for heap dumps; ensure the directory exists and
 7# has sufficient space
 8-XX:HeapDumpPath=/mnt/sdc/elastic/elasticsearch/data
 9 
10# specify an alternative path for JVM fatal error logs
11-XX:ErrorFile=/mnt/sdc/elastic/elasticsearch/log/hs_err_pid%p.log
12 
13## GC logging
14-Xlog:gc*,gc+age=trace,safepoint:file=/mnt/sdc/elastic/elasticsearch/log/gc.log:utctime,level,pid,tags:filecount=32,filesize=64m

⚠️注意：
Xms 和 Xmx 设置为不超过总内存的 50%

2.10 重要系统配置

2.10.1 Swapping & Temporary directory

1sudo systemctl edit elasticsearch

添加如下内容：

1[Service]
2LimitMEMLOCK=infinity
3Environment=ES_TMPDIR=/mnt/sdc/elastic/elasticsearch/tmp

2.10.2 TCP retransmission timeout

在 /etc/sysctl.d 中创建一个新文件

1sudo vim /etc/sysctl.d/99-custom.conf

按以下格式，每行包含一个要设置的变量

1net.ipv4.tcp_retries2=5

不重启系统应用更改

1sudo sysctl -p /etc/sysctl.d/99-custom.conf

验证更改

1sudo sysctl net.ipv4.tcp_retries2

2.11 Elasticsearch 启动

2.11.1 启动 Elasticsearch

启动 Elasticsearch 服务

1sudo systemctl start elasticsearch

分离根证书用于测试

1sudo openssl pkcs12 -in /usr/share/elasticsearch/elastic-stack-ca.p12 -out /usr/share/elasticsearch/elastic-stack-ca.crt.pem -clcerts -nokeys

确保 Elasticsearch 正常运行

1sudo curl --cacert /usr/share/elasticsearch/elastic-stack-ca.crt.pem -u elastic:$ELASTIC_PASSWORD https://192.168.31.61:9200

查看 Elasticsearch 服务状态

1sudo systemctl status elasticsearch

重置 elastic 密码（最少6位）

1sudo /usr/share/elasticsearch/bin/elasticsearch-reset-password -u elastic -i

2.11.2 无法启动 Elasticsearch 解决

持续跟踪 elasticsearch 输出日志

1sudo journalctl -f -u elasticsearch

列出 Elasticsearch 日志条目

1sudo journalctl - elasticsearch

查看集群持续输出日志：

1tail -f /mnt/sdc/elastic/elasticsearch/log/elk-cluster.log

查看集群最后 N 行输出日志

1tail -n 30 /mnt/sdc/elastic/elasticsearch/log/elk-cluster.log

重置节点（使用 [elasticsearch-node](https://www.elastic.co/guide/en/elasticsearch/reference/current/node-tool.html) 命令，可以在节点上执行某些不安全的操作，这些操作只有在节点关闭时才能进行）

1sudo /usr/share/elasticsearch/bin/elasticsearch-node repurpose

2.12 设置第二个节点

2.12.1 节点2配置参考

 1cluster.name: elk-cluster
 2node.name: app2
 3
 4path.data: /mnt/sdc/elastic/elasticsearch/data
 5path.logs: /mnt/sdc/elastic/elasticsearch/log
 6
 7bootstrap.memory_lock: true
 8network.host: 192.168.31.62
 9discovery.seed_hosts: ["192.168.31.61", "192.168.31.62", "192.168.31.63", "192.168.31.64"]
10cluster.initial_master_nodes: ["app1", "app2", "app3"]
11
12xpack.security.enabled: true
13xpack.security.enrollment.enabled: true
14
15xpack.security.http.ssl:
16  enabled: true
17  keystore.path: certs/http.p12
18
19xpack.security.transport.ssl:
20  enabled: true
21  verification_mode: certificate
22  client_authentication: required
23  keystore.path: certs/elastic-certificates.p12
24  truststore.path: certs/elastic-certificates.p12
25
26http.host: 0.0.0.0
27transport.host: 192.168.31.62
28# Manual additional configuration
29node.roles: [ master ]

2.12.2 启动节点2

在第二个节点上启动 Elasticsearch：

1sudo systemctl start elasticsearch

查看节点2 连接节点1 进度

1sudo tail -f /mnt/sdc/elasticsearch/log/elk.log

查看节点2 运行状态

1sudo curl --cacert /etc/elasticsearch/certs/elastic-stack-ca.crt.pem -u elastic:$ELASTIC_PASSWORD https://app2:9200

2.12.3 查看节点状态

查看当前所有节点状态

1sudo curl --cacert /etc/elasticsearch/certs/elastic-stack-ca.crt.pem -u elastic:$ELASTIC_PASSWORD https://app1:9200/_cat/nodes?v

查看集群健康状态

1sudo curl --cacert /etc/elasticsearch/certs/elastic-stack-ca.crt.pem -u elastic:$ELASTIC_PASSWORD https://app1:9200/_cluster/health?pretty

也可浏览器直接访问👇

1https://192.168.31.61:9200/_cluster/health

2.13 设置其它节点

2.13.1 节点3配置参考

 1cluster.name: elk-cluster
 2node.name: app3
 3
 4path.data: /mnt/sdc/elastic/elasticsearch/data
 5path.logs: /mnt/sdc/elastic/elasticsearch/log
 6
 7bootstrap.memory_lock: true
 8network.host: 192.168.31.63
 9discovery.seed_hosts: ["192.168.31.61", "192.168.31.62", "192.168.31.63", "192.168.31.64"]
10cluster.initial_master_nodes: ["app1", "app2", "app3"]
11
12xpack.security.enabled: true
13xpack.security.enrollment.enabled: true
14
15xpack.security.http.ssl:
16  enabled: true
17  keystore.path: certs/http.p12
18
19xpack.security.transport.ssl:
20  enabled: true
21  verification_mode: certificate
22  client_authentication: required
23  keystore.path: certs/elastic-certificates.p12
24  truststore.path: certs/elastic-certificates.p12
25
26http.host: 0.0.0.0
27transport.host: 192.168.31.63
28# Manual additional configuration
29node.roles: [ master, data, voting_only ]

2.13.2 节点4配置参考

 1cluster.name: elk-cluster
 2node.name: app4
 3
 4path.data: /mnt/sdc/elastic/elasticsearch/data
 5path.logs: /mnt/sdc/elastic/elasticsearch/log
 6
 7bootstrap.memory_lock: true
 8network.host: 192.168.31.64
 9discovery.seed_hosts: ["192.168.31.61", "192.168.31.62", "192.168.31.63", "192.168.31.64"]
10
11xpack.security.enabled: true
12xpack.security.enrollment.enabled: true
13
14xpack.security.http.ssl:
15  enabled: true
16  keystore.path: certs/http.p12
17
18xpack.security.transport.ssl:
19  enabled: true
20  verification_mode: certificate
21  client_authentication: required
22  keystore.path: certs/elastic-certificates.p12
23  truststore.path: certs/elastic-certificates.p12
24
25http.host: 0.0.0.0
26transport.host: 192.168.31.64
27# Manual additional configuration
28node.roles: [ data ]

2.14 后续配置

停止 Elasticsearch

1sudo systemctl stop elasticsearch

注释掉所有master 节点以下配置

1#cluster.initial_master_nodes: ["app1", "app2", "app3"]

重新启动 Elasticsearch

1sudo systemctl start elasticsearch

查看 Elasticsearch 服务状态

1sudo systemctl status elasticsearch

III. Kibana 安装

3.1 登录 sudo 用户

1su user

3.2 创建工作目录

1mkdir kibana-install-files
2cd kibana-install-files

3.3 下载并手动安装 Kibana

Kibana 的最新稳定版本可在 Download Kibana 页面找到。其他版本可在 Past Releases page 页面找到。

1wget https://artifacts.elastic.co/downloads/kibana/kibana-8.12.0-x86_64.rpm
2wget https://artifacts.elastic.co/downloads/kibana/kibana-8.12.0-x86_64.rpm.sha512
3shasum -a 512 -c kibana-8.12.0-x86_64.rpm.sha512 
4sudo rpm --install kibana-8.12.0-x86_64.rpm

3.4 创建数据目录

1sudo mkdir -p /mnt/sdc/elastic/kibana/{data,log}

1cd /mnt/sdc/elastic/
2sudo chown -R kibana:kibana ./kibana/
3sudo chmod -R 2750 ./kibana/

3.5 创建证书存放目录

1sudo mkdir /etc/kibana/certs
2sudo chmod 750 /etc/kibana/certs

3.6 Kibana 安全配置

参考文档：Configure security in Kibana

在 kibana.yml 配置文件中设置 xpack.security.encryptionKey 属性。

1xpack.security.encryptionKey: "something_at_least_32_characters"

可选：配置 Kibana 的会话过期设置（参考文档：Session management ）。

1# 会话空闲超时
2xpack.security.session.idleTimeout: "30m"
3# 会话寿命
4xpack.security.session.lifespan: "1h"
5# 会话清理间隔
6xpack.security.session.cleanupInterval: "30m"

3.7 在 Kibana 和 Elasticsearch 之间设置 TLS 加密

3.7.1 加密 Kibana 和 Elasticsearch 之间的流量

1elasticsearch.ssl.certificateAuthorities: $KBN_PATH_CONF/elasticsearch-ca.pem

3.7.2 加密浏览器和 Kibana 之间的流量

参考文档1：Mutual TLS with Elasticsearch

参考文档2：elasticsearch-certutil

为 Kibana 生成服务器证书和私钥（参考文档：Elastic Discuss ）。

1sudo /usr/share/elasticsearch/bin/elasticsearch-certutil cert --pem \
2--ca /usr/share/elasticsearch/elastic-stack-ca.p12 --ca-pass "capasswd" \
3--name kibana-server \
4--dns app1 \
5--silent

📢 参数说明：
–name 指定生成证书的名称。
–dns 指定以逗号分隔的 DNS 名称列表（域名或主机名）。
–ip 指定以逗号分隔的 IP 地址列表（域名或主机名解析后的 IP）。

解压缩 certificate-bundle.zip 文件，复制 kibana-server.crt 和 kibana-server.key 到证书目录

1sudo cp ~/kibana/elasticsearch-ca.pem /etc/kibana/certs/
2sudo cp ~/kibana-server/kibana-server.crt /etc/kibana/certs/
3sudo cp ~/kibana-server/kibana-server.key /etc/kibana/certs/
4sudo chmod -R 660 /etc/kibana/certs/*

打开 kibana.yml，添加以下几行，配置 Kibana 访问服务器证书和加密私钥。

1server.ssl.certificate: $KBN_PATH_CONF/kibana-server.crt
2server.ssl.key: $KBN_PATH_CONF/kibana-server.key

在 kibana.yml 中添加以下一行，为入站连接启用 TLS。

1server.ssl.enabled: true

3.8 服务账户令牌

3.8.1 创建服务账户令牌

以下命令通过 REST API 为 elastic/kibana 服务帐户（Service accounts ）创建一个名为 kibana_token 的服务帐户令牌

1sudo curl -X POST --cacert /etc/elasticsearch/certs/elastic-stack-ca.crt.pem -u elastic:$ELASTIC_PASSWORD https://app1:9200/_security/service/elastic/kibana/credential/token/kibana_token

ℹ️ 输出是不记名令牌，它是 Base64 编码的字符串：
{"created":true,"token":{"name":"kibana_token","value":"AAEAAWV.....RDJycHlR"}}

3.8.2 将 token 值添加到 kibana-keystore

1sudo /usr/share/kibana/bin/kibana-keystore add elasticsearch.serviceAccountToken

查看服务账户

1sudo /usr/share/kibana/bin/kibana-keystore list

3.8.3 REST API 令牌管理

获取服务帐户凭据 API（Get service account credentials ）

1sudo curl -X GET --cacert /etc/elasticsearch/certs/elastic-stack-ca.crt.pem -u elastic:$ELASTIC_PASSWORD https://app1:9200/_security/service/elastic/kibana/credential

删除服务帐户令牌（Delete service account token ）

1sudo curl -X DELETE --cacert /etc/elasticsearch/certs/elastic-stack-ca.crt.pem -u elastic:$ELASTIC_PASSWORD https://app1:9200/_security/service/elastic/kibana/credential/token/<token_name>

3.9 Kibana 配置参考

 1server.host: "192.168.31.61"
 2server.publicBaseUrl: "https://192.168.31.61:5601"
 3
 4server.ssl.enabled: true
 5server.ssl.certificate: /etc/kibana/certs/kibana-server.crt
 6server.ssl.key: /etc/kibana/certs/kibana-server.key
 7
 8elasticsearch.hosts: ["https://192.168.31.61:9200", "https://192.168.31.62:9200", "https://192.168.31.63:9200", "https://192.168.31.64:9200"]
 9
10elasticsearch.serviceAccountToken: "AAEAAWV.....RDJycHlR"
11
12elasticsearch.ssl.certificateAuthorities: [ "/etc/kibana/certs/elasticsearch-ca.pem" ]
13elasticsearch.ssl.verificationMode: full
14
15# Default configuration
16logging:
17  appenders:
18    file:
19      type: file
20      fileName: /kibana/log/kibana.log
21      layout:
22        type: json
23  root:
24    appenders:
25      - default
26      - file
27
28path.data: /kibana/data
29pid.file: /run/kibana/kibana.pid
30
31i18n.locale: "zh-CN"
32monitoring.ui.ccs.enabled: false
33
34# Manual additional configuration
35xpack.security.encryptionKey: "uyN6Y4D3xfPrZMhAwRBTcnK9p7eG5EJC"
36xpack.security.session.idleTimeout: "30m"
37xpack.security.session.lifespan: "1h"
38xpack.security.session.cleanupInterval: "30m"
39# Optional configuration
40xpack.reporting.roles.enabled: false

IV. Logstash 安装

4.1 登录 sudo 用户

1su user

4.2 创建工作目录

1mkdir logstash-install-files
2cd logstash-install-files

4.3 下载并手动安装 Logstash

Logstash 的最新稳定版本可在 Download Logstash 页面找到。其他版本可在 Past Releases page 页面找到。

1wget https://artifacts.elastic.co/downloads/logstash/logstash-8.12.0-x86_64.rpm
2wget https://artifacts.elastic.co/downloads/logstash/logstash-8.12.0-x86_64.rpm.sha512
3shasum -a 512 -c logstash-8.12.0-x86_64.rpm.sha512 
4sudo rpm --install logstash-8.12.0-x86_64.rpm

4.4 创建数据目录

1sudo mkdir -p /mnt/sdc/elastic/logstash/{data,log}

1cd /mnt/sdc/elastic/
2sudo chown -R logstash:logstash ./logstash/
3sudo chmod -R 755 ./logstash/

4.5 创建证书存放目录

1sudo mkdir /etc/logstash/certs

4.6 Logstash 配置参考

1path.data: /mnt/sdc/elastic/logstash/data
2path.logs: /mnt/sdc/elastic/logstash/log
3
4path.config: /etc/logstash/conf.d/*.conf

4.7 保护与 Elasticsearch 连接安全

4.7.1 配置 Logstash 以使用 TLS/SSL 加密

参考文档：Configuring Logstash to use TLS/SSL encryption

复制在 2.8.1 步骤中生成的 CA 证书到 Logstash 证书目录

1sudo scp ~/kibana/elasticsearch-ca.pem root@app2:/etc/logstash/certs/

在 logstash.conf 文件中配置 ssl 和 cacert 选项

1output {
2  elasticsearch {
3    ...
4    ssl => true
5    cacert => '/etc/logstash/certs/elasticsearch-ca.pem' 
6  }
7}

4.7.2 使用 API Key 授予访问权限

您可以使用 API 密钥来授予对 Elasticsearch 资源的访问权限，而不是使用用户名和密码。

参考文档：Grant access using API keys

创建用于发布的 API Key

 1POST /_security/api_key
 2{
 3  "name": "logstash_host001", 
 4  "role_descriptors": {
 5    "logstash_writer": { 
 6      "cluster": ["manage_index_templates", "monitor", "manage_ilm"],
 7      "index": [
 8        {
 9          "names": ["*"],
10          "privileges": ["write","create","create_index","manage","manage_ilm"]
11        }
12      ]
13    }
14  }
15}

返回值类似如下：

1{
2  "id": "4ffGWocBFIhKxP1f5xaL",
3  "name": "logstash_host001",
4  "api_key": "TG-S2SvdTOqa5w6OvKIZEg",
5  "encoded": "NGZmR1dvY0JGSWhLeFAxZjV4YUw6VEctUzJTdmRUT3FhNXc2T3ZLSVpFZw=="
6}

Elasticsearch 插件配置示例

1output {
2  elasticsearch {
3    ...
4    api_key => "4ffGWocBFIhKxP1f5xaL:TG-S2SvdTOqa5w6OvKIZEg" 
5  }
6}

💡Tips:
api_key 格式： id:api_key

4.8 Logstash pipeline 配置

示例管道配置

1sudo vim /etc/logstash/conf.d/logstash-example.conf

参考配置内容：

 1input {
 2    file {
 3        path => "/tmp/logs/example1.log"
 4        start_position => "beginning"
 5        sincedb_path => "/dev/null"
 6        add_field => {
 7            "log_type" => "logstash-example1"
 8        }
 9        stat_interval => "2"
10    }
11}
12
13output {
14  if [log_type] == "logstash-example1" {
15    elasticsearch {
16        hosts => ["https://app1:9200", "https://app2:9200", "https://app3:9200", "https://app4:9200"]
17        index => "logstash-example1@-%{+YYYY.MM.dd}--000001"
18        manage_template => false
19        action => "create"
20        ssl => true
21        cacert => "/etc/logstash/certs/elasticsearch-ca.pem"
22        api_key => "4ffGWocBFIhKxP1f5xaL:TG-S2SvdTOqa5w6OvKIZEg"
23    }
24  }
25}

检查配置文件格式

1sudo /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/logstash-example.conf -t

V. Filebeat 安装

5.1 登录 sudo 用户

1su user

5.2 创建工作目录

1mkdir filebeat-install-files
2cd filebeat-install-files

5.3 下载并手动安装 Filebeat

Filebeat 的最新稳定版本可在 Download Filebeat 页面找到。其他版本可在 Past Releases page 页面找到。

1wget https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-8.12.0-x86_64.rpm
2wget https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-8.12.0-x86_64.rpm.sha512
3shasum -a 512 -c filebeat-8.12.0-x86_64.rpm.sha512 
4sudo rpm --install filebeat-8.12.0-x86_64.rpm

5.4 创建数据目录

1sudo mkdir -p /mnt/sdc/elastic/filebeat/{data,log}

1cd /mnt/sdc/elastic/
2sudo chmod -R 750 ./filebeat/

5.5 创建证书存放目录

1sudo mkdir /etc/filebeat/certs
2sudo chmod 750 /etc/filebeat/certs

5.6 与 Logstash 安全通信

参考文档1：Secure communication with Logstash

参考文档2：Filebeat与Logstash配置ssl加密通信

5.6.1 为 Logstash 生成证书

为 Logstash 生成证书和私钥

1sudo /usr/share/elasticsearch/bin/elasticsearch-certutil cert --pem \
2--ca elastic-stack-ca.p12 \
3--name logstash \
4--dns app2 \
5--out logstash-certificate-bundle.zip \
6--silent

此命令生成一个 logstash-certificate-bundle.zip 文件，解压该文件：

1sudo unzip /usr/share/elasticsearch/logstash-certificate-bundle.zip

内容如下：

1/logstash
2|_ logstash.crt
3|_ logstash.key

logstash 与 beats 启动 ssl 加密通信时，仅支持使用 PKCS8 的密钥格式，将logstash.key 转换为 pkcs#8 格式：

1sudo openssl pkcs8 -inform PEM -in ~/logstash/logstash.key -topk8 -nocrypt -outform PEM -out ~/logstash/logstash.pkcs8.key

将 logstash.crt 和 logstash.pkcs8.key 到 logstash 证书目录。

1sudo scp ~/kibana/elasticsearch-ca.pem root@appX:/etc/logstash/certs
2sudo scp ~/logstash/logstash.crt root@appX:/etc/logstash/certs
3sudo scp ~/logstash/logstash.pkcs8.key root@appX:/etc/logstash/certs
4sudo chmod -R 660 /etc/logstash/certs/*

5.6.2 为 Filebeat 生成证书

创建实例文件

 1instances:
 2  - name: "app1" 
 3    dns: 
 4      - "app1"
 5  - name: "app2" 
 6    dns: 
 7      - "app2"
 8  - name: "app3" 
 9    dns: 
10      - "app3"
11  - name: "app4" 
12    dns: 
13      - "app4"

为 Filebeat 生成证书和私钥

1sudo /usr/share/elasticsearch/bin/elasticsearch-certutil cert --pem \
2--ca elastic-stack-ca.p12 \
3--in ~/instances-filebeat.yml \
4--out filebeat-certificate-bundle.zip \
5--silent

此命令生成一个 filebeat-certificate-bundle.zip 文件，解压该文件：

1sudo unzip /usr/share/elasticsearch/filebeat-certificate-bundle.zip -d ~/filebeat-certificate/

内容如下：

 1filebeat-certificate/
 2├── app1
 3│   ├── app1.crt
 4│   └── app1.key
 5├── app2
 6│   ├── app2.crt
 7│   └── app2.key
 8├── app3
 9│   ├── app3.crt
10│   └── app3.key
11└── app4
12    ├── app4.crt
13    └── app4.key

复制证书文件到各 Filebeat 节点目录。

1sudo scp ~/kibana/elasticsearch-ca.pem root@appX:/etc/filebeat/certs
2sudo scp ~/filebeat-certificate/appX/appX.crt root@appX:/etc/filebeat/certs
3sudo scp ~/filebeat-certificate/appX/appX.key root@appX:/etc/filebeat/certs

5.7 使用 SSL 相互认证:

5.7.1 配置 Filebeat 以使用 SSL

1output.logstash:
2  hosts: ["app2:5044"]
3  ssl.certificate_authorities: ["/etc/filebeat/certs/elasticsearch-ca.pem"]
4  ssl.certificate: "/etcfilebeat/certs/appX.crt"
5  ssl.key: "/etcfilebeat/certs/appX.key"

5.7.2 配置 Logstash pipeline 以使用 SSL

 1input {
 2  beats {
 3    port => 5044
 4    ssl => true
 5    ssl_certificate_authorities => ["/etc/logstash/certs/elasticsearch-ca.pem"]
 6    ssl_certificate => "/etc/logstash/certs/logstash.crt"
 7    ssl_key => "/etc/logstash/certs/logstash.pkcs8.key"
 8    ssl_verify_mode => "force_peer"
 9  }
10}

5.8 验证测试

参考文档：Validate the Logstash server’s certificate

5.8.1 验证 Logstash 服务器的证书

1sudo curl --cacert /etc/filebeat/certs/elasticsearch-ca.pem --cert /etc/filebeat/certs/app1.crt --key /etc/filebeat/certs/app1.key -v https://app2:5044

如果测试成功，您将收到一个空响应错误：

 1* About to connect() to app2 port 5044 (#0)
 2*   Trying 192.168.31.62...
 3* Connected to app2 (192.168.31.62) port 5044 (#0)
 4* Initializing NSS with certpath: sql:/etc/pki/nssdb
 5*   CAfile: /etc/filebeat/certs/elasticsearch-ca.pem
 6  CApath: none
 7* NSS: client certificate from file
 8* 	subject: CN=app1
 9* 	start date: Feb 24 14:25:00 2024 GMT
10* 	expire date: Feb 23 14:25:00 2027 GMT
11* 	common name: app1
12* 	issuer: CN=Elastic Certificate Tool Autogenerated CA
13* SSL connection using TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
14* Server certificate:
15* 	subject: CN=logstash
16* 	start date: Feb 24 13:56:23 2024 GMT
17* 	expire date: Feb 23 13:56:23 2027 GMT
18* 	common name: logstash
19* 	issuer: CN=Elastic Certificate Tool Autogenerated CA
20> GET / HTTP/1.1
21> User-Agent: curl/7.29.0
22> Host: app2:5044
23> Accept: */*
24> 
25* Empty reply from server
26* Connection #0 to host app2 left intact
27curl: (52) Empty reply from server

5.8.2 测试 Filebeat 与 Logstash 的连接

Logstash pipeline 配置参考

 1input {
 2  beats {
 3  port => 5044
 4  ssl => true
 5  ssl_certificate_authorities => ["/etc/logstash/certs/elasticsearch-ca.pem"]
 6  ssl_certificate => "/etc/logstash/certs/logstash.crt"
 7  ssl_key => "/etc/logstash/certs/logstash.pkcs8.key"
 8  ssl_verify_mode => "force_peer"
 9}
10}
11
12output {
13  if [log_type] == "logstash-example1" {
14  elasticsearch {
15  hosts => ["https://app1:9200", "https://app2:9200", "https://app3:9200", "https://app4:9200"]
16  index => "logstash-example1@-%{+YYYY.MM.dd}-000001"
17  manage_template => false
18  action => "create"
19  ssl => true
20  cacert => "/etc/logstash/certs/elasticsearch-ca.pem"
21  api_key => "4ffGWocBFIhKxP1f5xaL:TG-S2SvdTOqa5w6OvKIZEg"
22}
23} else if [log_type] == "logstash-example2" {
24  elasticsearch {
25  hosts => ["https://app1:9200", "https://app2:9200", "https://app3:9200", "https://app4:9200"]
26  index => "logstash-example2@-%{+YYYY.MM.dd}-000001"
27  manage_template => false
28  action => "create"
29  ssl => true
30  cacert => "/etc/logstash/certs/elasticsearch-ca.pem"
31  api_key => "4ffGWocBFIhKxP1f5xaL:TG-S2SvdTOqa5w6OvKIZEg"
32}
33}
34}

filebeat.yml 配置参考（官方文档：filebeat.reference.yml ）

 1path.data: /mnt/sdc/elastic/filebeat/data
 2path.logs: /mnt/sdc/elastic/filebeat/log
 3
 4filebeat.inputs:
 5- type: filestream
 6  id: demo-logstash-example2
 7  enabled: true
 8  backoff.init: 1s
 9  backoff.max: 10s
10  prospector.scanner.check_interval: 10s
11  close.on_state_change.inactive: 5m
12  paths:
13    - /tmp/logs/example2.log
14  fields:
15    log_type: logstash-example2
16  fields_under_root: true
17  parsers:
18    - multiline:
19        type: pattern
20        pattern: '^\[?(?:\d\d){1,2}[-\/](?:0?[1-9]|1[0-2])[-\/](?:(?:0[1-9])|(?:[12][0-9])|(?:3[01])|[1-9])[T ](?:2[0123]|[01]?[0-9]):(?:[0-5][0-9]):(?:(?:[0-5]?[0-9]|60)(?:[:.,][0-9]+)?)\s*(?:[0-9]+.[0-9]+)?\]?'
21        negate: true
22        match: after
23
24output.logstash:
25  hosts: ["appX:5044"]
26  ssl.certificate_authorities: ["/etc/filebeat/certs/elasticsearch-ca.pem"]
27  ssl.certificate: "/etc/filebeat/certs/appY.crt"
28  ssl.key: "/etc/filebeat/certs/appY.key"

测试配置（参考文档：Filebeat command reference ）

1sudo /usr/share/filebeat/bin/filebeat -c /etc/filebeat/filebeat.yml test config

测试输出

1sudo /usr/share/filebeat/bin/filebeat -c /etc/filebeat/filebeat.yml test output

如果 Filebeat 作为服务运行，请先停止服务。然后在前台运行 Filebeat 来测试设置，这样就可以快速查看发生的任何错误：

1sudo /usr/share/filebeat/bin/filebeat -c /etc/filebeat/filebeat.yml -e -v

5.9 应用索引生命周期管理

5.9.1 创建 Logstash ILM

 1PUT _ilm/policy/logstash_ilm_policy
 2{
 3  "policy": {
 4    "phases": {
 5      "hot": {
 6        "min_age": "0ms",
 7        "actions": {
 8          "rollover": {
 9            "max_primary_shard_size": "50gb",
10            "max_age": "30d"
11          },
12          "set_priority": {
13            "priority": 100
14          }
15        }
16      },
17      "warm": {
18        "min_age": "60d",
19        "actions": {
20          "forcemerge": {
21            "max_num_segments": 1,
22            "index_codec": "best_compression"
23          },
24          "readonly": {},
25          "set_priority": {
26            "priority": 50
27          }
28        }
29      },
30      "cold": {
31        "min_age": "120d",
32        "actions": {
33          "set_priority": {
34            "priority": 0
35          }
36        }
37      },
38      "delete": {
39        "min_age": "180d",
40        "actions": {
41          "delete": {
42            "delete_searchable_snapshot": true
43          }
44        }
45      }
46    }
47  }
48}

5.9.2 应用 ILM

 1input {
 2  beats {
 3    port => 5044
 4    ssl => true
 5    ssl_certificate_authorities => ["/etc/logstash/certs/elasticsearch-ca.pem"]
 6    ssl_certificate => "/etc/logstash/certs/logstash.crt"
 7    ssl_key => "/etc/logstash/certs/logstash.pkcs8.key"
 8    ssl_verify_mode => "force_peer"
 9  }
10}
11
12output {
13  if [log_type] == "logstash-example1" {
14    elasticsearch {
15      hosts => ["https://app1:9200", "https://app2:9200", "https://app3:9200", "https://app4:9200"]
16      index => "logstash-example1@-%{+YYYY.MM.dd}-000001"
17      manage_template => false
18      action => "create"
19      ssl => true
20      cacert => "/etc/logstash/certs/elasticsearch-ca.pem"
21      api_key => "4ffGWocBFIhKxP1f5xaL:TG-S2SvdTOqa5w6OvKIZEg"
22    }
23  } else if [log_type] == "logstash-example2" {
24    elasticsearch {
25      hosts => ["https://app1:9200", "https://app2:9200", "https://app3:9200", "https://app4:9200"]
26      index => "logstash-example2@-%{+YYYY.MM.dd}-000001"
27      manage_template => false
28      action => "create"
29      ssl => true
30      cacert => "/etc/logstash/certs/elasticsearch-ca.pem"
31      api_key => "4ffGWocBFIhKxP1f5xaL:TG-S2SvdTOqa5w6OvKIZEg"
32    }
33  } else if [log_type] == "logstash-example3" {
34    elasticsearch {
35      hosts => ["https://app1:9200","https://app2:9200","https://app3:9200", "https://app4:9200"]
36      data_stream => "false"
37      ilm_rollover_alias => "logstash-example3@"
38      ilm_pattern => "{now/d}-000001"
39      ilm_policy => "logstash_ilm_policy"
40      ssl => true
41      cacert => "/etc/logstash/certs/elasticsearch-ca.pem"
42      api_key => "4ffGWocBFIhKxP1f5xaL:TG-S2SvdTOqa5w6OvKIZEg"
43    }
44  }
45}

VI. Metricbeat 安装

6.1 登录 sudo 用户

1su user

6.2 创建工作目录

1mkdir metricbeat-install-files
2cd metricbeat-install-files

6.3 下载并手动安装 Metricbeat

Metricbeat 的最新稳定版本可在 Download Metricbeat 页面找到。其他版本可在 Past Releases page 页面找到。

1wget https://artifacts.elastic.co/downloads/beats/metricbeat/metricbeat-8.12.0-x86_64.rpm
2wget https://artifacts.elastic.co/downloads/beats/metricbeat/metricbeat-8.12.0-x86_64.rpm.sha512
3shasum -a 512 -c metricbeat-8.12.0-x86_64.rpm.sha512
4sudo rpm --install metricbeat-8.12.0-x86_64.rpm

6.4 创建数据目录

1sudo mkdir -p /mnt/sdc/elastic/metricbeat/{data,log}

1cd /mnt/sdc/elastic/
2sudo chmod -R 750 ./metricbeat/

6.5 创建证书存放目录

1sudo mkdir /etc/metricbeat/certs
2sudo chmod 750 /etc/metricbeat/certs

6.6 保护与 Logstash 连接安全

6.6.1 配置 Logstash Pipeline 以使用 TLS/SSL 加密

参考文档：Configuring Logstash to use TLS/SSL encryption

复制在 2.8.1 步骤中生成的 CA 证书到 Logstash 证书目录

1sudo scp ~/kibana/elasticsearch-ca.pem root@app2:/etc/logstash/certs/

在 logstash-example.conf 文件中配置 ssl 和 cacert 选项

 1input {
 2  beats {
 3    port => 5044
 4    ssl => true
 5    ssl_certificate_authorities => ["/etc/logstash/certs/elasticsearch-ca.pem"]
 6    ssl_certificate => "/etc/logstash/certs/logstash.crt"
 7    ssl_key => "/etc/logstash/certs/logstash.pkcs8.key"
 8    ssl_verify_mode => "force_peer"
 9  }
10}
11
12output {
13  elasticsearch {
14    ...
15    ssl => true
16    cacert => '/etc/logstash/certs/elasticsearch-ca.pem' 
17    api_key => "4ffGWocBFIhKxP1f5xaL:TG-S2SvdTOqa5w6OvKIZEg" 
18  }
19}

6.6.2 配置 Metricbeat 以使用 SSL

1output.logstash:
2  hosts: ["app2:5044"]
3  ssl.certificate_authorities: ["/etc/filebeat/certs/elasticsearch-ca.pem"]
4  ssl.certificate: "/etcfilebeat/certs/appX.crt"
5  ssl.key: "/etcfilebeat/certs/appX.key"

6.7 创建 Metricbeat ILM

 1PUT _ilm/policy/metricbeat_ilm_policy
 2{
 3  "policy": {
 4    "phases": {
 5      "hot": {
 6        "min_age": "0ms",
 7        "actions": {
 8          "rollover": {
 9            "max_primary_shard_size": "50gb",
10            "max_age": "30d"
11          },
12          "set_priority": {
13            "priority": 100
14          }
15        }
16      },
17      "warm": {
18        "min_age": "60d",
19        "actions": {
20          "forcemerge": {
21            "max_num_segments": 1,
22            "index_codec": "best_compression"
23          },
24          "readonly": {},
25          "set_priority": {
26            "priority": 50
27          }
28        }
29      },
30      "cold": {
31        "min_age": "120d",
32        "actions": {
33          "set_priority": {
34            "priority": 0
35          }
36        }
37      },
38      "delete": {
39        "min_age": "180d",
40        "actions": {
41          "delete": {
42            "delete_searchable_snapshot": true
43          }
44        }
45      }
46    }
47  }
48}

6.8 验证测试

参考文档：Validate the Logstash server’s certificate

6.8.1 验证 Logstash 服务器的证书

1sudo curl --cacert /etc/filebeat/certs/elasticsearch-ca.pem --cert /etc/filebeat/certs/app1.crt --key /etc/filebeat/certs/app1.key -v https://app2:5044

6.8.2 测试 Metricbeat 与 Logstash 的连接

Logstash pipeline 配置参考

 1input {
 2  beats {
 3    port => 5044
 4    ssl => true
 5    ssl_certificate_authorities => ["/etc/logstash/certs/elasticsearch-ca.pem"]
 6    ssl_certificate => "/etc/logstash/certs/logstash.crt"
 7    ssl_key => "/etc/logstash/certs/logstash.pkcs8.key"
 8    ssl_verify_mode => "force_peer"
 9  }
10}
11
12output {
13  if [log_type] == "metricbeat" {
14    elasticsearch {
15      hosts => ["https://app1:9200", "https://app2:9200", "https://app3:9200", "https://app4:9200"]
16      data_stream => "false"
17      ilm_rollover_alias => "metricbeat"
18      ilm_pattern => "{now/d}-000001"
19      ilm_policy => "logstash_ilm_policy"
20      ssl => true
21      cacert => "/etc/logstash/certs/elasticsearch-ca.pem"
22      api_key => "4ffGWocBFIhKxP1f5xaL:TG-S2SvdTOqa5w6OvKIZEg"
23    }
24  }
25}

metricbeat.yml 配置参考（官方文档：metricbeat.reference.yml ）

 1path.data: /metricbeat/data
 2path.logs: /metricbeat/log
 3
 4metricbeat.config.modules:
 5  path: ${path.config}/modules.d/*.yml
 6  reload.enabled: false
 7
 8setup.template.settings:
 9  index.number_of_shards: 1
10  index.codec: best_compression
11
12fields:
13    log_type: metricbeat
14fields_under_root: true
15
16output.logstash:
17  hosts: ["app2:5044"]
18  ssl.certificate_authorities: ["/etc/filebeat/certs/elasticsearch-ca.pem"]
19  ssl.certificate: "/etc/filebeat/certs/app1.crt"
20  ssl.key: "/etc/filebeat/certs/app1.key"
21
22processors:
23  - add_host_metadata: ~
24  - add_cloud_metadata: ~
25  - add_docker_metadata: ~
26  - add_kubernetes_metadata: ~

测试配置（参考文档：Metricbeat command reference ）

1sudo /usr/share/metricbeat/bin/metricbeat -c /etc/metricbeat/metricbeat.yml test config

测试输出

1sudo /usr/share/metricbeat/bin/metricbeat -c /etc/metricbeat/metricbeat.yml test output

测试所有已配置的模块设置

1sudo /usr/share/metricbeat/bin/metricbeat test modules
2sudo /usr/share/metricbeat/bin/metricbeat -c /etc/metricbeat/metricbeat.yml test modules

ℹ️ 测试指定模块设置
语法：modules [MODULE_NAME] [METRICSET_NAME]
测试：sudo /usr/share/metricbeat/bin/metricbeat -c /etc/metricbeat/metricbeat.yml test modules system cpu

如果 Metricbeat 作为服务运行，请先停止服务。然后在前台运行 Metricbeat 来测试设置，这样就可以快速查看发生的任何错误：

1sudo /usr/share/metricbeat/bin/metricbeat -c /etc/metricbeat/metricbeat.yml -e -v

I. 前提条件#

1.1 网络要求#

1.2 安装顺序#

1.3 节点规划#

1.4 服务器配置#

II. Elasticsearch 安装#

2.1 登录 sudo 用户#

2.2 Hosts 修改#

2.3 创建工作目录#

2.4 导入 Elasticsearch GPG 密钥#

2.5 下载并手动安装 RPM#

2.6 创建数据目录#

2.7 在节点之间配置 TLS#

2.7.1 创建证书存放目录#

2.7.2 生成证书授权#

2.7.3 将密码存储到密钥存储中#

2.8 在 HTTP 层启用 TLS#

2.8.1 生成 HTTP 证书#

2.8.2 拷贝证书至各节点#

2.8.3 修改各节点证书权限#

2.8.4 将密码存储到密钥存储中#

2.9 Elasticsearch 配置#

2.9.1 备份默认置文件#

2.9.2 节点1配置参考#

2.9.3 jvm.options 配置#

2.10 重要系统配置#

2.10.1 Swapping & Temporary directory#

2.10.2 TCP retransmission timeout#

2.11 Elasticsearch 启动#

2.11.1 启动 Elasticsearch#

2.11.2 无法启动 Elasticsearch 解决#

2.12 设置第二个节点#

2.12.1 节点2配置参考#

2.12.2 启动节点2#

2.12.3 查看节点状态#

2.13 设置其它节点#

2.13.1 节点3配置参考#

2.13.2 节点4配置参考#

2.14 后续配置#

III. Kibana 安装#

3.1 登录 sudo 用户#

3.2 创建工作目录#

3.3 下载并手动安装 Kibana#

3.4 创建数据目录#

3.5 创建证书存放目录#

3.6 Kibana 安全配置#

3.7 在 Kibana 和 Elasticsearch 之间设置 TLS 加密#

3.7.1 加密 Kibana 和 Elasticsearch 之间的流量#

3.7.2 加密浏览器和 Kibana 之间的流量#

3.8 服务账户令牌#

3.8.1 创建服务账户令牌#

3.8.2 将 token 值添加到 kibana-keystore#

3.8.3 REST API 令牌管理#

3.9 Kibana 配置参考#

IV. Logstash 安装#

4.1 登录 sudo 用户#

4.2 创建工作目录#

4.3 下载并手动安装 Logstash#

4.4 创建数据目录#

4.5 创建证书存放目录#

4.6 Logstash 配置参考#

4.7 保护与 Elasticsearch 连接安全#

4.7.1 配置 Logstash 以使用 TLS/SSL 加密#

4.7.2 使用 API Key 授予访问权限#

4.8 Logstash pipeline 配置#

V. Filebeat 安装#

5.1 登录 sudo 用户#

5.2 创建工作目录#

5.3 下载并手动安装 Filebeat#

5.4 创建数据目录#

5.5 创建证书存放目录#

5.6 与 Logstash 安全通信#

5.6.1 为 Logstash 生成证书#

5.6.2 为 Filebeat 生成证书#

5.7 使用 SSL 相互认证:#

5.7.1 配置 Filebeat 以使用 SSL#

5.7.2 配置 Logstash pipeline 以使用 SSL#

5.8 验证测试#

5.8.1 验证 Logstash 服务器的证书#

5.8.2 测试 Filebeat 与 Logstash 的连接#

I. 前提条件

1.1 网络要求

1.2 安装顺序

1.3 节点规划

1.4 服务器配置

II. Elasticsearch 安装

2.1 登录 sudo 用户

2.2 Hosts 修改

2.3 创建工作目录

2.4 导入 Elasticsearch GPG 密钥

2.5 下载并手动安装 RPM

2.6 创建数据目录

2.7 在节点之间配置 TLS

2.7.1 创建证书存放目录

2.7.2 生成证书授权

2.7.3 将密码存储到密钥存储中

2.8 在 HTTP 层启用 TLS

2.8.1 生成 HTTP 证书

2.8.2 拷贝证书至各节点

2.8.3 修改各节点证书权限

2.8.4 将密码存储到密钥存储中

2.9 Elasticsearch 配置

2.9.1 备份默认置文件

2.9.2 节点1配置参考

2.9.3 jvm.options 配置

2.10 重要系统配置

2.10.1 Swapping & Temporary directory

2.10.2 TCP retransmission timeout

2.11 Elasticsearch 启动

2.11.1 启动 Elasticsearch

2.11.2 无法启动 Elasticsearch 解决

2.12 设置第二个节点

2.12.1 节点2配置参考

2.12.2 启动节点2

2.12.3 查看节点状态

2.13 设置其它节点

2.13.1 节点3配置参考

2.13.2 节点4配置参考

2.14 后续配置

III. Kibana 安装

3.1 登录 sudo 用户

3.2 创建工作目录

3.3 下载并手动安装 Kibana

3.4 创建数据目录

3.5 创建证书存放目录

3.6 Kibana 安全配置

3.7 在 Kibana 和 Elasticsearch 之间设置 TLS 加密

3.7.1 加密 Kibana 和 Elasticsearch 之间的流量

3.7.2 加密浏览器和 Kibana 之间的流量

3.8 服务账户令牌

3.8.1 创建服务账户令牌

3.8.2 将 token 值添加到 kibana-keystore

3.8.3 REST API 令牌管理

3.9 Kibana 配置参考

IV. Logstash 安装

4.1 登录 sudo 用户

4.2 创建工作目录

4.3 下载并手动安装 Logstash

4.4 创建数据目录

4.5 创建证书存放目录

4.6 Logstash 配置参考

4.7 保护与 Elasticsearch 连接安全

4.7.1 配置 Logstash 以使用 TLS/SSL 加密

4.7.2 使用 API Key 授予访问权限

4.8 Logstash pipeline 配置

V. Filebeat 安装

5.1 登录 sudo 用户

5.2 创建工作目录

5.3 下载并手动安装 Filebeat

5.4 创建数据目录

5.5 创建证书存放目录

5.6 与 Logstash 安全通信

5.6.1 为 Logstash 生成证书

5.6.2 为 Filebeat 生成证书

5.7 使用 SSL 相互认证:

5.7.1 配置 Filebeat 以使用 SSL

5.7.2 配置 Logstash pipeline 以使用 SSL

5.8 验证测试

5.8.1 验证 Logstash 服务器的证书

5.8.2 测试 Filebeat 与 Logstash 的连接