K8S(15)监控实战-ELK 收集 K8S 内应用日志

1 收集 K8S 日志方案

K8S 系统里的业务应用是高度“动态化”的，随着容器编排的进行，业务容器在不断的被创建、被摧毁、被漂移、被扩缩容…

我们需要这样一套日志收集、分析的系统：

收集 – 能够采集多种来源的日志数据（流式日志收集器）
传输 – 能够稳定的把日志数据传输到中央系统（消息队列）
存储 – 可以将日志以结构化数据的形式存储起来（搜索引擎）
分析 – 支持方便的分析、检索方法，最好有 GUI 管理系统（ web ）
警告 – 能够提供错误报告，监控机制（监控系统）

1.1 传统ELk模型缺点：

Logstash 使用 Jruby 语言开发，吃资源，大量部署消耗极高
业务程序与 Logstash 耦合过松，不利于业务迁移
日志收集与 ES 耦合又过紧，（ Logstash ）易打爆（ ES ）、丢数据
在容器云环境下，传统 ELk 模型难以完成工作

1.2 K8S 容器日志收集模型

2 制作 TOMCAT 底包

2.1 准备 TOMCAT 底包

2.1.1 下载 TOMCAT8

cd /opt/src/
wget https://archive.apache.org/dist/tomcat/tomcat-8/v8.5.50/bin/apache-tomcat-8.5.50.tar.gz
mkdir /data/dockerfile/tomcat
tar xf apache-tomcat-8.5.50.tar.gz -C /data/dockerfile/tomcat
cd /data/dockerfile/tomcat

2.1.2 简单配置 TOMCAT

删除自带网页

rm -rf apache-tomcat-8.5.50/webapps/*

关闭 AJP 端口

AJP 端口为前端 Apache 时调用，故前端采用 Nginx 时不需要。

tomcat]# vim apache-tomcat-8.5.50/conf/server.xml
  <!-- <Connector port="8009" protocol="AJP/1.3" redirectPort="8443" /> -->

修改日志类型

删除 3manager，4host-manager 的 Handlers

tomcat]# vim apache-tomcat-8.5.50/conf/logging.properties
handlers = 
  [1catalina.org.apache.juli.AsyncFileHandler](http://1catalina.org.apache.juli.asyncfilehandler/), 
  [2localhost.org.apache.juli.AsyncFileHandler](http://2localhost.org.apache.juli.asyncfilehandler/), 
java.util.logging.ConsoleHandler

日志级别改为 INFO

1catalina.org.apache.juli.AsyncFileHandler.level = INFO
2localhost.org.apache.juli.AsyncFileHandler.level = INFO
java.util.logging.ConsoleHandler.level = INFO

注释所有关于 3manager，4host-manager 日志的配置

# 3manager.org.apache.juli.AsyncFileHandler.level = FINE
# 3manager.org.apache.juli.AsyncFileHandler.directory = ${catalina.base}/logs
# 3manager.org.apache.juli.AsyncFileHandler.prefix = manager.
# 3manager.org.apache.juli.AsyncFileHandler.encoding = UTF-8
# 4host-manager.org.apache.juli.AsyncFileHandler.level = FINE
# 4host-manager.org.apache.juli.AsyncFileHandler.directory = ${catalina.base}/logs
# 4host-manager.org.apache.juli.AsyncFileHandler.prefix = host-manager.
# 4host-manager.org.apache.juli.AsyncFileHandler.encoding = UTF-8

2.2 准备 Docker 镜像

2.2.1 创建 Dockerfile

cat >Dockerfile <<'EOF'
From harbor.zq.com/public/jre:8u112
RUN /bin/cp /usr/share/zoneinfo/Asia/Shanghai /etc/localtime &&\
  echo 'Asia/Shanghai' >/etc/timezone
ENV CATALINA_HOME /opt/tomcat
ENV LANG zh_CN.UTF-8
ADD apache-tomcat-8.5.50/ /opt/tomcat
ADD config.yml /opt/prom/config.yml
ADD jmx_javaagent-0.3.1.jar /opt/prom/jmx_javaagent-0.3.1.jar
WORKDIR /opt/tomcat
ADD entrypoint.sh /entrypoint.sh
CMD ["/bin/bash","/entrypoint.sh"]
EOF

2.2.2 准备 Dockerfile 所需文件

JVM 监控所需 JAR 包

wget  -O jmx_javaagent-0.3.1.jar https://repo1.maven.org/maven2/io/prometheus/jmx/jmx_prometheus_javaagent/0.3.1/jmx_prometheus_javaagent-0.3.1.jar

jmx_agent 读取的配置文件

cat >config.yml <<'EOF'
---
rules:
 - pattern: '.*'
EOF

容器启动脚本

cat  >entrypoint.sh <<'EOF'
#!/bin/bash

# Pod ip:port 监控规则传给jvm监控客户端
M_OPTS="-Duser.timezone=Asia/Shanghai -javaagent:/opt/prom/jmx_javaagent-0.3.1.jar=$(hostname -i):${M_PORT:-"12346"}:/opt/prom/config.yml"

C_OPTS=${C_OPTS}    # 启动追加参数

MIN_HEAP=${MIN_HEAP:-"128m"} # JAVA 虚拟机初始化时的最小内存
MAX_HEAP=${MAX_HEAP:-"128m"} # JAVA 虚拟机初始化时的最大内存

# 年轻代，GC 回收
JAVA_OPTS=${JAVA_OPTS:-"-Xmn384m -Xss256k -Duser.timezone=GMT+08  -XX:+DisableExplicitGC -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:+CMSParallelRemarkEnabled -XX:+UseCMSCompactAtFullCollection -XX:CMSFullGCsBeforeCompaction=0 -XX:+CMSClassUnloadingEnabled -XX:LargePageSizeInBytes=128m -XX:+UseFastAccessorMethods -XX:+UseCMSInitiatingOccupancyOnly -XX:CMSInitiatingOccupancyFraction=80 -XX:SoftRefLRUPolicyMSPerMB=0 -XX:+PrintClassHistogram  -Dfile.encoding=UTF8 -Dsun.jnu.encoding=UTF8"}

CATALINA_OPTS="${CATALINA_OPTS}"

JAVA_OPTS="${M_OPTS} ${C_OPTS} -Xms${MIN_HEAP} -Xmx${MAX_HEAP} ${JAVA_OPTS}"

sed -i -e "1a\JAVA_OPTS=\"$JAVA_OPTS\"" -e "1a\CATALINA_OPTS=\"$CATALINA_OPTS\"" /opt/tomcat/bin/catalina.sh

cd /opt/tomcat && /opt/tomcat/bin/catalina.sh run 2>&1 >> /opt/tomcat/logs/stdout.log # 日志文件

EOF

2.2.3 构建 Docker

docker build . -t harbor.zq.com/base/tomcat:v8.5.50
docker push       harbor.zq.com/base/tomcat:v8.5.50

2.3 构建 Dubbo 消费者服务（Tomcat 版）

2.3.1 通过 Jenkins 构建 Dubbo 消费者镜像（Tomcat 版）

新建 Jenkins Pipeline 流水线：
- item name：tomcat-demo
- 选择 -> 流水线
勾选： Discard old builds
- Days to keep builds ：3
- Max # of builds to keep ：30
勾选： This project is parameterized

注意：全部勾选 Trim the string
- 1：Add Parameter -> String Parameter
  - Name: app_name
  - Default Value :
  - Description : project name e g: dubbo-demo-web
- 2: Add Parameter -> String Parameter
  - Name: image_name
  - Default Value :
  - Description : project docker image name. e g: app/dubbo-demo-web
- 3: Add Parameter -> String Parameter
  - Name: git_repo
  - Default Value :
  - Description : project git repository. e g: git@gitee.com:outmanzzq/dubbo-demo-web.git
- 4: Add Parameter -> String Parameter
  - Name: git_ver
  - Default Value : tomcat
  - Description : git commit id of the project
- 5: Add Parameter -> String Parameter
  - Name: add_tag
  - Default Value :
  - Description : project docker image tag. date_timestamp recommended e.g: 20220428_2052
- 6: Add Parameter -> String Parameter
  - Name: target_dir
  - Default Value : ./dubbo-client/target
  - Description : the relative path of file such as jar or war package. e.g: ./dubbo-client/target
- 7: Add Parameter -> String Parameter
  - Name: mvn_dir
  - Default Value : ./
  - Description : project maven directory. e.g: ./
- 8: Add Parameter -> String Parameter
  - Name: mvn_cmd
  - Default Value : mvn clean package -Dmaven.test.skip=true
  - Description : maven command. e.g: mvn clean package -e -q -Dmaven.test.skip=true
- 9：Add Parameter -> Choice Parameter
  - Name: base_image
  - Choices : base/tomcat:v7.0.94 base/tomcat:v8.5.50 base/tomcat:v9.0.17
  - Description : project base image list in harbor.zq.com
- 9：Add Parameter -> Choice Parameter
  - Name: maven
  - Choices : 3.6.1 3.2.5 2.2.1
  - Description : different maven edition
- 10：Add Parameter -> String Parameter**
  - Name: root_url
  - Default Value : ROOT
  - Description : webapp dir.
Pipeline -> Pipeline script

pipeline {
  agent any 
    stages {
      stage('pull') { //get project code from repo 
        steps {
          sh "git clone ${params.git_repo} ${params.app_name}/${env.BUILD_NUMBER} && cd ${params.app_name}/${env.BUILD_NUMBER} && git checkout ${params.git_ver}"
        }
      }
      stage('build') { //exec mvn cmd
        steps {
          sh "cd ${params.app_name}/${env.BUILD_NUMBER}  && /var/jenkins_home/maven-${params.maven}/bin/${params.mvn_cmd}"
        }
      }
      stage('unzip') { //unzip target/*.war -c target/project_dir
        steps {
          sh "cd ${params.app_name}/${env.BUILD_NUMBER} && cd ${params.target_dir} && mkdir project_dir && unzip *.war -d ./project_dir"
        }
      }
      stage('image') { //build image and push to registry
        steps {
          writeFile file: "${params.app_name}/${env.BUILD_NUMBER}/Dockerfile", text: """FROM harbor.zq.com/${params.base_image}
ADD ${params.target_dir}/project_dir /opt/tomcat/webapps/${params.root_url}"""
          sh "cd  ${params.app_name}/${env.BUILD_NUMBER} && docker build -t harbor.zq.com/${params.image_name}:${params.git_ver}_${params.add_tag} . && docker push harbor.zq.com/${params.image_name}:${params.git_ver}_${params.add_tag}"
        }
      }
    }
}

Jenkins => 流水线：tomcat-demo =>新建 build，参数如图：

build 完成，Harbor 仓库 -> app/dubbo-demo-web 将生成 Dubbo 消费者 Tomcat 镜像：

harbor.zq.com:180/app/dubbo-demo-web:tomcat_20220428_2200

将镜像替换到 K8S -> test 空间 -> dubbo-demo-consumer -> dp.yaml (或直接通过 Dashboard 修改 Deployment 配置)

[root@hdss7-200 test]# vim /data/k8s-yaml/test/dubbo-demo-consumer/dp.yaml 
kind: Deployment
apiVersion: extensions/v1beta1
metadata:
  name: dubbo-demo-consumer
  namespace: test
  labels:
    name: dubbo-demo-consumer
spec:
  replicas: 1
  selector:
    matchLabels:
      name: dubbo-demo-consumer
  template:
    metadata:
      labels:
        app: dubbo-demo-consumer
        name: dubbo-demo-consumer
      annotations:
        prometheus_io_scrape: "true"
        prometheus_io_port: "12346"
        prometheus_io_path: "/"
    spec:
      containers:
      - name: dubbo-demo-consumer
        # 原 Apollo 镜像
        #image: harbor.zq.com/app/dubbo-demo-consumer:apollo_20220423_2223
        # 替换 tomcat 镜像
        image: harbor.zq.com/app/dubbo-demo-web:tomcat_20220428_2200
        ports:
        - containerPort: 8080
          protocol: TCP
        - containerPort: 20880
          protocol: TCP
        env:
        - name: JAR_BALL
          value: dubbo-client.jar
        - name: C_OPTS
          value: -Denv=fat -Dapollo.meta=http://apollo-testconfig.zq.com
        imagePullPolicy: IfNotPresent
      imagePullSecrets:
      - name: harbor
      restartPolicy: Always
      terminationGracePeriodSeconds: 30
      securityContext:
        runAsUser: 0
      schedulerName: default-scheduler
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 1
      maxSurge: 1
  revisionHistoryLimit: 7
  progressDeadlineSeconds: 600

说明：

因 Dubbo Tomcat 消费者镜像依赖 Apollo 配置中心，故需要 Test 空间 Apollo 服务。

重新应用配置，并检查结果

任意 K8S 控制节点应用 DP 配置
```
[root@hdss7-21 ~]# kubectl apply -f http://k8s-yaml.zq.com/test/dubbo-demo-consumer/dp.yaml
```
浏览器访问：http://dubbo-testdemo.zq.com/hello?name=noah

3 部署 ElasticSearch

官网: https://www.elastic.co/
Github: https://github.com/elastic/elasticsearch

部署 hdss7-12.host.com上：

3.1 安装 ElasticSearch

3.1.1 下载二进制包

cd /opt/src
wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-6.8.6.tar.gz
tar xf elasticsearch-6.8.6.tar.gz -C /opt/
ln -s /opt/elasticsearch-6.8.6/ /opt/elasticsearch
cd /opt/elasticsearch

3.1.2 配置 elasticsearch.yml

mkdir -p /data/elasticsearch/{data,logs}

cat >config/elasticsearch.yml <<'EOF'
cluster.name: es.zq.com
node.name: hdss7-12.host.com
path.data: /data/elasticsearch/data
path.logs: /data/elasticsearch/logs
bootstrap.memory_lock: true
network.host: 10.4.7.12
http.port: 9200
EOF

3.2 优化其他设置

3.2.1 设置 JVM 参数

elasticsearch]# vi config/jvm.options
# 根据环境设置，-Xms 和-Xmx 设置为相同的值，推荐设置为机器内存的一半左右
-Xms512m 
-Xmx512m

3.2.2 创建普通用户

useradd -s /bin/bash -M es
chown -R es.es /opt/elasticsearch-6.8.6
chown -R es.es /data/elasticsearch/

3.2.3 调整文件描述符

vim /etc/security/limits.d/es.conf
es hard nofile 65536
es soft fsize unlimited
es hard memlock unlimited
es soft memlock unlimited

3.2.4 调整内核参数

sysctl -w vm.max_map_count=262144
echo "vm.max_map_count=262144" > /etc/sysctl.conf
sysctl -p

3.3 启动 ES

3.3.1 启动 ES 服务

]# su -c "/opt/elasticsearch/bin/elasticsearch -d" es
]# netstat -luntp|grep 9200
tcp6    0   0 10.4.7.12:9200     :::*          LISTEN   16784/java

3.3.1 调整 ES 日志模板

curl -H "Content-Type:application/json" -XPUT http://10.4.7.12:9200/_template/k8s -d '{
 "template" : "k8s*",
 "index_patterns": ["k8s*"], 
 "settings": {
  "number_of_shards": 5,
  "number_of_replicas": 0
 }
}'

# 生产为 3 份副本集，本 ES 为单节点，不能配置副本集

4 部署 Kafka 和 Kafka-manager

官网: http://kafka.apache.org/
Github: https://github.com/apache/kafka

HDSS7-11.host.com上：

4.1 但节点安装 Kafka

4.1.1 下载包

cd /opt/src
wget https://archive.apache.org/dist/kafka/2.2.0/kafka_2.12-2.2.0.tgz
tar xf kafka_2.12-2.2.0.tgz -C /opt/
ln -s /opt/kafka_2.12-2.2.0/ /opt/kafka
cd /opt/kafka

4.1.2 修改配置

mkdir /data/kafka/logs -p
cat >config/server.properties <<'EOF'
log.dirs=/data/kafka/logs
zookeeper.connect=localhost:2181    # zk消息队列地址 
log.flush.interval.messages=10000
log.flush.interval.ms=1000
delete.topic.enable=true
host.name=hdss7-11.host.com
EOF

4.1.3 启动 Kafka

bin/kafka-server-start.sh -daemon config/server.properties
]# netstat -luntp|grep 9092
tcp6    0   0 10.4.7.11:9092     :::*          LISTEN   34240/java

4.2 获取 Kafka-manager 的 Docker 镜像

Github: https://github.com/yahoo/kafka-manager

运维主机 HDSS7-200.host.com 上：

Kafka-manager 是 Kafka 的一个 Web 管理页面,非必须

4.2.1 方法一通过 Dockerfile 获取

1 准备 Dockerfile

cat >/data/dockerfile/kafka-manager/Dockerfile <<'EOF'
FROM hseeberger/scala-sbt

ENV ZK_HOSTS=10.4.7.11:2181 \
     KM_VERSION=2.0.0.2

RUN mkdir -p /tmp && \
    cd /tmp && \
    wget https://github.com/yahoo/kafka-manager/archive/${KM_VERSION}.tar.gz && \
    tar xxf ${KM_VERSION}.tar.gz && \
    cd /tmp/kafka-manager-${KM_VERSION} && \
    sbt clean dist && \
    unzip  -d / ./target/universal/kafka-manager-${KM_VERSION}.zip && \
    rm -fr /tmp/${KM_VERSION} /tmp/kafka-manager-${KM_VERSION}

WORKDIR /kafka-manager-${KM_VERSION}

EXPOSE 9000
ENTRYPOINT ["./bin/kafka-manager","-Dconfig.file=conf/application.conf"]
EOF

2 制作 Docker 镜像

cd /data/dockerfile/kafka-manager

docker build . -t harbor.od.com/infra/kafka-manager:v2.0.0.2

docker push harbor.zq.com/infra/kafka-manager:latest

构建过程极其漫长,大概率会失败,因此可以通过第二种方式下载构建好的镜像

但构建好的镜像写死了 ZK 地址,要注意传入变量修改 ZK 地址

4.2.2 直接下载 Docker 镜像

docker pull hlebalbau/kafka-manager:2.0.0.2
docker tag 56a3140b4555 harbor.zq.com/infra/kafka-manager:v2.0.0.2
docker push harbor.zq.com/infra/kafka-manager:v2.0.0.2

4.3 部署 Kafka-manager

mkdir /data/k8s-yaml/kafka-manager
cd /data/k8s-yaml/kafka-manager

4.3.1 准备 Deployment 清单

cat >deployment.yaml <<'EOF'
kind: Deployment
apiVersion: extensions/v1beta1
metadata:
  name: kafka-manager
  namespace: infra
  labels: 
    name: kafka-manager
spec:
  replicas: 1
  selector:
    matchLabels: 
      name: kafka-manager
  template:
    metadata:
      labels: 
        app: kafka-manager
        name: kafka-manager
    spec:
      containers:
      - name: kafka-manager
        image: harbor.zq.com/infra/kafka-manager:v2.0.0.2
        ports:
        - containerPort: 9000
          protocol: TCP
        env:
        - name: ZK_HOSTS
          value: zk1.od.com:2181
        - name: APPLICATION_SECRET
          value: letmein
        imagePullPolicy: IfNotPresent
      imagePullSecrets:
      - name: harbor
      restartPolicy: Always
      terminationGracePeriodSeconds: 30
      securityContext: 
        runAsUser: 0
      schedulerName: default-scheduler
  strategy:
    type: RollingUpdate
    rollingUpdate: 
      maxUnavailable: 1
      maxSurge: 1
  revisionHistoryLimit: 7
  progressDeadlineSeconds: 600
EOF

4.3.2 准备 SVC 资源清单

cat >service.yaml <<'EOF'
kind: Service
apiVersion: v1
metadata: 
  name: kafka-manager
  namespace: infra
spec:
  ports:
  - protocol: TCP
    port: 9000
    targetPort: 9000
  selector: 
    app: kafka-manager
EOF

4.3.3 准备 Ingress 资源清单

cat >ingress.yaml <<'EOF'
kind: Ingress
apiVersion: extensions/v1beta1
metadata: 
  name: kafka-manager
  namespace: infra
spec:
  rules:
  - host: km.zq.com
    http:
      paths:
      - path: /
        backend: 
          serviceName: kafka-manager
          servicePort: 9000
EOF

4.3.4 应用资源配置清单

任意一台运算节点上：

kubectl apply -f http://k8s-yaml.zq.com/kafka-manager/deployment.yaml
kubectl apply -f http://k8s-yaml.zq.com/kafka-manager/service.yaml
kubectl apply -f http://k8s-yaml.zq.com/kafka-manager/ingress.yaml

4.3.5 解析域名

Hdss7-11.host.com 上

~]# vim /var/named/zq.com.zone
km    A   10.4.7.10
~]# systemctl restart named
~]# dig -t A km.zq.com @10.4.7.11 +short
10.4.7.10

4.3.6 浏览器访问

http://km.zq.com

添加集群

1651140892943

brokerViewThreadPoolSize =4

offsetCacheThreadPoolSize =4

kafkaAdminClientThreadPoolSize =4

查看集群信息**

5 部署 Filebeat

官方下载地址: https://www.elastic.co/downloads/beats/filebeat

运维主机HDSS7-200.host.com上：

5.1 制作 Docker 镜像

mkdir /data/dockerfile/filebeat
cd /data/dockerfile/filebeat

5.1.1 准备 Dockerfile

cat >Dockerfile <<'EOF'
FROM debian:jessie
# 如果更换版本,需在官网下载同版本 LINUX64-BIT 的 sha 替换 FILEBEAT_SHA1
ENV FILEBEAT_VERSION=7.5.1
ENV FILEBEAT_SHA1=daf1a5e905c415daf68a8192a069f913a1d48e2c79e270da118385ba12a93aaa91bda4953c3402a6f0abf1c177f7bcc916a70bcac41977f69a6566565a8fae9c  
RUN set -x && \
 apt-get update && \
 apt-get install -y wget && \
 wget https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-${FILEBEAT_VERSION}-linux-x86_64.tar.gz -O /opt/filebeat.tar.gz && \
 cd /opt && \
 echo "${FILEBEAT_SHA1} filebeat.tar.gz" | sha512sum -c - && \
 tar xzvf filebeat.tar.gz && \
 cd filebeat-* && \
 cp filebeat /bin && \
 cd /opt && \
 rm -rf filebeat* && \
 apt-get purge -y wget && \
 apt-get autoremove -y && \
 apt-get clean && rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/*
COPY filebeat.yaml /etc/
COPY docker-entrypoint.sh /
ENTRYPOINT ["/bin/bash","/docker-entrypoint.sh"]
EOF

5.1.2 准备 Filebeat 配置文件

cat >filebeat.yaml << EOF
filebeat.inputs:
- type: log
  fields_under_root: true
  fields:
    topic: logm-PROJ_NAME
  paths:
    - /logm/*.log
    - /logm/*/*.log
    - /logm/*/*/*.log
    - /logm/*/*/*/*.log
    - /logm/*/*/*/*/*.log
  scan_frequency: 120s
  max_bytes: 10485760
  multiline.pattern: 'MULTILINE'
  multiline.negate: true
  multiline.match: after
  multiline.max_lines: 100
- type: log
  fields_under_root: true
  fields:
    topic: logu-PROJ_NAME
  paths:
    - /logu/*.log
    - /logu/*/*.log
    - /logu/*/*/*.log
    - /logu/*/*/*/*.log
    - /logu/*/*/*/*/*.log
    - /logu/*/*/*/*/*/*.log
output.kafka:
  hosts: ["10.4.7.11:9092"]
  topic: k8s-fb-ENV-%{[topic]}
  version: 2.0.0      # Kafka 版本超过 2.0，默认写 2.0.0 
  required_acks: 0
  max_message_bytes: 10485760
EOF

5.1.3 准备启动脚本

cat >docker-entrypoint.sh <<'EOF'
#!/bin/bash

ENV=${ENV:-"test"}                    # 定义日志收集的环境
PROJ_NAME=${PROJ_NAME:-"no-define"}   # 定义项目名称
MULTILINE=${MULTILINE:-"^\d{2}"}      # 多行匹配，以 2 个数据开头的为一行，反之

# 替换配置文件中的内容
sed -i 's#PROJ_NAME#'${PROJ_NAME}'#g' /etc/filebeat.yaml
sed -i 's#MULTILINE#'${MULTILINE}'#g' /etc/filebeat.yaml
sed -i 's#ENV#'${ENV}'#g'             /etc/filebeat.yaml

if [[ "$1" == "" ]]; then
     exec filebeat  -c /etc/filebeat.yaml 
else
    exec "$@"
fi
EOF

5.1.4 构建镜像

docker build . -t harbor.zq.com/infra/filebeat:v7.5.1
docker push  harbor.zq.com/infra/filebeat:v7.5.1

5.2 以边车模式运行 Pod

5.2.1 准备资源配置清单

使用 dubbo-demo-consumer 的镜像,以边车模式运行 Filebeat

]# cat > /data/k8s-yaml/test/dubbo-demo-consumer/dp.yaml <<EOF
kind: Deployment
apiVersion: extensions/v1beta1
metadata:
  name: dubbo-demo-consumer
  namespace: test
  labels: 
    name: dubbo-demo-consumer
spec:
  replicas: 1
  selector:
    matchLabels: 
      name: dubbo-demo-consumer
  template:
    metadata:
      labels: 
        app: dubbo-demo-consumer
        name: dubbo-demo-consumer
      annotations:
        blackbox_path: "/hello?name=health"
        blackbox_port: "8080"
        blackbox_scheme: "http"
        prometheus_io_scrape: "true"
        prometheus_io_port: "12346"
        prometheus_io_path: "/"
    spec:
      containers:
      - name: dubbo-demo-consumer
        image: harbor.zq.com/app/dubbo-demo-web:tomcat_20220428_2200
        ports:
        - containerPort: 8080
          protocol: TCP
        - containerPort: 20880
          protocol: TCP
        env:
        - name: JAR_BALL
          value: dubbo-client.jar
        - name: C_OPTS
          value: -Denv=fat -Dapollo.meta=http://apollo-testconfig.zq.com
        imagePullPolicy: IfNotPresent
#--------新增内容--------
        volumeMounts:
        - mountPath: /opt/tomcat/logs
          name: logm
      - name: filebeat
        image: harbor.zq.com/infra/filebeat:v7.5.1
        imagePullPolicy: IfNotPresent
        env:
        - name: ENV
          value: test             # 测试环境
        - name: PROJ_NAME
          value: dubbo-demo-web   # 项目名
        volumeMounts:
        - mountPath: /logm        
          name: logm
      volumes:
      - emptyDir: {} # 随机在宿主机找目录创建,容器删除时一起删除
        name: logm
#--------新增结束--------
      imagePullSecrets:
      - name: harbor
      restartPolicy: Always
      terminationGracePeriodSeconds: 30
      securityContext: 
        runAsUser: 0
      schedulerName: default-scheduler
  strategy:
    type: RollingUpdate
    rollingUpdate: 
      maxUnavailable: 1
      maxSurge: 1
  revisionHistoryLimit: 7
  progressDeadlineSeconds: 600
EOF

5.2.2 应用资源清单

任意 Node 节点

kubectl apply -f http://k8s-yaml.zq.com/test/dubbo-demo-consumer/dp.yaml

5.2.3 验证

浏览器访问 http://km.zq.com ,看到 Kafaka-manager 里，Topic 打进来，即为成功

[

进入 dubbo-demo-consumer 的 Filebeat 容器中,查看 logm 目录下是否有日志

kubectl -n test exec -it dobbo...... -c filebeat /bin/bash
ls /logm

# -c参数指定 Pod 中的 Filebeat 容器
# /logm 是 Filebeat 容器挂载的目录

6 部署 Logstash