Cara Setup Agent Log di cluster Kubernetes : fluent bit agent

Introduction

Pada kesempatan kali ini, penulis akan membawakan sebauh artikel yang berjedul Cara Setup Agent Log di Cluster Kubernetes : menggunakan fluent bit agent, hal ini di latar belakangi karena adanya keperluan utnuk menyimpan loggin pada aplikasi kita, yang dimana sangat penting untuk kita gunakan tentunya.

Step 1 – Prepare Manifests Fluent Bit

Ada beberapa manfiests yang perlu disiapkan untuk men-deploy agent fluent-bit ini, sama halnya dengan aplikasi atau yang lainya untuk mekanisme, dan berikut list manifestsnya apa saja:

tree production/fluent-bit production/fluent-bit ├── fluent-bit-cluster-role.yaml ├── fluent-bit-cm.yaml ├── fluent-bit-crbinding.yaml ├── fluent-bit-daemonset.yaml ├── fluent-bit-rbac.yaml ├── fluent-bit-svc-acc.yaml ├── parsers.conf └── values.yaml

Adapaun fungsi-fungsi manifests tersebut sebagai berikut:

Cluster role: Cluster Role adalah peran yang didefinisikan dalam Kubernetes untuk memberikan izin akses tingkat cluster kepada subjek tertentu.
ConfigMaps: ConfigMaps adalah objek di Kubernetes yang digunakan untuk menyimpan konfigurasi non-confidential, seperti variabel lingkungan, file konfigurasi, atau nilai-nilai konfigurasi lainnya.
CR Binding: CR Binding adalah proses mengaitkan atau menghubungkan Custom Resource (CR) dengan sumber daya atau layanan tertentu di dalam cluster Kubernetes
Daemonset: Sama halnya dengan deployment atau POD tapi sifatnya daemon set hanya digunakan untuk agent2 seperti halnya ini.
RBAC: RBAC adalah model keamanan di Kubernetes yang memungkinkan administrator mengontrol akses pengguna atau pod ke sumber daya cluster.
Service Account: Service Account adalah identitas yang digunakan oleh pod atau layanan di dalam cluster Kubernetes.

Dan berikut ini adalah isi file manifests-manifests yang ada diatas:

fluent-bit-cluster-role.yaml

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: fluent-bit-read
  namespace: log
rules:
- apiGroups: [""]
  resources:
  - namespaces
  - pods
  verbs: ["get", "list", "watch"]

fluent-bit-cm.yaml

apiVersion: v1
kind: ConfigMap
metadata:
  name: fluent-bit-config
  namespace: log
  labels:
    k8s-app: fluent-bit
data:
  # Configuration files: server, input, filters and output
  # ======================================================
  fluent-bit.conf: |
    [SERVICE]
        Flush                     1
        Log_Level                 info
        Daemon                    off
        Parsers_File              parsers.conf
        HTTP_Server               On
        HTTP_Listen               0.0.0.0
        HTTP_Port                 2020

    @INCLUDE input-kubernetes.conf
    @INCLUDE filter-kubernetes.conf
    @INCLUDE output-graylog.conf

  input-kubernetes.conf: |
    [INPUT]
        Name               tail
        Tag                kube.*
        Path               /var/log/containers/*.log
        Exclude_Path       /var/log/containers/*_kube-system_*.log,/var/log/containers/*_vault_*.log,/var/log/containers/*_arms-prom_*.log,/var/log/containers/*_log_*.log
        Parser             docker
        DB                 /var/log/flb_graylog.db
        DB.Sync            Normal
        Docker_Mode        On
        Buffer_Chunk_Size  512KB
        Buffer_Max_Size    5M
        Rotate_Wait        30
        Mem_Buf_Limit      30MB
        Skip_Long_Lines    On
        Refresh_Interval   10

  filter-kubernetes.conf: |
    [FILTER]
        Name                kubernetes
        Match               kube.*
        Merge_Log           On
        Merge_Log_Key       log
        Keep_Log            Off
        K8S-Logging.Parser  On
        K8S-Logging.Exclude On	
        Annotations         Off
        Labels              On	
    

  output-graylog.conf: |
    [OUTPUT]
        Name                    gelf
        Match                   *
        Host                    IP tujuan
        Port                    12201 / Port tujuan
        Mode                    tcp
        Gelf_Short_Message_Key  log 

  parsers.conf: |
    [PARSER]
        Name   apache
        Format regex
        Regex  ^(?<host>[^ ]*) [^ ]* (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^\"]*?)(?: +\S*)?)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^\"]*)" "(?<agent>[^\"]*)")?$
        Time_Key time
        Time_Format %d/%b/%Y:%H:%M:%S %z

    [PARSER]
        Name   json
        Format json
        Time_Key time
        Time_Format %d/%b/%Y:%H:%M:%S %z

    [PARSER]
        Name        docker
        Format      json
        Time_Key    time
        Time_Format %Y-%m-%dT%H:%M:%S.%L
        Time_Keep   On

    [PARSER]
        Name        syslog
        Format      regex
        Regex       ^\<(?<pri>[0-9]+)\>(?<time>[^ ]* {1,2}[^ ]* [^ ]*) (?<host>[^ ]*) (?<ident>[a-zA-Z0-9_\/\.\-]*)(?:\[(?<pid>[0-9]+)\])?(?:[^\:]*\:)? *(?<message>.*)$
        Time_Key    time
        Time_Format %b %d %H:%M:%S

fluent-bit-crbinding.yaml

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: fluent-bit-read
  namespace: log
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: fluent-bit-read
subjects:
- kind: ServiceAccount
  name: fluent-bit
  namespace: log

fluent-bit-daemonset.yaml

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: fluent-bit
  namespace: log
  labels:
    k8s-app: fluent-bit-logging
    version: v1
    kubernetes.io/cluster-service: "true"
spec:
  selector:
    matchLabels:
      k8s-app: fluent-bit-logging
  template:
    metadata:
      labels:
        k8s-app: fluent-bit-logging
        version: v1
        kubernetes.io/cluster-service: "true"
      annotations:
        prometheus.io/scrape: "true"
        prometheus.io/port: "2020"
        prometheus.io/path: /api/v1/metrics/prometheus
    spec:
      containers:
      - name: fluent-bit
        image: fluent/fluent-bit:1.5.0
        resources:
          requests:
            cpu: "32m"
            memory: "32Mi"
          limits:
            cpu: "126m"
            memory: "126Mi"
        imagePullPolicy: Always
        ports:
          - containerPort: 2020
        volumeMounts:
        - name: varlog
          mountPath: /var/log
        - name: varlibdockercontainers
          mountPath: /var/lib/docker/containers
          readOnly: true
        - name: fluent-bit-config
          mountPath: /fluent-bit/etc/
      terminationGracePeriodSeconds: 10
      volumes:
      - name: varlog
        hostPath:
          path: /var/log
      - name: varlibdockercontainers
        hostPath:
          path: /var/lib/docker/containers
      - name: fluent-bit-config
        configMap:
          name: fluent-bit-config
      serviceAccountName: fluent-bit
      tolerations:
      - key: node-role.kubernetes.io/master
        operator: Exists
        effect: NoSchedule
      - operator: "Exists"
        effect: "NoExecute"
      - operator: "Exists"
        effect: "NoSchedule"

fluent-bit-rbac.yaml

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: fluent-bit-cluster-role
rules:
  - apiGroups: [""]
    resources: ["nodes"]
    verbs: ["get", "list", "watch"]
  - apiGroups: [""]
    resources: ["nodes/proxy"]
    verbs: ["get"]
  - apiGroups: [""]
    resources: ["pods"]
    verbs: ["list", "get"]
  - apiGroups: [""]
    resources: ["namespaces"]
    verbs: ["list", "get"]
  - apiGroups: [""]
    resources: ["configmaps"]
    verbs: ["list", "get"]

fluent-bit-svc-acc.yaml

apiVersion: v1
kind: ServiceAccount
metadata:
  name: fluent-bit
  namespace: log

dan yang terakhir adalah parsers.conf

[PARSER]
    Name   docker
    Format json
    Time_Key time
    Time_Format %Y-%m-%dT%H:%M:%S.%L
    Time_Keep On

Step 2 – Deploy Manifest fluent-bit

Untuk urutanya sendiri adalah : Service account, CR, RBAC parsers.conf dan yang terakhir adalah Daemonset karena dia running sebagai Docker/container, menggunakan perintah

kubectl apply -f "file-manifests"

Check