Leveraging Fluent Bit on Tanzu Kubernetes Grid to send VEBA logs to vRealize Log Insight

Introduction

With the recent v0.5 release of the VMware Event Broker Appliance (VEBA) project, the ability of deploying the core component, the VMware Event Router, via a Helm chart to an existing Kubernetes cluster was introduced. William Lam introduced this and all the other great new features and enhancements in his corresponding blog post. This earned very positive feedback from the community and opens even more doors in terms of flexibility as well as extensibility.

Speaking of which, extensibility…

During the v0.5 pre-release team meeting, William came up with the idea to let VEBA sending logs to our log analysis solutions VMware vRealize Log Insight or vRealize Log Insight Cloud. He asked me if I would do it and of course I found it a great idea. This post is a good addition to my previous post “Monitoring the VMware Event Broker Appliance with vRealize Operations Manager”¹, in which I’m describing how I used cAdvisor² to get resource usage and performance data from the various VEBA components and ultimately to monitor them.

At this point, I have to be clear that I didn’t used the VEBA appliance this time. Instead, I used Helm charts to deploy every component like e.g. the mentioned Event Router or the Event Processor to my Tanzu Kubernetes Grid - TKGs cluster (also referred to as Guestcluster) running on VMware vSphere. However, I’m considering to write a small follow-up article which will cover the Appliance variant.

The Open Source projects Fluentd and Fluent Bit can be leveraged for multi-platform log processing and log forwarding. For my use case with VEBA, my requirements on the solution were basically simple. It should be lightweight, fast and if possible, with few or no dependencies. Comparing both mentioned projects with each other, it brought me multiple times back to Fluent Bit, even besides the fact, that good content on how to integrate Fluentd with TKG already exist.

The Hummingbird (Note: no technical relevance)

The power of symbols thrills me. It’s that power that makes you think or remember a specific brand, a company or an event. Just think about the Kubernetes steering wheel for example. If you feel home in the cloud native space and you see a steering wheel in real life nowadays, you will probably think of Kubernetes. This is the tremendous power of symbols³. That is why there are fields of study for Symbology and movies with famous actors who are playing characters like Robert Langdon 😉.

Now, if we look at the logos of both projects (Figure I), the maintainers did a great job in choosing a Carrier Pigeon⁴ for Fluentd and a Hummingbird⁵ for Fluent Bit. Fluentd supports an extensive library of over 1000 ⁶ plugins to provide extended support and functionality for anything related to log and data management. A Carrier Pigeon is known for it’s impressive performance e.g. in terms of distance covered…and it carries out data 😃💡.

Hummingbirds in comparison to, are lightweight and PRETTY fast. They have wing-flapping rates between 12 and 80 beats per second (depending on their actual size). Fluent Bit is designed to perform fast and efficient with least amount of resource usage.

From the official Fluent Bit homepage:

A short comparison

Both projects are very strong and feature-rich and great articles already exist on “how to deploy” Fluentd to forward logs to e.g. vRealize Log Insight Cloud. I put some of them in the resource section below. The following table compares both projects in specific areas.

Source: Fluentd & Fluent Bit

Logging Primitives

Before we start deploying and leveraging Fluent Bit to read and forward logs to VMware’s vRealize Log Insight, I wanted to briefly explain or refer you to some of the logging primitives of Cloud Native Applications or Services itself as well as how Fluent Bit is and can be configured. In general the following applies:

Also, I found this good article during my research - Logging from Docker Containers to Elasticsearch with Fluent Bit - in which the author quoted the following statement from the Twelve-Factor App⁷ website regarding Logs.

Doesn’t it make absolutely sense?

Fluent Bit offers several Input plugins to collect logs and data from sources. From a high-level perspective, the structure of Fluent Bit looks as follows :

To structure the collected data, Fluent Bit provides Parsers which can be used to convert the collected data into a structured format. Also, Filters are configurable to e.g. append additional information or to filter out unwanted log statements. Links to even more details can also be found in the resource section.

Prerequisites

To get Fluent Bit in working state on your Kubernetes cluster, we do need a ServiceAccount with an assigned ClusterRole as well as with it’s appropriate ClusterRoleBinding. We will deploy Fluent Bit as a Kubernetes DaemonSet⁸ in a Namespace called fluent-bit. I’ve created the following repository on - rguske/fluent-bit-vmware-loginsight - which you can clone, make the appropriate adjustments to the files and apply everything easily onto your cluster. It contains the following four files which sequence of deplyoment can be identified based on the number at the beginning of the filename (except #4).

• 1-tkg-fluent-bit-preps.yaml
• 2-tkg-fluent-bit-configmap-cri.yaml
• 3-tkg-fluent-bit-ds.yaml
• 4-tkg-fluent-bit-configmap-docker.yaml

Repository

1. Clone the repository locally and change into the directory fluent-bit-vmware-loginsight: git clone https://github.com/rguske/fluent-bit-vmware-loginsight.git && cd fluent-bit-vmware-loginsight

Adjustments

2. Make your adjustments to the following files to meet your requirements and environment specifications:

• 2-tkg-fluent-bit-configmap-cri.yaml
• 3-tkg-fluent-bit-ds.yaml

Containerd as Runtime - CRI Parser used!

As mentioned at the beginning, I’m using a TKG cluster for my example respectively in my environment. VMware is using Containerd⁹ as it’s container runtime and therefore I had to adjust my configuration accordingly. If you are using Docker as your runtime on Kubernetes, use the file 4-tkg-fluent-bit-configmap-docker.yaml as an alternative.

Fluent Bit Configmap

The first adjustment you have to make is the replacement of your TKG clustername as well as the hostname of your vRealize Log Insight appliance.

File 2-tkg-fluent-bit-configmap-cri.yaml:

• Replace the data for your TKG cluster in the filter-record.conf: | section
  • Note: for non TKG cluster, just delete the record

filter-record.conf: |

1
2
3
4
5

[FILTER]
    Name                record_modifier
    Match               *
    Record tkg_instance tkg-veba
    Record tkg_cluster  tkg-veba

Enter your Log Insight server hostname in the output-syslog.conf section:

output-syslog.conf: |

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16

[OUTPUT]
    Name                 syslog
    Match                *
    Host                 vrli.jarvis.lab
    Port                 514
    Mode                 tcp
    Syslog_Format        rfc5424
    Syslog_Hostname_key  tkg_cluster
    Syslog_Appname_key   pod_name
    Syslog_Procid_key    container_name
    Syslog_Message_key   message
    syslog_msgid_key     msgid
    Syslog_SD_key        k8s
    Syslog_SD_key        labels
    Syslog_SD_key        annotations
    Syslog_SD_key        tkg

Fluent Bit DaemonSet

Basically, it’s not necessary to change the image for Fluent Bit but you could replace it with the one of your choice. I used the official Fluent Bit image which is at the time of writing this post version 1.6.9 - fluent/fluent-bit:1.6.9. You could also use e.g. the VMware Fluent Bit Image in version 1.5.3: registry.tkg.vmware.run/fluent-bit:v1.5.3_vmware.1. I’ve validated the functionality of both images during my tests.

More images:

• Official: https://hub.docker.com/r/fluent/fluent-bit
• Bitnami: https://hub.docker.com/r/bitnami/fluent-bit
• VMware Registry: registry.tkg.vmware.run/fluent-bit:v1.5.3_vmware.1

File 3-tkg-fluent-bit-ds.yaml:

• Make your edits at the spec section for the container:

1
2
3
4
5
6
7
8

[...]

spec:
  containers:
  - name: fluent-bit
    image: fluent/fluent-bit:1.6.9

[...]

Deploy Fluent Bit on Kubernetes

3. Start with applying the 1-tkg-fluent-bit-preps.yaml file to have the prerequisites (Namespace, ServiceAccount, etc.) done:

1
2
3
4
5
6

$ kubectl create -f 1-tkg-fluent-bit-preps.yaml

namespace/fluent-bit created
serviceaccount/fluent-bit created
clusterrole.rbac.authorization.k8s.io/fluent-bit-read created
clusterrolebinding.rbac.authorization.k8s.io/fluent-bit-read created

4. Next is to apply the configmap - 2-tkg-fluent-bit-configmap-cri.yaml - for Fluent Bit:

1
2

kubectl create -f 2-tkg-fluent-bit-configmap-cri.yaml
configmap/fluent-bit-config created

1. As mentioned in the Prerequisites section before, Fluent Bit will be deployed as a DaemonSet by using the 3-tkg-fluent-bit-ds.yaml file:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

kubectl create -f 3-tkg-fluent-bit-ds.yaml && kubectl wait pod --timeout=-1s --for=condition=Ready -l '!job-name' -n fluent-bit
daemonset.apps/fluent-bit created
pod/fluent-bit-dd4ql condition met

kubectl get daemonsets.apps -n fluent-bit

NAME         DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
fluent-bit   2         2         2       2            2           <none>          95s

kubectl get pods -n fluent-bit
NAME               READY   STATUS    RESTARTS   AGE
fluent-bit-9m7qq   1/1     Running   0          108s
fluent-bit-dd4ql   1/1     Running   0          108s

More information about the successful deployment and as a first indication that everything is working fine can be grabbed from the logs of one of the pods.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20

kubectl logs -n fluent-bit fluent-bit-dd4ql

Fluent Bit v1.6.9
* Copyright (C) 2019-2020 The Fluent Bit Authors
* Copyright (C) 2015-2018 Treasure Data
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io

[2021/01/12 14:05:10] [ info] [engine] started (pid=1)
[2021/01/12 14:05:10] [ info] [storage] version=1.0.6, initializing...
[2021/01/12 14:05:10] [ info] [storage] in-memory
[2021/01/12 14:05:10] [ info] [storage] normal synchronization mode, checksum disabled, max_chunks_up=128
[2021/01/12 14:05:11] [ info] [input:systemd:systemd.1] seek_cursor=s=f34691953f354fc19dbd06139503c160;i=156... OK
[2021/01/12 14:05:11] [ info] [filter:kubernetes:kubernetes.0] https=1 host=kubernetes.default.svc port=443
[2021/01/12 14:05:11] [ info] [filter:kubernetes:kubernetes.0] local POD info OK
[2021/01/12 14:05:11] [ info] [filter:kubernetes:kubernetes.0] testing connectivity with API server...
[2021/01/12 14:05:11] [ info] [filter:kubernetes:kubernetes.0] API server connectivity OK
[2021/01/12 14:05:11] [ info] [output:syslog:syslog.0] setup done for vrli.jarvis.lab:514
[2021/01/12 14:05:11] [ info] [http_server] listen iface=0.0.0.0 tcp_port=2020
[2021/01/12 14:05:11] [ info] [sp] stream processor started

What vRealize Log Insight shows

The log output from the Fluent Bit pods itself looks promising. Now let’s check what vRLI has to offer. Login and check the Hosts section under Administration:

My TKG cluster tkg-veba is showing up - nice!

The cluster hostname is not showing up anymore - so this isn’t applicable.

Now switching to Interactive Analytics to finally see what Fluent Bit is providing us and if our configuration is working as expected. I’ve filtered for hostname contains tkg-veba:

Monitoring Function Invocation Count

Coming to the end of this post, I’ve created a simple Dashboard which is showing the invocation count of two functions. Both functions were invoked on a DrsVmPoweredOn and DrsVmPoweredOff event. This is how the log output looks like for e.g. the PowerCLI tagging function and the Python restpost-fn.

Figure VI below is showing the dashboard and the invocation count of both functions based on the incoming logs delivered by Fluent Bit.

Stay healthy and stay safe!

Resources

• Fluentd Log Insight Plugin
• Configure log forwarding from Tanzu Kubernetes Grid (TKG) to vRealize Log Insight Cloud
• Introducing vRealize LogInsight Cloud Helm Chart for Kubernetes Logs
• Implementing Log Forwarding with Fluent Bit
• Fluent Bit - Configuration File/ Configmap
• Fluent Bit - Input Plugins
• Fluent Bit - Input Plugins Github
• Fluent Bit - Output Plugins
• Fluent Bit Log Level
• vRealize Log Insight - Ports
• Supported RFC’s LogInsight
• CRI Website