Leveraging Fluent Bit on Tanzu Kubernetes Grid to send VEBA logs to vRealize Log Insight
With the recent v0.5 release of the VMware Event Broker Appliance (VEBA) project, the ability of deploying the core component, the VMware Event Router, via a Helm chart to an existing Kubernetes cluster was introduced. William Lam introduced this and all the other great new features and enhancements in his corresponding blog post. This earned very positive feedback from the community and opens even more doors in terms of flexibility as well as extensibility.
Speaking of which, extensibility…
During the v0.5 pre-release team meeting, William came up with the idea to let VEBA sending logs to our log analysis solutions VMware vRealize Log Insight or vRealize Log Insight Cloud. He asked me if I would do it and of course I found it a great idea. This post is a good addition to my previous post “Monitoring the VMware Event Broker Appliance with vRealize Operations Manager”1, in which I’m describing how I used cAdvisor2 to get resource usage and performance data from the various VEBA components and ultimately to monitor them.
At this point, I have to be clear that I didn’t used the VEBA appliance this time. Instead, I used Helm charts to deploy every component like e.g. the mentioned Event Router or the Event Processor to my Tanzu Kubernetes Grid - TKGs cluster (also referred to as Guestcluster) running on VMware vSphere. However, I’m considering to write a small follow-up article which will cover the Appliance variant.
The Open Source projects Fluentd and Fluent Bit can be leveraged for multi-platform log processing and log forwarding. For my use case with VEBA, my requirements on the solution were basically simple. It should be lightweight, fast and if possible, with few or no dependencies. Comparing both mentioned projects with each other, it brought me multiple times back to Fluent Bit, even besides the fact, that good content on how to integrate Fluentd with TKG already exist.
The Hummingbird (Note: no technical relevance)
The power of symbols thrills me. It’s that power that makes you think or remember a specific brand, a company or an event. Just think about the Kubernetes steering wheel for example. If you feel home in the cloud native space and you see a steering wheel in real life nowadays, you will probably think of Kubernetes. This is the tremendous power of symbols3. That is why there are fields of study for Symbology and movies with famous actors who are playing characters like Robert Langdon 😉.
Now, if we look at the logos of both projects (Figure I), the maintainers did a great job in choosing a Carrier Pigeon4 for Fluentd and a Hummingbird5 for Fluent Bit. Fluentd supports an extensive library of over 1000 6 plugins to provide extended support and functionality for anything related to log and data management. A Carrier Pigeon is known for it’s impressive performance e.g. in terms of distance covered…and it carries out data 😃💡.
Hummingbirds in comparison to, are lightweight and PRETTY fast. They have wing-flapping rates between 12 and 80 beats per second (depending on their actual size). Fluent Bit is designed to perform fast and efficient with least amount of resource usage.
From the official Fluent Bit homepage:
A short comparison
Both projects are very strong and feature-rich and great articles already exist on “how to deploy” Fluentd to forward logs to e.g. vRealize Log Insight Cloud. I put some of them in the resource section below. The following table compares both projects in specific areas.
|Scope||Containers / Servers||Embedded Linux / Containers / Servers|
|Language||C & Ruby||C|
|Performance||High Performance||High Performance|
|Dependencies||Built as a Ruby Gem, it requires a certain number of gems.||Zero dependencies, unless some special plugin requires them|
|Plugins||More than 1000 plugins available||Around 70 plugins available|
|License||Apache License v2.0||Apache License v2.0|
Source: Fluentd & Fluent Bit
Before we start deploying and leveraging Fluent Bit to read and forward logs to VMware’s vRealize Log Insight, I wanted to briefly explain or refer you to some of the logging primitives of Cloud Native Applications or Services itself as well as how Fluent Bit is and can be configured. In general the following applies:
By default, container engines such as Docker capture the standard output or error and leverage the JSON-file driver on each host to write messages to files. Docker maintains a separate log file for each container and stores it in the /var/log/containers directory of the Docker host. Annotation for each log entry consists of the following:
- Log message
- Message origin - stdout or stderr
Also, I found this good article during my research - Logging from Docker Containers to Elasticsearch with Fluent Bit - in which the author quoted the following statement from the Twelve-Factor App7 website regarding
Doesn’t it make absolutely sense?
Fluent Bit offers several
Input plugins to collect logs and data from sources. From a high-level perspective, the structure of Fluent Bit looks as follows :
To structure the collected data, Fluent Bit provides
Parsers which can be used to convert the collected data into a structured format. Also,
Filters are configurable to e.g. append additional information or to filter out unwanted log statements. Links to even more details can also be found in the resource section.
To get Fluent Bit in working state on your Kubernetes cluster, we do need a
ServiceAccount with an assigned
ClusterRole as well as with it’s appropriate
ClusterRoleBinding. We will deploy Fluent Bit as a Kubernetes DaemonSet8 in a
fluent-bit. I’ve created the following repository on - rguske/fluent-bit-vmware-loginsight - which you can
clone, make the appropriate adjustments to the files and apply everything easily onto your cluster. It contains the following four files which sequence of deplyoment can be identified based on the number at the beginning of the filename (except #4).
- Clone the repository locally and change into the directory
git clone https://github.com/rguske/fluent-bit-vmware-loginsight.git && cd fluent-bit-vmware-loginsight
- Make your adjustments to the following files to meet your requirements and environment specifications:
Containerd as Runtime - CRI Parser used!
As mentioned at the beginning, I’m using a TKG cluster for my example respectively in my environment. VMware is using Containerd9 as it’s container runtime and therefore I had to adjust my configuration accordingly. If you are using Docker as your runtime on Kubernetes, use the file
4-tkg-fluent-bit-configmap-docker.yaml as an alternative.
Fluent Bit Configmap
The first adjustment you have to make is the replacement of your TKG clustername as well as the hostname of your vRealize Log Insight appliance.
- Replace the data for your TKG cluster in the
- Note: for non TKG cluster, just delete the record
Enter your Log Insight server hostname in the
Fluent Bit DaemonSet
Basically, it’s not necessary to change the image for Fluent Bit but you could replace it with the one of your choice. I used the official Fluent Bit image which is at the time of writing this post version 1.6.9 -
fluent/fluent-bit:1.6.9. You could also use e.g. the VMware Fluent Bit Image in version 1.5.3:
registry.tkg.vmware.run/fluent-bit:v1.5.3_vmware.1. I’ve validated the functionality of both images during my tests.
- Official: https://hub.docker.com/r/fluent/fluent-bit
- Bitnami: https://hub.docker.com/r/bitnami/fluent-bit
- VMware Registry: registry.tkg.vmware.run/fluent-bit:v1.5.3_vmware.1
- Make your edits at the
specsection for the container:
Deploy Fluent Bit on Kubernetes
- Start with applying the
1-tkg-fluent-bit-preps.yamlfile to have the prerequisites (
ServiceAccount, etc.) done:
- Next is to apply the configmap -
2-tkg-fluent-bit-configmap-cri.yaml- for Fluent Bit:
- As mentioned in the Prerequisites section before, Fluent Bit will be deployed as a DaemonSet by using the
More information about the successful deployment and as a first indication that everything is working fine can be grabbed from the logs of one of the pods.
What vRealize Log Insight shows
The log output from the Fluent Bit pods itself looks promising. Now let’s check what vRLI has to offer. Login and check the
Hosts section under
My TKG cluster
tkg-veba is showing up - nice!
tkg_clusterat some point in the Configmap which resulted in the following misbehavior - Figure III.
The cluster hostname is not showing up anymore - so this isn’t applicable.
Now switching to
Interactive Analytics to finally see what Fluent Bit is providing us and if our configuration is working as expected. I’ve filtered for
hostname contains tkg-veba:
Monitoring Function Invocation Count
Coming to the end of this post, I’ve created a simple Dashboard which is showing the invocation count of two functions. Both functions were invoked on a
DrsVmPoweredOff event. This is how the log output looks like for e.g. the PowerCLI tagging function and the Python restpost-fn.
Figure VI below is showing the dashboard and the invocation count of both functions based on the incoming logs delivered by Fluent Bit.
Stay healthy and stay safe!
- Fluentd Log Insight Plugin
- Configure log forwarding from Tanzu Kubernetes Grid (TKG) to vRealize Log Insight Cloud
- Introducing vRealize LogInsight Cloud Helm Chart for Kubernetes Logs
- Implementing Log Forwarding with Fluent Bit
- Fluent Bit - Configuration File/ Configmap
- Fluent Bit - Input Plugins
- Fluent Bit - Input Plugins Github
- Fluent Bit - Output Plugins
- Fluent Bit Log Level
- vRealize Log Insight - Ports
- Supported RFC’s LogInsight
- CRI Website