opensearch

In this post I will show you how to send Kubernetes logs to multiple outputs by using Fluentd but first, let’s do a recap. On the previous post I wrote about using Fluentd and FluentBit; in particular, I showed you how to use FluentBit for log collecting and forwarding and Fluentd for pushing these logs to a destination: Opensearch. Since Opensearch at the time didn’t have a Fluentd plugin, we had to craft our Dockerfile tailored for our use case by installing Ruby Gems and by specifing the latest Elasticsearch gem compatible with Opensearch....

This is a series. You can find part 1 here and part 2 here. On this last post I am going to cover the monitoring (with Prometheus and Grafana) for OpenSearch and Fluentd. FluentD If you recall from part-1, we set up a specific configuration for Prometheus in Fluentd main-fluentd-conf.yaml kind: ConfigMap apiVersion: v1 metadata: name: fluentd-es-config namespace: logging labels: addonmanager.kubernetes.io/mode: Reconcile data: fluent.conf: |-<source> type forward bind 0.0.0.0 port 32000 </source> [....

This is a series. You can find part 1 here and part 3 here. We were left on part-1 with a fully functional log pipeline. As I said, we’re going to check how to achieve a single shared index in OpenSearch. Shared Index If you recall, on the previous part we set a field on each document we were indexing in OpenSearch by leveraging the FluentD record_transformer plugin. main-fluentd-conf.yaml kind: ConfigMap apiVersion: v1 metadata: name: fluentd-es-config namespace: logging labels: addonmanager....

This is a series. You can find part 2 here and part 3 here. Managing multiple Kubernetes clusters is not so easy; even more managing the logs that are produced from these clusters. The architecture that I want to show you is still a WIP but on the right track. Let’s start from this scenario: 15 Kubernetes clusters (that we will call Tenants) where Spring Boot based microservices are running. We need to provide to the developers a central logging dashboard where they can navigate and correlate logs; in this case we will use OpenSearch (formerly known as Open Distro for ElasticSearch)....