Hi there 馃憢

This blog is a sort “knowledge stash”. A log of my journey in the industry. You will find posts about Kubernetes, Docker, Spring Boot, Kafka, CI/CD Pipelines and other stuff. Writing things down helps memorizing, offloads brain capacity and can help others.

How to send logs to multiple outputs with Fluentd on Kubernetes (and how to avoid Ruby gems incompatibility)

In this post I will show you how to send Kubernetes logs to multiple outputs by using Fluentd but first, let鈥檚 do a recap. On the previous post I wrote about using Fluentd and FluentBit; in particular, I showed you how to use FluentBit for log collecting and forwarding and Fluentd for pushing these logs to a destination: Opensearch. Since Opensearch at the time didn鈥檛 have a Fluentd plugin, we had to craft our Dockerfile tailored for our use case by installing Ruby Gems and by specifing the latest Elasticsearch gem compatible with Opensearch....

October 22, 2022 路 8 min 路 Justin

Achieving multi-tenant metrics with Prometheus Agent: a first building block

This is the first post of a series of articles about multi-tenant metrics with Prometheus Agent; here you will read how to enable Prometheus Agent mode and how to ingest metrics to a central Prometheus instance but also the limitations of this setup. Let鈥檚 say you鈥檙e managing 5 - 10 to n Kubernetes clusters (or hopefully Vclusters) for your tenants and you want to provide metrics to the developers: cpu usage, memory usage, custom metrics etc....

July 25, 2022 路 7 min 路 Justin

LDAP authentication with Dex on Kubernetes with Vcluster (K3S)

We set up our Vcluster but now we want to give proper access to the developers by leveraging their existing ldap credentials. The idea is to use DEX as a federated openid provider and kubelogin as a plugin for oidc integration. Scenario We鈥檙e going to achieve something like this: the user issues a command targeting our vcluster鈥檚 api server (kubectl get pods, for example), kubelogin will open the default browser on the user鈥檚 machine and display a login page....

June 11, 2022 路 8 min 路 Justin

How we reduced Kubernetes Clusters Sprawl by adopting Vclusters: An Introduction

With the consistent increasing of projects we were experiencing a huge growth of Kubernetes clusters for dev and test environments; each project had its own cluster with its stack: Nginx, Prometheus, Opa Gatekeeper etc. Moreover, each cluster had a different Kubernetes version with legacy clusters still pinned to the 1.15.x version. This meant we had to manage a different set of stacks (Prometheus, Nginx etc) since the skew between versions were too big....

April 24, 2022 路 9 min 路 Justin

Secure Your Docker Images With Cosign (and OPA Gatekeeper)

We built CI/CD pipelines so far which have Docker images as output but how we make sure about the provenance of the workload we run on Kubernetes? How can be sure that the containers we are running are run from images built from our pipelines? One way to ensure trust with Docker images is to sign these images. We can sign them during our CI pipeline and then verify the signature at runtime when deploying....

January 19, 2022 路 9 min 路 Justin

Centralized (multi tenant) Logging with Kubernetes: Part 3

This is a series. You can find part 1 here and part 2 here. On this last post I am going to cover the monitoring (with Prometheus and Grafana) for OpenSearch and Fluentd. FluentD If you recall from part-1, we set up a specific configuration for Prometheus in Fluentd main-fluentd-conf.yaml kind: ConfigMap apiVersion: v1 metadata: name: fluentd-es-config namespace: logging labels: addonmanager.kubernetes.io/mode: Reconcile data: fluent.conf: |-<source> type forward bind 0.0.0.0 port 32000 </source> [....

December 12, 2021 路 3 min 路 Justin

Centralized (multi tenant) Logging with Kubernetes: Part 2

This is a series. You can find part 1 here and part 3 here. We were left on part-1 with a fully functional log pipeline. As I said, we鈥檙e going to check how to achieve a single shared index in OpenSearch. Shared Index If you recall, on the previous part we set a field on each document we were indexing in OpenSearch by leveraging the FluentD record_transformer plugin. main-fluentd-conf.yaml kind: ConfigMap apiVersion: v1 metadata: name: fluentd-es-config namespace: logging labels: addonmanager....

December 8, 2021 路 5 min 路 Justin

Kafka Producer Timing

In this post I want to summarize some important Kafka producer鈥檚 configurations that usually are not taken in consideration when dealing with Kafka. max.block.ms: This timeout controls how long the producer may block when entering the send() method. This configuration provides an upper bound timeout for time spent waiting for metadata from the broker. Hence this timeout can be triggered when the producer鈥檚 send buffer is full or when topic metadata is not available....

December 4, 2021 路 3 min 路 Justin

How I build and deploy applications with Tekton CI and ArgoCD

I have been using Tekton since the last year and it amazed me for how much is easy to bootstrap CI pipelines with it. I will not go deep about how Tekton works (take a look at the documentation here ), but it鈥檚 important to mention that it executes your CI code in isolated Docker containers: for example, if your pipeline is composed by the steps 鈥済it-clone鈥 and 鈥渄ocker-build鈥, each of these steps will execute its code inside a container, so for the first step we can use the alpine/git Docker image and for the docker-build we can use gcr....

November 27, 2021 路 13 min 路 Justin

Centralized (multi tenant) Logging with Kubernetes: Part 1

This is a series. You can find part 2 here and part 3 here. Managing multiple Kubernetes clusters is not so easy; even more managing the logs that are produced from these clusters. The architecture that I want to show you is still a WIP but on the right track. Let鈥檚 start from this scenario: 15 Kubernetes clusters (that we will call Tenants) where Spring Boot based microservices are running. We need to provide to the developers a central logging dashboard where they can navigate and correlate logs; in this case we will use OpenSearch (formerly known as Open Distro for ElasticSearch)....

November 21, 2021 路 8 min 路 Justin