Store Prometheus Metrics with Thanos, Azure Storage and Azure Kubernetes Service (AKS)
Published Jan 20 2022 05:52 PM 8,604 Views
Frequent Visitor

by Lucas Santos and Aaron Wislang

 

In our previous article we explored how to deploy Grafana and Prometheus on AKS. However, despite Prometheus being an excellent way to monitor your AKS cluster, it lacks some functionality to save metrics for future use and enable us to query historical data.

 

Prometheus is a metrics collector, it can be used to collect metrics from any source, and it can be used to generate graphs. However, it is not a data storage solution and therefore isn't not focused on storing data for historical purposes. Prometheus has its storage solution and supports both on-disk storage as well as remote locations. You can configure a lot of options but the data management options are limited. This is where Thanos can help.

 

There is also a configuration overhead when we want to scale our Prometheus deployment and make it highly available. This generally includes the use of a federated set-up, and the use of a shared storage solution which, in Kubernetes, is usually accompised with a shared PersistentVolume.

 

What is Thanos?

Thanos is an open source Prometheus setup with long term storage capabilities. It is not a new implementation of Prometheus, but a pre-built setup that has been designed to be used in production environments when long term storage is needed.

 

Storing metrics data for long term use requires it to be stored in a way that is optimized for that use. Long term storage may require unlimited retention periods, with ever-growing storage requirements.

 

We will need to address two issues. Firstly, unlimited data requires storage that will scale accordingly. Secondly, the more data we have the slower the querying will become, and for this downsampling and compaction techniques are used to reduce the size of the data and improve query times.

 

Thanos provides all of this and more out of the box in a single binary. You also don’t need to install all of those features, and you can have a subset of Thanos running in your cluster.

 

Thanos currently depends on Prometheus v2.2.1+ and an optional object storage if you want to store your data in a remote location. Several supported object storage clients are available. We will be using Azure Blob Storage.

 

Thanos Architecture

 

Thanos is composed of several components:

 

  • Thanos Sidecar: The sidecar runs alongside Prometheus server to gather the metrics that are stored on disk. It’s composed of a StoreAPI and a Shipper, the shipper is responsible for sending the metrics to the object storage.

  • Thanos Store Gateway: This component is responsible for querying the object storage and exposing a StoreAPI that is queried by the other components.

  • Query Layer: Provides all the components required to query the Data, including the Web UI and the API.

  • Compactor: Reads from object storage and compacts the data that’s not compacted yet. It’s completely independent of the other components.

  • Ruler: Provides the ruler API that is used to evaluate rules and alerts from the Prometheus Alertmanager.

The overall architecture can be described as follows (via: Thanos Quick Tutorial).

Thanos ArchitectureThanos Architecture

Thanos uses a mix of HTTP and gRPC requests. HTTP requests are mostly used to query Prometheus, whilst gRPC requests are mostly used within Thanos' Store API.

 

We'll be deploying our metrics server to an Azure Kubernetes Service (AKS) cluster. If you don’t have a running AKS cluster, take a look at the quickstart to Deploy an Azure Kubernetes Service cluster using the Azure CLI.

 

We’ll also need an Azure Storage Account, you can create one using the Azure Portal or the Azure CLI. You will also need the storage account access key which can also be retrieved using the Azure CLI.

 

Create a storage account.

az storage account create --name <name> --resource-group <resource-group>

Create a storage container called metrics.

az storage container create --name metrics --account-name <name>

Retrieve the storage account access key for later use.

az storage account keys list --account-name <name> --resource-group <resource-group> -o tsv --query "[0].value"

Thanos is designed to scale and extend vanilla Prometheus. Start by creating the Prometheus configuration that you will use to deploy the kube-prometheus-stack.

 

First, create a file called prometheus.yaml.

 

# prometheus.yaml 
grafana:
  adminPassword: admin 
   
prometheus:
  thanosService:
    enabled: true
   
  thanosServiceMonitor:
    enabled: true
    interval: 5s
   
  prometheusSpec:
    thanos:
      objectStorageConfig:
        key: thanos.yaml
        name: thanos-objstore-config
   
prometheusOperator:
  thanosImage:
    repository: quay.io/thanos/thanos
    version: v0.23.0
    tag: v0.23.0
   
kubelet:
  serviceMonitor:
    https: false

 

 

In this file, we’re first configuring the Prometheus Operator to use the Thanos image, and we’re also configuring the Grafana to use a password for the admin user (this is optional, but the default password is prom-operator). We’re also enabling the Thanos service and monitor (for Thanos metrics).

 

The thanos key is the configuration object for the storage object, the remote configuration that Thanos will use to upload the metrics to. In our case, this is the Azure Storage account that we created earlier. Thanos configuration for Azure object can be found here, but we’ll only need a handful of options.

 

Create a thanos.yaml file locally.

 

# thanos.yaml
type: AZURE
config:
  storage_account: '<storage-account-name>'
  storage_account_key: '<storage-account-key>'
  container: 'metrics'

 

 

Replace <storage-account-name> with your storage account name and <storage-account-key> with the storage account access key you retrieved earlier.

 

Make sure you are authenticated to your Azure Kubernetes Service cluster (e.g. via az aks get-credentials) before running the following kubectl commands below.

 

Create a new namespace called monitoring.

kubectl create ns monitoring

Make sure you are in the same directory as thanos.yaml, then create a secret called thanos-objstore-config.

kubectl -n monitoring create secret generic thanos-objstore-config --from-file=thanos.yaml=thanos.yaml

Add the helm repo for the Prometheus Community Helm Charts

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update

Run the helm installation for the Prometheus Operator.

helm upgrade --install prometheus prometheus-community/kube-prometheus-stack -n monitoring --values prometheus.yaml

After a while, you’ll see that the Prometheus Operator is installing the Prometheus and Thanos components.

Check out the result by querying the Prometheus.

kubectl --namespace monitoring get pods

You’ll see that there are three containers inside the prometheus-prometheus-kube-prometheus-prometheus-0 pod, one of them is thanos-sidecar. By default it will back up all the blocks that prometheus generates to our Azure Storage Account every two hours.

 

Other components

 

Installing the sidecar is the first step to create a complete Thanos deployment. It’s important to install both the Querier, and the Compactor along with the Store for a complete solution. Each of these components is installed as a separate deployment with its own network configuration.

 

In the following section you will create multiple kubernetes manifests. After you create each file, apply it to your cluster using the kubectl apply command (e.g. kubectl -f FILENAME.yaml).

 

Querier

 

The Querier is the layer that will allow us to query all Prometheus instances at once. It needs a Deployment that will be pointed to all sidecars, and it also needs its own Service to be able to be discovered and used.

 

Create the querier Deployment (don't forget to run kubectl apply -f querier-deployment.yaml).

 

 

# querier-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: thanos-query
  namespace: monitoring
  labels:
    app: thanos-query
spec:
  replicas: 1
  selector:
    matchLabels:
      app: thanos-query
  template:
    metadata:
      labels:
        app: thanos-query
    spec:
      containers:
        - name: thanos-query
          image: quay.io/thanos/thanos:v0.23.0
          args:
            - 'query'
            - '--log.level=debug'
            - '--query.replica-label=prometheus_replica'
            - '--store=prometheus-kube-prometheus-thanos-discovery.monitoring.svc:10901'
          resources:
            requests:
              cpu: '100m'
              memory: '64Mi'
            limits:
              cpu: '250m'
              memory: '256Mi'
          ports:
            - name: http
              containerPort: 10902
            - name: grpc
              containerPort: 10901
            - name: cluster
              containerPort: 10900

 

 

Note that the image name and version must match the image name and version we deployed on the helm chart above. When we activated the thanosService option, we created a new service discovery network that would allow us to query all the Prometheus instances in the cluster from a single point, this is the option we used to configure the Querier with.

 

We also need the Service that will be used to expose the Querier and the metrics about that component so we can also monitor it using a ServiceMonitor. These can be can split across multiple files, or delimited with --- in the same file as we have below.

 

Create the Service and ServiceMonitor.

 

# querier-service-servicemonitor.yaml
apiVersion: v1
kind: Service
metadata:
  name: thanos-query
  labels:
    app: thanos-query
    release: prometheus-operator
    jobLabel: thanos
  namespace: monitoring
spec:
  selector:
    app: thanos-query
  ports:
    - port: 9090
      protocol: TCP
      targetPort: http
      name: http-query
---
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: prom-thanos-query
  namespace: monitoring
spec:
  jobLabel: thanos
  selector:
    matchLabels:
      app: thanos-query
  namespaceSelector:
    matchNames:
      - 'monitoring'
  endpoints:
    - port: http-query
      path: /metrics
      interval: 5s

 

 

This is telling prometheus to scrape the metrics from the path /metrics at port 10902 from the Querier application. Lastly, we add the ServiceMonitor to monitor our Querier.

 

Upon running the above, you’ll see that the Querier is being installed.

 

Take a look at the logs for that deployment.

kubectl -n monitoring logs deploy/thanos-query

You should see output as follows:

level=debug ts=2022-01-01T19:56:05.927270012Z caller=main.go:65 msg="maxprocs: Updating GOMAXPROCS=[1]: using minimum allowed GOMAXPROCS"
ts=2022-01-01T19:56:05.928679219Z caller=log.go:168 level=debug msg="Lookback delta is zero, setting to default value" value=5m0s
level=info ts=2022-01-01T19:56:05.932400335Z caller=options.go:27 protocol=gRPC msg="disabled TLS, key and cert must be set to enable"
level=info ts=2022-01-01T19:56:05.93349084Z caller=query.go:618 msg="starting query node"
level=debug ts=2022-01-01T19:56:05.933743641Z caller=endpointset.go:320 component=endpointset msg="starting to update API endpoints" cachedEndpoints=0
level=debug ts=2022-01-01T19:56:05.933774441Z caller=endpointset.go:323 component=endpointset msg="checked requested endpoints" activeEndpoints=0 cachedEndpoints=0
level=info ts=2022-01-01T19:56:05.933864742Z caller=intrumentation.go:60 msg="changing probe status" status=healthy
level=info ts=2022-01-01T19:56:05.933887542Z caller=http.go:63 service=http/server component=query msg="listening for requests and metrics" address=0.0.0.0:10902
ts=2022-01-01T19:56:05.934207943Z caller=log.go:168 service=http/server component=query level=info msg="TLS is disabled." http2=false
level=info ts=2022-01-01T19:56:05.934254744Z caller=intrumentation.go:48 msg="changing probe status" status=ready
level=info ts=2022-01-01T19:56:05.934281344Z caller=grpc.go:127 service=gRPC/server component=query msg="listening for serving gRPC" address=0.0.0.0:10901

 

Store

The store runs along with the Querier to bring the data from our Object Storage to the queries. It’s composed of a StatefulSet, and a configuration that contains the configuration for the Store, which we previously created as a secret.

 

Create a StatefulSet.

 

# store-statefulset.yaml
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: thanos-store
  namespace: monitoring
  labels:
    app: thanos-store
spec:
  serviceName: 'thanos-store'
  replicas: 1
  selector:
    matchLabels:
      app: thanos-store
  template:
    metadata:
      labels:
        app: thanos-store
    spec:
      containers:
        - name: thanos-store
          image: quay.io/thanos/thanos:v0.23.0
          args:
            - 'store'
            - '--log.level=debug'
            - '--data-dir=/var/thanos/store'
            - '--objstore.config-file=/config/thanos.yaml'
          ports:
            - name: http
              containerPort: 10902
            - name: grpc
              containerPort: 10901
            - name: cluster
              containerPort: 10900
          volumeMounts:
            - name: config
              mountPath: /config/
              readOnly: true
            - name: data
              mountPath: /var/thanos/store
      volumes:
        - name: data
          emptyDir: {}
        - name: config
          secret:
            secretName: thanos-objstore-config

 

 

Create a ServiceMonitor.

 

# store-servicemonitor.yaml
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: thanos-store
  namespace: monitoring
  labels:
    release: prom-op
spec:
  jobLabel: thanos
  endpoints:
    - port: http
      path: /metrics
      interval: 30s
  selector:
    matchLabels:
      app: thanos-store

 

Compactor

 

The compactor is the service that will downsample the historical data. It’s recommended when you have a lot of incoming data in order to reduce the storage requirements. Just like the Querier component, it is composed of a StatefulSet and a Service. It takes configurations like the Store.

 

Create the StatefulSet.

 

# compactor-statefulset.yaml
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: thanos-compactor
  namespace: monitoring
  labels:
    app: thanos-compactor
spec:
  serviceName: 'thanos-compactor'
  replicas: 1
  selector:
    matchLabels:
      app: thanos-compactor
  template:
    metadata:
      labels:
        app: thanos-compactor
    spec:
      containers:
        - name: thanos-compactor
          image: quay.io/thanos/thanos:v0.23.0
          args:
            - 'compact'
            - '--log.level=debug'
            - '--data-dir=/var/thanos/store'
            - '--objstore.config-file=/config/thanos.yaml'
            - '--wait'
          ports:
            - name: http
              containerPort: 10902
          volumeMounts:
            - name: config
              mountPath: /config/
              readOnly: true
            - name: data
              mountPath: /var/thanos/store
      volumes:
        - name: data
          emptyDir: {}
        - name: config
          secret:
            secretName: thanos-objstore-config

 

 

Create the Service and the ServiceMonitor.

 

# compactor-service-servicemonitor.yaml
apiVersion: v1
kind: Service
metadata:
  name: thanos-compactor
  labels:
    app: thanos-compactor
  namespace: monitoring
spec:
  selector:
    app: thanos-compactor
  ports:
    - port: 10902
      name: http
---
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: thanos-compactor
  namespace: monitoring
  labels:
    release: prom-op
spec:
  jobLabel: thanos
  endpoints:
    - port: http
      path: /metrics
      interval: 30s
  selector:
    matchLabels:
      app: thanos-compactor

 

Querying Thanos

 

To query Thanos using the stack we installed, you need to port-forward your Grafana to your computer.

kubectl port-forward svc/prometheus-grafana 3000:80 -n monitoring

 

Open your browser at localhost:3000 and login with username admin and password admin.

 

Open the "Configurations" menu on the left hand side and click "Data sources" then click "Add data source".

 

Select "Prometheus" under "Time series databases". Name it "Prometheus Thanos" and set the "HTTP URL" to http://thanos-query:9090 and click "Save & test".

 

Congratulations! You have successfully deployed Thanos to your Azure Kubernetes Service cluster, with data stored in Azure Blob Storage.

 

You can now click on "Explore" on the left hand-side and select "Prometheus Thanos" from the drop-down menu to submit PromQL queries.

 

Additionally, select the "Prometheus Thanos" data source when using Dashboards such as Kubernetes API Server (Dashboards > Browse > General / Kubernetes / API Server) below:

 

Dashboard for Kubernetes API Server in GrafanaDashboard for Kubernetes API Server in Grafana

 

Don't forget to delete your cluster, and storage account, with the az group delete command to avoid any ongoing charges.

 

3 Comments
Co-Authors
Version history
Last update:
‎Jan 20 2022 06:00 PM
Updated by:
www.000webhost.com