# Install Tempo

> [HTML Version](install-tempo.html)

Tempo is a high-performance and scalable system for storing and analyzing traces. It provides complete visibility into interactions between components of distributed systems. Tempo eliminates the complexities associated with storing and retrieving traces from numerous sources in real-time.

Beforehand, install the [monitoring tools](install-monitoring-tools.md).

The installation of Tempo consists of the following steps:

1. [Download the Helm chart and the configuration file](#download-helm-chart-and-config-file).

2. [Create buckets in Minio](#create-bucket).

3. [Fill in the configuration file](#fill-config-file).

4. [Install the Tempo chart using Helm in the Kubernetes cluster](#install-chart-tempo).

5. [Set the parameters for sending traces](#traces).

6. [Configure the connection to Tempo in Grafana data sources](#configure-connection).

````
Начало внимание

````
When installing [Loki](install-loki.md) + Tempo, start with the Loki addon. This allows preserving cross-linking settings, as they are initially configured in the Datasource settings.

````
Конец внимание

````
After installing the Tempo add‑on, you can link its traces to Loki logs. This will allow you to diagnose errors faster, as you can click a link from the log to access the details of the desired trace. If you use the Deckhouse platform, you can [connect Tempo to the built‑in Loki module](#loki).

## Step 1: Download the Helm chart and the configuration file

To install via the internet, obtain the configuration file \[OBJECT\] by executing the command:

````
helm repo add brix365 https://charts.brix365.com  
helm repo update  
helm show values brix365/tempo > values-tempo.yaml

````


Getting configuration file for installation in a closed environment without internet access

1. On a computer with internet access, download the archive of the latest version of the Tempo chart from the BRIX repository using the command:

````
helm repo add brix365 https://charts.brix365.com  
helm repo update  
helm pull brix365/tempo

2. ````
Copy the obtained \[OBJECT\] chart archive to the server where the installation will take place.

3. Unpack the chart and copy the default configuration file \[OBJECT\] to \[OBJECT\]

````
tar -xf tempo-X.Y.Z.tgz  
cp tempo/values.yaml values-tempo.yaml
````

## ````
Step 2: Create buckets in Minio

1. Create \[OBJECT\] for MinIO:

````
mc alias set my\_alias http://minio.local accessKey secretKey

````
2. Create a bucket with the name \[OBJECT\]:

````
mc mb -p my\_alias/tempo --region=ru-central-1

## ````
Step 3: Fill in the configuration file

Fill in the configuration file for Tempo installation

````
  
 tempo:

    tempo:

      retention: 24h

      storage:

        trace:

          # tempo storage backend

          # refer https://grafana.com/docs/tempo/latest/configuration/

          ## Use s3 for example

          backend: s3

          # store traces in s3

          s3:

            bucket: tempo                        # store traces in this bucket

            endpoint: minio.local:9000           # api endpoint

            access\_key: access\_key               # optional. access key when using static credentials.

            secret\_key: secret\_key               # optional. secret key when using static credentials.

            insecure: true                       # optional. enable if endpoint is http

          #backend: local

          local:

            path: /var/tempo/traces

          wal:

            path: /var/tempo/wal

  jaeger:

    enabled: false

    fullnameOverride: "jaeger"

    provisionDataStore:

        cassandra: false

    storage:

      type: badger

    collector:

      enabled: false

    agent:

      enabled: true

      extraEnv:

        MEMORY\_MAX\_TRACES: 100000

        SPAN\_STORAGE\_TYPE: badger

        BADGER\_EPHEMERAL: false

        BADGER\_DIRECTORY\_VALUE: /badger/data

        BADGER\_DIRECTORY\_KEY: /badger/key

        REPORTER\_TYPE: grpc

        REPORTER\_GRPC\_HOST\_PORT: tempo:14250

        REPORTER\_GRPC\_RETRY\_MAX: 100

````


Filling in connection parameters to a private registry for installation in a closed environment without internet access

  
To connect to a private registry, follow these steps:

1. Download BRIX images and upload them to the local image registry. For more details, refer to  [Download BRIX images](downloadin-images-elma365.md).

2. Specify the address and path for the \[OBJECT\] parameter.

3. Specify the name of the secret with access rights to the private registry in the \[OBJECT\] parameter. The secret must be created manually and encoded in Base64.

````
 tempo:  
 # connection parameters for the private registry  
 # address and path for the private registry  
   repository: registry.example.com/tempo  
 #   tag: "2.3.0"  
 #   pullPolicy: IfNotPresent  
   pullSecrets:  
     - myRegistryKeySecretName  
 jaeger-all-in-one:  
 # connection parameters for the private registry  
   image:  
 # address and path for the private registry  
     repository: registry.example.com/all-in-one  
 #     versionOverride: "1.47"  
 #     pullPolicy: IfNotPresent  
   imagePullSecrets:  
     - myRegistryKeySecretName

````
Where the repository** **format is: address — \[OBJECT\].

## Step 4: Install the Tempo chart using Helm in the Kubernetes cluster

Install Tempo chart in \[OBJECT\].

For online installation:

````
helm upgrade --install -n monitoring tempo brix365/tempo -f values-tempo.yaml

````
For offline installation without internet access, navigate to the directory with the downloaded chart and execute:

````
helm upgrade --install tempo ./tempo -f values-tempo.yaml -n monitoring

## ````
Step 5: Set the parameters for sending traces

To display traces, set the \[OBJECT\] or \[OBJECT\] parameters on the BRIX application side.

````
brix365:  
  global:  
    monitoring:  
      jaegerAgentAddress: "jaeger-agent.monitoring.svc.cluster.local:5775"  
      jaegerCollectorEndpoint: "http://tempo.monitoring.svc.cluster.local:14268/api/traces"

1. ````
The preferred method is to specify the HTTP endpoint of the collector using the \[OBJECT\] parameter. In this case, traces will be sent directly to the collection system, bypassing intermediate components.

2. An alternative option is to use Jaeger Agent, specifying its address in the \[OBJECT\] parameter. Traces are then transmitted to the agent via the UDP protocol, after which it forwards them to the storage system.

3. If none of the parameters are specified, traces are automatically sent to port \[OBJECT\] of the local node, where the Jaeger agent is expected to run in DaemonSet mode.

## Step 6: Configure the connection to Tempo in Grafana data sources

The Tempo data source is added during installation. Verify it using the path:  **Home > Administration > Data source > Tempo**.

URL: **http://tempo:3200**

**(clip0086.png)**

For convenient viewing, a cross-link to Tempo is added to the Loki data source.

````
(Query: \$\{\_\_value.raw\}; Regex: (?:trace\\.traceid|traceid)(?:":|=)"(\\w+)")

````
 **(clip0087.png)**

Configuring Derived Fields will facilitate viewing of the trace's reference connection in Tempo.

**(clip0088.png)**

## Link Tempo traces to the logs of the Deckhouse built-in Loki module 

When using the Deckhouse platform, you can collect logs and view them in the Grafana interface using its [built-in Loki module](install-loki.md#loki-module). Once you install Tempo, it will be added to the Grafana data sources for viewing traces. To display links to Tempo traces in the logs from the Loki module, create a custom data source called `GrafanaAdditionalDatasource`.



To do this:



1. Obtain the token from the `httpHeaderValue1` parameter using the command:

````
kubectl get GrafanaAdditionalDatasource d8-loki -o yaml

````


2. Use the following command to apply the configuration to link Loki logs to Tempo traces, specifying the received token instead of the `<token>` value:

````
kubectl apply -f - <<EOF  
\# Configure log collection from all pods  
apiVersion: deckhouse.io/v1  
kind: GrafanaAdditionalDatasource  
metadata:  
name: loki  
spec:  
access: Proxy  
basicAuth: false  
jsonData:  
derivedFields:  
\- datasourceUid: tempo  
matcherRegex: "(?:trace.traceid|traceid)(?:\\":|=)\\"(\\\\w+)\\""  
url: '\$\$\{\_\_value.raw\}'  
name: traceid  
httpHeaderName1: Authorization  
maxLines: 5000  
timeInterval: 30s  
tlsSkipVerify: true   
secureJsonData:   
httpHeaderValue1: Bearer <token>   
type: loki   
url: https://loki.d8-monitoring:3100  
EOF

````