# Kubernetes cluster

> [HTML Version](kubernetes-deckhouse-air-gap.html)

BRIX is installed in a Kubernetes cluster. To manage it, we recommend using the [Deckhouse](https://deckhouse.io/) platform built on Open Source components. 

Deckhouse is [certified by CNCF](https://landscape.cncf.io/?view-mode=card&classify=category&sort-by=name&sort-direction=asc&item=platform--certified-kubernetes-distribution--flant-deckhouse#app-definition-and-development--application-definition-image-build ). Besides Kubernetes, this platform incorporates additional modules for monitoring, traffic balancing, autoscaling, secure access, and more. The modules are pre‑configured, integrated with each other, and ready to use. The management of all cluster components and the platform, as well as their updates, are fully automated.

You can check the supported Kubernetes versions for the platform in [the official Deckhouse documentation](https://deckhouse.io/products/kubernetes-platform/documentation/v1/supported_versions.html).

This article outlines the deployment of a Kubernetes cluster consisting of a single master node.

The installation consists of five steps:

1. [Prepare infrastructure](#infrastructure-preparation).

2. [Prepare configuration file](#preparation-config-file).

3. [Install the Kubernetes cluster based on Deckhouse](#installation).

4. [Set up Deckhouse](#setting).

5. [Install Helm](#installation-helm).

## Step 1: Prepare infrastructure

To deploy a Kubernetes cluster based on the Deckhouse platform, you will need:

1. A personal computer.

A computer from which the installation will be carried out. It is only needed to launch the Deckhouse installer and will not be part of the cluster.

System requirements:

- OS: Windows 10+, macOS 10.15+, Linux (Ubuntu 18.04+, Fedora 35+).

- Installed Docker for running the Deckhouse installer.

- Access to a proxying registry or to a private container image storage with Deckhouse container images.

- SSH access by key to the node that will be the master node of the future cluster.

2. Master node.

A server (physical server or virtual machine) that will be the master node (master node) of the future cluster.

During the installation process, the Deckhouse installer, launched on a personal computer, will connect to the master node via SSH, install the necessary packages, configure the Kubernetes control plane, and deploy Deckhouse.

System requirements:

- At least 12 CPU cores.

- At least 16 GB of RAM.

- At least 200 GB of disk space.

- A [supported OS](https://deckhouse.io/documentation/v1/supported_versions.html).

- Access to a proxying registry or to a private container image storage with Deckhouse container images.

````
Начало внимание

````
Deckhouse only supports Bearer token authentication scheme in the **registry**.

````
Конец внимание

- ````
Access to a proxy server for downloading OS deb/rpm packages (if necessary).

- SSH access from a personal computer using key.

- The node should not have container runtime packages installed, such as containerd or Docker.

**Important**: Installation directly from the master node is currently not supported. The installer in the form of a Docker image cannot be run on the same node where the master node is planned to be deployed, as the node should not have container runtime packages installed, such as containerd or Docker. In the absence of management nodes, install Docker on any other node of the future cluster, run the Docker image of the installer, install Deckhouse, and then remove the Docker image of the installer from the node along with Docker.

3. Downloading Deckhouse images to the local image registry.

You can deploy a Kubernetes cluster using Deckhouse in a closed environment without internet access. For this, first download the Deckhouse platform images on a computer with internet access and upload them to the local image registry. Read more in [Download Deckhouse Images](downloading-images-deckhouse.md).

## Step 2: Prepare configuration file

To install Deckhouse, prepare a YAML configuration file for installation. To obtain the YAML configuration file, use the [Getting started](https://deckhouse.io/gs/) service on the Deckhouse website. The service will generate an up-to-date YAML file for the current version of the platform.

1. Generate the YAML file using the [Getting started](https://deckhouse.io/gs/) service by following these steps:

2. Choose the infrastructure: Bare Metal.

3. Read the installation information.

4. Specify the template for the cluster’s DNS names. In our case it is **%s.example.com**.

5. Save `config.yml`.

2. Make the necessary changes in `config.yml` by doing the following:

1. Set the cluster’s pod address space in **podSubnetCIDR**.

2. Set the cluster’s Service address space in **serviceSubnetCIDR**.

3. Specify the required Kubernetes version in **kubernetesVersion**.

4. Check the update channel in **releaseChannel** (EarlyAccess).

5. Check the domain name template in **publicDomainTemplate** (**%s.example.com**).  
It is used to form domains for system applications in the cluster. For example, **Grafana** for the template **%s.example.com** will be accessible as **grafana.example.com**.

6. Check the operation mode of the cni-flannel module in **podNetworkMode**.  
Flannel operation mode, permissible values are **VXLAN** (if your servers have L3 connectivity) or **HostGW** (for L2 networks).

7. Specify the local network to be used by cluster nodes in **internalNetworkCIDRs**.  
List of internal networks of cluster nodes, for example, `'192.168.1.0/24'`, used for communication between Kubernetes components (kube-apiserver, kubelet, etc.).

### Sample primary cluster configuration file

Below are samples of the primary cluster configuration file: config.yml. This file uses the following parameters for both online and offline installations:

- `podSubnetCIDR`: internal pod addresses that should not overlap with your office or cloud network.

- `serviceSubnetCIDR`: service addresses within the cluster that should not overlap with your office or cloud network.

- `internalNetworkCIDRs`: networks from which access to the cluster API is permitted.

- `releaseChannel`: Deckhouse Virtualization Platform release channel. Available channels are: Alpha, Beta, Early Access, Stable, and Rock Solid. For a detailed description, see the [official Deckhouse documentation](https://deckhouse.io/products/virtualization-platform/documentation/about/release-channels.html).



**For installation via the internet**

````
apiVersion: deckhouse.io/v1  
kind: ClusterConfiguration  
clusterType: Static  
podSubnetCIDR: 10.111.0.0/16  
serviceSubnetCIDR: 10.222.0.0/16  
kubernetesVersion: "Automatic"  
clusterDomain: "cluster.local"  
\---  
apiVersion: deckhouse.io/v1  
kind: InitConfiguration  
deckhouse:  
  imagesRepo: registry.deckhouse.com/deckhouse/ce  
  registryDockerCfg: eyJhdXRocyI6IHsgInJlZ2lzdHJ5LmRlY2tob3VzZS5ydSI6IHt9fX0K  
\---  
apiVersion: deckhouse.io/v1alpha1  
kind: ModuleConfig  
metadata:  
  name: deckhouse  
spec:  
  version: 1  
  enabled: true  
  settings:  
    bundle: Default  
    releaseChannel: EarlyAccess  
    logLevel: Info  
\---  
apiVersion: deckhouse.io/v1alpha1  
kind: ModuleConfig  
metadata:  
  name: global  
spec:  
  version: 2  
  settings:  
    modules:  
      publicDomainTemplate: "%s.elewise.local"  
\---  
apiVersion: deckhouse.io/v1alpha1  
kind: ModuleConfig  
metadata:  
  name: user-authn  
spec:  
  version: 2  
  enabled: true  
  settings:  
    controlPlaneConfigurator:  
      dexCAMode: DoNotNeed  
    publishAPI:  
      enabled: true  
      https:  
        mode: Global  
        global:  
          kubeconfigGeneratorMasterCA: ""  
\---  
apiVersion: deckhouse.io/v1alpha1  
kind: ModuleConfig  
metadata:  
  name: cni-cilium  
spec:  
  version: 1  
  enabled: true  
  settings:  
    tunnelMode: VXLAN  
\---  
apiVersion: deckhouse.io/v1  
kind: StaticClusterConfiguration  
internalNetworkCIDRs:  
\- 192.168.1.0/24
````

````
**For offline installation without internet access**

````
Начало внимание

````
To generate the YAML file using the [Getting started](https://deckhouse.io/gs/) service, choose the **Private environment** infrastructure.

````
Конец внимание

````
Set the following parameters in the **InitConfiguration** resource:

- **devBranch**: If there are no images in the isolated private repository containing update channel information, use the exact Deckhouse image tag to install Deckhouse Platform. For example, if you want to install release v1.46.3, use the image` registry.example.com/images/deckhouse/install:v1.46.3`. Also, specify `devBranch: v1.46.3` instead of `releaseChannel: XXX`.

- **imagesRepo**: `<PROXY\_REGISTRY>/<DECKHOUSE\_REPO\_PATH>/<DECKHOUSE\_REVISION>` — the address of the Deckhouse image in the private repository, considering the edition used. In this article, the images were uploaded to `registry.example.com/images/deckhouse`. Read more in [Download Deckhouse Images](downloading-images-deckhouse.md).

- **registryDockerCfg**: `<BASE64>` — access rights to the private repository, encrypted in Base64. Examples of filling in **registryDockerCfg** can be found in the official [Deckhouse Kubernetes Platform documentation](https://deckhouse.io/documentation/v1/deckhouse-faq.html). In this article, anonymous access to Deckhouse images in an external registry is allowed. Generate **registryDockerCfg** by executing the following command:

````
echo -n "\{\\"auths\\": \{ \\"registry.example.com:443/images/deckhouse\\": \{\}\}\}" | base64

- ````
**registryScheme**: specify the protocol (HTTP or HTTPS) used by the private repository.

- **registryCA**: oot SSL certificate that can verify the SSL certificate of the private registry, for example, if the storage uses a self-signed certificate. If you are using a non-self-signed certificate or the storage works via the HTTP protocol, remove this parameter.

Example of the initial cluster configuration file, `config.yml`:

````
apiVersion: deckhouse.io/v1  
kind: ClusterConfiguration  
clusterType: Static  
podSubnetCIDR: 10.111.0.0/16  
serviceSubnetCIDR: 10.222.0.0/16  
kubernetesVersion: "1.23"  
clusterDomain: "cluster.local"  
\---  
apiVersion: deckhouse.io/v1  
kind: InitConfiguration  
deckhouse:  
  devBranch: v1.46.3  
  configOverrides:  
    global:  
      modules:  
        publicDomainTemplate: "%s.example.com"  
    cniFlannelEnabled: true  
    cniFlannel:  
      podNetworkMode: VXLAN  
  imagesRepo: registry.example.com:443/images/deckhouse  
  registryDockerCfg: eyJhdXRocyI6IHsgInJlZ2lzdHJ5LmV4YW1wbGUuY29tOjQ0My9pbWFnZXMvZGVja2hvdXNlIjoge319fQ==  
  registryScheme: HTTPS  
  registryCA: |  
    -----BEGIN CERTIFICATE-----  
    MIIFBzCCGu+gAwIBAgIUBZ37mm02QGGcmd5pZvWwnpCfQUowDQYGKoZIhvcNAQEL  
    BQAwHjEcMBoGA1UEAwwTaW1hZ2VzLnByb2FjdG9yLnBybzAeFw0yMjA5MjkxNDUw  
    ...  
    9UpckrwxPhctmln5/Awd/2gcaRAxI3qBL7SyDFT0YpnGcAiGPY4Z67HhZ7h6y+2F  
    fQDSXli0r61/Fenkh5OLMihLYTm+5gjZlG1LCXpaGIpjAf16Q+3/pIqapQ==  
    -----END CERTIFICATE-----  
\---  
apiVersion: deckhouse.io/v1  
kind: StaticClusterConfiguration  
internalNetworkCIDRs:  
  - 192.168.1.0/24
````

## ````
Step 3: Install the Kubernetes cluster based on Deckhouse

The installation of Deckhouse Platform Community Edition involves setting up a cluster (using a Docker-image-based installer) consisting of a single master node. The Deckhouse installer is available as a container image, which requires the configuration files and SSH keys for accessing the master node. It is assumed that the SSH key used is `~/.ssh/id\_rsa`. The installer is based on the [dhctl](https://github.com/deckhouse/deckhouse/tree/main/dhctl/) utility.

1. Start the installer.

**Important**: Direct installation from the master node is currently not supported. The installer, in the form of a Docker image, cannot be run on the same node where the master node deployment is planned, as container runtime packages like containerd or docker should not be installed on the node.

The installer is run on a personal computer prepared in the [infrastructure preparation](#infrastructure-preparation) step. On the PC, navigate to the directory with the configuration file `config.yml`, prepared during the [configuration file preparation](#preparation-config-file) step.

To launch the installer via the internet:

````
sudo docker run --pull=always -it -v "\$PWD/config.yml:/config.yml" -v "\$HOME/.ssh/:/tmp/.ssh/" registry.deckhouse.io/deckhouse/ce/install:stable bash

````
**For offline installation without internet access**

  
Execute the command:

````
sudo docker run --pull=always -it -v "\$PWD/config.yml:/config.yml" -v "\$HOME/.ssh/:/tmp/.ssh/" example.com:443/images/deckhouse/install:v1.46.3 bash

````
where:

\[OBJECT\] is the version of the release being installed.

2. Install Deckhouse. To do this, execute the command inside the installer container:

````
dhctl bootstrap --ssh-user=<username> --ssh-host=<master\_ip> --ssh-agent-private-keys=/tmp/.ssh/id\_rsa \\  
\--config=/config.yml \\  
\--ask-become-pass

````
where:

- ````
\[OBJECT\]````
. In the````
 \[OBJECT\]````
 parameter specify the name of the user who generated the SSH key for installation.

- ````
\[OBJECT\]````
.````
 ````
This is the IP address of the master node prepared during the [infrastructure preparation](#infrastructure-preparation) step.

- `<ask-become-pass>`. This is an optional parameter. Use it if a password is required to access the node.

The installation process may take 15-30 minutes with a good connection.

Possible errors when installing Deckhouse

  
Errors may occur during Deckhouse installation. Let's see how to resolve them. 

1. **Port access error** —  \[OBJECT\].

To resolve this error, follow the steps below:

1. Verify that access to all ports specified in [the official Deckhouse documentation](https://deckhouse.io/products/kubernetes-platform/documentation/v1/network_security_setup.html) is configured correctly.

2. Start the installer again and [execute the Deckhouse installation command](#install-deckhouse).

3. Wait for the Deckhouse installation to complete.

4. If the error occurs again, start the installer and execute the Deckhouse installation command with the \[OBJECT\] flag.   
Example:

````
dhctl bootstrap --ssh-user=<username> --ssh-host=<master\_ip> --ssh-agent-private-keys=/tmp/.ssh/id\_rsa --preflight-skip-availability-ports-check \\.  
 --config=/config.yml.  
 --ask-become-pass

2. ````
**Error with access permissions for the user who started the installation (**\[OBJECT\]**) **—  \[OBJECT\].

To resolve it, follow the steps below:

1. In the \[OBJECT\] file, verify that the user has sudo permissions on the host where Deckhouse is installed. To grant the rights, add:

````
<user\_name> ALL=(ALL) ALL````
 

  
The \[OBJECT\] records the account of the user who runs the installation.

2. Start the installer again and execute the Deckhouse installation command.

3. Wait for the Deckhouse installation to complete.

4. If the error occurs again, start the installer and execute the Deckhouse installation command with the \[OBJECT\] flag.  
Example:

````
dhctl bootstrap --ssh-user=<username> --ssh-host=<master\_ip> --ssh-agent-private-keys=/tmp/.ssh/id\_rsa --preflight-skip-sudo-allowed \\  
\--config=/config.yml.  
\--ask-become-pass

3. ````
**Error when re-running the install script** — \[OBJECT\].

The error occurs if files from an incomplete Deckhouse installation are left on the master node. To delete the files, run the command on the master node:

````
sudo bash /var/lib/bashible/cleanup\_static\_node.sh --yes-i-am-sane-and-i-understand-what-i-am-doing
````

## ````
Step 4: Set up Deckhouse

Connect via SSH to the master node prepared during the [infrastructure preparation](#infrastructure-preparation) step. Perform the following steps:

1. ### Remove taint restrictions from the master node.

In this article, the Kubernetes cluster based on Deckhouse consists of a single node. Allow Deckhouse components to work on the master node. To do this, execute the following command:

````
sudo -i d8 k patch nodegroup master --type json -p '\[\{"op": "remove", "path": "/spec/nodeTemplate/taints"\}\]'

2. ### ````
Increase the number of pods on the master node.

Increase the maximum number of pods on the NodeGroup master nodes by executing the following command:

````
sudo -i d8 k patch nodegroup master --type=merge -p '\{"spec":\{"kubelet":\{"maxPods":200\}\}\}'

3. ````
Add Local Path Provisioner.

By default, storageclass is absent in Deckhouse. Create a custom resource **LocalPathProvisioner**, allowing Kubernetes users to use local storage on nodes. Perform the following actions:

1. Create a \[OBJECT\] file on the master node containing the configuration for **LocalPathProvisioner**.

2. Set the desired Reclaim policy (Retain by default). In this article, the parameter \[OBJECT\] is set to \[OBJECT\] (PVs are deleted after PVCs are deleted).

Example of a \[OBJECT\] file:

````
apiVersion: deckhouse.io/v1alpha1  
kind: LocalPathProvisioner  
metadata:  
  name: localpath-deckhouse-system  
spec:  
  nodeGroups:  
  - master  
  path: "/opt/local-path-provisioner"  
  reclaimPolicy: Delete

3. ````
Apply the \[OBJECT\] file in Kubernetes by executing the command:

````
sudo kubectl apply -f local-path-provisioner.yaml

4. ````
Set the created **LocalPathProvisioner** as the default storageclass (default-class) by executing the following command:

````
sudo -i d8 k patch mc global --type merge -p '\{"spec": \{"settings":\{"defaultClusterStorageClass":"localpath-deckhouse-system"\}\}\}'

4. ````
Add Ingress Nginx Controller.

Deckhouse installs and manages the NGINX Ingress Controller using Custom Resources. If there is more than one node for deploying the Ingress controller, it is installed in a fail-safe mode and takes into account all the features of cloud and bare metal infrastructure implementations, as well as different types of Kubernetes clusters.

1. Create an \[OBJECT\] file on the master node containing the Ingress controller configuration by executing the following command:

````
\# section describing the parameters of the nginx ingress controller  
\# used version of the Deckhouse API  
apiVersion: deckhouse.io/v1  
kind: IngressNginxController  
metadata:  
  name: nginx  
spec:  
  # name of the Ingress class for serving Ingress NGINX  
  ingressClass: nginx  
    # how traffic enters from the external world  
  inlet: HostPort  
  hostPort:  
    httpPort: 80  
    httpsPort: 443  
  # describes which nodes the component will be located on  
  # you might want to change  
  nodeSelector:  
    node-role.kubernetes.io/control-plane: ""  
  tolerations:  
  - effect: NoSchedule  
    key: node-role.kubernetes.io/control-plane  
    operator: Exists

2. ````
Apply the ````
\[OBJECT\]````
 file in Kubernetes by executing the command:

````
sudo kubectl create -f ingress-nginx-controller.yml

5. ````
Add a user for access to the cluster web interface.

1. Add the \[OBJECT\] file on the master node containing the description of the user account and access rights:

````
apiVersion: deckhouse.io/v1  
kind: ClusterAuthorizationRule  
metadata:  
  name: admin  
spec:  
  # list of Kubernetes RBAC accounts  
  subjects:  
  - kind: User  
    name: admin@deckhouse.io  
  # predefined access level template  
  accessLevel: SuperAdmin  
  # allow the user to do kubectl port-forward  
  portForwarding: true  
\---  
\# section describing the parameters of the static user  
\# used version of the Deckhouse API  
apiVersion: deckhouse.io/v1  
kind: User  
metadata:  
  name: admin  
spec:  
  # user's email  
  email: admin@deckhouse.io  
  # this is the hash of the password xgnv5gkggd, generated now  
  # generate your own or use this one, but only for testing  
  # echo "xgnv5gkggd" | htpasswd -BinC 10 "" | cut -d: -f2  
  # you might want to change  
  password: '\$2a\$10\$4j4cUeyonCfX7aDJyqSHXuAxycsf/sDK0T4n9ySQ7.owE34L1uXTm'

2. ````
Apply the ````
\[OBJECT\]````
 file by executing the following command on the master node:

````
sudo kubectl create -f user.yml

6. ````
Allow reassigning privilege policy for the running pods:

````
sudo kubectl label namespace brix365 security.deckhouse.io/pod-policy=privileged --overwrite

````
If the namespace has not yet been created, add it with the following command:

````
kubectl create namespace brix365

7. ````
If necessary, configure DNS for the internal domain. Deckhouse has the kube-dns module for this purpose. Enable it with the command:

````
sudo -i d8 system module enable kube-dns

````
If the module is already enabled, use the command to configure it:

````
sudo -i d8 k edit mc kube-dns

````
Specify the DNS settings in the configuration file.

Example of a minimal configuration specifying your DNS server and domain

````
apiVersion: deckhouse.io/v1alpha1  
kind: ModuleConfig  
metadata:  
  name: kube-dns  
spec:  
  version: 1  
  enabled: true  
  settings:  
    upstreamNameservers:  
    - 8.8.8.8  
    - 8.8.4.4  
    hosts:  
    - domain: one.example.com  
      ip: 192.168.0.1  
    - domain: two.another.example.com  
      ip: 10.10.0.128  
    stubZones:  
    - zone: consul.local  
      upstreamNameservers:  
      - 10.150.0.1  
    enableLogs: true  
    clusterDomainAliases:  
    - foo.bar  
    - baz.qux
````

````
For more information about the kube-dns module, see the [official Deckhouse documentation](https://deckhouse.io/modules/kube-dns/).

## Step 5: Install Helm

You can install HELM by downloading a specific version manually or by using a script. Both methods are described in detail in [the official documentation of the HELM package manager](https://helm.sh/docs/intro/install/). 

To download and install the tool manually, do the following:

1. To install Helm, visit the [Helm](https://github.com/helm/helm/releases) releases page and download the \[OBJECT\] archive of the required version.

For installation via the internet:

````
wget https://get.helm.sh/helm-vX.Y.Z-linux-amd64.tar.gz

````
**For offline installation without internet access**

  
On a computer with internet access, go to the Helm releases page and download the \[OBJECT\] archive of the desired version by executing the command:

````
wget https://get.helm.sh/helm-vX.Y.Z-linux-amd64.tar.gz

````
Copy the downloaded archive to the master node.

2. Unpack the archive and move the helm binary file:

````
tar -zxvf helm-vX.Y.Z-linux-amd64.tar.gz  
mv linux-amd64/helm /usr/local/bin/helm
````