# BRIX AI diagnostics

> [HTML Version](cortex-troubleshooting.html)

When encountering errors or issues in **BRIX AI**, run diagnostics to troubleshoot the problem. You can review the diagnostic files to fix issues on your own or attach them to a technical support ticket for analysis.

Several diagnostic methods are available:

- [Diagnostic script](#script).

- [Manual diagnostic commands](#commands).

- [HAR files collection in browser](#har-files).

## Running diagnostic script

1. Go to the **BRIX AI** installation folder:

````
cd ~/elma-cortex

2. ````
If you are running diagnostics for the first time, set up permissions:

````
chmod +x ./diagnose.sh

3. ````
Run the diagnostic script. By default, it collects logs for the last 30 days:

````
./diagnose.sh

````
The script generates an archive named `<project-name>-diagnostics-<timestamp>.tar.gz` containing all collected diagnostic data.

To customize the diagnostic data collection, run the script using the parameters described below.

### Diagnostic script parameters

| | | |
|------|------|------|
| **Parameter** | **Description** | **Default value** |
| \[OBJECT\] | Project or Docker Swarm stack name for diagnostics | Retrieved from the `.env` environment variables file. If no name is specified, `elma-cortex` is used |
| \[OBJECT\] | Time period for log collection | Format:<br>- `30m`: 30 minutes.<br>- `1h`: 1 hour.<br>- `2h`: 2 hours.<br>- `24h`: 24 hours.<br>- `7d`: 7 days.<br>Default: 30 days |
| \[OBJECT\] | Name of the output diagnostic file | Specify the file name without the `.tar.gz` extension.<br>Format: `<project-name>-diagnostics-<timestamp>` |
| \[OBJECT\] | Display the list of available parameters | — |


### Diagnostic data collection examples

Here are examples of commands used to run diagnostics:

- For the last two hours:

````
./diagnose.sh --logs-since 2h

- ````
For the last 24 hours with a custom file name:

````
./diagnose.sh --logs-since 24h --output issue-diagnostics

- ````
For a week with a custom project name configured in environment variables:

````
./diagnose.sh --project-name my-cortex --logs-since 7d

### ````
Diagnostic file contents

The diagnostic script gathers operational data about **BRIX AI** into several folders, which are then packaged into an archive:

- [system](#system): System information about the installation and environment.

- [logs](#logs): Service logs.

- [containers and services](#containers-services): 

	- Information about containers when using Docker Compose.

	- Information about services when using Docker Swarm.

- [config](#config): Docker and environment configuration. Standard sensitive data such as passwords and API keys in the environment configuration file are automatically replaced with `\*\*\*MASKED\*\*\*`.

````
начало внимание

````
Before submitting diagnostic files to technical support, ensure they do not contain sensitive data.

````
конец внимание

````
Below is a summary of the data collected by the diagnostic script.

1. **System information**

| | |
|------|------|
| **File** | **Content** |
| \[OBJECT\] | OS version, memory, disk space, CPU information |
| \[OBJECT\] | Docker version and configuration |
| \[OBJECT\] | Docker Compose service status (only when using Docker Compose) |
| \[OBJECT\] | Swarm nodes and cluster information (only when using Docker Swarm) |
| \[OBJECT\] | Stack services and replicas (only when using Docker Swarm) |
| \[OBJECT\] | Task history including failures (only when using Docker Swarm) |
| \[OBJECT\] | Running and stopped containers |
| \[OBJECT\] | Docker network configuration |
| \[OBJECT\] | Docker volume information |
| \[OBJECT\] | Docker secret metadata (only when using Docker Swarm) |
| \[OBJECT\] | Service health checks |
| \[OBJECT\] | Recent Docker events |


2. **Service logs**.

Log files for all **BRIX AI** services:

- `gamma-apps.log`: Main app service.

- `gamma-tasks.log`: Background task worker.

- `caddy.log`: Reverse proxy.

- `mongo-db.log`: MongoDB.

- `rabbit.log`: RabbitMQ.

- `redis.log`: Redis.

- `minio.log`: MinIO S3 storage.

- `qdrant.log`: Vector database.

3. **Container or service details**:

- JSON data on configuration, status, and metadata for each running container or service.

- Docker task data (only when using Docker Swarm).

4. **Configuration**:

- The `docker-compose.yml` files when using Docker Compose or `stack-\*.yml`** **when using Docker Swarm.

- The `env-sanitized.txt` file: Environment configuration with masked passwords.

- The `install.md` file: Installation documentation.

### Troubleshooting the diagnostic script

Here are common issues you might encounter when using the diagnostic script.

1. **Permission denied error**.

If you get a permission error, run these commands:

````
chmod +x diagnose.sh  
./diagnose.sh

2. ````
**Script not found in Docker Compose**.

Make sure you are in the correct directory when running the script:

````
ls -la diagnose.sh 

````
If the script is missing from the directory, copy it from the installation archive.

3. **Docker permissions issues**.

If Docker commands fail due to permission errors, you can:

- Run the script using `sudo`:

````
sudo ./diagnose.sh

- ````
Add the user to the `docker` group. You will need to log in again afterward:

````
sudo usermod -aG docker \$USER

## ````
Manual diagnostic commands

Here are commands you can use to collect diagnostic data manually.

### Diagnostics in Docker Compose

1. Check service status:

````
docker compose ps --all

2. ````
View logs for a specific service:

````
docker compose logs gamma-apps --since 1h

3. ````
View combined logs:

````
docker compose logs --since 1h

4. ````
Get container details:

````
docker inspect \$(docker compose ps -q gamma-apps)

5. ````
Check service readiness:

````
docker exec \$(docker compose ps -q gamma-apps) curl -sf http://localhost:8080/api/int/v1/info/application/health

### ````
Diagnostics in Docker Swarm

1. Get the list of stack services:

````
docker stack services elma-cortex

2. ````
Get the list of stack tasks, including failed ones:

````
docker stack ps elma-cortex --no-trunc

3. ````
View service logs:

````
docker service logs elma-cortex\_gamma-apps --since 1h

4. ````
Get service details:

````
docker service inspect elma-cortex\_gamma-apps

5. ````
Check Swarm node status:

````
docker node ls

## ````
Collecting HAR files in browser

HAR files record network interaction between the browser and the server. They help diagnose API issues, authentication errors, and UI failures. Generating a HAR file depends on your browser and is described in its documentation.

A HAR file includes:

- HTTP requests and responses.

- Request and response headers.

- Request and response bodies.

- Timing details.

- Cookie values.

**Важно**: HAR files include sensitive data such as:

- Authentication tokens and cookies.

- Session data.

- Personal data from API response.

Before sending HAR files to technical support::

1. Ensure that the file does not contain sensitive information.

2. Record activity for the minimum time required to reproduce the issue.

3. Whenever possible, use a test account.