side menu

SeaweedFS cluster

This article describes how to install SeaweedFS as an S3 object storage for BRIX.

 

Before installation, please review the following information:

Topology and replication in SeaweedFS

 

The recommended SeaweedFS topology for BRIX is a distributed configuration. Using replication settings, you can determine how many copies of your data will be stored and where.

 

To do this, use the replication parameter in the XYZ format when configuring cluster servers. Where:

 

  • X is the number of additional copies in different data centers.
  • Y is the number of additional copies in different racks.
  • Z is the number of additional copies in a single rack.

 

The total number of copies = one primary copy + the sum of all numbers specified in the replication parameter. Examples of this parameter are available in the table.

Parameter

Total copies

Where stored

000

1

One primary copy (volume) without replication

001

2

Two copies in a single rack

010

2

Two copies in different racks in a single data center

020

3

Three copies in different racks in a single data center

100

2

Two copies in different data centers

Let's take a closer look at some examples of the replication parameter:

 

  1. replication=001:

 

  • Total: two copies (primary + 1).
  • Stored in a single rack.
  • If you have three virtual machines, this replication does not guarantee that each virtual machine will receive its own copy.

 

  1. replication=002:

 

  • Total: three copies (primary + 2).
  • Stored in a single rack.
  • If there are three virtual machines, each will have one copy.

 

  1. replication=020:

 

  • Total: three copies (primary + 2).
  • Stored in three different racks (rack1, rack2, rack3).
  • If there are three virtual machines, each will have one copy.
  • Limitation: if one of three racks fails, the cluster will stop.

 

Read more about replication in the official SeaweedFS documentation.

Install SeaweedFS

 

Consists of several steps:

 

  1. Prepare servers.
  2. Install Docker and Docker Compose.
  3. Install SeaweedFS.
  4. Configure TLS/SSL in SeaweedFS.
  5. Install MC Client.
  6. Start the SeaweedFS service.
  7. Configure a connection to SeaweedFS.
  8. Create buckets.
  9. HAproxy configuration.
  10. Connect to SeaweedFS.

 

Step 1: Prepare servers

 

Create three servers (nodes) with sequentially numbered hostnames, for example:

 

  • seaweedfs-server1.your_domain.
  • seaweedfs-server2.your_domain.
  • seaweedfs-server3.your_domain.

 

Step 2: Install Docker and Docker Compose

 

On the created servers, install:

  • Docker according to the instructions for your operating system on the official website.
  • Docker Compose according to the instructions for your operating system on the official website.

 

Step 3: Install SeaweedFS

 

  1. Create a directory for mounting the disk on all servers:

 

mkdir -p /opt/seaweedfs/data/{master,volume,filer}

  1. Create the /opt/seaweedfs/s3.json file on all servers:

{
 "identities": [
   {
     "name": "admin",
     "credentials": [
       {
         "accessKey": "elma365user",
         "secretKey": "SecretPassword"
       }
     ],
     "actions": ["Admin", "Read", "Write"]
   }
 ]
}

  1. By default, SeaweedFS uses the LevelDB database. It supports multiple file server replicas that will automatically synchronize with some limitations. For more information, see the official SeaweedFS documentation.

If there are limitations or if there are a large number of file server replicas, we recommend using an external data store, such as PostgreSQL or MySQL.

Let's look at how to configure such an external store in PostgreSQL. Create a database in PostgreSQL and link the configuration to the filer component: /opt/seaweedfs/filer.toml.

Example:

#/etc/seaweedfs/filer.toml
[leveldb2]
enabled = false
[postgres2]
enabled = true
createTable = """
 CREATE TABLE IF NOT EXISTS "%s" (
   dirhash   BIGINT,
   name      VARCHAR(65535),
   directory VARCHAR(65535),
   meta      bytea,
   PRIMARY KEY (dirhash, name)
 )
"""
hostname = "hostname"
port = 5432
username = "username"
password = "password"
database = "database"
scheme = "scheme"
sslmode = "disable"

  1. Create a docker-compose.yml file on all servers.

Example configuration for the first server

version: "3.9"
services:
 master:
   image: chrislusf/seaweedfs:4.06
   command: >
     master
     -garbageThreshold=0.3
     -volumeSizeLimitMB=1024
     -ip=192.168.1.101
     -defaultReplication=010
     -peers=192.168.1.101:9333,192.168.1.102:9333,192.168.1.103:9333
     -ip.bind=0.0.0.0
   ports:
     - "9333:9333"
     - "19333:19333"
   volumes:
     - /opt/seaweedfs/data/master:/data
   restart: unless-stopped
   healthcheck:
     test: ["CMD", "curl", "-f", "http://localhost:9333/cluster/status"
     interval: 30s
     timeout: 10s
     retries: 3
 volume:
   image: chrislusf/seaweedfs:4.06
   command: >
     volume
     -fileSizeLimitMB=1024
     -mserver=192.168.1.101:9333,192.168.1.102:9333,192.168.1.103:9333
     -ip=192.168.1.101
     -ip.bind=0.0.0.0
     -dir=/data
     -dataCenter=dc1
     -rack=rack1
   ports:
     - "8080:8080"
     - "18080:18080"
   depends_on:
     - master
   volumes:
     -  /opt/seaweedfs/data/volume:/data
   restart: unless-stopped
 filer:
   image: chrislusf/seaweedfs:4.06
   command: >
     filer
     -defaultReplicaPlacement=010
     -master=192.168.1.101:9333,192.168.1.102:9333,192.168.1.103:9333
     -ip=192.168.1.101
     -ip.bind=0.0.0.0
     -s3
     -s3.config=/etc/seaweedfs/s3.json
     -s3.port=8333
   ports:
     - "8888:8888"
     - "8333:8333"
     - "18888:18888"
   depends_on:
     - master
     - volume
   volumes:
     - /opt/seaweedfs/data/filer:/data
     - /opt/seaweedfs/s3.json:/etc/seaweedfs/s3.json:ro
    - /opt/seaweedfs/filer.toml:/etc/seaweedfs/filer.toml:ro
   restart: unless-stopped
 worker:
   image: chrislusf/seaweedfs:4.06
   command: >
     worker
     -admin=192.168.1.101:9333,192.168.1.102:9333,192.168.1.103:9333
   depends_on:
   - master

 

 

Example configuration for the second server

version: "3.9"
services:
 master:
   image: chrislusf/seaweedfs:4.06
   command: >
     master
     -garbageThreshold=0.3
     -volumeSizeLimitMB=1024
     -ip=192.168.1.102
     -defaultReplication=010
     -peers=192.168.1.101:9333,192.168.1.102:9333,192.168.1.103:9333
     -ip.bind=0.0.0.0
   ports:
     - "9333:9333"
     - "19333:19333"
   volumes:
     - /opt/seaweedfs/data/master:/data
   restart: unless-stopped
   healthcheck:
     test: ["CMD", "curl", "-f", "http://localhost:9333/cluster/status"
     interval: 30s
     timeout: 10s
     retries: 3
 volume:
   image: chrislusf/seaweedfs:4.06
   command: >
     volume
     -fileSizeLimitMB=1024
     -mserver=192.168.1.101:9333,192.168.1.102:9333,192.168.1.103:9333
     -ip=192.168.1.102
     -ip.bind=0.0.0.0
     -dir=/data
     -dataCenter=dc1
     -rack=rack2
   ports:
     - "8080:8080"
     - "18080:18080"
   depends_on:
     - master
   volumes:
     -  /opt/seaweedfs/data/volume:/data
   restart: unless-stopped
 filer:
   image: chrislusf/seaweedfs:4.06
   command: >
     filer
     -defaultReplicaPlacement=010
     -master=192.168.1.101:9333,192.168.1.102:9333,192.168.1.103:9333
     -ip=192.168.1.102
     -ip.bind=0.0.0.0
     -s3
     -s3.config=/etc/seaweedfs/s3.json
     -s3.port=8333
   ports:
     - "8888:8888"
     - "8333:8333"
     - "18888:18888"
   depends_on:
     - master
     - volume
   volumes:
     - /opt/seaweedfs/data/filer:/data
     - /opt/seaweedfs/s3.json:/etc/seaweedfs/s3.json:ro
    - /opt/seaweedfs/filer.toml:/etc/seaweedfs/filer.toml:ro
   restart: unless-stopped
 worker:
   image: chrislusf/seaweedfs:4.06
   command: >
     worker
     -admin=192.168.1.101:9333,192.168.1.102:9333,192.168.1.103:9333
   depends_on:
   - master

 

 

Example configuration for the third server

version: "3.9"
services:
 master:
   image: chrislusf/seaweedfs:4.06
   command: >
     master
     -garbageThreshold=0.3
     -volumeSizeLimitMB=1024
     -ip=192.168.1.103
     -defaultReplication=010
     -peers=192.168.1.101:9333,192.168.1.102:9333,192.168.1.103:9333
     -ip.bind=0.0.0.0
   ports:
     - "9333:9333"
     - "19333:19333"
   volumes:
     - /opt/seaweedfs/data/master:/data
   restart: unless-stopped
   healthcheck:
     test: ["CMD", "curl", "-f", "http://localhost:9333/cluster/status"
     interval: 30s
     timeout: 10s
     retries: 3
 volume:
   image: chrislusf/seaweedfs:4.06
   command: >
     volume
     -fileSizeLimitMB=1024
     -mserver=192.168.1.101:9333,192.168.1.102:9333,192.168.1.103:9333
     -ip=192.168.1.103
     -ip.bind=0.0.0.0
     -dir=/data
     -dataCenter=dc1
     -rack=rack3
   ports:
     - "8080:8080"
     - "18080:18080"
   depends_on:
     - master
   volumes:
     -  /opt/seaweedfs/data/volume:/data
   restart: unless-stopped
 filer:
   image: chrislusf/seaweedfs:4.06
   command: >
     filer
     -defaultReplicaPlacement=010
     -master=192.168.1.101:9333,192.168.1.102:9333,192.168.1.103:9333
     -ip=192.168.1.103
     -ip.bind=0.0.0.0
     -s3
     -s3.config=/etc/seaweedfs/s3.json
     -s3.port=8333
   ports:
     - "8888:8888"
     - "8333:8333"
     - "18888:18888"
   depends_on:
     - master
     - volume
   volumes:
     - /opt/seaweedfs/data/filer:/data
     - /opt/seaweedfs/s3.json:/etc/seaweedfs/s3.json:ro
    - /opt/seaweedfs/filer.toml:/etc/seaweedfs/filer.toml:ro
   restart: unless-stopped
 worker:
   image: chrislusf/seaweedfs:4.06
   command: >
     worker
     -admin=192.168.1.101:9333,192.168.1.102:9333,192.168.1.103:9333
   depends_on:
   - master

 

 

Step 4: Configure TLS/SSL support in SeaweedFS

 

To enable TLS/SSL support in SeaweedFS on each server:

 

  1. Save the certificate file and private key file in the /opt/seaweedfs/ssl directory.

 

  1. Rename the server certificate file to cert.pem.

 

  1. Rename the private key file to key.pem.

 

  1. When using self-signed certificates, save the root CA file in the /opt/seaweedfs/certs directory.

 

  1. Prepare the security settings file: security.toml:

[jwt.signing]
key = "MASTERVOLUMESECRET"
[jwt.filer_signing]
key = "FILERSECRET"
[grpc]
ca = "/etc/seaweedfs/certs/ca.pem"
[grpc.master]
cert = "/etc/seaweedfs/certs/cert.pem"
key = "/etc/seaweedfs/certs/key.pem"
[grpc.volume]
cert = "/etc/seaweedfs/certs/cert.pem"
key = "/etc/seaweedfs/certs/key.pem"
[grpc.filer]
cert = "/etc/seaweedfs/certs/cert.pem"
key = "/etc/seaweedfs/certs/key.pem"
[grpc.client]
cert = "/etc/seaweedfs/certs/cert.pem"
key = "/etc/seaweedfs/certs/key.pem"
[https.client]
enabled = true
ca = "/etc/seaweedfs/certs/ca.pem"
cert = "/etc/seaweedfs/certs/cert.pem"
key = "/etc/seaweedfs/certs/key.pem"
[https.volume]
cert = "/etc/seaweedfs/certs/cert.pem"
key = "/etc/seaweedfs/certs/key.pem"
ca = "/etc/seaweedfs/certs/ca.pem"
[https.master]
cert = "/etc/seaweedfs/certs/cert.pem"
key = "/etc/seaweedfs/certs/key.pem"
ca = "/etc/seaweedfs/certs/ca.pem"
[https.filer]
cert = "/etc/seaweedfs/certs/cert.pem"
key = "/etc/seaweedfs/certs/key.pem"
ca = "/etc/seaweedfs/certs/ca.pem"

 

Example configuration for the first server with TLS

version: "3.9"
services:
 master:
   image: chrislusf/seaweedfs:4.06
   command: >
     master
     -garbageThreshold=0.3
     -volumeSizeLimitMB=1024
     -ip=192.168.1.101
     -defaultReplication=010
     -peers=192.168.1.101:9333,192.168.1.102:9333,192.168.1.103:9333
     -ip.bind=0.0.0.0
   ports:
     - "9333:9333"
     - "19333:19333"
   volumes:
     - /opt/seaweedfs/data/master:/data
     - /opt/seaweedfs/certs:/etc/seaweedfs/certs:ro
     - /opt/seaweedfs/security.toml:/etc/seaweedfs/security.toml:ro
   restart: unless-stopped
   healthcheck:
     test: ["CMD", "curl", "-f", "http://localhost:9333/cluster/status"
     interval: 30s
     timeout: 10s
     retries: 3
 volume:
   image: chrislusf/seaweedfs:4.06
   command: >
     volume
     -fileSizeLimitMB=1024
     -mserver=192.168.1.101:9333,192.168.1.102:9333,192.168.1.103:9333
     -ip=192.168.1.101
     -ip.bind=0.0.0.0
     -dir=/data
   ports:
     - "8080:8080"
     - "18080:18080"
   depends_on:
     - master
   volumes:
     -  /opt/seaweedfs/data/volume:/data
     - /opt/seaweedfs/certs:/etc/seaweedfs/certs:ro
     - /opt/seaweedfs/security.toml:/etc/seaweedfs/security.toml:ro
   restart: unless-stopped
 filer:
   image: chrislusf/seaweedfs:4.06
   command: >
     filer
     -defaultReplicaPlacement=010
     -master=192.168.1.101:9333,192.168.1.102:9333,192.168.1.103:9333
     -ip=192.168.1.101
     -ip.bind=0.0.0.0
     -s3
     -s3.config=/etc/seaweedfs/s3.json
     -s3.port.https=8334
     -s3.cacert.file=/etc/seaweedfs/certs/ca.pem
     -s3.cert.file=/etc/seaweedfs/certs/cert.pem
     -s3.key.file=/etc/seaweedfs/certs/key.pem
   ports:
     - "8888:8888"
     - "8334:8334"
     - "18888:18888"
   depends_on:
     - master
     - volume
   volumes:
     - /opt/seaweedfs/data/filer:/data
     - /opt/seaweedfs/s3.json:/etc/seaweedfs/s3.json:ro
     - /opt/seaweedfs/certs:/etc/seaweedfs/certs:ro
     - /opt/seaweedfs/security.toml:/etc/seaweedfs/security.toml:ro
    - /opt/seaweedfs/filer.toml:/etc/seaweedfs/filer.toml:ro
   restart: unless-stopped
 worker:
   image: chrislusf/seaweedfs:4.06
   command: >
     worker
     -admin=192.168.1.101:9333,192.168.1.102:9333,192.168.1.103:9333
   depends_on:
   - master
   volumes:
     - /opt/seaweedfs/certs:/etc/seaweedfs/certs:ro
     - /opt/seaweedfs/security.toml:/etc/seaweedfs/security.toml:ro

 

 

Example configuration for the second server with TLS

version: "3.9"
services:
 master:
   image: chrislusf/seaweedfs:4.06
   command: >
     master
     -garbageThreshold=0.3
     -volumeSizeLimitMB=1024
     -ip=192.168.1.102
     -defaultReplication=010
     -peers=192.168.1.101:9333,192.168.1.102:9333,192.168.1.103:9333
     -ip.bind=0.0.0.0
   ports:
     - "9333:9333"
     - "19333:19333"
   volumes:
     - /opt/seaweedfs/data/master:/data
     - /opt/seaweedfs/certs:/etc/seaweedfs/certs:ro
     - /opt/seaweedfs/security.toml:/etc/seaweedfs/security.toml:ro
   restart: unless-stopped
   healthcheck:
     test: ["CMD", "curl", "-f", "http://localhost:9333/cluster/status"
     interval: 30s
     timeout: 10s
     retries: 3
 volume:
   image: chrislusf/seaweedfs:4.06
   command: >
     volume
     -fileSizeLimitMB=1024
     -mserver=192.168.1.101:9333,192.168.1.102:9333,192.168.1.103:9333
     -ip=192.168.1.102
     -ip.bind=0.0.0.0
     -dir=/data
   ports:
     - "8080:8080"
     - "18080:18080"
   depends_on:
     - master
   volumes:
     -  /opt/seaweedfs/data/volume:/data
     - /opt/seaweedfs/certs:/etc/seaweedfs/certs:ro
     - /opt/seaweedfs/security.toml:/etc/seaweedfs/security.toml:ro
   restart: unless-stopped
 filer:
   image: chrislusf/seaweedfs:4.06
   command: >
     filer
     -defaultReplicaPlacement=010
     -master=192.168.1.101:9333,192.168.1.102:9333,192.168.1.103:9333
     -ip=192.168.1.102
     -ip.bind=0.0.0.0
     -s3
     -s3.config=/etc/seaweedfs/s3.json
     -s3.port.https=8334
     -s3.cacert.file=/etc/seaweedfs/certs/ca.pem
     -s3.cert.file=/etc/seaweedfs/certs/cert.pem
     -s3.key.file=/etc/seaweedfs/certs/key.pem
   ports:
     - "8888:8888"
     - "8334:8334"
     - "18888:18888"
   depends_on:
     - master
     - volume
   volumes:
     - /opt/seaweedfs/data/filer:/data
     - /opt/seaweedfs/s3.json:/etc/seaweedfs/s3.json:ro
     - /opt/seaweedfs/certs:/etc/seaweedfs/certs:ro
     - /opt/seaweedfs/security.toml:/etc/seaweedfs/security.toml:ro
    - /opt/seaweedfs/filer.toml:/etc/seaweedfs/filer.toml:ro
   restart: unless-stopped

 

 

Example configuration for the third server with TLS

version: "3.9"
services:
 master:
   image: chrislusf/seaweedfs:4.06
   command: >
     master
     -garbageThreshold=0.3
     -volumeSizeLimitMB=1024
     -ip=192.168.1.103
     -defaultReplication=010
     -peers=192.168.1.101:9333,192.168.1.102:9333,192.168.1.103:9333
     -ip.bind=0.0.0.0
   ports:
     - "9333:9333"
     - "19333:19333"
   volumes:
     - /opt/seaweedfs/data/master:/data
     - /opt/seaweedfs/certs:/etc/seaweedfs/certs:ro
     - /opt/seaweedfs/security.toml:/etc/seaweedfs/security.toml:ro
   restart: unless-stopped
   healthcheck:
     test: ["CMD", "curl", "-f", "http://localhost:9333/cluster/status"
     interval: 30s
     timeout: 10s
     retries: 3
 volume:
   image: chrislusf/seaweedfs:4.06
   command: >
     volume
     -fileSizeLimitMB=1024
     -mserver=192.168.1.101:9333,192.168.1.102:9333,192.168.1.103:9333
     -ip=192.168.1.103
     -ip.bind=0.0.0.0
     -dir=/data
   ports:
     - "8080:8080"
     - "18080:18080"
   depends_on:
     - master
   volumes:
     -  /opt/seaweedfs/data/volume:/data
     - /opt/seaweedfs/certs:/etc/seaweedfs/certs:ro
     - /opt/seaweedfs/security.toml:/etc/seaweedfs/security.toml:ro
   restart: unless-stopped
 filer:
   image: chrislusf/seaweedfs:4.06
   command: >
     filer
     -defaultReplicaPlacement=010
     -master=192.168.1.101:9333,192.168.1.102:9333,192.168.1.103:9333
     -ip=192.168.1.103
     -ip.bind=0.0.0.0
     -s3
     -s3.config=/etc/seaweedfs/s3.json
     -s3.port.https=8334
     -s3.cacert.file=/etc/seaweedfs/certs/ca.pem
     -s3.cert.file=/etc/seaweedfs/certs/cert.pem
     -s3.key.file=/etc/seaweedfs/certs/key.pem
   ports:
     - "8888:8888"
     - "8334:8334"
     - "18888:18888"
   depends_on:
     - master
     - volume
   volumes:
     - /opt/seaweedfs/data/filer:/data
     - /opt/seaweedfs/s3.json:/etc/seaweedfs/s3.json:ro
     - /opt/seaweedfs/certs:/etc/seaweedfs/certs:ro
     - /opt/seaweedfs/security.toml:/etc/seaweedfs/security.toml:ro
    - /opt/seaweedfs/filer.toml:/etc/seaweedfs/filer.toml:ro
   restart: unless-stopped

 

Step 5: Install MC Client

 

Download the latest stable SeaweedFS Client binary and install it on your system:

wget https://dl.min.io/client/mc/release/linux-amd64/mc
chmod +x mс
sudo mv mc /usr/local/bin/

Step 6: Start the SeaweedFS service

 

Run the manifest:

docker-compose -f docker-compose.yml up -d

 

Step 7: Configure a connection to SeaweedFS

 

Create an alias for SeaweedFS:

/usr/local/bin/mc alias set seaweedfs http://seaweedfs.your_domain:8333 elma365user SecretPassword

Step 8: Create buckets

 

Important: Bucket names in S3 must follow the format s3elma365*. In this article, the example uses selma365 as bucket name, elma365user as user, and SecretPassword as password. When setting this up, configure this information according to your organization's security policy.

 

Examples of bucket names: s3elma365; s3elma365-dev; s3elma365-prod.

 

For BRIX operation create a bucket named s3elma365 using the command:

/usr/local/bin/mc mb -p seaweedfs/s3elma365 --region=eu-central-1

 

Step 9: HAproxy configuration

 

In this article, user traffic arrives at HAproxy via the connection seaweedfs.your_domain:8333 and is evenly balanced between the SeaweedFS cluster servers. To do this, configure the settings according to the Configure HAProxy for S3 article.

 

Step 10: Connect to SeaweedFS

 

Parameters for connecting to SeaweedFS:

 

  • address: seaweedfs.your_domain:8333.
  • bucket: s3elma365.
  • region: eu-central-1.
  • access key ID: elma365user.
  • secret access key: SecretPassword.
  • upload method: PUT.
  • enable SSL: No.

 

If SeaweedFS expects a connection using TLS/SSL, specify:

  • enable SSL: Yes.
  • address: seaweedfs.your_domain:8334 connection port.

 

 

configure-rabbitmq.html minio-cluster.html