Home / How To / Configure MinIO server to use storage architecture

Configure MinIO server to use storage architecture



In this tutorial I will explain how to set up a MinIO server to use storage architecture. Like anyone who does not already know what MinIO is: it is a high-performance distributed object storage system. It is software defined, runs on industry standard hardware and is 100% open source. It is deliberately built to serve objects as a single-layer architecture to achieve all the necessary functionality without compromise. The result is seen as a cloud-language object server that is at the same time scalable and light.

As the world of cloud technology has evolved more and more mature, will it think about why we need MinIO in the first place?

Keep in mind that if you serve your solution in the cloud, you can stop using solution storage such as AWS S3, Azure Blob Storage and Alibaba OSS. The same applies to the concept if your solution is still in place because Minio works as an alternative to the storage architecture in the same way as the cloud storage service provided.

MinIO object storage

1. How does it work?

In a simple concept, Minio is available in two parts – the client part and the server part. This concept also includes a dashboard via web ui or file browser. Each client and server side is relatively easy to set up and if you are familiar with CLI (Command Line Interface), it would be easy to understand.

But when we design it to a production level, everything must be distributed, which means that the solution provided must ensure that it works well on a large scale, self-expanded growth and high availability ready. With this in mind, minio has called its own concept Distributed delete code.

MinIO server scheme

This concept is a reliable approach to share data across multiple devices and retrieve it, even when some of the devices are not available. By using this concept, you can lose by half of the devices and still have your data guaranteed

For this guide, I will show you how to install and configure the MinIO server as a distributed delete code. Then we take a quick look at the client side on how to use the MinIO service as an end user.

2. Installation phase

For the installation phase, I configure two servers as a mini-cluster to prepare the configuration of distributed deletion code.

We are now listing 4 hard drives that we will use to partition it as a blocking device for mini-use. Because our architecture decided to set up multiple servers, the minimum unit that must be for a server is 2, but if you use a single server, the minimum requirement for the units is 1. Detailed requirements required for deletion code design can be seen here.

Below are the steps:

 [[email protected] ~]# fdisk -l 

Disk /dev/sda: 107.4 GB, 107374182400 bytes, 209715200 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0x000a4fd8

Device Boot Start End Blocks Id System
/dev/sda1 * 2048 2099199 1048576 83 Linux
/dev/sda2 2099200 209715199 103808000 8e Linux LVM

Disk /dev/sdb: 8589 MB, 8589934592 bytes, 16777216 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes

Disk /dev/sdc: 8589 MB, 8589934592 bytes, 16777216 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes

Disk /dev/sdd: 8589 MB, 8589934592 bytes, 16777216 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes

Disk /dev/sde: 8589 MB, 8589934592 bytes, 16777216 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes

Disk /dev/mapper/centos-root: 104.1 GB, 104144568320 bytes, 203407360 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes

Disk /dev/mapper/centos-swap: 2147 MB, 2147483648 bytes, 4194304 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes

As you can see above, there are four devices mounted in our server with 8 GB size each.

Then we create a partition from each device and then create a dedicated directory that is mounted on each partition we create. Below are the steps.

 
[[email protected] ~]# fdisk /dev/sdb
Welcome to fdisk (util-linux 2.23.2).

Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.

Device does not contain a recognized partition table
Building a new DOS disklabel with disk identifier 0x4217c4d9.

Command (m for help): p

Disk /dev/sdb: 8589 MB, 8589934592 bytes, 16777216 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0x4217c4d9

Device Boot Start End Blocks Id System

Command (m for help): n
Partition type:
p primary (0 primary, 0 extended, 4 free)
e extended
Select (default p): p
Partition number (1-4, default 1):
First sector (2048-16777215, default 2048):
Using default value 2048
Last sector, +sectors or +size{K,M,G} (2048-16777215, default 16777215):
Using default value 16777215
Partition 1 of type Linux and of size 8 GiB is set

Command (m for help): p

Disk /dev/sdb: 8589 MB, 8589934592 bytes, 16777216 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0x4217c4d9

Device Boot Start End Blocks Id System
/dev/sdb1 2048 16777215 8387584 83 Linux

Command (m for help): w
The partition table has been altered!

Calling ioctl() to re-read partition table.
Syncing disks.

Command (m for help): q

[[email protected] ~]# ls /dev/sdb*
/dev/sdb /dev/sdb1
[[email protected] ~]# mkfs.xfs -f /dev/sdb1
meta-data=/dev/sdb1 isize=512 agcount=4, agsize=524224 blks
= sectsz=512 attr=2, projid32bit=1
= crc=1 finobt=0, sparse=0
data = bsize=4096 blocks=2096896, imaxpct=25
= sunit=0 swidth=0 blks
naming =version 2 bsize=4096 ascii-ci=0 ftype=1
log =internal log bsize=4096 blocks=2560, version=2
= sectsz=512 sunit=0 blks, lazy-count=1
realtime =none extsz=4096 blocks=0, rtextents=0
[[email protected] ~]#
[[email protected] ~]# mkdir -p /opt/drive1
[[email protected] ~]# mkdir -p /opt/drive2
[[email protected] ~]# mkdir -p /opt/drive3
[[email protected] ~]# mkdir -p /opt/drive4
[[email protected] ~]#
[[email protected] ~]# mount /dev/sdb1 /opt/drive1
[[email protected] ~]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/centos-root 97G 3.8G 94G 4% /
devtmpfs 1.9G 0 1.9G 0% /dev
tmpfs 1.9G 0 1.9G 0% /dev/shm
tmpfs 1.9G 8.6M 1.9G 1% /run
tmpfs 1.9G 0 1.9G 0% /sys/fs/cgroup
/dev/sda1 1014M 145M 870M 15% /boot
tmpfs 379M 0 379M 0% /run/user/0
/dev/sdb1 8.0G 33M 8.0G 1% /opt/drive1
[[email protected] ~]#

When you are done, repeat the same process to create a partition on the remaining drives and then mount it to each directory we have created. As a final result, you should finally output as below: –

 [[email protected] ~]# mount /dev/sdb1 /opt/drive1 
[[email protected] ~]# mount /dev/sdc1 /opt/drive2
[[email protected] ~]# mount /dev/sdd1 /opt/drive3
[[email protected] ~]# mount /dev/sde1 /opt/drive4
[[email protected] ~]#
[[email protected] ~]#
[[email protected] ~]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/centos-root 97G 3.8G 94G 4% /
devtmpfs 1.9G 0 1.9G 0% /dev
tmpfs 1.9G 0 1.9G 0% /dev/shm
tmpfs 1.9G 8.6M 1.9G 1% /run
tmpfs 1.9G 0 1.9G 0% /sys/fs/cgroup
/dev/sda1 1014M 145M 870M 15% /boot
tmpfs 379M 0 379M 0% /run/user/0
/dev/sdb1 8.0G 33M 8.0G 1% /opt/drive1
/dev/sdc1 8.0G 33M 8.0G 1% /opt/drive2
/dev/sdd1 8.0G 33M 8.0G 1% /opt/drive3
/dev/sde1 8.0G 33M 8.0G 1% /opt/drive4

Ok, since a prerequisite for devices is made for server 1, repeat the same configuration on server 2 as above.

3. Configuration phase

Now that both server configurations are complete, let’s continue installing the minio service. First download the mini package as below:

 [[email protected] ~]# wget https://dl.min.io/server/minio/release/linux-amd64/minio && chmod +x minio 
--2019-09-29 22:23:57-- https://dl.min.io/server/minio/release/linux-amd64/minio
Resolving dl.min.io (dl.min.io)... 178.128.69.202
Connecting to dl.min.io (dl.min.io)|178.128.69.202|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 43831296 (42M) [application/octet-stream]
Saving to: ‘minio’

3% [=> ] 1,335,296 106KB/s eta 6m 33s

Now repeat the same as above on server 2.

When everything is ready, let’s start the minio configuration. We will define MINIO_ACCESS_KEY and MINIO_SECRET_KEY as authentication access. The configuration is as below: –

 [[email protected] ~]# export MINIO_ACCESS_KEY=shahril && export MINIO_SECRET_KEY=shahril123 
[[email protected] ~]# ./minio server http://10.124.12.{141..142}:9000/opt/drive{1..4}
Waiting for a minimum of 4 disks to come online (elapsed 0s)

Waiting for a minimum of 4 disks to come online (elapsed 2s)

Waiting for a minimum of 4 disks to come online (elapsed 3s)

Waiting for a minimum of 4 disks to come online (elapsed 3s)

Waiting for all other servers to be online to format the disks.
Status: 8 Online, 0 Offline.
Endpoint: http://10.124.12.141:9000 http://10.124.12.142:9000
AccessKey: shahril
SecretKey: shahril123

Browser Access:
http://10.124.12.141:9000 http://10.124.12.142:9000

Command-line Access: https://docs.min.io/docs/minio-client-quickstart-guide
$ mc config host add myminio http://10.124.12.141:9000 shahril shahril123

Object API (Amazon S3 compatible):
Go: https://docs.min.io/docs/golang-client-quickstart-guide
Java: https://docs.min.io/docs/java-client-quickstart-guide
Python: https://docs.min.io/docs/python-client-quickstart-guide
JavaScript: https://docs.min.io/docs/javascript-client-quickstart-guide
.NET: https://docs.min.io/docs/dotnet-client-quickstart-guide

Now the configuration is complete on server 1, repeat the same steps on server 2 for the configuration.

When everything is ready, we can test the results

4. Test phase

When everything is ready, let’s start seeing the usefulness of mini services. As shown in the configuration above, we can access the dashboard for its user interface via the browser. Let’s log in to our example http://10.124.12.141:9000 with access key shahril and secret key shahril123 as configured.

The result is shown as below:Advertisement

Advertisement

MinIO login

When done, it will redirect us to the dashboard. Let’s now create our first bucket.

Click on the icon folder with the plus button and name our first bucket as my love. Examples as below:

MinIO Browser - Bucket dashboard

Bucket

When you are done, you will notice that a new bucket is created and displayed on the left panel as the screenshot below.

To-do-before-I-die-listAdvertisement

When you are done, you will notice that a new bucket is created and displayed on the left panel as the screenshot below.

Then let’s add some files from your local page to insert into the bucket

Add files to MinIO bucket

You will notice that the new file has been uploaded in the bucket shown below.

File uploaded to MinIO Bucket

To ensure that the concept is distributed is well implemented. Let’s do a simple test by accessing the minio dashboard via another server. The URL of the other server is http://10.124.12.142:9000

MinIO Dashboard

As expected, the bucket and files we have inserted are also in other servers URLs shown above.

Now we’ll do a new test. This time we use another workstation that accesses the mini-server using the client console called mc.

From the client side, we create a file and then upload it to the existing bucket.

Then as an end result, we expect to see from the dashboard that the new file that is uploaded from the client side automatically exists.

First open the client workstation and download the mini-client package. An example is shown below:

 [[email protected] ~]# wget https://dl.min.io/client/mc/release/linux-amd64/mc 
--2019-09-30 11:47:38-- https://dl.min.io/client/mc/release/linux-amd64/mc
Resolving dl.min.io (dl.min.io)... 178.128.69.202
Connecting to dl.min.io (dl.min.io)|178.128.69.202|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 16592896 (16M) [application/octet-stream]
Saving to: ‘mc’

100%[==============================================================================>] 16,592,896 741KB/s in 1m 59s

2019-09-30 11:49:37 (137 KB/s) - ‘mc’ saved [16592896/16592896]

[[email protected] ~]# chmod +x mc

Then configure from the client side to access the dedicated bucket using create access key and secret. Examples as follows:

 [[email protected] ~]# ./mc config host add myminio http://10.124.12.142:9000 shahril shahril123 
mc: Configuration written to `/root/.mc/config.json`. Please update your access credentials.
mc: Successfully created `/root/.mc/share`.
mc: Initialized share uploads `/root/.mc/share/uploads.json` file.
mc: Initialized share downloads `/root/.mc/share/downloads.json` file.
Added `myminio` successfully.

Once configured, you should be able to see the contents of the existing bucket. Examples as follows:

 [[email protected] ~]# ./mc ls myminio 
[2019-09-30 11:16:25 +08] 0B mylove/ [[email protected] ~]# ./mc ls myminio/mylove/
[2019-09-30 11:16:25 +08] 55KiB myself.jpg

Create now or upload existing file from the client side of the bucket. Examples as below: –

 [[email protected] ~]# ./mc cp new_file.txt myminio/mylove 
new_file.txt: 38 B / 38 B ???????????????????????????????????????????????????????????????? 100.00% 1.02 KiB/s 0s
[[email protected] ~]# [[email protected] ~]# ./mc ls myminio/mylove/
[2019-09-30 11:16:25 +08] 55KiB myself.jpg
[2019-09-30 11:58:16 +08] 38B new_file.txt

When you are done, as expected when updating from the dashboard page via any of the server’s URLs, you should see the new file appear there as shown below.

File uploaded with MinIO consoleAdvertisement

You should see the full link to the image when you click on the share icon on your right side as below. This is the unique link for each object inside the bucket that you can use on the application page via curl or API.

Share items

Thumbs up! now we have successfully configured and configured our own on-site hosting service using Minio. For more detailed details, you can check its documentation here


Source link