Home / How To / How to install Elastic Stack (Elasticsearch, Logstah and Kibana) on CentOS 8

How to install Elastic Stack (Elasticsearch, Logstah and Kibana) on CentOS 8



Elasticsearch is an open source search engine based on Lucene, developed in Java. It provides a distributed and multitenant full text search engine with an HTTP Dashboard web interface (Kibana). The information is requested, retrieved and stored with a JSON document schema. Elasticsearch is a scalable search engine that can be used to search for all types of text documents, including log files. Elasticsearch is the heart of & # 39; Elastic Stack & # 39; or ELK Stack.

Logstash is an open source tool for managing events and logs. It provides real-time pipelining for data collection. Logstash collects your log data, converts data into JSON documents and stores them in Elasticsearch.

Kibana is an open source data visualization tool for Elasticsearch. Kibana provides a beautiful dashboard web interface. It lets you manage and visualize data from Elasticsearch. It is not only beautiful, but also powerful.

In this tutorial, we will show you step by step how to install and configure & # 39; Elastic Stack & # 39; on the CentOS 8 server. We will install and configure Elasticsearch, Logstash and Kibana. And then set Beats & # 39; filebeat & # 39; on Ubuntu clients and CentOS systems.

Prerequisites

  • CentOS 8 64 bits with 4 GB RAM ̵
    1; elk-master
  • CentOS 8 64 bits with 1 GB RAM – client01
  • Ubuntu 18.04 64 bits with 1 GB RAM – client02 [19659009] What to do:

    • Add Elastic Repository to CentOS 8 Server
    • Install and configure Elasticsearch
    • Install and configure Kibana Dashboard [19659007] Setting Nginx as a reverse proxy for Kibana
    • Install and configure Logstash
    • Installing and Configuring Filebeat
    • Testing

    Step 1 – Add Elastic Repository

    First we will add Elasticsearc key and store to CentOS 8 server. With the elasticsearch repository provided by elastic.co, we can install elastic products including Elasticsearch, Logstash, Kibana and Beats.

    Add the elastic key to the system with the following command.

      rpm - import https://artifacts.elastic.co/GPG-KEY-elasticsearch 12919659021 ?? When you go to directory & # 39; /etc/yum.repos.d' and creates a new archive file & # 39; elasticsearch.repo & # 39 ;. [19659020] cd /etc/yum.repos.d/strong19459016achtevim elasticsearch.repo 

    Paste the elasticsearch layer below.

    [elasticsearch-7.x]
    name = Elasticsearch archive for 7.x packages
    baseurl = https: // artifacts .elastic.co / packages / 7.x / yum
    gpgcheck = 1
    gpgkey = https: / /artifacts.elastic.co/GPG-KEY-elasticsearch
    enabled = 1
    autorefresh = 1
    type = rpm-md

    Save and close.

    Now check all available repositories on the system using the dnf command below.

      dnf repolist 

    And you will get the elastic research repository that has b added to the CentOS 8 server.

     Download storage list with the command dnf

    As a result, you can install Elastic products like Elasticsearch, Logstash and Kibana, etc.

    Step 2 – Install and configure Elasticsearch

    In this step we will install and configure Elasticsearch.

    Install the Elasticsearch package with the dnf command below.

      sudo dnf install elasticsearch-y 

    Once the installation is complete, go to the & # 39; / etc / elasticsearch & # 39; directory and edit the configuration file & # 39; elasticsearch.yml & # 39; with the vim editor.

      cd / etc / elasticsearch / 
    vim elasticsearch.yml

    Deselect the following rows and change the value for each row as below.

      network.host: 127.0.0.1 
    http.port: 9200

    Save and close.

    Optional: [19659002] You can set elasticsearch by editing the JVM configuration file & # 39; jvm.options & # 39; and set the high-size base d on how much memory you have.

    Edit the JVM configuration & # 39; jvm.options & # 39; with the vim editor.

      vim jvm.options 

    Change the min / max size via the Xms and Xmx configuration as below.

      -Xms512m 
    -Xmx512m

    Save and close.

    Then load the systemd manager configuration and add the elasticsearch service to the start time.

      sudo systemctl daemon-reload 
    sudo systemctl enable elasticsearch

    Then start the elasticsearch service.

      sudo systemctl start elasticsearch 

     Configure and start elastic search

    As a result, elasticsearch is running behind the local IP address & # 39; 127.0.0.1 & # 39; with the default port & # 39; 9200 & # 39; on the CentOS 8 server.

    You can check elasticsearch using the curl command below.

      curl -XGET & # 39; http://127.0.0.1:9200/?pretty & # 39; 

    And below is the result you will get.

     Test cachection for Elasticsearch

    Step 3 – Install and configure Kibana Dashboard

    After installing elasticsearch we will install and configure Kibana Dashboard on server CentOS 8.

    Install Kibana dashboard with the command dnf below .

      sudo dnf install kibana 

    When the installation is complete, go to the & # 39; / etc / kibana & # 39; directory and edit the configuration file & # 39; kibana.yml & # 39 ;.

      cd / etc / kibana / 
    vim kibana.yml

    Uncheck and change some line configuration as below.

      server.port: 5601 
    server.host: "127.0.0.1"
    elasticsearch.url: "http://127.0.0.1:9200"ebrit19659021] Save and close.

    Then add the kibana service to the system startup and start the kibana service.

      sudo systemctl enable kibana 
    sudo systemctl start kibana

     Install Kibana Dashboard [19659002] The Kibana service is running on the CentOS 8 server, check it with the following commands.

      systemctl status kibana 
    netstat -plntu

    And you will get the result as below.

     Checking the status of Kibana service

    As a result, the Kibana service is running and running the standard TCP port "5601 & # 39;

    Step 4 - Configure Nginx as a reverse proxy for Kibana

    In this step we will install the Nginx web server and set it as a reverse proxy for Kibana Dashboard.

    Install Nginx and httpd-tools using the dnf command below.

      sudo dnf install nginx httpd-tools 

    When the installation is complete, go to the & # 39; /etc/nginx/conf.d' directory and create a new configuration file & # 39; kibana.conf & # 39; [19659019] cd /etc/nginx/conf.d/vud19459016achtevim kibana.conf

    Paste the following configuration.

      server {
    listen 80;

    server name elk.hakase-labs.io;

    authentic_basic "Restricted Access";
    auth_basic_user_file /etc/nginx/.kibana-user;emand19459016]
    location / {
    proxy_pass http: // 127 .0.0.1: 5601; ebrit19459016] proxy_http_version 1.1;
    proxy_set_header Upgrade $ http_upgrade;
    proxy_set_header Connection & # 39; upgrade & # 39 ;;
    proxy_set_header Host6 host01] proxy_cache_bypass $ http_upgrade;
    }
    }

    Save and close.

    Next, we need to create the basic authentication to secure Kibana access. Change the "hakase" user with your own and run the htpasspwd command as below.

      sudo htpasswd -c /etc/nginx/.kibana-user hakase 
    TYPE YOUR PASSWORD

    Type your password and test nginx configuration.

      nginx -t 

    Make sure there is no error.

    Now add the nginx service to the system boot and start the nginx service.

      systemctl enable nginx 
    systemctl start nginx [19659021]   Configure Nginx as a reverse proxy

    As a result, Nginx installation and configuration as a reverse proxy for Kibana Dashboard has been completed.

    Step 5 - Install and Configure Logstash

    In this step we will install and configure the logstash log sender. We will install logstash, set input beats, set syslog filtering with logstash plugin called & # 39; grok & # 39; and then set the output to elasticsearch.

    Install logstash with the dnf command below.

      sudo dnf install logstash 

    When the installation is complete, go to the directory & # 39; / etc / logstash & # 39; and edit the JVM configuration file & # 39; jvm.options & # 39; with the vim editor.

      cd / etc / logstash / 
    vim jvm.options

    Change the min / max size via the Xms and Xmx configuration as below.

      -Xms512m 
    -Xmx512m

    Save and close.

    Next, go to the & # 39; / etc / logstash / conf.d & # 39; directory and create the beats input configuration file called & # 39; input-beat.conf & # 39 ;.

      cd /etc/logstash/conf.d/vud19459016??vim input-beat.conf 

    Paste the following configuration.

      input {
    beats {
    port => 5044
    }
    }

    Save and close.

    Now create & # 39; syslog-filter.conf & # 39; configuration file.

      vim syslog-filter.conf 

    Paste the following configuration.

      filter {
    if [type] == "syslog" {
    grok {
    match => {"message" => "% {SYSLOGTIMESTAMP: syslog_timestamp}% {SYSLOGHOST: syslog_hostname}% { DATA: syslog_program} (?: [%{POSINT:syslog_pid}])?:% {GREEDYDATA: syslog_message} "}
    add_10 => 1965] Add_field => [ "received_from", "%{host}" ]
    }
    date {
    match => [19659109]}
    }
    }

    Save and close.

    And then create the initial configuration for elasticsearch & # 39; output-elasticsearch.conf & # 39 ;.

      vim output-elasticsearch.conf 

    Paste the following configuration.

      output {
    elasticsearch {hosts => ["127.0.0.1:9200"]
    hosts => "127.0.0.1:9200"
    manage_template => false
    index => "% {[@metadata][beat]} -% {+ YYYY.MM.dd} "
    document_type =>"% {[@metadata][type]} "
    }
    }

    Save and close.

    Then add the logstash service to the system boot and start the logstash service.

      systemctl enable logstash 
    systemctl start logstash

    <img src = "https://www.howtoforge.com/images/how-to-install-elastic-stack-on-centos-8/9. png "alt =" Configure Logstash
    "width =" 750 "height =" 266 "style =" display: block; margin-left: auto; margin-right: auto; "/>

    The Logstash service is running, check using the following commands.

      systemctl status logstash 
    netstat -plntu

    And you will get the result as below.

     Check logstash status

    As a result, the logstash log driver is running and running on the CentOS 8 server with the default TCP port & # 39; 5044 & # 39; And the basic Elastic Stack installation is complete, and we are ready to send and monitor our logs to the Elastic (ELK Stack) server.

    Step 6 - Install Filebeat on Client

    In this step, we will show you how to set the filebeat on Ubuntu and the CentOS system. We will install a filebeat and configure to send logs from both servers to Logstash on the resilient server.

    - Install Filebeat on CentOS 8

    Add the elasticsearch key to the CentOS 8 system with the following command.

      rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearchebrit1919909021Now goes to directory & # 39; /etc/yum.repos.d&#39; and creates the & # 39; elasticsearch.repo & # 39; with the vim editor. 

      cd /etc/yum.repos.d/strong19459016??vim elasticsearch.repo 

    Paste the following configuration.

    [elasticsearch-7.x]
    name = Elasticsearch repository for 7.x packages
    baseurl = https: // artifacts .elastic.co / package / 7.x / yum
    gpgcheck = 1
    gpgkey = https: // artifacts.elastic.co/GPG-KEY-elasticsearch
    enabled = 1
    autorefresh = 1
    type = rpm-md

    Save and close.

    Now install filebeat using the dnf command below.

      sudo dnf install filebeat 

    Please wait until the filebeat installation is complete.

     Install Filebeat

    - Install Filebeat on Ubuntu 18.04

    First install the apt-transport-https packages.

      sudo apt install apt-transport-https 

    Then add the elasticsearch key and archive with the following commands.

      wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add - 
    echo "deb https://artifacts.elastic.co/packages/7.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-7.x.list??19659021??Nu updates all repositories and installs filebeat to the ubuntu system using the apt command below.

      sudo apt update 
    sudo apt install filebeat

    Please wait until the filebeat installation is complete.

     Install apz https transport

    - Configure Filebeat

    The filebeat configuration found in the & # 39; / etc / filebeat & # 39; directory. Go to the filbeat directory and edit the configuration file & # 39; filebeat.yml & # 39;

      cd / etc / filebeat / 
    vim filebeat.yml

    Now disable the standard output for elasticsearch as below.

      #output. elasticsearch: 
    # Array of hosts to connect to.
    # Hosts: ["127.0.0.1:9200"]

    Then enable the logstash output and enter the logstash host's IP address.

      output.logstash: 
    # Logstash hosts
    hosts: ["10.5.5.25:5044"]

    Save and close

    Then we need to enable filebeat modules. Run the filebeat command below to get a list of filebeat modules.

      filebeat module list 

    Enable & # 39; system & # 39; module with the following command.

      with the configuration file & # 39; modules.d / system.yml & # 39 ;. 

    Edit the configuration of the system module with vim editor.

      cd / etc / filebeat / 
    vim modules.d / system.yml

    Uncomment the path for the syslog file and the ssh permission file.

    For the CentOS system:

      # Syslog 
    syslog:
    enabled: true
    var.paths: ["/var/log/messages"]

    # Authorization logs
    auth:
    enabled: true
    var.paths: ["/var/log/secure"]

    For the Ubuntu system:

      # Syslog 
    syslog:
    enabled: true
    var.paths: ["/var/log/syslog"]

    # Authorization logs
    authorization :
    enabled: true
    var.paths: ["/var/log/auth.log"]

    Save and close.

    Now add the filebeat service to the system boot and start the service.

      systemctl enable filebeat 
    systemctl start filebeat

    The filebeat service is running, you can check with the following command.

      systemctl status filebeat 

    And you will see the result as below.

    Below is the result from the CentOS 8 server.

     results from the CentOS server

    And below is from Ubuntu Server 18.04.

     Results from Ubuntu 18.04 server

    As a result, the relationship between filebeat and logstash service on the elastic stack server's IP address has been established .

    Step 7 - Test

    Open your web browser and type the Elastic Stack installation domain name in the address bar.

    http: / /elk.hakase-labs.io/??19659002??Now log in to the Kibana Dashboard using the basic authentication account you created.

     Kibana Login

    And you will get the Kibana Dashboard as below.

     Kibana Dashboard

    Now connect to elasticsearch index data that is created automatically after file type connected to logstash. Click the & # 39; link to your Elasticsearch index & # 39;

    Create " filebeat - * " index pattern and click the " Next Step " button.

     Create filebeat index pattern

    Select filter @timestamp and click " Create index pattern ".

     Select filter name

    And " filebeat - * " index pattern created, click " Discover " on the left.

     Filebeat in the Discover menu

    And you get log data from filebeat clients as below.

    Logs for CentOS 8 systems.

     Logs of the CentOS 8 system

    Log for the Ubuntu system.

     Logs of the Ubuntu system

    As a result, log data defined in the filebeat system module has been sent to elas tic stack server.

    And the installation and configuration of Elastic Stack on CentOS 8 has been completed.

    Reference


Source link