Managing logs with ELK


#1

ELK is a famous and powerful log tool. ELK stands for Elasticsearch, Logstash and Kibana.

Preview:

Prerequisites:

[root@iZ259zncm66Z kibana-6.4.0-linux-x86_64]# java -version
java version "1.8.0_25"
Java(TM) SE Runtime Environment (build 1.8.0_25-b17)
Java HotSpot(TM) 64-Bit Server VM (build 25.25-b02, mixed mode)

Install ELK:

download and unzip the following files.

wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-6.4.0.tar.gz
wget https://artifacts.elastic.co/downloads/logstash/logstash-6.4.0.tar.gz
wget https://artifacts.elastic.co/downloads/kibana/kibana-6.4.0-linux-x86_64.tar.gz

Configuration:

  1. Run elasticsearch.

cd path/to/elasticsearch
bin/elasticsearch
curl http://localhost:9200/

  1. Run Kibana

cd path/to/kibana
open config/kibana.yml and set elasticsearch.url to point at your Elasticsearch instance, set server.host to external ip
bin/kibana
curl http://localhost:5601

3.Run Logstash on each cosmos node.

cd path/to/logstash
create a cosmos.conf.
bin/logstash -f cosmos.conf

cosmos.conf:

input {
  file {
     path => "/root/.gaiad/out.log"
     codec => multiline {
      pattern => "^[\s\}]"
      what => "previous"
    }
  }
}
filter {
  if [message] =~ ".*Block\{.*" {
    dissect {
      mapping => {
        "message" => "%{level}[%{logtime}]"
      }
    }   
    kv { }
  }else if [message] =~ "^E" {
    dissect {
      mapping => {
        "message" => "%{level}[%{logtime}]"
      }
    }
    kv { } 
  }else{
    drop { }
  }
}

output {
    elasticsearch {
        hosts => ["127.0.0.1:9200"]  #edit this value to your elasticsearch host
    }
   #stdout { codec => rubydebug }
}

Then you can manage your logs in one log management server. (Perhaps you will need wait a couple of minutes to see logs on kibana)

Additional:

i droped some logs like precommited in cosmos.conf. if you want to see them, replace drop {} with:

dissect {
      mapping => {
            "message" => "%{level}[%{logtime}]"
      }
}
kv { } 

I hope you will enjoy this amazing tool.


#2

Big fan of Kibana myself. Thanks for sharing this!


#3

Can you also share some of the queries you are running to alert on?


#4

Sure, Here are some examples.

1 Blocks Count

2 Error Peers Count

3.Error Peers Location

4 Log For Block #height

query:

BlockHeight:158023

result:


5 Logs For Block Range:

query:

BlockHeight:[158023 TO 158024]

6 Block with transactions

7 Block with evidence ("" means no evidence )

8 Top errors from Info level

If you have any suggestions be free to write a post here.