AWS ElasticSearch를 이용한 CloudFront 로그 분석(Amazon Linux/ES)

ElasticSearch 설치

ElasticSearch: 5.4

https://www.elastic.co/guide/en/beats/libbeat/current/getting-started.html

EC2 Discovery Plugin(Cluster)

https://www.elastic.co/guide/en/elasticsearch/plugins/current/discovery-ec2.html

elasticsearch-head Plugin(5.x site plugin 지원하지 않음)

https://github.com/mobz/elasticsearch-head

Limit

vim /etc/security/limits.conf

ec2-user soft memlock unlimited

ec2-user hard memlock unlimited

Permission 설정(chown ec2-user:ec2-user elastcsearch-x.x.x)

vim /etc/elasticsearch/jvm.options

-Xms2g

-Xmx2g

Cluster Config

Config(vim /etc/elasticsearch/elasticsearch.yml)

node.name: master

network.host: 0.0.0.0

http.port: 9200

discovery.zen.ping.unicast.hosts: ["172.31.19.132", "172.31.30.4"]

#

# Prevent the "split brain" by configuring the majority of nodes (total number of master-eligible nodes / 2 + 1):

#

discovery.zen.minimum_master_nodes: 2

#

# For more information, consult the zen discovery module documentation.

#

# ---------------------------------- Gateway -----------------------------------

#

# Block initial recovery after a full cluster restart until N nodes are started:

#

#gateway.recover_after_nodes: 3

#

# For more information, consult the gateway module documentation.

#

# ---------------------------------- Various -----------------------------------

#

# Require explicit names when deleting indices:

#

#action.destructive_requires_name: true

cloud:

        aws:

                access_key: AKIAJHxxxxxxxQ

                secret_key: iIwUrMxxxxxxxxxN5Ar1JzeX8x5

                region: ap-northeast-2

discovery:

        type: ec2

node 상태 확인: curl -XGET 'localhost:9200/_cat/nodes?v&pretty'

cluster 상태 확인: curl -XGET 'localhost:9200/_cat/health?v&pretty'

172.31.30.4              2          58   3    0.14    0.12     0.05 mdi       *(matster)    node-2

172.31.19.132            3          61  12    0.41    0.16     0.06 mdi       -      node-1

실행

service elasticsearch start/stop/status

테스트

GET: curl -XGET http://localhost:9200/classes?pretty 

PUT: curl -XPUT http://localhost:9200/classes?pretty

DELETE: curl -XDELETE http://localhost:9200/classes?pretty // curl -XDELETE http://localhost:9200/*

POST: curl -XPOST http://localhost:9200/classes/class/1/?pretty -d '{"title":"Algorithm"}'

POST: curl -XPOST http://localhost:9200/classes/class/1/?pretty -d @class.json

Logstash 설치

Logstash: https://www.elastic.co/guide/en/logstash/2.3/installing-logstash.html

bin/logstash --version

bin/plugin list

bin/plugin update

Amazon ES 플러그인 설치

bin/logstash-plugin install logstash-output-amazon_es

Config

vim /usr/share/logstash/*.conf

input {
  s3 {
    bucket => "leedoing-log-es2"
    prefix => ""
    region => "ap-northeast-2"
    access_key_id => "AKIAJTK7PN7PNT5L******A"
    secret_access_key => "EQLDNeVdbt*********kyGd77xBqegZcnST0QG4"
  }
}
filter {
  grok {
    match => { "message" => "%{DATE_EU:date}\t%{TIME:time}\t%{WORD:x_edge_location}\t(?:%{NUMBER:sc_bytes:int}|-)\t%{IPORHOST:c_ip}\t%{WORD:cs_method}\t%{HOSTNAME:cs_host}\t%{NOTSPACE:cs_uri_stem}\t%{NUMBER:sc_status:int}\t%{GREEDYDATA:referrer}\t%{GREEDYDATA:User_Agent}\t%{GREEDYDATA:cs_uri_stem}\t%{GREEDYDATA:cookies}\t%{WORD:x_edge_result_type}\t%{NOTSPACE:x_edge_request_id}\t%{HOSTNAME:x_host_header}\t%{URIPROTO:cs_protocol}\t%{INT:cs_bytes:int}\t%{GREEDYDATA:time_taken}\t%{GREEDYDATA:x_forwarded_for}\t%{GREEDYDATA:ssl_protocol}\t%{GREEDYDATA:ssl_cipher}\t%{GREEDYDATA:x_edge_response_result_type}" }
  }
 
  mutate {
    add_field => [ "listener_timestamp", "%{date} %{time}" ]
  }
 
  date {
    match => [ "listener_timestamp", "yy-MM-dd HH:mm:ss" ]
    target => "@timestamp"
  }
 
  geoip {
    source => "c_ip"
  }
 
  useragent {
    source => "User_Agent"
    target => "useragent"
  }
 
  mutate {
    remove_field => ["date", "time", "listener_timestamp", "cloudfront_version", "message", "cloudfront_fields", "User_Agent"]
  }
}
output {
        elasticsearch {
                hosts => "localhost:9200"
                index => "logstash-%{+YYYY.MM.dd}"
        }
}
 

실행

bin/logstash -f *.conf --debug

Logstash Config(AWS ES)

참고: https://gist.github.com/s-fujimoto/ed2cd96b2efafde57a23995f6a15ce13

인덱스 확인: curl 'localhost:9200/_cat/indices?v'

Kibana 설치

vim /etc/yum.repos.d/kibana.repo

[kibana-4.4]

name=Kibana repository for 4.4.x packages

baseurl=http://packages.elastic.co/kibana/4.4/centos

gpgcheck=1

gpgkey=http://packages.elastic.co/GPG-KEY-elasticsearch

enabled=1

yum -y install kibana

vim /opt/kibana/config/kibana.yml

server.host: "0.0.0.0"

systemctl start kibana

chkconfig kibana on

실행

/bin/kibana serve

kibana 확인(default port 5601)

참고

https://aws.amazon.com/ko/blogs/korea/category/amazon-elasticsearch-service/

https://aws.amazon.com/ko/blogs/database/get-started-with-amazon-elasticsearch-service-how-many-data-instances-do-i-need/

https://aws.amazon.com/ko/blogs/database/get-started-with-amazon-elasticsearch-service-how-many-shards-do-i-need/

CloudWatch Logs-Lambda-ES

https://gist.github.com/torgeir/aa3c28c336fc977b9ebf10784bc1a666

추가 ELB LOG Config

input {
    s3 {
        bucket => "clipper-logs-elb"
        prefix => "gu8/05/14"
        region => "ap-northeast-2"
        type => "elb"
        access_key_id => "AKIKVQ"
        secret_access_key => "ypwsdqunKVdUkQ5"
    }
}
filter {
    grok {
        match => [ "message", "%{ELB_ACCESS_LOG}" ]
    }
 
    date {
        match => [ "timestamp", "ISO8601" ]
        target => "@timestamp"
    }
 
    geoip {
        source => "clientip"
    }
 
    mutate {
        remove_field => ["timestamp", "message"]
    }