Posted in Data, ElasticSearch, Kibana

Elasticsearch, Kibana, Logstash Oh My!

I don’t have access to our production logs. That stinks. We’ve been discussing internally on how to gain access, so with a little searching I found this wonderful gem called logstasher. It provides a way for our rails app to convert its logs to JSON to allow logstash to grab and send to elasticsearch then the Cou de Gras! —Kibana. Kibana provides a visual interface to view and query against elasticsearch.

Installation on my local was more involved since the installation instructions were not well documented. I had to figure it out between installing the gem and finally installing the above apps. The gem says that kibana comes with logstash but I couldn’t get that to work.

For MacOS–El Capitan:

$brew install elasticsearch

$brew install logstash

$brew install kibana

$vi ~/path/to/kibana.yml

uncomment the default line for elasticsearch url

In your rails app, add gem ‘logstasher’ to the Gemfile and bundle. Add the following to your development.rb file:

#Enable the logstasher logs for the current env

config.logstasher.enabled = true

Create a name_that_makes_sense.conf file in directory where you will maintain your logstasher configurations(you could add it to your app root) and add these lines:

input {

file {

type => “rails”

path => “/Users/username/app_name/log/development.log”

codec => json {

charset => “UTF-8”

}

}

}

output {

# Print each event to stdout.

stdout {

codec => rubydebug

}

elasticsearch {

hosts => [“localhost:9200”]

}

}

After you save that, start all the services if you havent already(you will need to restart kibana if you setup the yml file after you already started it):

$elasticsearch & #you can browse to localhost:9200 and see if you get a response

$kibana & #kibana has its on webserver with the latest release

$logstash -f path/to/name_that_makes_sense.conf

$rails s

Then open a browser at http://localhost:5601

The kibana application will not let you configure any indices until your logstash setup is sending data to elasticsearch, so the app needs to serve some request before it has data to search and query. It will not initialize and convert an existing log.

Have Fun!

Advertisements

Author:

I love finding resources that help me do life.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s