The Metric Network

Kibana: Part of this complete breakfast metrics solution

Kibana is complemented by a range of free and open-source tools that can help you aggregate and deliver metrics, analytics and vital performance data. Here you'll find a log analysis infrastructure that can be dropped directly into an existing centralized syslog system, without affecting legacy applications.

Parts List Your analytics pantry

  • Syslog-ng - Syslogd
  • Logstash - Powerful log pipeline
  • RabbitMQ or Redis - For queuing messages
  • Elasticsearch - Full text document storage and search
  • Graphite - From Orbitz, Scalable real-time graphing
  • Statsd - From Etsy, counts occurrences of fields and ships to graphite
  • Graphital - A ruby daemon to send host level performance data to graphite
  • Kibana - A browser based log analysis front end for Logstash and Elasticsearch

How it all fits together Like a well oiled, uh, something that gets oiled.

The setup Getting your logs into Logstash

  • Network nodes log to syslog, in this case Syslog-ng.
  • Syslog-ng delivers logs to a centralized syslog server, also Syslog-ng. The central syslog server has 2 outputs:
    • TCP is used to forward events to Logstash and to allow for seperation of the Logstash instance if needed. Logstash could also simply read the files that are outputted by syslog-ng. Why not just archive everything in Elasticsearch? That's an option, however flat files allow for longer historical retention; Compressed flat files are 10% or less of the size of Elasticsearch documents.
  • Logstash uses an AMQP output to a RabbitMQ exchanged named 'Logstash'. At this stage, Logstash's only role is to queue logs for processing by a filtering cluster. Without significant volume logs could be processed/filtered at this level.
  • Queuing, with RabbitMQ or Redis, allows for horizontal scaling as needed.

The real work So I'm here, whats goin on?

  • A cluster of Logstash filter workers pull input from RabbitMQ. The Logstash filters are the heavy lifters; they process syslog messages into easy to analyze fields. For example, a web server log line might use grok to break the event down into request, response, client and agent. The process of breaking an event into pieces is usually handled by Logstash's grok filter, but Logstash has lots of useful filters for working with events.
  • Logstash has 2 outputs at this level:
    • Full events are sent to a second RabbitMQ exchange called 'Elasticsearch'
    • Selected fields are sent to Statsd (more on this in the Pretty Lines section)
  • Logstash pulls from the 'Elasticsearch' exchange and indexes events into their final resting place, Elasticsearch.
  • Kibana talks to Elasticsearch and presents a friendly face to the recently parsed event data
Why the 2nd AMQP exchange?

Why don't the Logstash filter workers write directly to Elasticsearch? Good question. The answer involves pretty lines. Logstash will stop reading from the 'Logstash' AMQP exchange if it can't write to Elasticsearch, such as when it's under heavy load. When this happens, filtering can't happen, and the real-time data that Statsd relies on can not be delivered.

Pretty lines powered by Graphite

  • The other Logstash filter output: Statsd
  • Logstash picks out some key fields from events, such the HTTP 'response' field, and fires them off to statsd
  • Statsd counts up the occurences of each unique value it recieves and sends the totals onto Graphite.
  • Graphite charts values and displays them as time series data to which can be applied apply filters and functions
  • A world away from log processing: Graphital is collecting host level vitals and providing them to Graphite
  • Graphite allows for dragging and dropping of graphs on top of each other. Statsd output can be directly compared to host level vital signs to correlate events