Contents

Fluentd

Fluentd

Build Your Unified Logging Layer

Fluentd is an open source data collector for unified logging layer.

Fluentd allows you to unify data collection and consumption for a better use and understanding of data.

www.fluentd.org

Unified Logging Layer

Fluentd decouples data sources from backend systems by providing a unified logging layer in between.

Simple yet Flexible

Fluentd’s 500+ plugins connect it to many data sources and outputs while keeping its core simple.

Plugins are written in Ruby and hosted in rubygems

Source

Proven

5,000+ data-driven companies rely on Fluentd. Its largest user currently collects logs from 50,000+ servers. Testimonials

Log data is for machines

https://www.fluentd.org/blog/unified-logging-layer

Architecture

Architecture

Life of a fluentd event

Event Structure

  • tag: Specifies the origin where an event comes from. It is used for message routing.

  • time: Specifies the time when an event happens with nanosecond resolution.

  • record: Specifies the actual log as a JSON object.

https://livebook.manning.com/book/logging-in-action/chapter-2/v-10/14

Configuration File

The configuration file is the fundamental piece to connect all things together, as it allows to define which Inputs or listeners Fluentd will have and set up common matching rules to route the Event data to a specific Output.

<source>
  @type http
  port 44444
  bind 0.0.0.0
</source>

# accept all log events regardless of tag and write them to the console
<match **>
    @type stdout
</match>

docker run -p 44444:44444 -v $(pwd)/conf:/fluentd/etc fluent/fluentd:edge-debian -c /fluentd/etc/helloworld.conf
curl -X POST -d 'json={"json":"message"}' http://127.0.0.1:44444/sample.test

Processing Events

When a Setup is defined, the Router Engine contains several predefined rules to apply to different input data. Internally, an Event will to pass through a chain of procedures that may alter its lifecycle.

Plugin types

Fluentd has nine (9) types of plugins:

  • Input

  • Parser

  • Filter

  • Output

  • Formatter

  • Storage

  • Service Discovery

  • Buffer

  • Metrics

https://docs.fluentd.org/input

Examples

previous

Logstash