FluentBit to SigNoz

If you use fluentBit to collect logs in your stack with this tutotrial you will be able to send logs from fluentBit to SigNoz.

At SigNoz we use opentelemetry collector to recieve logs which supports the fluentforward protocol. So you can forward your logs from your fluentBit agent to opentelemetry collector using fluentforward protocol.

Collect Logs Using FluentBit in SigNoz cloud

  • Add otel collector binary to your VM by following this guide.

  • Add fluentforward reciever to your config.yaml

    receivers:
      fluentforward:
        endpoint: 0.0.0.0:24224
    

    Here we have used port 24224 for listening in fluentforward protocol, but you can change it to a port you want. You can read more about fluentforward receiver here.

  • Modify your config.yaml and add the above receiver

    service:
        ....
        logs:
            receivers: [otlp, fluentforward]
            processors: [batch]
            exporters: [otlp]
    
  • Add the following to your fluentBit config to forward the logs to otel collector.

    [OUTPUT]
      Name          forward
      Match         *
      Host          localhost
      Port          24224
    

In this config we are forwarding the logs to the otel collector which is listening on port 24224. Also we are assuming that you are running the fluentBit binary on the host. If not, the value of host might change depending on your environment.

  • Once you make this changes you can restart fluentBit and otel-binary, and you will be able to see the logs in SigNoz.
  • To properly transform your existing log model into opentelemetry log model you can use the different processors provided by opentelemetry. link

eg:-

processors:
  logstransform:
    operators:
      - type: trace_parser
        trace_id:
          parse_from: attributes.trace_id
        span_id:
          parse_from: attributes.span_id
      - type: remove
        field: attributes.trace_id
      - type: remove
        field: attributes.span_id

The operations in the above processor will parse the trace_id and span_id from log to opentelemetry log model and remove them from attributes.

Collect Logs Using FluentBit in Self-Hosted SigNoz

  • Add fluentforward reciever to your otel-collector-config.yaml which is present inside deploy/docker/clickhouse-setup

    receivers:
      fluentforward:
        endpoint: 0.0.0.0:24224
    

    Here we have used port 24224 for listening in fluentforward protocol, but you can change it to a port you want. You can read more about fluentforward receiver here.

  • Update the pipleline for logs by making the following change in otel-collector-config.yaml

    service:
      ...
    
      logs:
        receivers: [ otlp, fluentforward ]
        processors: [ batch ]
        exporters: [  clickhouselogsexporter ]
    

    Here we are updating the logs pipeline which will collect logs from fluentforward and otlp receiver, processing it using batch processor and export it to clickhouse.

  • Expose the port in port for otel-collector in docker-compose.yaml file present in deploy/docker/clickhouse-setup

    otel-collector:
      ...
      ports:
        - "24224:24224"
    
  • Change the fluentBit config to forward the logs to otel collector.

    [INPUT]
      Name   dummy
      Tag    dummy.log
      Dummy {"message": "mylog", "trace_id": "0000000000000000f4dbb3edd765f620", "span_id": "43222c2d51a7abe3"}
    
    [OUTPUT]
      Name          forward
      Match         *
      Host          <otel-collector-host>
      Port          24224
    

    In this example we are generating sample logs and then forwarding them to the otel collector which is listening on port 24224. <otel-collector-host> has to be replaced by the host where otel-collector is running. For more info check troubleshooting.

  • Once you make this changes you can restart fluentBit and SignNoz, and you will be able to see the logs in SigNoz.

  • To properly transform your existing log model into opentelemetry log model you can use the different processors provided by opentelemetry. link

    eg:-

    processors:
      logstransform:
        operators:
          - type: trace_parser
            trace_id:
              parse_from: attributes.trace_id
            span_id:
              parse_from: attributes.span_id
          - type: remove
            field: attributes.trace_id
          - type: remove
            field: attributes.span_id
    

    The operations in the above processor will parse the trace_id and span_id from log to opentelemetry log model and remove them from attributes.

Was this page helpful?