fluent-plugin-sflow

This branch is valid for Fluentd 0.14.x or later

See 0.12.x branch for Fluentd 0.12.x.

Overview

Fluentd input plugin that acts as sFlow v2/v4/v5 collector.

Including a ruby wrapper of sflowtool which processes sFlow datagrams to report in JSON format.

Features

fluent-plugin-sflow supports many packet formats below. Basically, it can process anything sflowtool can do. ✨

See sflowtool document for more details.

Sample Type

• Flow Sample
• Counter Sample

Address Family

• IPv4
• IPv6

Extended Data Type

• Switch
• Router
• Gateway_v2
• Gateway
• User
• Url
• mplsLabelStack
• Mpls
• Nat
• NatPort
• MplsTunnel
• MplsVC
• MplsFTN
• MplsLDP_FEC
• VlanTunnel
• WifiPayload
• WifiRx
• WifiTx
• Aggregation
• Socket4
• ProxySocket4
• Socket6
• ProxySocket6
• Decap
• VNI
• TCPInfo

Counter Data Type

• ethernet
• tokenring
• vg
• vlan
• 80211
• processor
• radio
• OFPort
• portName
• OVSDP
• host_hid
• adaptors
• host_parent
• host_cpu
• host_mem
• host_dsk
• host_nio
• host_ip
• host_icmp
• host_tcp
• host_udp
• host_vnode
• host_vcpu
• host_vmem
• host_vdsk
• host_vnio
• host_gpu_nvml
• bcm_tables
• memcache
• memcache2
• http
• JVM
• JMX
• APP
• APP_RESOURCE
• APP_WORKERS
• VDI
• LACP
• SFP

Requirement

• Fluentd: 0.14.x or lator
  • See fluentd-0.12 branch for Fluentd 0.12.x.
• gcc and make
• Ruby dev package

Install

You need to install development tools beforehand.

📝 If you see any problem on other platforms rather than Debian, Ubuntu, or CentOS, please open a new issue.

Debian Linux or Ubuntu Linux

$ sudo apt install -y gcc make

If you're using ruby installed by apt, install ruby-dev.
( Skip if your ruby is installed by rbenv )

$ sudo apt install -y ruby-dev

CentOS Linux

$ sudo dnf install -y redhat-rpm-config gcc make

If you're using ruby installed by dnf ( or yum ), install ruby-devel.
( Skip if your ruby is installed by rbenv )

$ sudo dnf install -y ruby-devel

fluent-plugin-sflow

$ gem install fluent-plugin-sflow

You can also use fluent-gem or td-agent-gem instead of gem command.

Configuration

<source>
@type sflow
tag example.sflow
bind 0.0.0.0
port 6343
</source>

bind

IP address on which this plugin will accept sFlow. (Default: '0.0.0.0')

port

UDP port number on which this plugin will accept sFlow. (Default: 6343)

Record Example

Flow Sample

{"datagram_source_ip": "10.1.2.1","datagram_size": 240,"unix_seconds_utc": 1502113217,"datagram_version": 5,"agent_sub_id": 0,"agent": "10.1.2.1","packet_sequence_no": 53,"sys_up_time": 3808877,"samples_in_packet": 1,"sample_type_tag": "0:1","sample_type": "flow_sample","sample_sequence_no": 44,"source_id": "0:513","mean_skip_count": 5,"sample_pool": 225,"drop_events": 0,"input_port": 513,"output_port": 512,"flow_block_tag": "0:1002","header_protocol": 1,"sampled_packet_size": 102,"stripped_bytes": 4,"header_len": 98,"header_bytes": "02-05-86-71-7D-03-08-00-27-12-99-09-08-00-45-00-00-54-B0-61-40-00-40-01-79-44-0A-01-02-02-01-00-04-01-08-00-DF-97-04-12-03-AD-D2-1A-87-59-00-00-00-00-F6-61-02-00-00-00-00-00-10-11-12-13-14-15-16-17-18-19-1A-1B-1C-1D-1E-1F-20-21-22-23-24-25-26-27-28-29-2A-2B-2C-2D-2E-2F-30-31-32-33-34-35-36-37","dst_mac": "020586717d03","src_mac": "080027129909","ip_size": 84,"ip_tot_len": 84,"src_ip": "10.1.2.2","dst_ip": "1.0.4.1","ip_protocol": 1,"ip_tos": 0,"ip_ttl": 64,"ip_id": 25008,"icmp_type": 8,"icmp_code": 0,"in_vlan": 0,"in_priority": 0,"out_vlan": 0,"out_priority": 0,"next_hop": "202.249.2.169","src_subnet_mask": 32,"dst_subnet_mask": 22}

Counters Sample

{"datagram_source_ip": "10.1.2.1","datagram_size": 204,"unix_seconds_utc": 1502113415,"datagram_version": 5,"agent_sub_id": 0,"agent": "10.1.2.1","packet_sequence_no": 54,"sys_up_time": 3810403,"samples_in_packet": 1,"sample_type_tag": "0:2","sample_type": "counters_sample","sample_sequence_no": 5,"source_id": "0:512","counter_block_tag": "0:2","if_index": 512,"network_type": 6,"if_speed": 10000000000,"if_direction": 1,"if_status": 3,"if_in_octets": 0,"if_in_ucast_pkts": 8,"if_in_multicast_pkts": 121,"if_in_broadcast_pkts": 0,"if_in_discards": 0,"if_in_errors": 0,"if_in_unknown_protos": 0,"if_out_octets": 130454,"if_out_ucast_pkts": 1060,"if_out_multicast_pkts": 0,"if_out_broadcast_pkts": 95,"if_out_discards": 0,"if_out_errors": 0,"if_promiscuous_mode": 0,"dot3_stats_alignment_errors": 0,"dot3_stats_fcserrors": 0,"dot3_stats_single_collision_frames": 0,"dot3_stats_multiple_collision_frames": 0,"dot3_stats_sqetest_errors": 0,"dot3_stats_deferred_transmissions": 0,"dot3_stats_late_collisions": 0,"dot3_stats_excessive_collisions": 0,"dot3_stats_internal_mac_transmit_errors": 0,"dot3_stats_carrier_sense_errors": 0,"dot3_stats_frame_too_longs": 0,"dot3_stats_internal_mac_receive_errors": 0,"dot3_stats_symbol_errors": 0}

Benchmark

Here is a quick benchmark on Macbook Pro Mid 2015:

• sFlow v5 flow samples, 5.88 samples per packet in average
• Fluentd v0.14.17
• 13161 sFlow v5 records/s

TODO

• Add more tests
  • Please send me the sFlow pcap file if your device is not supported. 🌀
    1. Archive your pcap in .zip or .gz
    2. Open a new issue
    3. Attach the archive

Contributing

Please report issues or enhancement requests to GitHub issues. For questions or feedbacks write to my twitter @codeout.

Or send a pull request to fix.

Copyright and License

Copyright (c) 2017-2021 Shintaro Kojima. Code released under the MIT license.

Code includes a part of sflowtool which is distributed in the InMon sFlow License.