A little while back I created a bit of code to help get data from linux systems in real time where the Splunk Universal Forwarder could not be installed. At the time we had a few limitations the biggest problem being time stamps were never parsed only “current” time on the indexer could be used. Want to try out version 2 lets get started! First let me explain what we are doing
If you manage a Splunk environment with high rate sources such as a Palo Alto firewall or Web Proxy you will notice that events are not evenly distributed over the indexers because the the data is not evenly balanced across your aggregation tier. The reasons for this are boiled down to “time based load balancing” in Larger environments the universal forwarder may not be able to split by time to distribute a high load. So what is an admin to do? Lets look for a connection load balancing solution. We need to find a way to switch from “SYSLOG” to HTTP(s) so we can utilize a proper load balancer. How will we do this?
- Using containers we will dedicate one or more instance of RSYSLOG for each “type” of data,
- Use a custom plugin to package and forward batches of events over http(s)
- Use a load balancer configured for least connected round robin to balance the batches of events
What you need
- At least two indexers with http event collector, more = better. The “benefits” of this solution require collection on the indexer dedicated collectors will not be a adequate substitute
- One load balancer, I use HA Proxy
- One syslog collection server with rsyslog 8.24+ host I use LXC instances hosted on proxmox. Optimal deployment will utilize 1 collector per source technology. For example 1 instance collecting for Cisco IOS and another for Palo Alto Firewalls. Using advanced configuration and filters you can combine several low volume source.
- A GUID if you need one generated there are many ways this one is quick and easy https://www.guidgenerator.com/online-guid-generator.aspx
Basic Setup
- Follow docs, to setup HTTP event collector on your indexers, note if your indexers are clustered docs does not cover this, you must create the configuration manually be sure to generate a unique GUID manually. Clusters environments can use the sample configuration below:
- Follow documentation for your load balancer of choice to create a http VIP with https back end servers. HEC listens on 8088 by default
- Grab the code and configuration examples from bitbucket
- Deploy the script omsplunkhec.py to /opt/rsyslog/ ensure the script is executable
- Review rsyslogd.d.conf.example and your configuration in /etc/rsyslog.d/00-splunkhec.conf replace the GUID and IP with your correct values
- Restart rsyslog
What to expect, My hope data balance Zen.
HTTP Event Collector inputs.conf example deployed via master-apps
[http] disabled=0 port=8088 # [http://SM_rsyslog_routerboard] disabled=0 index=main token=DAA61EE1-F8B2-4DB1-9159-6D7AA5220B21 indexes=main,summary
Example /etc/rsyslog.d/00-splunk.conf
This example will listen on 514 TCP and UDP sending events via http, be sure to replace the GUID and ip address
module(load="imudp") input(type="imudp" port="514" ruleset="default_file") module(load="imptcp") input(type="imptcp" port="514" ruleset="default_file") module(load="omprog") ruleset(name="default_file"){ $RulesetCreateMainQueue action(type="omprog" binary="/opt/rsyslog/omsplunkhec.py DAA61EE1-F8B2-4DB1-9159-6D7AA5220B21 192.168.100.70 --sourcetype=syslog --index=main" template="RSYSLOG_TraditionalFileFormat") stop }
Example HAProxy Configuration 1.7 /etc/haproxy/haproxy.cfg
global log /dev/log local0 log /dev/log local1 notice chroot /var/lib/haproxy stats socket /run/haproxy/admin.sock mode 660 level admin stats timeout 30s user haproxy group haproxy daemon # Default SSL material locations ca-base /etc/ssl/certs crt-base /etc/ssl/private # Default ciphers to use on SSL-enabled listening sockets. # For more information, see ciphers(1SSL). ssl-default-bind-ciphers kEECDH+aRSA+AES:kRSA+AES:+AES256:RC4-SHA:!kEDH:!LOW:!EXP:!MD5:!aNULL:!eNULL defaults log global mode http option httplog option dontlognull timeout connect 5000 timeout client 50000 timeout server 50000 errorfile 400 /etc/haproxy/errors/400.http errorfile 403 /etc/haproxy/errors/403.http errorfile 408 /etc/haproxy/errors/408.http errorfile 500 /etc/haproxy/errors/500.http errorfile 502 /etc/haproxy/errors/502.http errorfile 503 /etc/haproxy/errors/503.http errorfile 504 /etc/haproxy/errors/504.http listen stats bind *:1936 mode http log global maxconn 10 clitimeout 100s srvtimeout 100s contimeout 100s timeout queue 100s stats enable stats hide-version stats refresh 30s stats show-node stats auth admin:password stats uri /haproxy?stats frontend localnodes bind *:8088 mode http default_backend nodes backend nodes mode http balance leastconn option forwardfor http-request set-header X-Forwarded-Port %[dst_port] http-request add-header X-Forwarded-Proto https if { ssl_fc } option httpchk server idx2 192.168.100.52:8088 ssl verify none check server idx1 192.168.100.51:8088 ssl verify none check