Having great and informative data will make for some hefty lookups. I’ve heard from a few customers that run into this rather than plan for it so let us talk about the levers we need to pull.
Don’t wait around upgrade to Splunk Enterprise 6.5.2+ Now is the time
Don’t wait any longer upgrade to Splunk Enterprise Security 4.5.1 the dev team invested in improvements to assets and identities lookups that also improve by decreasing the size of the merged lookups.
Update server.conf on the indexers and search head cluster peers.
max_content_length = 1610612736 # 1.5 GB
Update distsearch.conf to better replication on the SH/SHC
# 1.5 GB with encoding room this will increase the memory utilization while decreasing CPU utilization
maxMemoryBundleSize = 1700
#1.5 GB to match server.conf on the other side
maxBundleSize = 1536
Ok, I said posts in threes so here it is. We all know RYSLOG config is much more painful than syslog-ng but for reasons beyond all of our control, it is readily available for more customers than syslog-ng is today. Thanks to Splunk users I want to share a couple links to better doc to make this not so awful
Do blog posts come in threes, keep watching to find out? Yesterday I gave you the run down on a new way to collect syslog. Today I’m going to spend some time on a simple low cost and performant way to collect flow data.
At least two indexers with http event collector, more = better. For this use case it is not appropriate to utilize dedicated HEC servers.
One http load balancer, I use HA proxy. You can certainly use the same one from our rsyslog configuration.
Optional one UDP load balancer such as NGNIX. I am not documenting this setup at this time.
One ubuntu 16.04 VM
Follow docs, to setup HTTP event collector on your indexers, note if your indexers are clustered docs does not cover this, you must create the configuration manually be sure to generate a unique GUID manually. Clusters environments can use the sample configuration below: IMPORTANT ensure your data indexes AND _internal are allowed for the token
A little while back I created a bit of code to help get data from linux systems in real time where the Splunk Universal Forwarder could not be installed. At the time we had a few limitations the biggest problem being time stamps were never parsed only “current” time on the indexer could be used. Want to try out version 2 lets get started! First let me explain what we are doing
If you manage a Splunk environment with high rate sources such as a Palo Alto firewall or Web Proxy you will notice that events are not evenly distributed over the indexers because the the data is not evenly balanced across your aggregation tier. The reasons for this are boiled down to “time based load balancing” in Larger environments the universal forwarder may not be able to split by time to distribute a high load. So what is an admin to do? Lets look for a connection load balancing solution. We need to find a way to switch from “SYSLOG” to HTTP(s) so we can utilize a proper load balancer. How will we do this?
Using containers we will dedicate one or more instance of RSYSLOG for each “type” of data,
Use a custom plugin to package and forward batches of events over http(s)
Use a load balancer configured for least connected round robin to balance the batches of events
What you need
At least two indexers with http event collector, more = better. The “benefits” of this solution require collection on the indexer dedicated collectors will not be a adequate substitute
One load balancer, I use HA Proxy
One syslog collection server with rsyslog 8.24+ host I use LXC instances hosted on proxmox. Optimal deployment will utilize 1 collector per source technology. For example 1 instance collecting for Cisco IOS and another for Palo Alto Firewalls. Using advanced configuration and filters you can combine several low volume source.
Follow docs, to setup HTTP event collector on your indexers, note if your indexers are clustered docs does not cover this, you must create the configuration manually be sure to generate a unique GUID manually. Clusters environments can use the sample configuration below:
Follow documentation for your load balancer of choice to create a http VIP with https back end servers. HEC listens on 8088 by default
Grab the code and configuration examples from bitbucket
Deploy the script omsplunkhec.py to /opt/rsyslog/ ensure the script is executable
Review rsyslogd.d.conf.example and your configuration in /etc/rsyslog.d/00-splunkhec.conf replace the GUID and IP with your correct values
What to expect, My hope data balance Zen.
HTTP Event Collector inputs.conf example deployed via master-apps