Data Streams – Fill the data river with events from Splunk

I’ve had this in the bucket for a while waiting for the right time to share. There is a growing demand to develop “real time” analytic capability using machine data. Some great things are being created in labs their problem coming out of the lab is generally the inability to get events from the source systems, immediately following by difficulty normalizing events. If you’ve been working with these systems for very long and also worked with Splunk you may share my opinion that the Universal Forwarder, and the Schema at read power of Splunk is simply unmatched. How can we leverage the power of Splunk without reinventing the wheel, the axel, and the engine.

Credits

  • Liu-yuan Lai, Engineer, Splunk https://conf.splunk.com/session/2015/conf2015_LYuan_Splunk_BigData_DistributedProcessingwithSpark.pdf
  • Splunk App for CEF https://splunkbase.splunk.com/app/1847/

Back in 2015 I attended a short conf presentation that introduced me to the concepts and the value of Spark like engines. Last year our new CEF app introduced the idea message distribution can be executed on the indexer allowing very large scale processing with Splunk.

Introducing Integration Kit (IntKit)

The solution adds three interesting abilities to Splunk using “summarizing searches” to distribute events via a durable message bus.

  1. Send raw events using durable message queue
  2. Send reformated events using an arbitrary schema
  3. Send “Data Model” schema eliminating the need to build parsing logic for each type of source on the receiving side.

But what about other solutions

  • Syslog Output using the heavy forwarder
    • Syslog is not a reliable delivery protocol unable to resend lost events can cause backup on the UF
  • CEF 2.0
    • Great tool limited to single line events or reformating also allows for data loss.

The tools consist of a message formatter currently preparing a _json field, other formats such as xml or csv could be implemented and a producer that will place the message into the kafka queue (other queues can also be implemented)

example

[code lang=text]
| datamodel Network_Traffic All_Traffic search
| fields + _raw,All_Traffic.*
| generatejsonmsg suppress_empty=true suppress_unknown=true suppress_stringnull=true output_field=_json
include_metadata=true include_fields=true include_raw=false sort_fields=true sort_mv=true
| ProduceKafkamsgCommand bootstrap_servers="localhost:9092" topic="topicname" msgfield="_json"
| stats count
[/code]

What does this do:

  1. Using the datamodel command gather all Network_Traffic events
  2. Keep only _raw and the data model fields
  3. generate a _json field containing the fields in json format omit empty strings, “null”, sort the values of mv fields
  4. Send the message to kafka using a bootstrap server (localhost) topic “topicname”

This project is slightly above science project. That is poorly documented and mostly functional.  I expect it will fit in well with the ecosystem its helping. Please submit enhancements to make it better including documentation if you use it.

Using systemd to squash THP and start splunk enterprise

Updated Jan, 16, 2018 user security issue

Updated Jan 19,2018 using forking type for splunk

Fixing INIT Scripts

If you are currently or prefer using init script startup to remain as close to “out of box” configuration as possible be aware of a serious security risk present in the traditional startup method.  REF: https://www.splunk.com/view/SP-CAAAP3M To mitigate the issue and address THP/Ulimits consider moving to a field modified version of the script. https://bitbucket.org/snippets/rfaircloth-splunk/Gek8My

Going forward using SYSTEMD

 

The concept presented in this post, as well as the original inspiration, have some risks. Using alternatives to the vendor provided init scripts have support risks including loss of the configuration by future upgrades. Each operating system vendor has their own specific guidance on how to do this, each automation vendor has example automation scripts as well. Picking an approach that is appropriate for your environment is up to you.

THP the bain of performance for so many things in big data is often left on by default and is slightly difficult to disable. As a popular Splunk answers post and Splunk consultants include Marquis have found the best way to ensure ulimit and THP settings are properly configured is to modify the init scripts. This is a really crafty and reliable way to ensure THP is disabled for Splunk, it works on all Linux operating systems regardless of how services are started.

I’m doing some work with newer operating systems and wanted to explore how systemd really works and changes what is possible in managing a server. Lets face it systemd has not gotten the best of receptions in the community, after all it moved our cheese, toys and the ball all at once. It seems to be here to stay what if we could use its powers for good in relation to Splunk. Let’s put an end to THP and start Splunk the systemd native way.

Note: the following config file is present for readability and google. Downloadable text file is available https://bitbucket.org/snippets/rfaircloth-splunk/ze7rqL

Create the file /etc/systemd/system/disable-transparent-huge-pages.service


[Unit]
Description=Disable Transparent Huge Pages

[Service]
Type=oneshot
ExecStart=/bin/sh -c “echo never >/sys/kernel/mm/transparent_hugepage/enabled”
ExecStart=/bin/sh -c “echo never >/sys/kernel/mm/transparent_hugepage/defrag”
RemainAfterExit=true
[Install]
WantedBy=multi-user.target

Verify THP and defrag is presently enabled to avoid a false sense of success

# cat /sys/kernel/mm/transparent_hugepage/enabled

[always] madvise never

# cat /sys/kernel/mm/transparent_hugepage/defrag

[always] madvise never

Enable and start the unit to disable THP

# systemctl enable disable-transparent-huge-pages.service

# systemctl start disable-transparent-huge-pages.service

# cat /sys/kernel/mm/transparent_hugepage/enabled

always madvise [never]

# cat /sys/kernel/mm/transparent_hugepage/defrag

always madvise [never]

Reboot and repeat the verification to ensure the process is enforced

Note: the following config file is present for readability and google. Downloadable text file is available https://bitbucket.org/snippets/rfaircloth-splunk/xe7rqj

create the unit file /etc/systemd/system/splunk.service

<p class="p1"><span class="s1">[unit]</span></p>
<p class="p1"><span class="s1">After=network.target</span></p>
<p class="p1"><span class="s1">Wants=network.target</span></p>
<p class="p1"><span class="s1">Description=Splunk Enterprise</span></p>
<p class="p1"><span class="s1">[Service]</span></p>
<p class="p1"><span class="s1">Type=forking</span></p>
<p class="p1"><span class="s1">RemainAfterExit=False</span></p>
<p class="p1"><span class="s1">User=splunk</span></p>
<p class="p1"><span class="s1">Group=splunk</span></p>
<p class="p1"><span class="s1">ExecStart=/opt/splunk/bin/splunk start --answer-yes --no-prompt --accept-license</span></p>
<p class="p1"><span class="s1">ExecStop=/opt/splunk/bin/splunk stop</span></p>
<p class="p1"><span class="s1">PIDFile=/opt/splunk/var/run/splunk/splunkd.pid</span></p>
<p class="p1"><span class="s1">Restart=always</span></p>
<p class="p1"><span class="s1">TimeoutSec=300</span></p>
<p class="p1"><span class="s1">#ulimit -Sn 65535</span></p>
<p class="p1"><span class="s1">#ulimit -Hn 65535</span></p>
<p class="p1"><span class="s1">LimitNOFILE=65535</span></p>
<p class="p1"><span class="s1">#ulimit -Su 20480</span></p>
<p class="p1"><span class="s1">#ulimit -Hu 20480</span></p>
<p class="p1"><span class="s1">LimitNPROC=20480</span></p>
<p class="p1"><span class="s1">#ulimit -Hf unlimited</span></p>
<p class="p1"><span class="s1">#ulimit -Sf unlimited</span></p>
<p class="p1"><span class="s1">LimitFSIZE=infinity</span></p>
<p class="p1"><span class="s1">LimitCORE=infinity</span></p>
<p class="p1"><span class="s1">[Install]</span></p>
<p class="p1"><span class="s1">WantedBy=multi-user.target</span></p>

# systemctl enable splunk.service

# systemctl start splunk.service

Verify the ulimits have been applied via splunk logs

#cat /opt/splunk/var/log/splunk/splunkd.log | grep ulimit

Reboot and repeate all verifications

Bonus material, kill Splunk (lab env only) and watch systemd bring it back

# killall splunk

# ps aux | grep splunk

You just noticed splunkd was brought back to up when it died without using systemctl stop. This means using splunk start|stop is not valid when systemd started Splunk.

Splunk OS Data on boarding – best practices updated

I’ve updated my best practices a bit and moved the implementation guides from confluence out to the bitbuckets in markdown so they can be more easily referenced on any platform or secured environments where PDFs might be discouraged.

Each repo will contain a README.md and one or more INSTALL.md files with the implementation guides. If you find an issue have a better practice or other enhancement, please open an issue in the repositories tracker.

Trust but verify – your logs might be lieing to you aka, the case for network tap traffic monitoring

I really do “get” it, logging and monitoring can be very costly, we all agree not nearly as costly as a breach. Each organization is struggling to ensure they log enough to see detection and value while being good stewards of their company budget. It has been a day reading vault 7 leaks and I see honestly not much that surprises me. I do see something worth a strong restatement, that is an encouragement to rethink what you log and how you log it. The CIA has a very cool (sorry hacker at heart) tool we have known about for some time but have not been able to talk about. Their tool “Drillbit” allows the creation of a covert tunnel using common cisco gear in such a way typical monitoring and logging using IDS and firewalls will not identify. American companies should note criminal gangs and foreign governments certainly have similar capabilities. Splunk has your back if you are willing to let us. Using Splunk Stream and proper sensor placement we can collect data from the inside and outside of your firewall that can be used to identify covert tunnels. Detection should be performed three approaches. The danger these leaks are presenting you is an increased awareness of the effectiveness of these techniques, encouraging the advancement of commodity cybercrime toolkits with ever more difficult to detect features. Don’t use cisco, sorry bad news is almost every major gear vendor has been exploited with similar approaches

  • Static Rules such as
    • This not that “Stream identified traffic not in firewall logs”
    • New patterns in DNS, NTP, GRE flows
    • Change/login to firewall or switch not associated with a change record
  • Threat List Enrichment and detection
    • Source and Destination traffic on quality threat lists. Traffic for protocols other than http(s) and DNS should be treated with high or critical priority
  • Machine Learning
    • Anomalous  egress traffic by source from network devices
    • Anomalous admin connections by source to network devices

Automating Splunk deployment for RedHat/Centos (poor man’s edition)

I pulled this out of the archives , on request notice this was originally developed for Splunk 6.2.x and RHEL 7.0. Please review the details make sure it is suitable for you and TEST. If I can talk you out of doing things this way I would. Salt is a great way to manage app config its free and just awesome.

Title: Splunk Universal Forwarder Version 6.2.3+ Red Hat Enterprise Linux 7

Author: Ryan Faircloth 

Summary: Using repositories for version managment of the Splunk Universal Forwarder assists in ensuring managed Red Hat and compatible linux  systems are using the approved version of the software at all times. 

[TOC]

## Setup the repository server ##
1. Install createrepo  and nginx ``` yum install createrepo apache2 ```
3. Create a user to work with the repository 

	``` 
	sudo adduser  repouser 
	```  
3. Change user to our repouser user all commands for the repository should be executed using this ID 

	 ``` 
	 sudo su - repouser
	 ```

## Generate GPG Keys ##
1. Change user to our repouser user all commands for the repository should be executed using this ID 

	```
	sudo su - repouser 
	```
2. Create the default configuration for gpg by running the command 

	```
	gpg --list-keys
	``` 
3. Edit ~/.gnupg/gpg.conf 
	* uncomment the line ``` no-greeting ```
	* add the following content to the end of the file 
	
	```
	# Prioritize stronger algorithms for new keys.
	default-preference-list SHA512 SHA384 SHA256 SHA224 AES256 AES192 AES CAST5 BZIP2 ZLIB ZIP UNCOMPRESSED
	# Use a stronger digest than the default SHA1 for certifications.
	cert-digest-algo SHA512
	```
	
4. Generate a new key with the command ``` gpg --gen-key ```
5. Select the folowing options ON CENTOS/RHEL this procedure must be executed on the console or SSH having logged in as the repouser
	1. Type of key "(1) RSA and RSA (default)"
	2. Key size "4096"
	3. Expires "10y"
	4. Confirm "Y"
	5. Real Name "Splunk local repository"
	6. Email address on repository contact this generally should be an alias or distribution list
	7. Leave the comment blank
	8. Confirm and "O" to Okay
	9. Leave passphrase blank and confirm, a key will be generated not the sub KEY ID in the following example * E507D48E * 

	```
	gpg: checking the trustdb
	gpg: 3 marginal(s) needed, 1 complete(s) needed, PGP trust model
	gpg: depth: 0  valid:   1  signed:   0  trust: 0-, 0q, 0n, 0m, 0f, 1u
	gpg: next trustdb check due at 2025-05-24
	pub   4096R/410E1699 2015-05-27 [expires: 2025-05-24]
    	  Key fingerprint = 7CB8 81A9 E07F DA7B 83FF  2E1B 8B31 DA83 410E 1699
	uid                  Splunk local repository <repo@example.com>
	sub   4096R/E507D48E 2015-05-27 [expires: 2025-05-24]
	```	
10. Export the signing keys public component save this content for use later

	```
	gpg --export --armor KEY_ID >~/repo.pub 
	``` 
11. Install the new key into the RPM database

```
sudo cp ~/repo.pub /etc/pki/rpm-gpg/RPM-GPG-KEY-splunkrepo
sudo rpm --import /etc/pki/rpm-gpg/RPM-GPG-KEY-splunkrepo
```
12. Configure RPM signing with the new key

```
echo "%_signature gpg" > ~/.rpmmacros
echo "%_gpg_name splunkrepo" >> ~/.rpmmacros
``

13. Create a repository

```
mkdir /opt/splunkrepo
cp splunkforwarder*.rpm /opt/splunkrepo
createrepo /opt/splunkrepo
```
14. Configure the local repository create the following configuration /etc/yum.repos.d/splunk.repo

```
[splunkrepo]
name=splunk repository
baseurl=file:///opt/splunkrepo/
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-splunkrepo
enabled=1
```
15. Test the local repository by installing splunkforwarder

```
sudo yum update
sudo yum install splunkforwader
```
Note: Create a configuration RPM refer to https://fedoraproject.org/wiki/How_to_create_an_RPM_package and https://www.redhat.com/promo/summit/2010/presentations/summit/opensource-for-it-leaders/thurs/pwaterma-2-rpm/RPM-ifying-System-Configurations.pdf for more information do not run as root sudo to repouser
17.  Prepare the rpm tree ``` rpmdev-setuptree ```
18.  Create a spec file with the following content ~/splunkforwarder-baseconfig.spec

```
#--------------------------------------------------------------------------
# This spec file is Copyright 2010, My Company, Inc.
#--------------------------------------------------------------------------
Summary: My Company general configuration RPM
Name: splunkforwarder-baseconfig
Version: 1
Release: 3
License: Copyright 2010, My Company, Inc.
Group: MyCompany/Configs
Packager: Packager Name <my-email@mycompany.com>
requires: splunkforwarder
BuildArch: noarch
%description
This RPM provides general services and security configuration for My Company.

%triggerin -- splunkforwarder
/opt/splunkforwarder/bin/splunk enable boot-start --accept-license --answer-yes 
service splunk stop
if [ -d "/opt/splunkforwarder/etc/apps/org_all_deploymentclient/local" ]
then
    echo "Directory /opt/splunkforwarder/etc/apps/org_all_deploymentclient/local exists."
else
    mkdir -p /opt/splunkforwarder/etc/apps/org_all_deploymentclient/local    
fi
echo #Base deployment configuration >/opt/splunkforwarder/etc/apps/org_all_deploymentclient/local/deploymentclient.conf
echo [deployment-client] >>/opt/splunkforwarder/etc/apps/org_all_deploymentclient/local/deploymentclient.conf
#echoclientName  >>/opt/splunkforwarder/etc/apps/org_all_deploymentclient/local/deploymentclient.conf
echo [deployment-client] >>/opt/splunkforwarder/etc/apps/org_all_deploymentclient/local/deploymentclient.conf
echo targetUri = ds.example.com:8089 >>/opt/splunkforwarder/etc/apps/org_all_deploymentclient/local/deploymentclient.conf

service splunk start


%triggerun -- splunkforwarder
if [ $1 -eq 0 -a $2 -gt 0 ] ; then
 /opt/splunkforwarder/bin/splunk stop
 /opt/splunkforwarder/bin/splunk disable boot-start
 rm -Rf /opt/splunkforwarder/etc/apps/org_all_deploymentclient
fi
%files

```
18.  Build the RPM ``` rpmbuild -sign -ba splunkforwarder-baseconfig.spec ```
19. Copy the RPM to the repository cp ~/rpmbuild/RPMS/noarch/splunkforwarder-baseconfig-1-3.noarch.rpm / /opt/splunkrepo
20. Update repository DB ``` createrepo /opt/splunkrepo ```
21. Test the rpms

```
yum update
yum install splunkforwarder-baseconfig
```
22. Configure a web server (Apache) for use as a repository server
23.  Set permissions on the repository folder ``` chmod -R 755 /opt/splunkrepo ```
24. Create the web server configuration file with the following contents /etc/http/conf.d/splunkrepo.conf

```
Alias /splunkrepo/ "/opt/splunkrepo/"

<Directory "/opt/splunkrepo">
   Options Indexes FollowSymLinks MultiViews
   AllowOverride All
   Require local
   Order allow,deny
   Allow from all
</Directory>
```
25. Reload (or restart) the web server ```service httpd reload ```
26. Test `` lynx http://localhost/splunkrepo/repodata/repomd.xml ```
27. Enable the new repository on the first test client

```
sudo yum-config-manager --add-repo http://localhost/splunkrepo
sudo yum update
sudo yum install splunkforwarder-baseconfig
```

Splunk the server and the enpoint aka “All the Things”

I’m sharing something today that has been available thanks to many in white papers and presentations dealing with identification of malicious code and activities in your windows event data. Shout out to everyone from our “friends” at the NSA, to Splunk .Conf presenters and malwarearcheology.com just to name a few.

The PDF attached is a portion of the next evolution of the Use Case Repository I maintain at Splunk. Along with the reference TAs and inputs, this will allow you to quickly and consistently collect very valuable data supporting security use cases at multiple levels of maturity. If it seems like too much don’t work Splunk Pro Services and partners are able to help you get this visibility just contact your account team.

Standard disclaimer, this is a blog post, I built the content from public non-warrantied information, and this is still public non-warrantied information, your situation might not match the advice given.

PT005-Microsoft-Windows

Good Assets and Identities make for big bundles

Having great and informative data will make for some hefty lookups. I’ve heard from a few customers that run into this rather than plan for it so let us talk about the levers we need to pull.

  • Don’t wait around upgrade to Splunk Enterprise 6.5.2+ Now is the time
  • Don’t wait any longer upgrade to Splunk Enterprise Security 4.5.1 the dev team invested in improvements to assets and identities lookups that also improve by decreasing the size of the merged lookups.
  • Update server.conf on the indexers and search head cluster peers.

[httpServer]

max_content_length = 1610612736 # 1.5 GB

  • Update distsearch.conf to better replication on the SH/SHC
[replicationSettings]
# 1.5 GB with encoding room this will increase the memory utilization while decreasing CPU utilization
maxMemoryBundleSize = 1700 
#1.5 GB to match server.conf on the other side
maxBundleSize = 1536

RSYSLOG read the docs

Ok, I said posts in threes so here it is. We all know RYSLOG config is much more painful than syslog-ng but for reasons beyond all of our control, it is readily available for more customers than syslog-ng is today. Thanks to Splunk users I want to share a couple links to better doc to make this not so awful

  • RedHat https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/html/System_Administrators_Guide/s1-basic_configuration_of_rsyslog.html
  • Usenix https://www.usenix.org/system/files/login/articles/06_lang-online.pdf

thank you @mattymo and @lowell via Splunk Slack chat

Unbelievably simple (ipfix|(net|j|s)flow) collection

Do blog posts come in threes, keep watching to find out? Yesterday I gave you the run down on a new way to collect syslog. Today I’m going to spend some time on a simple low cost and performant way to collect flow data.

  • At least two indexers with http event collector, more = better. For this use case it is not appropriate to utilize dedicated HEC servers.
  • One http load balancer, I use HA proxy. You can certainly use the same one from our rsyslog configuration.
  • Optional one UDP load balancer such as NGNIX. I am not documenting this setup at this time.
  • One ubuntu 16.04 VM

Basic Setup

  1. Follow docs, to setup HTTP event collector on your indexers, note if your indexers are clustered docs does not cover this, you must create the configuration manually be sure to generate a unique GUID manually. Clusters environments can use the sample configuration below: IMPORTANT ensure your data indexes AND _internal are allowed for the token
  2. [http] 
    disabled=0
    port=8088
    #
    [http://streamfwd]
    disabled=0
    index=main
    token=DAA61EE1-F8B2-4DB1-9159-6D7AA5220B21
    indexes=_internal,main
  3. Follow documentation for your load balancer of choice to create a http VIP with https back end servers. HEC listens on 8088 by default.
  4. Install stream for the independent per Docs
  5. Kill stream if its running “killall -9 streamfwd”
  6. Remove the init script
    1. update-rc.d -f streamfwd remove”
    2. rm /etc/init.d/streamfwd
  7. Create a new service unit file for systemd /etc/systemd/system/streamfwd.service
    [Unit]
    Description= Splunk Stream Dedicated Service
    After=syslog.target network.target
    [Service]
    Type=simple
    ExecStart=/opt/streamfwd/bin/streamfwd -D
  8. Enable the new service “systemctl enable streamfwd”
  9. Create/update the streamfwd.conf replacing GUID VIP and INTERFACE
    1. [streamfwd]
      
      httpEventCollectorToken = <GUID>
      
      indexer.0.uri= <HEC VIP>
      netflowReceiver.0.ip = <INTERFACE TO BIND>
      netflowReceiver.0.port = 9995
      netflowReceiver.0.decoder = netflow
  10. Create/update the inputs.conf ensure the URL is correct for the location of your stream app
  11. [streamfwd://streamfwd]
    
    splunk_stream_app_location = https://192.168.100.62:8000/en-us/custom/splunk_app_stream/
    
    stream_forwarder_id=infra_netflow
  12. Start the streamfwd “systemctl start streamfwd”
  13. Login to the search head where Splunk App for Stream is Installed
  14. Navigate to Splunk App for Stream –> Configuration –> Distributed Forwarder Managment
  15. Click Create New Group
  16. Enter Name as “INFRA_NETFLOW”
  17. Enter a Description
  18. Click Next
  19. Enter “INFRA_NETFLOW” as the rule and click next
  20. Click Finish without selecting options
  21. Navigate to Splunk App for Stream –> Configuration –> Configure Streams
  22. Click New Stream select netflow as the protocol (this is correct for netflow/sflow/jflow/ipfix
  23. Enter Name as “INFRA_NETFLOW”
  24. Enter a Description and click next
  25. No Aggregation and click next
  26. Deselect any fields NOT interesting for your use case and click next
  27. Optional develop filters to reduce noise from high traffic devices and click next
  28. Select the index for this collection and click enable then click next
  29. Select only the Infra_netflow group and Create_Stream
  30. Configure your NETFLOW generator to send records to the new streamfwd

Validation! search the index configured in step 27

Building a more perfect Syslog Collection Infrastructure

A little while back I created a bit of code to help get data from linux systems in real time where the Splunk Universal Forwarder could not be installed. At the time we had a few limitations the biggest problem being time stamps were never parsed only “current” time on the indexer could be used.  Want to try out version 2 lets get started! First let me explain what we are doing

If you manage a Splunk environment with high rate sources such as a Palo Alto firewall or Web Proxy you will notice that events are not evenly distributed over the indexers because the the data is not evenly balanced across your aggregation tier. The reasons for this are boiled down to “time based load balancing” in Larger environments the universal forwarder may not be able to split by time to distribute a high load. So what is an admin to do? Lets look for a connection load balancing solution. We need to find a way to switch from “SYSLOG” to HTTP(s) so we can utilize a proper load balancer. How will we do this?

  1. Using containers we will dedicate one or more instance of RSYSLOG for each “type” of data,
  2. Use a custom plugin to package and forward batches of events over http(s)
  3. Use a load balancer configured for least connected round robin to balance the batches of events

What you need

  • At least two indexers with http event collector, more = better. The “benefits” of this solution require collection on the indexer dedicated collectors will not be a adequate substitute
  • One load balancer, I use HA Proxy
  • One syslog collection server with rsyslog 8.24+ host I use LXC instances hosted on proxmox. Optimal deployment will utilize 1 collector per source technology. For example 1 instance collecting for Cisco IOS and another for Palo Alto Firewalls. Using advanced configuration and filters you can combine several low volume source.
  • A GUID if you need one generated there are many ways this one is quick and easy https://www.guidgenerator.com/online-guid-generator.aspx

Basic Setup

  1. Follow docs, to setup HTTP event collector on your indexers, note if your indexers are clustered docs does not cover this, you must create the configuration manually be sure to generate a unique GUID manually. Clusters environments can use the sample configuration below:
  2. Follow documentation for your load balancer of choice to create a http VIP with https back end servers. HEC listens on 8088 by default
  3. Grab the code and configuration examples from bitbucket
    1. Deploy the script omsplunkhec.py to /opt/rsyslog/ ensure the script is executable
    2. Review rsyslogd.d.conf.example and your configuration in /etc/rsyslog.d/00-splunkhec.conf replace the GUID and IP with your correct values
    3. Restart rsyslog

What to expect, My hope data balance Zen.

HTTP Event Collector inputs.conf example deployed via master-apps

[http] 
disabled=0
port=8088
#
[http://SM_rsyslog_routerboard]
disabled=0
index=main
token=DAA61EE1-F8B2-4DB1-9159-6D7AA5220B21
indexes=main,summary

Example /etc/rsyslog.d/00-splunk.conf

This example will listen on 514 TCP and UDP sending events via http, be sure to replace the GUID and ip address

module(load="imudp")
input(type="imudp" port="514" ruleset="default_file")
module(load="imptcp")
input(type="imptcp" port="514" ruleset="default_file")
module(load="omprog")

ruleset(name="default_file"){
    $RulesetCreateMainQueue    
    action(type="omprog"
       binary="/opt/rsyslog/omsplunkhec.py DAA61EE1-F8B2-4DB1-9159-6D7AA5220B21 192.168.100.70 --sourcetype=syslog --index=main" 
       template="RSYSLOG_TraditionalFileFormat")
    stop
}

Example HAProxy Configuration 1.7 /etc/haproxy/haproxy.cfg

 

global
        log /dev/log    local0
        log /dev/log    local1 notice
        chroot /var/lib/haproxy
        stats socket /run/haproxy/admin.sock mode 660 level admin
        stats timeout 30s
        user haproxy
        group haproxy
        daemon
        # Default SSL material locations
        ca-base /etc/ssl/certs
        crt-base /etc/ssl/private
        # Default ciphers to use on SSL-enabled listening sockets.
        # For more information, see ciphers(1SSL).
        ssl-default-bind-ciphers kEECDH+aRSA+AES:kRSA+AES:+AES256:RC4-SHA:!kEDH:!LOW:!EXP:!MD5:!aNULL:!eNULL
defaults
        log     global
        mode    http
        option  httplog
        option  dontlognull
        timeout connect 5000
        timeout client  50000
        timeout server  50000
        errorfile 400 /etc/haproxy/errors/400.http
        errorfile 403 /etc/haproxy/errors/403.http
        errorfile 408 /etc/haproxy/errors/408.http
        errorfile 500 /etc/haproxy/errors/500.http
        errorfile 502 /etc/haproxy/errors/502.http
        errorfile 503 /etc/haproxy/errors/503.http
        errorfile 504 /etc/haproxy/errors/504.http
listen  stats   
        bind            *:1936
        mode            http
        log             global
        maxconn 10
        clitimeout      100s
        srvtimeout      100s
        contimeout      100s
        timeout queue   100s
        stats enable
        stats hide-version
        stats refresh 30s
        stats show-node
        stats auth admin:password
        stats uri  /haproxy?stats
frontend localnodes
    bind *:8088
    mode http
    default_backend nodes
backend nodes
    mode http
    balance leastconn
    option forwardfor
    http-request set-header X-Forwarded-Port %[dst_port]
    http-request add-header X-Forwarded-Proto https if { ssl_fc }
    option httpchk
    server idx2 192.168.100.52:8088 ssl verify none check 
    server idx1 192.168.100.51:8088 ssl verify none check 
%d bloggers like this: