Tag Archives: Splunk

Using systemd to squash THP and start splunk enterprise

The concept presented in this post, as well as the original inspiration, have some risks. Using alternatives to the vendor provided init scripts have support risks including loss of the configuration by future upgrades. Each operating system vendor has their own specific guidance on how to do this, each automation vendor has example automation scripts as well. Picking an approach that is appropriate for your environment is up to you.

THP the bain of performance for so many things in big data is often left on by default and is slightly difficult to disable. As a popular Splunk answers post and Splunk consultants include Marquis have found the best way to ensure ulimit and THP settings are properly configured is to modify the init scripts. This is a really crafty and reliable way to ensure THP is disabled for Splunk, it works on all Linux operating systems regardless of how services are started.

I’m doing some work with newer operating systems and wanted to explore how systemd really works and changes what is possible in managing a server. Lets face it systemd has not gotten the best of receptions in the community, after all it moved our cheese, toys and the ball all at once. It seems to be here to stay what if we could use its powers for good in relation to Splunk. Let’s put an end to THP and start Splunk the systemd native way.

Create the file /etc/systemd/system/disable-transparent-huge-pages.service

Description=Disable Transparent Huge Pages

ExecStart=/bin/sh -c “echo never >/sys/kernel/mm/transparent_hugepage/enabled”
ExecStart=/bin/sh -c “echo never >/sys/kernel/mm/transparent_hugepage/defrag”

Verify THP and defrag is presently enabled to avoid a false sense of success

# cat /sys/kernel/mm/transparent_hugepage/enabled

[always] madvise never

# cat /sys/kernel/mm/transparent_hugepage/defrag

[always] madvise never

Enable and start the unit to disable THP

# systemctl enable disable-transparent-huge-pages.service

# systemctl start disable-transparent-huge-pages.service

# cat /sys/kernel/mm/transparent_hugepage/enabled

always madvise [never]

# cat /sys/kernel/mm/transparent_hugepage/defrag

always madvise [never]

Reboot and repeat the verification to ensure the process is enforced

create the unit file /etc/systemd/system/splunk.service

ExecStart=/opt/splunk/bin/splunk start --no-prompt --answer-yes --accept-license
ExecStop=/opt/splunk/bin/splunk stop


#ulimit -Sn 65535
#ulimit -Hn 65535
#ulimit -Su 20480
#ulimit -Hu 20480
#ulimit -Hf unlimited
#ulimit -Sf unlimited

# systemctl enable splunk.service

# systemctl start splunk.service

Verify the ulimits have been applied via splunk logs

#cat /opt/splunk/var/log/splunk/splunkd.log | grep ulimit

Reboot and repeate all verifications

Bonus material, kill Splunk (lab env only) and watch systemd bring it back

# killall splunk

# ps aux | grep splunk

Splunk the server and the enpoint aka “All the Things”

I’m sharing something today that has been available thanks to many in white papers and presentations dealing with identification of malicious code and activities in your windows event data. Shout out to everyone from our “friends” at the NSA, to Splunk .Conf presenters and malwarearcheology.com just to name a few.

The PDF attached is a portion of the next evolution of the Use Case Repository I maintain at Splunk. Along with the reference TAs and inputs, this will allow you to quickly and consistently collect very valuable data supporting security use cases at multiple levels of maturity. If it seems like too much don’t work Splunk Pro Services and partners are able to help you get this visibility just contact your account team.

Standard disclaimer, this is a blog post, I built the content from public non-warrantied information, and this is still public non-warrantied information, your situation might not match the advice given.


Building a more perfect Syslog Collection Infrastructure

A little while back I created a bit of code to help get data from linux systems in real time where the Splunk Universal Forwarder could not be installed. At the time we had a few limitations the biggest problem being time stamps were never parsed only “current” time on the indexer could be used.  Want to try out version 2 lets get started! First let me explain what we are doing

If you manage a Splunk environment with high rate sources such as a Palo Alto firewall or Web Proxy you will notice that events are not evenly distributed over the indexers because the the data is not evenly balanced across your aggregation tier. The reasons for this are boiled down to “time based load balancing” in Larger environments the universal forwarder may not be able to split by time to distribute a high load. So what is an admin to do? Lets look for a connection load balancing solution. We need to find a way to switch from “SYSLOG” to HTTP(s) so we can utilize a proper load balancer. How will we do this?

  1. Using containers we will dedicate one or more instance of RSYSLOG for each “type” of data,
  2. Use a custom plugin to package and forward batches of events over http(s)
  3. Use a load balancer configured for least connected round robin to balance the batches of events


What you need

  • At least two indexers with http event collector, more = better. The “benefits” of this solution require collection on the indexer dedicated collectors will not be a adequate substitute
  • One load balancer, I use HA Proxy
  • One syslog collection server with rsyslog 8.24+ host I use LXC instances hosted on proxmox. Optimal deployment will utilize 1 collector per source technology. For example 1 instance collecting for Cisco IOS and another for Palo Alto Firewalls. Using advanced configuration and filters you can combine several low volume source.
  • A GUID if you need one generated there are many ways this one is quick and easy https://www.guidgenerator.com/online-guid-generator.aspx

Basic Setup

  1. Follow docs, to setup HTTP event collector on your indexers, note if your indexers are clustered docs does not cover this, you must create the configuration manually be sure to generate a unique GUID manually. Clusters environments can use the sample configuration below:
  2. Follow documentation for your load balancer of choice to create a http VIP with https back end servers. HEC listens on 8088 by default
  3. Grab the code and configuration examples from bitbucket
    1. Deploy the script omsplunkhec.py to /opt/rsyslog/ ensure the script is executable
    2. Review rsyslogd.d.conf.example and your configuration in /etc/rsyslog.d/00-splunkhec.conf replace the GUID and IP with your correct values
    3. Restart rsyslog

What to expect, My hope data balance Zen.


HTTP Event Collector inputs.conf example deployed via master-apps


Example /etc/rsyslog.d/00-splunk.conf

This example will listen on 514 TCP and UDP sending events via http, be sure to replace the GUID and ip address

input(type="imudp" port="514" ruleset="default_file")
input(type="imptcp" port="514" ruleset="default_file")

       binary="/opt/rsyslog/omsplunkhec.py DAA61EE1-F8B2-4DB1-9159-6D7AA5220B21 --sourcetype=syslog --index=main" 

Example HAProxy Configuration 1.7 /etc/haproxy/haproxy.cfg


        log /dev/log    local0
        log /dev/log    local1 notice
        chroot /var/lib/haproxy
        stats socket /run/haproxy/admin.sock mode 660 level admin
        stats timeout 30s
        user haproxy
        group haproxy
        # Default SSL material locations
        ca-base /etc/ssl/certs
        crt-base /etc/ssl/private
        # Default ciphers to use on SSL-enabled listening sockets.
        # For more information, see ciphers(1SSL).
        ssl-default-bind-ciphers kEECDH+aRSA+AES:kRSA+AES:+AES256:RC4-SHA:!kEDH:!LOW:!EXP:!MD5:!aNULL:!eNULL
        log     global
        mode    http
        option  httplog
        option  dontlognull
        timeout connect 5000
        timeout client  50000
        timeout server  50000
        errorfile 400 /etc/haproxy/errors/400.http
        errorfile 403 /etc/haproxy/errors/403.http
        errorfile 408 /etc/haproxy/errors/408.http
        errorfile 500 /etc/haproxy/errors/500.http
        errorfile 502 /etc/haproxy/errors/502.http
        errorfile 503 /etc/haproxy/errors/503.http
        errorfile 504 /etc/haproxy/errors/504.http
listen  stats   
        bind            *:1936
        mode            http
        log             global
        maxconn 10
        clitimeout      100s
        srvtimeout      100s
        contimeout      100s
        timeout queue   100s
        stats enable
        stats hide-version
        stats refresh 30s
        stats show-node
        stats auth admin:password
        stats uri  /haproxy?stats
frontend localnodes
    bind *:8088
    mode http
    default_backend nodes
backend nodes
    mode http
    balance leastconn
    option forwardfor
    http-request set-header X-Forwarded-Port %[dst_port]
    http-request add-header X-Forwarded-Proto https if { ssl_fc }
    option httpchk
    server idx2 ssl verify none check 
    server idx1 ssl verify none check 

Making Splunk Certified Apps

As a developer of “Apps” for the Splunk platform; I have been very eager to automate more tedious tasks including build and static code analysis. Today our very awesome development community has access to a new tool App Inspect. The new python based extensible framework will allow your automated build process to validate key issues and prepare for formal certification for Public apps on Splunk Base, or assure quality for internally developed apps. The process example can easily be ported to the tool section of your choice allowing for effective version control and testing of applications built on the Splunk platform.

To help you get started I’ve developed an example using our partner’s tools at Atlassian.

  • Bitbucket repository containing the source
  • CMAKE build script for packaging and versioning
  • Bitbucket pipelines integration using docker to ensure a clean package and execute validation
  • Publish to AWS S3 as a package repository before manual publishing to Splunk Base

Getting started review https://bitbucket.org/Splunk-SecPS/seckit_sa_geolocation this is my first and most complete example.

  • CMakeLists.txt controls the build process
  • src/ contains the applications source
  • src/default/app.conf.in is the template for app.conf our build will update this file with the correct version tag supplied by git
  • bitbucket-pipelines.yml controls the pipelines automated integration process
    • Retrieve and deploy the latest docker image with build tools and app inspect
    • Package the app
    • Push to S3

Try it yourself!

Syncing up shclusterapps

This one is short and sweet, when building a Splunk search head cluster we often will create a search head unattached to indexers to “stage” .spl deployments, configure THEN update shcluster/apps and push the following rsync command does this for you and obeys the golden rule to avoid default core apps. The list is correct as of 6.4.1 update as needed for new versions and be sure to exclude anything like an “app” containing deployment client

rsync –verbose –progress –stats –recursive –times –perms \
–exclude alert_logevent \
–exclude launcher \
–exclude SplunkForwarder \
–exclude alert_webhook \
–exclude learned \
–exclude splunk_httpinput \
–exclude appsbrowser \
–exclude legacy \
–exclude SplunkLightForwarder \
–exclude framework \
–exclude sample_app \
–exclude splunk_management_console \
–exclude gettingstarted \
–exclude search \
–exclude*_deploymentclient** \
–exclude introspection_generator_addon \
–exclude splunk_archiver \
–exclude user-prefs \
/opt/splunk/etc/apps/* /opt/splunk/etc/shcluster-test/apps

Ghost Detector (CVE-2015-7547)


Just in case you need need yet another reason to utilize passive DNS analytic, a new significant vulnerability is out for GLIBC. Have stream? You can monitor your queries for this IOC


Update: the attack requires both A and AAAA records. Only show possible attacks with both involved. This should return zero results. If results are returned there “may” be something of interest drill into the answers involved to determine if they are malicious based on the CVE above.

index=streams sourcetype=stream:dns (query_type=A OR query_type=AAAA)
search index=streams sourcetype=stream:dns (query_type=A OR query_type=AAAA)
| rare limit=20 dest
| fields + dest | format
| stats max(bytes_in) max(bytes_out) max(bytes) values(query_type) as qt by src,dest,query
| where mvcount(qt)>=2
| sort – max*
| lookup domain_segments_lookup domain as query OUTPUT privatesuffix as domain
| lookup alexa_lookup_by_str domain OUTPUT rank
| where isnull(rank)

Don’t have stream yet? Deploy in under 20 minutes.

Dealing with bad threat data

Every now and then a threat data provider will include invalid entries in their threat list creating loads of false positives in Enterprise Security. For “reasons” namely performance ES will append new entries to the internal threat system but does not remove entries no longer present in a source. You can easily clear an entire threat collection which will allow your system to reload from the current sources.

splunk stop
splunk clean inputdata threatlist
splunk clean inputdata threat_intelligence_manager
splunk start
splunk clean kvstore -app DA-ESS-ThreatIntelligence -collection

Common values for collection are http_intel and domain_intel