Using systemd to squash THP and start splunk enterprise

Updated Jan, 16, 2018 user security issue

Updated Jan 19,2018 using forking type for splunk

Updated Oct 2019 for format issues after wordpress upgrade

Fixing INIT Scripts

If you are currently or prefer using init script startup to remain as close to “out of box” configuration as possible be aware of a serious security risk present in the traditional startup method.  REF: To mitigate the issue and address THP/Ulimits consider moving to a field modified version of the script.

Going forward using SYSTEMD

The concept presented in this post, as well as the original inspiration, have some risks. Using alternatives to the vendor provided init scripts have support risks including loss of the configuration by future upgrades. Each operating system vendor has their own specific guidance on how to do this, each automation vendor has example automation scripts as well. Picking an approach that is appropriate for your environment is up to you.

THP the bain of performance for so many things in big data is often left on by default and is slightly difficult to disable. As a popular Splunk answers post and Splunk consultants include Marquis have found the best way to ensure ulimit and THP settings are properly configured is to modify the init scripts. This is a really crafty and reliable way to ensure THP is disabled for Splunk, it works on all Linux operating systems regardless of how services are started.

I’m doing some work with newer operating systems and wanted to explore how systemd really works and changes what is possible in managing a server. Lets face it systemd has not gotten the best of receptions in the community, after all it moved our cheese, toys and the ball all at once. It seems to be here to stay what if we could use its powers for good in relation to Splunk. Let’s put an end to THP and start Splunk the systemd native way.

Note: the following config file is present for readability and google. Downloadable text file is available

Create the file /etc/systemd/system/disable-transparent-huge-pages.service

Description=Disable Transparent Huge Pages

ExecStart=/bin/sh -c “echo never >/sys/kernel/mm/transparent_hugepage/enabled”
ExecStart=/bin/sh -c “echo never >/sys/kernel/mm/transparent_hugepage/defrag”

Verify THP and defrag is presently enabled to avoid a false sense of success

# cat /sys/kernel/mm/transparent_hugepage/enabled

[always] madvise never

# cat /sys/kernel/mm/transparent_hugepage/defrag

[always] madvise never

Enable and start the unit to disable THP

# systemctl enable disable-transparent-huge-pages.service

# systemctl start disable-transparent-huge-pages.service

# cat /sys/kernel/mm/transparent_hugepage/enabled

always madvise [never]

# cat /sys/kernel/mm/transparent_hugepage/defrag

always madvise [never]

Reboot and repeat the verification to ensure the process is enforced

Note: the following config file is present for readability and google. Downloadable text file is available

create the unit file /etc/systemd/system/splunk.service

#2018-01-19 Switched to forking indexers with no web port exit differentl than search heads
Description=Splunk Enterprise


ExecStart=/opt/splunk/bin/splunk start --answer-yes --no-prompt --accept-license
ExecStop=/opt/splunk/bin/splunk stop


#ulimit -Sn 65535
#ulimit -Hn 65535
#ulimit -Su 20480
#ulimit -Hu 20480
#ulimit -Hf unlimited
#ulimit -Sf unlimited

# systemctl enable splunk.service

# systemctl start splunk.service

Verify the ulimits have been applied via splunk logs

#cat /opt/splunk/var/log/splunk/splunkd.log | grep ulimit

Reboot and repeate all verifications

Bonus material, kill Splunk (lab env only) and watch systemd bring it back

# killall splunk

# ps aux | grep splunk

You just noticed splunkd was brought back to up when it died without using systemctl stop. This means using splunk start|stop is not valid when systemd started Splunk.

Splunk Universal Forwarder Version 6.2.3+ Ubuntu 15.04

Author: Ryan Faircloth

Summary: Using repositories for version managment of the Splunk Universal Forwarder assists in ensuring managed Ubuntu systems are using the approved version of the software at all times.

Setup the repository server

  1. Install reprepro and nginx

    sudo apt-get install reprepro nginx packaging-dev -y

  2. Create a user to work with the repository

    adduser --disabled-password --disabled-login --home /srv/reprepro --group reprepro

  3. Change user to our reprepro user all commands for the repository should be executed using this ID

    sudo su - reprepro

Generate GPG Keys

  1. Change user to our reprepro user all commands for the repository should be executed using this ID

    sudo su - reprepro 
  2. Create the default configuration for gpg by running the command

    gpg --list-keys

  3. Edit ~/.gnupg/gpg.conf
    • uncomment the line no-greeting
    • add the following content to the end of the file
    # Prioritize stronger algorithms for new keys.
    default-preference-list SHA512 SHA384 SHA256 SHA224 AES256 AES192 AES CAST5 BZIP2 ZLIB ZIP UNCOMPRESSED
    # Use a stronger digest than the default SHA1 for certifications.
    cert-digest-algo SHA512
  4. Generate a new key with the command gpg --gen-key

  5. Select the folowing options
    1. Type of key “(1) RSA and RSA (default)”
    2. Key size “4096”
    3. Expires “10y”
    4. Confirm “Y”
    5. Real Name “Splunk local repository”
    6. Email address on repository contact this generally should be an alias or distribution list
    7. Leave the comment blank
    8. Confirm and “O” to Okay
    9. Leave passphrase blank and confirm, a key will be generated not the sub KEY ID in the following example * E507D48E *

    gpg: checking the trustdb
    gpg: 3 marginal(s) needed, 1 complete(s) needed, PGP trust model
    gpg: depth: 0 valid: 1 signed: 0 trust: 0-, 0q, 0n, 0m, 0f, 1u
    gpg: next trustdb check due at 2025-05-24
    pub 4096R/410E1699 2015-05-27 [expires: 2025-05-24]
    Key fingerprint = 7CB8 81A9 E07F DA7B 83FF 2E1B 8B31 DA83 410E 1699
    uid Splunk local repository <>
    sub 4096R/E507D48E 2015-05-27 [expires: 2025-05-24]

  6. Export the signing keys public component save this content for use later

    gpg --export --armor KEY_ID >~/

Configure Prerepro

  1. Change user to our reprepro user all commands for the repository should be executed using this ID sudo su - reprepro

  2. Create the directory structure sudo mkdir -p /srv/reprepro/ubuntu/{conf,dists,incoming,indices,logs,pool,project,tmp}

  3. Change directories to the new repository cd /srv/reprepro/ubuntu/

  4. Edit the file /srv/reprepro/ubuntu/conf/distributions

  5. Update the file contents

    Origin: SplunkEnterprise
    Label: SplunkEnterprise
    Codename: ponies
    Architectures: i386 amd64 source    
    Components: main
    Description: Splunk Enterprise and Universal Forwarders for Debian based systems
    SignWith: YOUR-KEY-ID 
  6. Edit the file /srv/reprepro/ubuntu/conf/options

  7. Update the file contents

    basedir .

Load the packages

Load the packages using the following commands syntax replace package.deb with the correct path to the splunkforwarder deb file

reprepro -S utils -P standard includedeb ponies package.deb

Setup the web server

  1. Create the file /etc/nginx/sites-available/vhost-packages.conf

  2. Use the following content replacing package.local with the fqdn of the repository host
    server {
      listen 80;
      server_name packages.internal;
      access_log /var/log/nginx/packages-access.log;
      error_log /var/log/nginx/packages-error.log;
      location / {
        root /srv/reprepro;
        index index.html;
      location ~ /(.*)/conf {
        deny all;
      location ~ /(.*)/db {
        deny all;
  3. Increase the server name hash bucket by creating the following file /etc/nginx/conf.d/server_names_hash_bucket_size.conf

  4. Use the following content server_names_hash_bucket_size 64;

  5. Enable the new configuration

    sudo ln -s /etc/nginx/sites-available/vhost-packages.conf /etc/nginx/sites-enabled/vhost-packages.conf
    sudo service nginx reload

Configure the repository

  1. Edit the file
  2. Use the following content
    deb http://packages.internal/ubuntu/ ponies main
  3. Import the public key
    sudo apt-key add /tmp/
  4. Update the repository cache
    sudo apt-get update 

Install the Splunk Universal Forwarder

Run the following command

sudo apt-get install splunkforwarder

Configure the universal forwarder

  • Using best practices to manually create the org_deploymentclient configuration app
  • Using RPM based configuration package
  • Using Configuration Managment system such as Puppet or Chef

Create and install a configuration package for the Universal Forwarder

In the following procedure “org” should be replace with the abbreviate of the organization using the configuration.

  1. Create the paths /srv/reprepro/org_debs/

  2. Create the path for the first version of the package ie mkdir org-splunk-ufconfig-1

  3. Change to the new directory

  4. Create the following structure

    ├── DEBIAN
    │   ├── control (file)
    │   ├── postinst (file)
    │   ├── preinst (file)
    │   └── prerm (file)
    └── opt
        └── splunkforwarder
            └── etc
                └── apps
                    └── org_all_deploymentclient
                        └── default
                            ├── deploymentclient.conf (file)
  5. Edit the DEBIAN/control file as follows

    Package: org-splunk-ufconfig
    Section: base
    Priority: standard
    Version: 1
    Architecture: all
    Maintainer: Your Name <>
    Depends: splunkforwarder (>=6.0.0)
    Description: <insert up to 60 chars description>
    <insert long description, indented with spaces>

  6. Edit the DEBIAN/postinst

    /opt/splunkforwarder/bin/splunk enable boot-start -user splunk --accept-license --answer-yes
    service splunk start    
  7. Edit the DEBIAN/preinst
    if [ -f "$file" ]
        echo "$file found."
        service splunk stop
        echo "$file not found."
  8. Edit the DEBIAN/prerm
    if [ -f "$file" ]
        echo "$file found."
        service splunk stop
        /opt/splunkforwarder/bin/splunk disable boot-start
        echo "$file not found."
  9. Update the contents of deploymentclient.conf with the appropriate information for you installation

  10. Add additional content as required for your deployment

  11. Change directories up to the parent of org-splunk-ufconfig–1

  12. Create the debian package with the command dpkg-deb --build org-splunk-ufconfig-1/

  13. Change to the repository directory /srv/reprepro/ubuntu

  14. Store the new package in the repository

    reprepro -S utils -P standard includedeb ponies /srv/reprepro/org_debs/org-splunk-ufconfig-1.deb

  15. Install the new package on the client using the command sudo apt-get install org-splunk-ufconfig this will install the splunk forwarder package if has not yet been installed.