Lets Encrypt and get an A for A Great Splunk TLS config

Setting up SSL/TLS on Splunk doesn’t have to be super hard or costly. While running Splunk in cloud providers has many benefits there are some hassles like provisioning certificates we can better manage using let’s encrypt. This method of installing browser trusted certificates can help to keep your administrative costs down in large Splunk deployments such as MssP services.

Expanding on prior work https://www.splunk.com/blog/2016/08/12/secure-splunk-web-in-five-minutes-using-lets-encrypt.html

NGINX

First we are going to install NGINX we will use this as a front end reverse proxy. Why, we can renew our certs with minimal own time in the future, OCSP stapling (improved page load times) and other things (future posts)

#centos

yum install nginx

#ubuntu

apt-get install nginx

Second setup a new vhost for the splunk reverse proxy. Any request to http will be redirected to https except for requests related to certificate management.

map $uri $redirect_https {

    /.well-known/                      0;

    default                            1;

}

server {

    listen       80;

    server_name  hf-scan.splunk.example.com;

    root /usr/share/nginx/html;

    if ($redirect_https = 1) {

       return 301 https://$server_name$request_uri;

    }

#    return       301 $scheme://hf-scan.splunk.example.com$request_uri;

}

server {

    

    listen 443 ssl http2;

    server_name hf-scan.splunk.example.com;

    root /usr/share/nginx/html;

    index index.html index.htm;

   location / {

        proxy_pass_request_headers on;

        proxy_set_header x-real-IP $remote_addr;

        proxy_set_header x-forwarded-for $proxy_add_x_forwarded_for;

        proxy_set_header host $host;

        proxy_pass https://127.0.0.1:8000;

        add_header Strict-Transport-Security “max-age=31536000; includeSubDomains” always;

      }

    

    

    ssl_certificate     /etc/letsencrypt/live/hf-scan.splunk.example.com/fullchain.pem;

    ssl_certificate_key /etc/letsencrypt/live/hf-scan.splunk.example.com/privkey.pem;

    ssl_protocols       TLSv1.2;

    ssl_ciphers         HIGH:!aNULL:!MD5;

    ssl_dhparam /etc/nginx/ssl/dhparam.pem;

    ssl_session_cache shared:SSL:50m;

    ssl_session_timeout 1d;

    ssl_session_tickets off;

    ssl_prefer_server_ciphers on;

    ssl_stapling on;

    ssl_stapling_verify on;

    resolver 8.8.8.8 8.8.4.4 valid=300s;

    resolver_timeout 5s;

    add_header Strict-Transport-Security “max-age=31536000; includeSubDomains” always;

}

Setup a deploy hook script this will prepare the cert files as splunk needs them and will also be used on renewal. Save this script as /etc/letsencrypt/renewal-hooks/deploy/splunk.sh

#!/bin/bash
#deploy to /etc/letsencrypt/renewal-hooks/deploy/splunk.sh
#when requesting a cert add "--deploy-hook /etc/letsencrypt/renewal-hooks/deploy/splunk.sh" to the command
dir=/opt/splunk/etc/auth/ssl
if [[ ! -e $dir ]]; then
    mkdir -p $dir
elif [[ ! -d $dir ]]; then
    echo "$dir already exists but is not a directory" 1>&2
fi
openssl rsa -aes256 -in $RENEWED_LINEAGE/privkey.pem -out $dir/protected.pem -passout pass:password
if [[ ! -f $dir/protected.pem ]]; then
    exit 1
fi
cat $dir/protected.pem $RENEWED_LINEAGE/fullchain.pem > $dir/server.pem
cp $RENEWED_LINEAGE/fullchain.pem $dir/
cp $RENEWED_LINEAGE/privkey.pem $dir/
chown splunk:splunk $dir/*
systemctl restart splunk

Request the certificate note correct the webroot folder for your platform and the certificate with the fqdn of your server

certbot certonly –webroot -w /var/www/html –hsts -d hf-scan.splunk.example.com –noninteractive –agree-tos –email your@example.com –deploy-hook /etc/letsencrypt/renewal-hooks/deploy/splunk.sh

Setup Splunk

Update /opt/splunk/etc/system/local/web.conf

[settings]

enableSplunkWebSSL = true

#sendStrictTransportSecurityHeader = true

sslVersions = tls1.2

cipherSuite = TLSv1.2:!NULL-SHA256:!AES128-SHA256:!ADH-AES128-SHA256:!ADH-AES256-SHA256:!ADH-AES128-GCM-SHA256:!ADH-AES256-GCM-SHA384

privKeyPath =  /opt/splunk/etc/auth/ssl/privkey.pem

caCertPath = /opt/splunk/etc/auth/ssl/fullchain.pem

Update /opt/splunk/etc/system/local/server.conf

[general]

serverName = hf-scan.splunk.example.com

[sslConfig]

sslVersions = tls1.2

sslVersionsForClient = tls1.2

serverCert = $SPLUNK_HOME/etc/auth/ssl/server.pem

sslRootCAPath = $SPLUNK_HOME/etc/auth/ssl/fullchain.pem

dhFile = /opt/splunk/etc/auth/ssl/dhparam.pem

sendStrictTransportSecurityHeader = true

allowSslCompression = false

cipherSuite = TLSv1.2:!NULL-SHA256:!AES128-SHA256:!ADH-AES128-SHA256:!ADH-AES256-SHA256:!ADH-AES128-GCM-SHA256:!ADH-AES256-GCM-SHA384

useClientSSLCompression = false

useSplunkdClientSSLCompression = false

Test

  • Option 1 SSL labs, limited to port 443 (don’t forget about 8089)
  • Option 2 testssl.sh CLI based doesn’t share data no letter grade (management likes letters)
  • Option 3 High Tech Bridge https://www.htbridge.com/ssl allows testing multiple ports similar coverage to ssllabs less well known

Renew certs

Setup a cron job to run the following command at least once per week in your scheduled change window. If a certificate renewal is required splunk will be restarted

certbot renew –webroot  -w /usr/share/nginx/html

Can we even patch this Spectre/Meltdown oh and AV also

Isn’t it great when things are in meltdown and you can’t patch yet because your waiting on another patch?

Microsoft has stated you can’t patch until AV goes first

http://www.zdnet.com/article/windows-meltdown-spectre-fix-how-to-check-if-your-av-is-blocking-microsoft-patch/

https://support.microsoft.com/en-us/help/4072699/january-3-2018-windows-security-updates-and-antivirus-software

Bottom line if your AV vendor hasn’t update to set this registry to give the update permissions to install or you don’t use AV and instead use an application whitelist approach for security the patch won’t apply. You can use splunk to track down hosts that will refuse to apply the patch by adding this monitor to splunk and well Splunking the results

Key="HKEY_LOCAL_MACHINE" Subkey="SOFTWARE\Microsoft\Windows\CurrentVersion\QualityCompat" Value="cadca5fe-87d3-4b96-b7fb-a231484277cc" Type="REG_DWORD”
Data="0x00000000

Add the following to the inputs.conf applied to all windows system and ensure the server class is set to restart the UF and happy Splunking

 

[WinRegMon://HKLMSoftwareMSWindowsQualityCompat]
index = epintel
baseline = 1
disabled = 0
hive = \\REGISTRY\\MACHINE\\Software\\Microsoft\\Windows\\CurrentVersion\\QualityCompat\\.*
proc = .*
type = delete|create|set|rename