Splunk OS Data on boarding – best practices updated

I’ve updated my best practices a bit and moved the implementation guides from confluence out to the bitbuckets in markdown so they can be more easily referenced on any platform or secured environments where PDFs might be discouraged.

Each repo will contain a README.md and one or more INSTALL.md files with the implementation guides. If you find an issue have a better practice or other enhancement, please open an issue in the repositories tracker.

Trust but verify – your logs might be lieing to you aka, the case for network tap traffic monitoring

I really do “get” it, logging and monitoring can be very costly, we all agree not nearly as costly as a breach. Each organization is struggling to ensure they log enough to see detection and value while being good stewards of their company budget. It has been a day reading vault 7 leaks and I see honestly not much that surprises me. I do see something worth a strong restatement, that is an encouragement to rethink what you log and how you log it. The CIA has a very cool (sorry hacker at heart) tool we have known about for some time but have not been able to talk about. Their tool “Drillbit” allows the creation of a covert tunnel using common cisco gear in such a way typical monitoring and logging using IDS and firewalls will not identify. American companies should note criminal gangs and foreign governments certainly have similar capabilities. Splunk has your back if you are willing to let us. Using Splunk Stream and proper sensor placement we can collect data from the inside and outside of your firewall that can be used to identify covert tunnels. Detection should be performed three approaches. The danger these leaks are presenting you is an increased awareness of the effectiveness of these techniques, encouraging the advancement of commodity cybercrime toolkits with ever more difficult to detect features. Don’t use cisco, sorry bad news is almost every major gear vendor has been exploited with similar approaches

  • Static Rules such as
    • This not that “Stream identified traffic not in firewall logs”
    • New patterns in DNS, NTP, GRE flows
    • Change/login to firewall or switch not associated with a change record
  • Threat List Enrichment and detection
    • Source and Destination traffic on quality threat lists. Traffic for protocols other than http(s) and DNS should be treated with high or critical priority
  • Machine Learning
    • Anomalous  egress traffic by source from network devices
    • Anomalous admin connections by source to network devices

Automating Splunk deployment for RedHat/Centos (poor man’s edition)

I pulled this out of the archives , on request notice this was originally developed for Splunk 6.2.x and RHEL 7.0. Please review the details make sure it is suitable for you and TEST. If I can talk you out of doing things this way I would. Salt is a great way to manage app config its free and just awesome.

Title: Splunk Universal Forwarder Version 6.2.3+ Red Hat Enterprise Linux 7

Author: Ryan Faircloth 

Summary: Using repositories for version managment of the Splunk Universal Forwarder assists in ensuring managed Red Hat and compatible linux  systems are using the approved version of the software at all times. 


## Setup the repository server ##
1. Install createrepo  and nginx ``` yum install createrepo apache2 ```
3. Create a user to work with the repository 

	sudo adduser  repouser 
3. Change user to our repouser user all commands for the repository should be executed using this ID 

	 sudo su - repouser

## Generate GPG Keys ##
1. Change user to our repouser user all commands for the repository should be executed using this ID 

	sudo su - repouser 
2. Create the default configuration for gpg by running the command 

	gpg --list-keys
3. Edit ~/.gnupg/gpg.conf 
	* uncomment the line ``` no-greeting ```
	* add the following content to the end of the file 
	# Prioritize stronger algorithms for new keys.
	default-preference-list SHA512 SHA384 SHA256 SHA224 AES256 AES192 AES CAST5 BZIP2 ZLIB ZIP UNCOMPRESSED
	# Use a stronger digest than the default SHA1 for certifications.
	cert-digest-algo SHA512
4. Generate a new key with the command ``` gpg --gen-key ```
5. Select the folowing options ON CENTOS/RHEL this procedure must be executed on the console or SSH having logged in as the repouser
	1. Type of key "(1) RSA and RSA (default)"
	2. Key size "4096"
	3. Expires "10y"
	4. Confirm "Y"
	5. Real Name "Splunk local repository"
	6. Email address on repository contact this generally should be an alias or distribution list
	7. Leave the comment blank
	8. Confirm and "O" to Okay
	9. Leave passphrase blank and confirm, a key will be generated not the sub KEY ID in the following example * E507D48E * 

	gpg: checking the trustdb
	gpg: 3 marginal(s) needed, 1 complete(s) needed, PGP trust model
	gpg: depth: 0  valid:   1  signed:   0  trust: 0-, 0q, 0n, 0m, 0f, 1u
	gpg: next trustdb check due at 2025-05-24
	pub   4096R/410E1699 2015-05-27 [expires: 2025-05-24]
    	  Key fingerprint = 7CB8 81A9 E07F DA7B 83FF  2E1B 8B31 DA83 410E 1699
	uid                  Splunk local repository <repo@example.com>
	sub   4096R/E507D48E 2015-05-27 [expires: 2025-05-24]
10. Export the signing keys public component save this content for use later

	gpg --export --armor KEY_ID >~/repo.pub 
11. Install the new key into the RPM database

sudo cp ~/repo.pub /etc/pki/rpm-gpg/RPM-GPG-KEY-splunkrepo
sudo rpm --import /etc/pki/rpm-gpg/RPM-GPG-KEY-splunkrepo
12. Configure RPM signing with the new key

echo "%_signature gpg" > ~/.rpmmacros
echo "%_gpg_name splunkrepo" >> ~/.rpmmacros

13. Create a repository

mkdir /opt/splunkrepo
cp splunkforwarder*.rpm /opt/splunkrepo
createrepo /opt/splunkrepo
14. Configure the local repository create the following configuration /etc/yum.repos.d/splunk.repo

name=splunk repository
15. Test the local repository by installing splunkforwarder

sudo yum update
sudo yum install splunkforwader
Note: Create a configuration RPM refer to https://fedoraproject.org/wiki/How_to_create_an_RPM_package and https://www.redhat.com/promo/summit/2010/presentations/summit/opensource-for-it-leaders/thurs/pwaterma-2-rpm/RPM-ifying-System-Configurations.pdf for more information do not run as root sudo to repouser
17.  Prepare the rpm tree ``` rpmdev-setuptree ```
18.  Create a spec file with the following content ~/splunkforwarder-baseconfig.spec

# This spec file is Copyright 2010, My Company, Inc.
Summary: My Company general configuration RPM
Name: splunkforwarder-baseconfig
Version: 1
Release: 3
License: Copyright 2010, My Company, Inc.
Group: MyCompany/Configs
Packager: Packager Name <my-email@mycompany.com>
requires: splunkforwarder
BuildArch: noarch
This RPM provides general services and security configuration for My Company.

%triggerin -- splunkforwarder
/opt/splunkforwarder/bin/splunk enable boot-start --accept-license --answer-yes 
service splunk stop
if [ -d "/opt/splunkforwarder/etc/apps/org_all_deploymentclient/local" ]
    echo "Directory /opt/splunkforwarder/etc/apps/org_all_deploymentclient/local exists."
    mkdir -p /opt/splunkforwarder/etc/apps/org_all_deploymentclient/local    
echo #Base deployment configuration >/opt/splunkforwarder/etc/apps/org_all_deploymentclient/local/deploymentclient.conf
echo [deployment-client] >>/opt/splunkforwarder/etc/apps/org_all_deploymentclient/local/deploymentclient.conf
#echoclientName  >>/opt/splunkforwarder/etc/apps/org_all_deploymentclient/local/deploymentclient.conf
echo [deployment-client] >>/opt/splunkforwarder/etc/apps/org_all_deploymentclient/local/deploymentclient.conf
echo targetUri = ds.example.com:8089 >>/opt/splunkforwarder/etc/apps/org_all_deploymentclient/local/deploymentclient.conf

service splunk start

%triggerun -- splunkforwarder
if [ $1 -eq 0 -a $2 -gt 0 ] ; then
 /opt/splunkforwarder/bin/splunk stop
 /opt/splunkforwarder/bin/splunk disable boot-start
 rm -Rf /opt/splunkforwarder/etc/apps/org_all_deploymentclient

18.  Build the RPM ``` rpmbuild -sign -ba splunkforwarder-baseconfig.spec ```
19. Copy the RPM to the repository cp ~/rpmbuild/RPMS/noarch/splunkforwarder-baseconfig-1-3.noarch.rpm / /opt/splunkrepo
20. Update repository DB ``` createrepo /opt/splunkrepo ```
21. Test the rpms

yum update
yum install splunkforwarder-baseconfig
22. Configure a web server (Apache) for use as a repository server
23.  Set permissions on the repository folder ``` chmod -R 755 /opt/splunkrepo ```
24. Create the web server configuration file with the following contents /etc/http/conf.d/splunkrepo.conf

Alias /splunkrepo/ "/opt/splunkrepo/"

<Directory "/opt/splunkrepo">
   Options Indexes FollowSymLinks MultiViews
   AllowOverride All
   Require local
   Order allow,deny
   Allow from all
25. Reload (or restart) the web server ```service httpd reload ```
26. Test `` lynx http://localhost/splunkrepo/repodata/repomd.xml ```
27. Enable the new repository on the first test client

sudo yum-config-manager --add-repo http://localhost/splunkrepo
sudo yum update
sudo yum install splunkforwarder-baseconfig

Splunk the server and the enpoint aka “All the Things”

I’m sharing something today that has been available thanks to many in white papers and presentations dealing with identification of malicious code and activities in your windows event data. Shout out to everyone from our “friends” at the NSA, to Splunk .Conf presenters and malwarearcheology.com just to name a few.

The PDF attached is a portion of the next evolution of the Use Case Repository I maintain at Splunk. Along with the reference TAs and inputs, this will allow you to quickly and consistently collect very valuable data supporting security use cases at multiple levels of maturity. If it seems like too much don’t work Splunk Pro Services and partners are able to help you get this visibility just contact your account team.

Standard disclaimer, this is a blog post, I built the content from public non-warrantied information, and this is still public non-warrantied information, your situation might not match the advice given.