What’s in a URL now you can Splunk that

Hunting we find URLs in logs both email and proxy that are interesting all the time. What will that URL return, if it redirects where is it going and what kind of content questions you might be asking. If you are not asking them now is the time to start. I’ve released a new add on to Splunk Base, a little adaptive response action that can be used with just Splunk Enterprise OR Splunk Enterprise Security to collect and index information about those URLs.


Ghost Detector (CVE-2015-7547)


Just in case you need need yet another reason to utilize passive DNS analytic, a new significant vulnerability is out for GLIBC. Have stream? You can monitor your queries for this IOC


Update: the attack requires both A and AAAA records. Only show possible attacks with both involved. This should return zero results. If results are returned there “may” be something of interest drill into the answers involved to determine if they are malicious based on the CVE above.

index=streams sourcetype=stream:dns (query_type=A OR query_type=AAAA)
search index=streams sourcetype=stream:dns (query_type=A OR query_type=AAAA)
| rare limit=20 dest
| fields + dest | format
| stats max(bytes_in) max(bytes_out) max(bytes) values(query_type) as qt by src,dest,query
| where mvcount(qt)>=2
| sort – max*
| lookup domain_segments_lookup domain as query OUTPUT privatesuffix as domain
| lookup alexa_lookup_by_str domain OUTPUT rank
| where isnull(rank)

Don’t have stream yet? Deploy in under 20 minutes.

Dealing with bad threat data

Every now and then a threat data provider will include invalid entries in their threat list creating loads of false positives in Enterprise Security. For “reasons” namely performance ES will append new entries to the internal threat system but does not remove entries no longer present in a source. You can easily clear an entire threat collection which will allow your system to reload from the current sources.

splunk stop
splunk clean inputdata threatlist
splunk clean inputdata threat_intelligence_manager
splunk start
splunk clean kvstore -app DA-ESS-ThreatIntelligence -collection

Common values for collection are http_intel and domain_intel

Building Reliable Syslog infrastructure on Centos 7 for Splunk



Preparation of a base infrastructure for high availability ingestion of syslog data with a default virtual server and configuration for test data on boarding. Reference technology specific on boarding procedures.


Multiple critical log sources require a reliable syslog infrastructure. The following attributes must be present for the solution

  • Enterprise supported linux such as RHEL, OR Centos
  • Syslog configuration which will not impact the logging of the host on which syslog is configured
  • External Load Balancing utilizing DNAT lacking available enterprise shared services NLB devices KEMP offers a free to use version of their product up to 20 Mbs suitable for many cases

Technical Environment

The following systems will be created utilizing physical or virtual systems. System specifications will vary due estimated load.

  • Centos 7.x (current) servers in n+1 configuration
    • Minimum 2 GB memory
    • Minimum 2 x 2.3 GHZ core
    • Mounts configure per enterprise standard with the following additions
      • /opt/splunk 40 GB XFS
      • /var/splunk-syslog 40 GB XFS
  • Dual interfaced load balancer configured for DNAT support.
  • Subnet with at minimum the number of unique syslog sources (technologies) additional space for growth is strongly advised
  • Subnet allocated for syslog servers

Solution Prepare the syslog-ng servers

The following procedure will be utilized to prepare the syslog-ng servers

  1. Install the base operating system and harden according to enterprise standards
  2. Provision and mount the application partitions /opt/splunk and /var/splunk-syslog according the estimates required for your environment.
    1. Note 1 typical configuration utilize noatime on both mounts
    2. Note 2 typical configuration utilizes no execute on the syslog moun
  3. Enable the EPEL repository for RHEL/CENTOS as the source for syslog-ng installation 
    yum -y install epel-release
    yum -y repolist
    yum -y update
  4. Install the syslog-ng software


    yum y install syslog-ng
  5. Replace /etc/syslog-ng/syslog-ng.conf
    @include "scl.conf"
    # syslog-ng configuration file.
    # SecKit template 
    # We utilize syslog-ng on Centos to allow syslog ingestion without 
    # interaction with the OS
    # Note: it also sources additional configuration files (*.conf)
    #    located in /etc/syslog-ng/conf.d/
    options {
        flush_lines (0);
        time_reopen (10);
        log_fifo_size (1000);
        chain_hostnames (off);
        use_dns (no);
        use_fqdn (no);
        create_dirs (no);
        keep_hostname (yes);
    # Source additional configuration files (.conf extension only)
    @include "/etc/syslog-ng/conf.d/*.conf"
  6. Create the following directories for modular configuration of syslog-ng
    mkdir -p /etc/syslog-ng/conf.d/splunk-0-source
    mkdir -p /etc/syslog-ng/conf.d/splunk-1-dest  
    mkdir -p /etc/syslog-ng/conf.d/splunk-2-filter  
    mkdir -p /etc/syslog-ng/conf.d/splunk-3-log  
    mkdir -p /etc/syslog-ng/conf.d/splunk-4-simple
  7. Create the Splunk master syslog-configuration /etc/syslog-ng/conf.d/splunk.conf
    # SecKit syslog template based on the work of Vladimir
    # Template from https://github.com/hire-vladimir/SA-syslog_collection/
    #### Global config ####
    options {
      # Specific file/directory permissions can be set
      # this is particularly needed, if Splunk UF is running as non-root
    @include "/etc/syslog-ng/conf.d/splunk-0-source/*.conf"
    @include "/etc/syslog-ng/conf.d/splunk-1-dest/*.conf"
    @include "/etc/syslog-ng/conf.d/splunk-2-filter/*.conf"
    @include "/etc/syslog-ng/conf.d/splunk-3-log/*.conf"
    @include "/etc/syslog-ng/conf.d/splunk-4-simple/*.conf"
  8. Create the catch all syslog collection source. /etc/syslog-ng/conf.d/splunk-4-simple/8100-default.conf
    #### Enable listeners ####
    source remote8100_default
    #### Log remote sources classification ####
    destination d_default_syslog {
    # catch all, all data that did not meet above criteria will end up here
    log {
  9. Ensure splunk can read from the syslog folders. The paths should exist at this point due to the dedicated mount
    chown -R splunk:splunk /var/splunk-syslog
    chmod -R 0755 /var/splunk-syslog
  10. Verify syslog-ng configuration no errors should be reported (no output)
    syslog-ng -s
  11. Update the systemd servics configuration to correctly support both rsyslog and syslog-ng edit /lib/systemd/system/syslog-ng.service


    ExecStart=/usr/sbin/syslog-ng -F -p /var/run/syslogd.pid
    ExecStart=/usr/sbin/syslog-ng -F -p /var/run/syslogd-ng.pid
  12. Create log rotation configuration /etc/logrotate.d/splunk-syslog
        rotate 4
        maxage 7
        /bin/kill -HUP `cat /var/run/syslogd-ng.pid 2> /dev/null` 2> /dev/null || true
  13. Resolve SELinux blocked actions
    semanage port -a -t syslogd_port_t -p tcp 8100
    semanage port -a -t syslogd_port_t -p udp 8100
    semanage fcontext -a -t var_log_t /var/splunk-syslog
    restorecon -v '/var/splunk-syslog'
    logger -d -P 8100 -n -p 1 "test2"
    cd /root
    mkdir selinux
    cd selinux
    audit2allow -M syslog-ng-modified -l -i /var/log/audit/audit.log
    #verify the file does not contain anything no related to syslog
    vim syslog-ng-modified.te
    semodule -i syslog-ng-modified.pp
  14. Allow firewall access to the new ports
    firewall-cmd --permanent --zone=public --add-port=8100/tcp 
    firewall-cmd --permanent --zone=public --add-port=8100/udp
    firewall-cmd --reload
  15. Enable and start syslog-ng
    systemctl enable syslog-ng
    systemctl start syslog-ng


Solution Prepare KEMP Loadbalancer

  • Deploy virtual load balancer to hypervisor with two virtual interfaces
    • #1 Enterprise LAN
    • #2 Private network for front end of syslog servers
  • Login to the load balancer web UI
  • Apply free or purchased license
  • Navigate to network setup
    • Set eth0 external ip
    • Set eth1 internal ip
  • Add the first virtual server (udp)
    • Navigate to Virtual Services –> Add New
    • set the virtual address
    • set port 514
    • set port name syslog-default-8100-udp
    • set protocol udp
    • Click Add this virtual service
    • Adjust virtual service settings
      • Force Layer 7
      • Transparency
      • set persistence mode source ip
      • set persistence time 6 min
      • set scheduling method lest connected
      • Use Server Address for NAT
      • Click Add new real server
        • Enter IP of syslog server 1
        • Enter port 8100
  • Add the first virtual server (tcp)
    • Navigate to Virtual Services –> Add New
    • set the virtual address
    • set port 514
    • set port name syslog-default-8100-tcp
    • set protocol tcp
    • Click Add this virtual service
    • Adjust virtual service settings
      • Service type Log Insight
      • Transparency
      • set scheduling method lest connected
      • TCP Connection only check port 8100
      • Click Add new real server
        • Enter IP of syslog server 1
        • Enter port 8100
  • Repeat the add virtual server process for additional resource servers


Update syslog server routing configuration

Update the default gateway of the syslog servers to utilize the NLB internal interface

Validation procedure

from a linux host utilize the following commands to validate the NLB and log servers are working together
logger -P 514 -T -n <vip_ip> "test TCP"
logger -P 514 -d -n <vip_ip> "test UDP"
verify the messages are logged in /var/splunk-syslog/default

Prepare Splunk Infrastructure for syslog

  • Follow procedure for deployment of the Universal Forwarder with deployment client ensure the client has has valid outputs and base configuration
  • Create the indexes syslog and syslog_unclassified
  • Deploy input configuration for the default input
host_regex = .*\/(.*)\.log
sourcetype = syslog
source = syslog_enterprise_default
index = syslog_unclassified
disabled = enabled


  • Validate the index contains data


When you have 100 problems, more logs are not the answer

big_fire_01 Often SIEM projects begin where log aggregation projects end. So many logs cut into organized stacks of wood ready to burn for value. I can be quoted on this “All logs can be presumed to have security value”. One project to build the worlds largest bonfire however is seldom the correct answer. What value you may ask? Value will be gained in one or more of these categories:

Continue reading “When you have 100 problems, more logs are not the answer”

Making Asset data useful with Splunk Enterprise Security CSC 1 Part 1


Update broken link 2017-10-04

Friend we need to talk, there is something important that you have been overlooking for a long time. Two years ago when you implemented your first SIEM you gave your consultant an excel file listing all of your servers on the corporate network. You promised you would spend time on it after the consultant left but, then you got the new FireEye. You didn’t forget but then the you got a new Next Gen firewall and after there was the new red team initiative.

It is time make a difference in the security posture of your organization. It is time to take a bite out of CSC #1a that’s not a typo we need to work on #1a, #2 can wait. so can #1b .It is time to work SANs critical control #1. I know the CMDB is out of date and doesn’t reflect today’s architecture. We can do a lot with a small amount of work, today I will share how to lay a foundation to address CSC 1: Inventory of Authorized (a) and Unauthorized Devices (b).


Objective 1: Identify the location of each asset using latitude, longitute, city state and zip
Objective 2: Identify the compliance zone for each network segment
Objective 3: Identify categories that can assist the analyst in review of events related to the network containing the source or destination
Objective 4: Identify the minimum priority of devices in a given network segment.

Code is provided via Security Kit Install the app “SecKit_SA_idm_common” on your ES Search head.

Don’t forget to update the app imports to include “SecKit_SA_.*”


  1. Update seckit_idm_pre_cidr_location.csv so that for each subnet in cidr notation define the location. On a very large campus it may be desirable to present a point on a specific building however in most cases it will be adequate to have a single lat/long pair for all subnets on a campus. Include all private and public spaces owned or managed by your organization do not include any public space not external spaces such as hosting providers and cloud services.
  2. Update seckit_idm_pre_cidr_category.csv note subnet in this case may be larger or smaller than used in locations. The most precise definition will be utilized by Splunk Identity Management within Enterprise Security. This may contain cloud address space if the ip space is not continually re-purposed
    1. Populate pci_cidr_domain we will overload this field for non PCI environments.
      1. PCI usage “wireless ORtrust|cardholder OR  trust|dmz OR empty (empty or default represents untrust
      2. Non PCI usage substitute other compliance in place of cardholder such as pii, sox, hippa, cip
    2. Populate cidr_priority
      1. low the most often used value should represent the majority of your devices
      2. medium common servers
      3. high devices of significant importance
      4. critical devices requiring immediate response such as
        1. A server whose demise would cause you to work on Christmas
        2. A server whose demise could cause the closure of the company even if you work on Christmas
    3. Populate cidr_category values provided here would apply to all devices in this network. I will list some very common categories I apply note each category needs to be pipe “|” separated and may not contain a space
      1. net_internal – internal IP space
      2. net_external – external IP space
      3. netid_ddd_ddd_ddd_ddd_bits – applied to each allocated subnet (smallest assigned unit.
      4. zone_string – where string is one of dmz, server, endpoint, storage management, wan, vip, nat
      5. facility_string – where string is the internal facility identification code
      6. facility_type_string – where string is a common identifier such as datacenter, store, office, warehouse, port, mine, airport, moonbase, cloud, dr, ship
      7. net_assignment_string – where string is static dyndhcp, dynvirt
    4. Run the saved search “seckit_idm_common_assets_networks_lookup_gen” and review the results in seckit_idm_common_assets_networks.csv you may run this report on demand as the lookup files above are changed or on a schedule of your choice.
    5. Enable the asset file in enterprise security by navigating to Configuration –> Enrichment –> Assets and Identities then clicking enable on “seckit_idm_common_assets_networks”

Bonus Objective

Enhance your existing server and network device assets list by integrating the following lookups and merging the OUTPUT fields with the device specific asset data.

  1. | lookup seckit_idm_pre_cidr_category_by_cidr_lookup cidr as lip OUTPUT cidr_pci_domain as pci_domain cidr_category as category
  2. | lookup idm_shared_cidr_location_lookup cidr as ip OUTPUT lat long city country




Share that search! Building a content pack for Splunk Enterprise Security 4.0+

Splunk has initial support for export of “content” which can be dashboards and correlation searches created by the user to share with another team. What if you need to be a little more complex for example including a lookup generating search? This will get a little more complicated but very doable by the average admin. Our mission here is to implement UC0029. What is UC0029 glad you ask Each new malware signature detected should be reviewed by a security analyst to determine if proactive steps can be taken to prevent infection. We will create this as a notable event so that we can provide evidence to audit that the process exists and was followed.

Source code will be provided so I will not detail step by step how objects will be created and defined for this post

UC0029 Endpoint new malware detected by signature


My “brand” is SecKit so you will see this identifier in content I have created alone or with my team here at Splunk. As per our best practice adopt your own brands and use appropriately for your content. There is no technical reason to replace the “brand” on third party content you elect to utilize.

Note ensure all knowledge objects are exported as all app’s owned by admin as you go

      • Create a DA-ESS-SecKit-EndpointProtection
        • This will contain ES specific content such as menus dashboards, and correlation searches
      • Create the working app SA-SecKit-EndpointProtection
        • This will contain props transforms lookups and scheduled searches created outside of ES
      • Create the lookup seckit_endpoint_malware_tracker this lookup will contain each signature as it is detected in the environment and some handy information such as the endpoint first detected, user involved and the most recent detection.
      • Create empty lookup CSV files
        • seckit_endpoint_malware_tracker.csv (note you will not ship this file in your content pack)
        • seckit_endpoint_malware_tracker.csv.default

Build and test the saved search SecKit Malware Tracker – Lookup Gen. This search will use tstats to find the first and last instance of all signatures in a time window and update the lookup if an earlier or later instance is found


      Build and test the correlation search UC0029-S01-V001 New malware signature detected. This search will find “new” signatures from the lookup we have created and create a notable event”Make it default” In both apps move content from local/ to default/ this will allow your users to customize the content without replacing the existing searches”Turn if off by default” It is best practice to ensure any load generating searches are disabled by default

        add disabled=1 to each savedsearches.conf stanza that does not end in”- Rule”add disabled=1 to each correleationsearches.conf

Create a spl (tar.gz) containing both apps createdWrite a blog post explaining what you did, how the searches work and share the code!Gain fame and respect maybe a fez or a cape

The source code


Bonus: Delegate administration of content app

  1. Using your favorite editor edit app/metadata/local.meta
  2. Update the following permisions adding “ess_admin” role

## access = read : [ * ], write : [ admin,role2,role3 ]
access = read : [ * ], write : [ admin,ess_admin ]

access = read : [ * ], write : [ admin,ess_admin ]

Get started with Splunk App Stream 6.4 for DNS Analysis

Passive DNS analysis is all the rage right now, the detection opportunities presented have been well discussed for some time. If your organization is like most now is the time you are being asked how you can implement these detection strategies. Leveraging your existing Splunk investment you can get started very quickly with less change to your organization than one might think. Here is what we will use older versions will work fine however the screen shots will be a bit off:

  •  Splunk Enterprise 6.3.1
  • Splunk App for Stream 6.4

We will assume Splunk Enterprise 6.3.1has already been installed.

Decide where to install your Stream App. Typically this will be the Enterprise Security search head. However if your ES search head is also a search head cluster you will need to use an AD-HOC search head,  dedicated search head or a deployment server. Current versions of Stream fully support installation on a Search Head Cluster.

Note: If using the deployment server (DS) you must configure the server to search the indexer or index cluster containing your stream data.

  1. Install Splunk App for Stream using the standard procedures located here.
  2. Copy the deployment TA to your deployment server if you installed on a search head. /opt/splunk/etc/deployment-apps/Splunk_TA_stream
  3. On your deployment server create a new folder to contain configuration for your stream dns server group.
    • mkdir -p Splunk_TA_stream_infra_dns/local
  4. Copy the inputs.conf from the default TA to the new TA for group management
    • cp Splunk_TA_stream/local/inputs.conf Splunk_TA_stream_infra_dns/local/
  5. Update the inputs.conf to include your forwarder group id
    • vi Splunk_TA_stream_infra_dns/local/inputs.conf
    • Alter “stream_forwarder_id =” to “stream_forwarder_id =infra_dns”
  6. Create a new server class “infra_stream_dns” include both the following apps and deploy to all DNS servers (Windows DNS or BIND)
    • Splunk_TA_stream
    • Splunk_TA_stream_infra_dns
  7. Reload your deployment server

Excellent at this point the Splunk Stream app will be deployed to all of your DNS servers and sit idle. The next few steps will prepare the environment to start collections

  • Create a new index I typically will create stream_dns and setup retention for 30 days.

Configure your deployment group

  1. Login to the search head with the Splunk App for Stream
  2. Navigate to Splunk App for Stream
  3. If this is your first time you may find you need to complete the welcome wizard .
  4. Click on Configure the “Distributed Forwarder Management”
    • stream_configure_dfm
  5. Click Create New Group as follows then click Next
    1. Name Infra_DNS
    2. Description Applied to All DNS servers
    3. Include Ephemeral Streams? No
  6. Enter “infra_dns” as this will ensure all clients deployed above will pickup this configuration from the Stream App
  7. Search for “Splunk_DNS” and select each match then Click Finish
    • stream_dns_aggs
  8. Click on Configuration then “Configure Streams”
    • stream_configure
  1. Click on New Stream
  2. Setup basic info as follows then click Next
    1. Protocol DNS
    2. Name “Infra_DNS”
    3. Description “Capture DNS on internal DNS servers”
    4. stream_configure_dns
  3. We will no use Aggregation so leave this as “No” and click Next
  4. The default fields will meet our needs so go ahead and click Next
  5. Optional Step: Create filters in most cases requests from the DNS server to the outside are not interesting as they are generated based on client requests that cannot be answer from the cache. Creating filters will reduce the total volume of data by approximately 50%
    1. Click create filter
    2. Select src_ip as the field
    3. Select “Not Regular Expression” as the type
    4. Provide a regex capture that will match  all DNS server IPs example “(172\.16\.0\.(19|20|21))” will match in my lab network.
      • stream_filter
    5. Click next
    6. Select only the Infra_DNS group and click Create Stream

At this point stream will deploy and begin collection however index selection is not permitted in this workflow so we need to go back and set it up now.

  1. Find Infra_DNS and click edit
  2. Select the index appropriate for your environment
  3. Click save

Ready to check your work? Run this search replace index=* with your index

index=* sourcetype=stream:dns | stats count by query | sort – count


Staying up to date with the updates, update smart with Microsoft and Secunia

Its been a busy year already Oracle’s Java, Adobe’s Flash, and so many Updates to Windows. Most users by how have heard they should keep their Windows PCs up to date to avoid infection. Unfortunately, our adversaries have heard the same speech and are trying to deceive through fake updates for your computer. First reliable companies will not notify you by email, instant message, or advertisement that your computer is out of date and needs an update. You may see email or advertisements for new versions or upgrades, and subscription renewals. Some leading software companies are helping us stay secure through automatic or seamless updates such as Google’s Chrome browser, the FireFox Browser, and Adobe’s Flash. Security updates for these productions will simply install in the backgroud without needing your help. You can keep yourself safer by taking a few steps to secure your computer.

Lets take care of our operating system first.

  • Open “Computer” on Windows 7 or “This PC” on Windows 8.1
  • Click on Control Panel in the menu bar.
  • Search for “Windows Update” (1) in control panel then select “Turn automatic updating on or off” (2)


  • Setup the options (1) (2) (3) as shown below then click ok (4)

Windows will now check all Microsoft Products daily for updates and install them as needed. You will be asked to reboot your computer to finish applying updates this is very important don’t put it off. Now what about non Microsoft programs? Secunia provides a product called PSI to help us with this task.

Update the rest of our software with Secuina

First download and install Secuina PSI it is very important for you to download from this link. There are a number of sites offering versions modified to include malware.

  • http://secunia.com/vulnerability_scanning/personal/ you will need to provide your name and email address.
  • Then look for the big “Download Now” button. “Try Now” is for a separate business grade product.


  • Run and install PSISetup.Exe, this is a simple next, next, finish, default choices will be best.
  • After you click finish the software will start to update your computer. I installed an old version of Java to demonstrate the process below:
  • After the updates complete you will see an updated list of software and your are done.

PSI will not upgrade software however, for example Adobe Acrobat XII (Future software) or the new Java JRE 1.8 will require you to visit the software vendor to download or purchase an upgrade at some point in the future.

Getting Started with KeePass Part 1

KeePass is a Open Source Information manager. KeePass is simple to install and has a wide variety of options of personal security however it does not directly integrate with any web browser. The significant plus with this solution is the cost. Free

Get started by downloading and installing the software from this site. 


  • Open KeyPass by clicking on your start menu then all programs then “Key Pass 2”
  • The first time you run the program you will be asked if  can automatically check for updates. Enable this option KeePass
  • KeePass will open up and look like this to start


  • We would like KeePass to start with windows so from the Tools Menu click Options
  • Click the integration tab and check “Run KeyPass at Windows Startup”
  • Click OkKeepass2

Now we are ready to create our first password database. For most users one database will be enough however it may make sense to create separate databases for information associated with a specific organization with a separate database for personal information.

  • From the file menu click “New”
  • Create a folder in documents “KeePass”
  • Name the database with a meaningful name such as “PersonalAccounts”Keepass3
  • Create a master pass phrase with at least 12 total characters, using two words 1 or more upper case letters 1 or more symbols and 1 or more numbers.Keepass4
  • 1-Enter a descriptive name for this database
  • 2-Enter a default username that is either a username or email address that you will typically use for your accounts
  • 3-Optional Pick a colorKeepass5
  • Click the Security Tab
  • Change the iterations value to 15000Keepass6
  • Click OK
  • First lets Add a Group under our Internet identities for social media right click on “Internet” click social group then “Add Group”
  • Keepass7
  • Name the group “Social Media” and Click OKKeepass8
  • Click Add Entry
  • keepass9
  • Fill out the entry with all of the information you have
    • 1- Title of the entry
    • 2- Your Username on this site
    • 3- Your password on this site (x2) if the password is less than 50 bit a strong password is advisable
    • 4- The URL to this site i.e. http://www.facebook.com
    • 5- Click OK
  • The entry will now be listed under the social media group
  • keepass10
  • The entry will not be listed under the social media groupkeepass11
  • Congratulations on creating your first entry! Open a webrowser for the site you just created
  • Return to KeePass and select your entry then choose copy username (green arrow or Control +B)
  • Go to your web browser and past in the username field
  • Return to KeePass and select your entry then choose copy password (red arrow or control +C)
  • Note you have 15 seconds to past the value in to the  correct location last pass will clear the clip board to protect your information
  • keepass12

Repeat the steps above for each web site or system you will use. When you are done with a work session choose “Lock Workspace” from the file menu to protect your information. Also don’t forget to save your database from the file menu after important changes.