Building Reliable Syslog infrastructure on Centos 7 for Splunk

 

Overview

Preparation of a base infrastructure for high availability ingestion of syslog data with a default virtual server and configuration for test data on boarding. Reference technology specific on boarding procedures.

Requirement

Multiple critical log sources require a reliable syslog infrastructure. The following attributes must be present for the solution

  • Enterprise supported linux such as RHEL, OR Centos
  • Syslog configuration which will not impact the logging of the host on which syslog is configured
  • External Load Balancing utilizing DNAT lacking available enterprise shared services NLB devices KEMP offers a free to use version of their product up to 20 Mbs suitable for many cases

Technical Environment

The following systems will be created utilizing physical or virtual systems. System specifications will vary due estimated load.

  • Centos 7.x (current) servers in n+1 configuration
    • Minimum 2 GB memory
    • Minimum 2 x 2.3 GHZ core
    • Mounts configure per enterprise standard with the following additions
      • /opt/splunk 40 GB XFS
      • /var/splunk-syslog 40 GB XFS
  • Dual interfaced load balancer configured for DNAT support.
  • Subnet with at minimum the number of unique syslog sources (technologies) additional space for growth is strongly advised
  • Subnet allocated for syslog servers

Solution Prepare the syslog-ng servers

The following procedure will be utilized to prepare the syslog-ng servers

  1. Install the base operating system and harden according to enterprise standards
  2. Provision and mount the application partitions /opt/splunk and /var/splunk-syslog according the estimates required for your environment.
    1. Note 1 typical configuration utilize noatime on both mounts
    2. Note 2 typical configuration utilizes no execute on the syslog moun
  3. Enable the EPEL repository for RHEL/CENTOS as the source for syslog-ng installation 
    yum -y install epel-release
    yum -y repolist
    yum -y update
    reboot
  4. Install the syslog-ng software

     

    yum y install syslog-ng
  5. Replace /etc/syslog-ng/syslog-ng.conf
    @version:3.5
    @include "scl.conf"
    
    # syslog-ng configuration file.
    #
    # SecKit template 
    # We utilize syslog-ng on Centos to allow syslog ingestion without 
    # interaction with the OS
    
    # Note: it also sources additional configuration files (*.conf)
    #    located in /etc/syslog-ng/conf.d/
    
    options {
        flush_lines (0);
        time_reopen (10);
        log_fifo_size (1000);
        chain_hostnames (off);
        use_dns (no);
        use_fqdn (no);
        create_dirs (no);
        keep_hostname (yes);
    };
    
    # Source additional configuration files (.conf extension only)
    @include "/etc/syslog-ng/conf.d/*.conf"
  6. Create the following directories for modular configuration of syslog-ng
    mkdir -p /etc/syslog-ng/conf.d/splunk-0-source
    mkdir -p /etc/syslog-ng/conf.d/splunk-1-dest  
    mkdir -p /etc/syslog-ng/conf.d/splunk-2-filter  
    mkdir -p /etc/syslog-ng/conf.d/splunk-3-log  
    mkdir -p /etc/syslog-ng/conf.d/splunk-4-simple
  7. Create the Splunk master syslog-configuration /etc/syslog-ng/conf.d/splunk.conf
    ################################################################################
    # SecKit syslog template based on the work of Vladimir
    # Template from https://github.com/hire-vladimir/SA-syslog_collection/
    ################################################################################
    
    ################################################################################
    #### Global config ####
    options {
      create-dirs(yes);
    
      # Specific file/directory permissions can be set
      # this is particularly needed, if Splunk UF is running as non-root
      owner("splunk");
      group("splunk");
      dir-owner("splunk");
      dir-group("splunk");
      dir-perm(0755);
      perm(0755);
    
      time-reopen(10);
      keep-hostname(yes);
      log-msg-size(65536);
    };
    
    @include "/etc/syslog-ng/conf.d/splunk-0-source/*.conf"
    @include "/etc/syslog-ng/conf.d/splunk-1-dest/*.conf"
    @include "/etc/syslog-ng/conf.d/splunk-2-filter/*.conf"
    @include "/etc/syslog-ng/conf.d/splunk-3-log/*.conf"
    @include "/etc/syslog-ng/conf.d/splunk-4-simple/*.conf"
  8. Create the catch all syslog collection source. /etc/syslog-ng/conf.d/splunk-4-simple/8100-default.conf
    ################################################################################
    #### Enable listeners ####
    source remote8100_default
    {
        udp(port(8100));
        tcp(port(8100));
    };
    
    #### Log remote sources classification ####
    destination d_default_syslog {
            file("/var/splunk-syslog/default/$HOST.log");
    };
    
    # catch all, all data that did not meet above criteria will end up here
    log {
            source(remote8100_default);
            destination(d_default_syslog);
            flags(fallback);
    };
  9. Ensure splunk can read from the syslog folders. The paths should exist at this point due to the dedicated mount
    chown -R splunk:splunk /var/splunk-syslog
    chmod -R 0755 /var/splunk-syslog
  10. Verify syslog-ng configuration no errors should be reported (no output)
    syslog-ng -s
  11. Update the systemd servics configuration to correctly support both rsyslog and syslog-ng edit /lib/systemd/system/syslog-ng.service

     

    find:
    ExecStart=/usr/sbin/syslog-ng -F -p /var/run/syslogd.pid
    replace:
    ExecStart=/usr/sbin/syslog-ng -F -p /var/run/syslogd-ng.pid
  12. Create log rotation configuration /etc/logrotate.d/splunk-syslog
    /var/splunk-syslog/*/*.log
    {
        daily
        compress
        delaycompress
        rotate 4
        ifempty
        maxage 7
        nocreate
        missingok
        sharedscripts
        postrotate
        /bin/kill -HUP `cat /var/run/syslogd-ng.pid 2> /dev/null` 2> /dev/null || true
        endscript
    }
  13. Resolve SELinux blocked actions
    semanage port -a -t syslogd_port_t -p tcp 8100
    semanage port -a -t syslogd_port_t -p udp 8100
    semanage fcontext -a -t var_log_t /var/splunk-syslog
    restorecon -v '/var/splunk-syslog'
    logger -d -P 8100 -n 127.0.0.1 -p 1 "test2"
    cd /root
    mkdir selinux
    cd selinux
    audit2allow -M syslog-ng-modified -l -i /var/log/audit/audit.log
    #verify the file does not contain anything no related to syslog
    vim syslog-ng-modified.te
    semodule -i syslog-ng-modified.pp
  14. Allow firewall access to the new ports
    firewall-cmd --permanent --zone=public --add-port=8100/tcp 
    firewall-cmd --permanent --zone=public --add-port=8100/udp
    firewall-cmd --reload
  15. Enable and start syslog-ng
    systemctl enable syslog-ng
    systemctl start syslog-ng

 

Solution Prepare KEMP Loadbalancer

  • Deploy virtual load balancer to hypervisor with two virtual interfaces
    • #1 Enterprise LAN
    • #2 Private network for front end of syslog servers
  • Login to the load balancer web UI
  • Apply free or purchased license
  • Navigate to network setup
    • Set eth0 external ip
    • Set eth1 internal ip
  • Add the first virtual server (udp)
    • Navigate to Virtual Services –> Add New
    • set the virtual address
    • set port 514
    • set port name syslog-default-8100-udp
    • set protocol udp
    • Click Add this virtual service
    • Adjust virtual service settings
      • Force Layer 7
      • Transparency
      • set persistence mode source ip
      • set persistence time 6 min
      • set scheduling method lest connected
      • Use Server Address for NAT
      • Click Add new real server
        • Enter IP of syslog server 1
        • Enter port 8100
  • Add the first virtual server (tcp)
    • Navigate to Virtual Services –> Add New
    • set the virtual address
    • set port 514
    • set port name syslog-default-8100-tcp
    • set protocol tcp
    • Click Add this virtual service
    • Adjust virtual service settings
      • Service type Log Insight
      • Transparency
      • set scheduling method lest connected
      • TCP Connection only check port 8100
      • Click Add new real server
        • Enter IP of syslog server 1
        • Enter port 8100
  • Repeat the add virtual server process for additional resource servers

 

Update syslog server routing configuration

Update the default gateway of the syslog servers to utilize the NLB internal interface

Validation procedure

from a linux host utilize the following commands to validate the NLB and log servers are working together
logger -P 514 -T -n <vip_ip> "test TCP"
logger -P 514 -d -n <vip_ip> "test UDP"
verify the messages are logged in /var/splunk-syslog/default

Prepare Splunk Infrastructure for syslog

  • Follow procedure for deployment of the Universal Forwarder with deployment client ensure the client has has valid outputs and base configuration
  • Create the indexes syslog and syslog_unclassified
  • Deploy input configuration for the default input
[monitor:///var/splunk-syslog/default/*.log]
host_regex = .*\/(.*)\.log
sourcetype = syslog
source = syslog_enterprise_default
index = syslog_unclassified
disabled = enabled

 

  • Validate the index contains data

 

When you have 100 problems, more logs are not the answer

big_fire_01 Often SIEM projects begin where log aggregation projects end. So many logs cut into organized stacks of wood ready to burn for value. I can be quoted on this “All logs can be presumed to have security value”. One project to build the worlds largest bonfire however is seldom the correct answer. What value you may ask? Value will be gained in one or more of these categories:

Continue reading “When you have 100 problems, more logs are not the answer”

Share that search! Building a content pack for Splunk Enterprise Security 4.0+

Splunk has initial support for export of “content” which can be dashboards and correlation searches created by the user to share with another team. What if you need to be a little more complex for example including a lookup generating search? This will get a little more complicated but very doable by the average admin. Our mission here is to implement UC0029. What is UC0029 glad you ask Each new malware signature detected should be reviewed by a security analyst to determine if proactive steps can be taken to prevent infection. We will create this as a notable event so that we can provide evidence to audit that the process exists and was followed.

Source code will be provided so I will not detail step by step how objects will be created and defined for this post

UC0029 Endpoint new malware detected by signature

 

My “brand” is SecKit so you will see this identifier in content I have created alone or with my team here at Splunk. As per our best practice adopt your own brands and use appropriately for your content. There is no technical reason to replace the “brand” on third party content you elect to utilize.

Note ensure all knowledge objects are exported as all app’s owned by admin as you go

      • Create a DA-ESS-SecKit-EndpointProtection
        • This will contain ES specific content such as menus dashboards, and correlation searches
      • Create the working app SA-SecKit-EndpointProtection
        • This will contain props transforms lookups and scheduled searches created outside of ES
      • Create the lookup seckit_endpoint_malware_tracker this lookup will contain each signature as it is detected in the environment and some handy information such as the endpoint first detected, user involved and the most recent detection.
      • Create empty lookup CSV files
        • seckit_endpoint_malware_tracker.csv (note you will not ship this file in your content pack)
        • seckit_endpoint_malware_tracker.csv.default

Build and test the saved search SecKit Malware Tracker – Lookup Gen. This search will use tstats to find the first and last instance of all signatures in a time window and update the lookup if an earlier or later instance is found

 

      Build and test the correlation search UC0029-S01-V001 New malware signature detected. This search will find “new” signatures from the lookup we have created and create a notable event”Make it default” In both apps move content from local/ to default/ this will allow your users to customize the content without replacing the existing searches”Turn if off by default” It is best practice to ensure any load generating searches are disabled by default

        add disabled=1 to each savedsearches.conf stanza that does not end in”- Rule”add disabled=1 to each correleationsearches.conf

Create a spl (tar.gz) containing both apps createdWrite a blog post explaining what you did, how the searches work and share the code!Gain fame and respect maybe a fez or a cape

The source code

https://bitbucket.org/rfaircloth-splunk/securitykit/src/1ea60c46b685622116e28e8f1660a6c63e7d9e96/base/ess/?at=master

Bonus: Delegate administration of content app

  1. Using your favorite editor edit app/metadata/local.meta
  2. Update the following permisions adding “ess_admin” role

## access = read : [ * ], write : [ admin,role2,role3 ]
[savedsearches]
access = read : [ * ], write : [ admin,ess_admin ]

[correlationsearches]
access = read : [ * ], write : [ admin,ess_admin ]

Splunk Universal Forwarder Version 6.2.3+ Ubuntu 15.04

Author: Ryan Faircloth

Summary: Using repositories for version managment of the Splunk Universal Forwarder assists in ensuring managed Ubuntu systems are using the approved version of the software at all times.

Setup the repository server

  1. Install reprepro and nginx


    sudo apt-get install reprepro nginx packaging-dev -y

  2. Create a user to work with the repository


    adduser --disabled-password --disabled-login --home /srv/reprepro --group reprepro

  3. Change user to our reprepro user all commands for the repository should be executed using this ID


    sudo su - reprepro

Generate GPG Keys

  1. Change user to our reprepro user all commands for the repository should be executed using this ID

    sudo su - reprepro 
    
  2. Create the default configuration for gpg by running the command

    gpg --list-keys

  3. Edit ~/.gnupg/gpg.conf
    • uncomment the line no-greeting
    • add the following content to the end of the file
    # Prioritize stronger algorithms for new keys.
    default-preference-list SHA512 SHA384 SHA256 SHA224 AES256 AES192 AES CAST5 BZIP2 ZLIB ZIP UNCOMPRESSED
    # Use a stronger digest than the default SHA1 for certifications.
    cert-digest-algo SHA512
    
  4. Generate a new key with the command gpg --gen-key

  5. Select the folowing options
    1. Type of key “(1) RSA and RSA (default)”
    2. Key size “4096”
    3. Expires “10y”
    4. Confirm “Y”
    5. Real Name “Splunk local repository”
    6. Email address on repository contact this generally should be an alias or distribution list
    7. Leave the comment blank
    8. Confirm and “O” to Okay
    9. Leave passphrase blank and confirm, a key will be generated not the sub KEY ID in the following example * E507D48E *


    gpg: checking the trustdb
    gpg: 3 marginal(s) needed, 1 complete(s) needed, PGP trust model
    gpg: depth: 0 valid: 1 signed: 0 trust: 0-, 0q, 0n, 0m, 0f, 1u
    gpg: next trustdb check due at 2025-05-24
    pub 4096R/410E1699 2015-05-27 [expires: 2025-05-24]
    Key fingerprint = 7CB8 81A9 E07F DA7B 83FF 2E1B 8B31 DA83 410E 1699
    uid Splunk local repository <repo@example.com>
    sub 4096R/E507D48E 2015-05-27 [expires: 2025-05-24]

  6. Export the signing keys public component save this content for use later


    gpg --export --armor KEY_ID >~/repo.pub

Configure Prerepro

  1. Change user to our reprepro user all commands for the repository should be executed using this ID sudo su - reprepro

  2. Create the directory structure sudo mkdir -p /srv/reprepro/ubuntu/{conf,dists,incoming,indices,logs,pool,project,tmp}

  3. Change directories to the new repository cd /srv/reprepro/ubuntu/

  4. Edit the file /srv/reprepro/ubuntu/conf/distributions

  5. Update the file contents

    Origin: SplunkEnterprise
    Label: SplunkEnterprise
    Codename: ponies
    Architectures: i386 amd64 source    
    Components: main
    Description: Splunk Enterprise and Universal Forwarders for Debian based systems
    SignWith: YOUR-KEY-ID 
    
  6. Edit the file /srv/reprepro/ubuntu/conf/options

  7. Update the file contents

    ask-passphrase
    basedir .
    

Load the packages

Load the packages using the following commands syntax replace package.deb with the correct path to the splunkforwarder deb file

reprepro -S utils -P standard includedeb ponies package.deb

Setup the web server

  1. Create the file /etc/nginx/sites-available/vhost-packages.conf

  2. Use the following content replacing package.local with the fqdn of the repository host
    server {
      listen 80;
      server_name packages.internal;
        
      access_log /var/log/nginx/packages-access.log;
      error_log /var/log/nginx/packages-error.log;
        
      location / {
        root /srv/reprepro;
        index index.html;
      }
        
      location ~ /(.*)/conf {
        deny all;
      }
        
      location ~ /(.*)/db {
        deny all;
      }
    }
    
  3. Increase the server name hash bucket by creating the following file /etc/nginx/conf.d/server_names_hash_bucket_size.conf

  4. Use the following content server_names_hash_bucket_size 64;

  5. Enable the new configuration

    sudo ln -s /etc/nginx/sites-available/vhost-packages.conf /etc/nginx/sites-enabled/vhost-packages.conf
    sudo service nginx reload
    

Configure the repository

  1. Edit the file
    /etc/apt/sources.list.d/packages.internal.list  
    
  2. Use the following content
    deb http://packages.internal/ubuntu/ ponies main
    
  3. Import the public key
    sudo apt-key add /tmp/repo.pub
    
  4. Update the repository cache
    sudo apt-get update 
    

Install the Splunk Universal Forwarder

Run the following command

sudo apt-get install splunkforwarder

Configure the universal forwarder

  • Using best practices to manually create the org_deploymentclient configuration app
  • Using RPM based configuration package
  • Using Configuration Managment system such as Puppet or Chef

Create and install a configuration package for the Universal Forwarder

In the following procedure “org” should be replace with the abbreviate of the organization using the configuration.

  1. Create the paths /srv/reprepro/org_debs/

  2. Create the path for the first version of the package ie mkdir org-splunk-ufconfig-1

  3. Change to the new directory

  4. Create the following structure

    ├── DEBIAN
    │   ├── control (file)
    │   ├── postinst (file)
    │   ├── preinst (file)
    │   └── prerm (file)
    └── opt
        └── splunkforwarder
            └── etc
                └── apps
                    └── org_all_deploymentclient
                        └── default
                            ├── deploymentclient.conf (file)
    
  5. Edit the DEBIAN/control file as follows


    Package: org-splunk-ufconfig
    Section: base
    Priority: standard
    Version: 1
    Architecture: all
    Maintainer: Your Name <you@email.com>
    Depends: splunkforwarder (>=6.0.0)
    Description: <insert up to 60 chars description>
    <insert long description, indented with spaces>

  6. Edit the DEBIAN/postinst

    #!/bin/bash
    /opt/splunkforwarder/bin/splunk enable boot-start -user splunk --accept-license --answer-yes
    service splunk start    
    
  7. Edit the DEBIAN/preinst
    #!/bin/bash
    file="/etc/init.d/splunk"
    if [ -f "$file" ]
    then
        echo "$file found."
        service splunk stop
    else
        echo "$file not found."
    fi
    
  8. Edit the DEBIAN/prerm
    #!/bin/bash
    file="/etc/init.d/splunk"
    if [ -f "$file" ]
    then
        echo "$file found."
        service splunk stop
        /opt/splunkforwarder/bin/splunk disable boot-start
    else
        echo "$file not found."
    fi
    
  9. Update the contents of deploymentclient.conf with the appropriate information for you installation

  10. Add additional content as required for your deployment

  11. Change directories up to the parent of org-splunk-ufconfig–1

  12. Create the debian package with the command dpkg-deb --build org-splunk-ufconfig-1/

  13. Change to the repository directory /srv/reprepro/ubuntu

  14. Store the new package in the repository

    reprepro -S utils -P standard includedeb ponies /srv/reprepro/org_debs/org-splunk-ufconfig-1.deb

  15. Install the new package on the client using the command sudo apt-get install org-splunk-ufconfig this will install the splunk forwarder package if has not yet been installed.

Splunk Universal Forwarder Version 6.2.3+ Microsoft System Center 2012 R2

Author: Ryan Faircloth

Summary: Rapid deployment of the universal forwarder in a production environment is possible with a minimal amount of risk for the customer. The installation of a universal forwarder can be performed at any time without impact to the production system and without reboot. A small caution is required in that if an existing MSI installation has created on reboot actions the installation of the Splunk universal forwarder or any other MSI may trigger a reboot by the SCCM client.

[TOC]

Overview

This guide will deploy the universal forwarder to all servers with a supported version of the Microsoft Windows Server operating system.

  • Create a new folder to contain Splunk related collections
  • Create one or more collection containing all systems which should receive the universal forwarder.
  • Create a collection containing all systems where any version of the universal forwarder -has been deployed
  • Create an application definition to deploy the universal forwarder without configuration
  • Create an application definition to deploy an upgrade to the universal forwarder without configuration
  • Create a package containing a powershell script to configure the universal forwarder
  • Deploy the configuration script using a task sequence

Prerequisite Steps

Task Responsible
Create CNAME for Deployment Server DNS Admin
Install Splunk Enterprise on Server Splunk Admin
Configure Splunk Instance as Deployment Server Splunk Admin

Step by Step

Create the deployment collection folder

  1. Navigate to Device Collections

  2. Right click

  3. Create new folder

  4. Name the new folder “Splunk Universal Forwarders”

  5. Navigate to the new folder

Create a collection for deployment

  1. Right click and choose "“Create New Device Collection”
  2. Name the collection “Splunk Deployment Collection for Servers”
  3. Select “All Desktop and Server Clients” as the limiting collection
    Create Device Collection
  4. Click Next
  5. Click Add to define the criteria used to determine which devices will receive the Universal Forwarder
  6. Click Query
  7. Name the Query “Server OS”
  8. Click Edit
  9. Click Show query language
  10. Enter the following query:
    sql
    select SMS_R_SYSTEM.ResourceID,
    SMS_R_SYSTEM.ResourceType,
    SMS_R_SYSTEM.Name,SMS_R_SYSTEM.SMSUniqueIdentifier,
    SMS_R_SYSTEM.ResourceDomainORWorkgroup,SMS_R_SYSTEM.Client
    from SMS_R_System
    inner join
    SMS_G_System_OPERATING_SYSTEM on SMS_G_System_OPERATING_SYSTEM.ResourceId = SMS_R_System.ResourceId
    where
    SMS_G_System_OPERATING_SYSTEM.ProductType = 2
    or SMS_G_System_OPERATING_SYSTEM.ProductType = 3
  11. Click OK
  12. Click OK again
  13. Enable Incremental Update by checking the box
  14. Click Next
  15. Click Next
  16. Click Close

> Note: the collection will contain zero members until the update collection background task completes

Create a collection of all successfully deployed universal forwarders

  1. Right click and choose “Create New Device Collection”
  2. Name the collection “Splunk Deployment Collection for Deployed Forwarders”

  3. Select “All Desktop and Server Clients” as the limiting collection

  4. Click Next

  5. Click Add to define the criteria used to determine which devices will receive the Universal Forwarder

  6. Click Query

  7. Name the Query “Server OS”

  8. Click Edit

  9. Click Show query language

  10. Enter the following query:

    sql
    Select
    SMS_R_SYSTEM.ResourceID,
    SMS_R_SYSTEM.ResourceType,
    SMS_R_SYSTEM.Name,
    SMS_R_SYSTEM.SMSUniqueIdentifier,
    SMS_R_SYSTEM.ResourceDomainORWorkgroup,
    SMS_R_SYSTEM.Client
    from SMS_R_System
    inner join
    SMS_G_System_ADD_REMOVE_PROGRAMS on SMS_G_System_ADD_REMOVE_PROGRAMS.ResourceID = SMS_R_System.ResourceId
    inner join
    SMS_G_System_INSTALLED_SOFTWARE on SMS_G_System_INSTALLED_SOFTWARE.ResourceID = SMS_R_System.ResourceId
    inner join
    SMS_G_System_ADD_REMOVE_PROGRAMS_64 on
    SMS_G_System_ADD_REMOVE_PROGRAMS_64.ResourceId = SMS_R_System.ResourceId
    where
    SMS_G_System_ADD_REMOVE_PROGRAMS.DisplayName = "UniversalForwarder"
    and SMS_G_System_ADD_REMOVE_PROGRAMS_64.DisplayName = "UniversalForwarder"
    or SMS_G_System_INSTALLED_SOFTWARE.ProductName = "UniversalForwarder"
    order by SMS_R_System.Name

  11. Click OK

  12. Click OK again

  13. Enable Incremental Update by checking the box

  14. Click Next

  15. Click Next

  16. Click Close

Note: the collection will contain zero members until the update collection background task completes

Create Application Definitions

Download both the 32bit and 64bit versions of the Splunk Universal Forwarder into the source folder structure used for SCCM deployment applications. Do this for all versions currently deployed as well as the new version to be deployed.

In general the locations are similar to the path:
\\servername\source\vendor\product\version\bitness
\\servername\source\Splunk\UniversalForwarder\6.2.3\x86

Create the application definition for the oldest deployed version of the Univeral Forwarder first.

  1. Navigate to Applications in the Software Library screen
  2. Right click and create a new folder for Splunk definitions
  3. Right click on the new folder and choose Create New Application
  4. Locate the 64 bit MSI for this product version
  5. Click Next
  6. Click Next again
  7. Update the definition with the following information
    • Name (Include version Number and bitness Version number i.e. Universal Forwarder 6.2.3 (x64)
    • Publisher
    • Version
    • Update the command line by removing “/q” and appending “/quiet AGREETOLICENSE=Yes”

      Note it is very important that /q is replaced by /quiet

  8. Click Next
  9. Click Next
  10. Click Close
  11. Right click on the new application definition and click properties
  12. Select the deployment type tab
  13. Select the first deployment and click edit
  14. Select the program tab
  15. update the uninstall command replacing /q with /quiet
  16. select the third browse next to product code and select the MSI
  17. Click requirements
  18. Click add
  19. Select category = device condition = operating system and provide the supported 64bit operating systems
  20. Create and additional requirements appropriate for your environment such as memory and disk space free
  21. Click OK
  22. Click OK again
  23. Add a new deployment type define the 32 bit MSI type using the information above
  24. Edit the new type using the information above to set the product MSI and verify requirements
  25. Select the supersedence tab
  26. click add
  27. Click Browse and select the oldest prior version of the application deployed to replace
  28. Map old deployment type to new ensuring the types match
  29. Click OK
  30. Add any other replacements required
  31. Verify your work and click OK

Repeat the application creation process for all versions of the UF in production If you are upgrading monitor your deployment progress You may continue with this procedure while the Universal Forwarder application is deployed.

Create a Configuration Script

  1. Create a source folder to contain the configuration script for example \\servername\source\splunk\scripts\UF_Config_V1
  2. The following script can be used as a template for the appropriate configuration for your site. At minimum the deployment server FQDN must be customized. Name the script configure.ps1
#Splunk Configuration Script for SCCM Task Sequence
#Locate Splunk based on the MSI registration

function Get-IniContent ($filePath)
{
$ini = @{}
$section="GLOBAL"
$CommentCount=0
switch -regex -file $FilePath
{
 
 "^\[(.+)\]" # Section
{
$section = $matches[1]
$ini[$section] = @{}
$CommentCount = 0
}
"^(\#.*)$" # Comment
{
$value = $matches[1]
$CommentCount = $CommentCount + 1
$name = "Comment" + $CommentCount
#$ini[$section][$name] = $value
}
"(.+?)\s*=(.*)" # Key
{
$name,$value = $matches[1..2]
$ini[$section][$name] = $value
}
}
return $ini
}

$location ="C:\Program Files\SplunkUniversalForwarder\"

#note if splunk may not be installed at the default location uncomment the following lines
#$list = Get-WmiOBject -Class Win32_Product | Where-Object {
# $_.Name -eq 'UniversalForwarder' -or $_.Name -eq 'Splunk' }

#$splunkprod = $list | where-Object { $_.InstallLocation }

#$location = $splunkprod.InstallLocation

$scriptappver = 2

$splunkcmd = $location + "bin\splunk.exe"
$staticapp = $location + "etc\apps\_static_all_universalforwarder\"
$staticdefault = $staticapp + "default\"
$staticlocal = $staticapp + "local\"

$staticdefault_dc = $staticdefault + "deploymentclient.conf"
$staticlocal_dc = $staticlocal + "deploymentclient.conf"
$staticdefault_app = $staticdefault + "app.conf"

if (!(Test-Path -Path $staticapp)) {new-item -ItemType Directory -Path $staticapp}

if (!(Test-Path -Path $staticdefault)) {new-item -ItemType Directory -Path $staticdefault}

if (!(Test-Path -Path $staticlocal)) {new-item -ItemType Directory -Path $staticlocal}

if (!(Test-Path -Path $staticdefault_app))
{
 new-item -path $staticdefault_app -ItemType File
 Add-Content -Path $staticdefault_app -Value "#Generated by scripting"
 #Add-Content -Path $staticdefault_app -Value "`r`n"
 Add-Content -Path $staticdefault_app -Value "[_static_all_universalforwarder]"
 Add-Content -Path $staticdefault_app -Value "author=Ryan Faircloth"
 Add-Content -Path $staticdefault_app -Value "description=Script Generated UF default configuration applied by SCCM"
 Add-Content -Path $staticdefault_app -Value "version=1"
 Add-Content -Path $staticdefault_app -Value "[ui]"
 Add-Content -Path $staticdefault_app -Value "is_visible = false"
}

$appconf = Get-IniContent $staticdefault_app
$appver = $appconf[“_static_all_universalforwarder”][“version”]

if ($appver -ne $scriptappver)
{
if (!(Test-Path -Path $staticdefault_dc))
{
 new-item -path $staticdefault_dc -ItemType File
 Add-Content -Path $staticdefault_dc -Value "#Generated by scripting"
 Add-Content -Path $staticdefault_dc -Value "[deployment-client]"
 Add-Content -Path $staticdefault_dc -Value "clientName=ScriptDeployed|"
 Add-Content -Path $staticdefault_dc -Value "[target-broker:deploymentServer]"
 Add-Content -Path $staticdefault_dc -Value "targetUri=srvsplunk.ad.domainname.com:8089"
 Add-Content -Path $staticdefault_dc -Value ""

}

& $splunkcmd "restart"
}

Create a Package to contain the configuration script

  1. Create a new package folder Splunk
  2. Create a new folder on a network share Splunk_config_vx where X is the version of the script and include a customized version of the config script provided
  3. Right click on the package folder create package
  4. Name the package Splunk Configuration Script v1
  5. Select the source folder
  6. Click Next
  7. Click do not create a program
  8. Click next
  9. Click next
  10. Click Close
  11. Right click on the package and click “Distribute Content” using appropriate options for the environment. Do not click deploy
  12. Create the Task Sequence
  13. Crea a new Task Sequence Folder “Splunk”
  14. Right click the Task Sequence Folder Create Task Sequence
  15. Name the task Splunk Config Vx
  16. Click Next
  17. Click Next
  18. Click Close
  19. Right click on the task sequence
  20. Click properties
  21. Click the advance tab
  22. Select suppress task sequence notifications and disable this task sequence on computers where it is deployed
  23. Right click on the task sequence and choose edit
  24. Click Add General —> powershell script
  25. Set the script name i.e. configure.ps1 and execution policy=bypass
  26. Click OK
  27. Right click on the task and deploy to the deployed collection created second above

Create the configuration task sequence

  1. Navigate to Software Library
  2. Navigate to Operating System Deployment
  3. Navigate to Task Sequence
  4. Optional Create a new folder called Splunk
  5. Right click and Create a new task sequence
  6. Select Custom Sequence
  7. Click Next
  8. Name the sequence i.e. Splunk Configuration Script Vx
  9. Click Next
  10. Click Next
  11. Click Close
  12. Right click on the task sequence
  13. Click properties
  14. Click the advanced tab
    • Select suppress task sequence notifications
    • disable this task sequence on computers where it is deployed
  15. Click Ok
  16. Right click on the task sequence and choose edit
    • Click Add General —> powershell script
    • Set the script name and execution policy=bypass
  17. Click OK
  18. Right click on the task and deploy to the deployed collection created second above