Dev Life: Splunk Add-ons like a developer

As a life long (seems that way) software developer come to Splunk I would like to have some of the properties of a Integrated Development Environment (IDE). This blog post walks you through setting up and experiencing my approach to development for Splunk I wrote a second post in this series creating an actual add-on for Splunk using this toolchain.

  • I can edit “code” i.e splunk conf in my editor and reload the code without restarting
  • Every time I build/debug I have a clean environment
  • I can run unit tests manually or automatically in a consistent way.
  • I can participate in VCS (i.e. git) if desired
  • I can consistently reproduce build and packaging including integration into a CI/CD process
  • I can leverage dependencies from other developed products.
  • Have ready access to common tools like add-on builder and eventgen

Setting up the environment Mac OSX

  • Install Brew
  • Install LibMagic “brew install libmagic”
  • Install python “brew install python”
  • Install pandoc “brew install pandoc”
  • Install moreutills “brew install moreutils”
  • Install jq “brew install jq”
  • Install lxml support “xcode-select –install”
  • Install git “brew install git”
  • Install git flow “brew install git-flow”
  • Install gitversion “brew install gitversion”
  • Install virtual env for python “sudo pip install virtualenv”
  • Install docker
  • Create the virtual env “virtualenv ~/venv/splservices”
  • Activate the new env “source ~/venv/splservices/bin/activate”
  • Install pip “sudo python easy_install pip”
  • Install our specific requirements “pip install -r”
  • I personally prefer the atom editor

Setup the local project

For demonstration purposes we are going to work with one of my recent add-ons for Splunk. A full tutorial on git is beyond the scope of this article we will simply clone the repo and start a feature branch.

  • Clone the repo “git clone”
  • Cd into the repo “cd ta-cef-for-splunk”
  • Initials git submodules “git submodule init”
  • Setup git flow “git flow init -d”
  • Start a new feature “git flow feature start myfeature”

Package and Test

Before we change anything we should verify we can recreate a successful build.

  • Build a package “make package”
  • Verify the package builds the last line will report something like this, path and version will vary.
slim package: [NOTE] Source package exported to "/Users/user/Downloads/ta-cef-for-splunk/out/packages/splunkbase/TA-cef-for-splunk-0.2.0-myfeature.1+17.tar.gz"
  • Test the package using Splunk’s appinspect “make package_test”
  • Verify the test report shows one failure. While developing this one failure is expected which is the version number does not conform to release rules for Splunk Base. Note: per the feature branch version clearly indicates this is a development build this is helpful in preventing accidental “escapes” to production
splunk-appinspect inspect out/packages/splunkbase/TA-cef-for-splunk-0.2.0-myfeature.1+17.tar.gz --data-format junitxml --output-file test-reports/TA-cef-for-splunk.xml --excluded-tags manual
Validating: TA-cef-for-splunk Version: 0.2.0-myfeature.1+17

A default value of 25 for max-messages will be used.
Splunk app packaging standards These checks validate that a Splunk app has been 
correctly packaged, and can be provided safely for package validation. 
    Check that the extracted Splunk App contains a default/app.conf file 
    that contains an [id] or [launcher] stanza with a version property that is 
    formatted as Major.Minor.Revision. 
        FAILURE: `Major.Minor.Revision` version numbering is required. 
            File: default/app.conf Line Number: 20 

TA-cef-for-splunk Report Summary:

       skipped: 176
       success:  9
  manual_check:  0
       failure:  1
       warning:  0
         error:  0
not_applicable:  3
         Total: 189

Please note that more issues could be found out later during the optional manual review process.

Interactive Development

Now for the good stuff how can we interactively lets fire up a Splunk Docker container with the latest version of Splunk and our local copy of the addon. “make docker_dev” wait for the text “Ansible playbook complete” to appear on terminal indicating Splunk is ready to work. Visit “” and login to a fresh copy of Splunk with the addon ready to go. The password will be “Changed!11” lets prove life by making a simple change to our addon.

  • Open atom or the editor of your choice
  • Navigate to <project>/src/TA-cef-for-splunk/default/props.conf
  • Add the “EVAL-alive=”yes”” to the [cef] stanza
  • Return to the running copy of Splunk and visit (click refresh)
  • Turn on the event gen “Settings –>Data Inputs –> SA Event-Gen then click enable
  • Wait about and minute and click disable
  • Go back to search and check for the alive field “index=* sourcetype=cef | head | table sourcetype,alive”

Further reading