I’ve updated the SecKit templates and guidance for Windows TA 6.0 no longer do you need to also deploy the TA for Microsoft DNS and TA for Microsoft AD!.
Check it out https://seckit.readthedocs.io/projects/splunk-ta-windows-seckit/en/latest/
I’ve updated the SecKit templates and guidance for Windows TA 6.0 no longer do you need to also deploy the TA for Microsoft DNS and TA for Microsoft AD!.
Check it out https://seckit.readthedocs.io/projects/splunk-ta-windows-seckit/en/latest/
Sometimes it is not noticed because there is no license charge associated with Splunk’s Universal forwarder internal logs and in some cases heavy forwarders. In very large deployments this can be a significant portion of storage used per day. Do you really need to keep those events around as long as the events associated with the Splunk Enterprise instances probably not.
It has been pointed out this change WILL impact license on recent versions of Splunk in older versions and customers with EAA agreements in place this is OK. If you are on a recent (not sure which version) this change will impact license.
The following changes will disable the Splunk Monitoring consoles built in forwarder monitoring feature. You can customize the searches but be aware this is not upgrade safe.
If you have any custom forwarder monitoring searches/dashboards/alerts they may be impacted.
The index we need to define is _internal_forwarder the following sample configuration will allow us to keep about 3 days of data from our forwarders adjust according to need.
[_internal_forwarder]
maxWarmDBCount = 200
frozenTimePeriodInSecs = 259200
quarantinePastSecs = 459200
homePath = $SPLUNK_DB/$_index_name/db
coldPath = $SPLUNK_DB/$_index_name/colddb
thawedPath = $SPLUNK_DB/$_index_name/thaweddb
maxHotSpanSecs = 43200
maxHotBuckets = 10
We need to create a new “TA” named “Splunk_TA_splunkforwarder we will CAREFULLY use the DS to push this to forwarders only. DO NOT push this to any Splunk Enterprise instance (CM/LM/MC/SH/IDX/deployer/ds) but you may push this to a “heavy” or intermediate forwarder. The app only needs two files in default app.conf and inputs.conf
#app.conf
[install]
state_change_requires_restart = true
is_configured = 0
state = enabled
build = 2
[launcher]
author = Ryan Faircloth
version = 1.0.0
[ui]
is_visible = 0
label = Splunk_UF Inputs
[package]
id = Splunk_TA_splunkforwarder
#inputs.conf
[monitor://$SPLUNK_HOME/var/log/splunk]
index = _internal_forwarder
First lets check positive make sure UFs have moved to the new index, we should get results.
index=_internal_forwarder source=*splunkforwarder*
Second lets check the negative make sure only UF logs got moved we should get no results
index= _internal_forwarder source=*splunk* NOT source=*splunkforwarder*
Splunk released a major update to the Splunk TA for Windows last month you may not have noticed but I think you should take a closer look. A few key things
If you are a SecKit for Windows user it is safe to upgrade just follow Splunk’s upgrade instructions. Need some guidance on good practices for Windows data on-boarding to Splunk be sure to checkout SecKit
While this is not a replacement for the upgrade notes you are probably wondering how will this impact my users.
Splunk’s SmartStore technology is a game changing advancement in data retention for Splunk Enterprise. Allowing Splunk to move least used data to an AWS for low cost “colder storage”.
We will review indexes.conf on the indexer and identify any references to the setting maxDataSize. Common historical practice has been to increase the size of this setting from the default of auto to an arbitrary large value or auto_high_volume. SmartStore is optimized and enforces the use of “auto” or 750mb as the max bucket size. This task should be completed at least 7 days prior cutover to SmartStore.
We will review indexes.conf and identify all indexes which continuously stream data. Common historical practice to leave this as default value which are very wide this increases the likely a user will retrieve buckets from S3 that do not actually meet their needs. We will determine a value of maxHotSpanSecs that will SmartStore to uncache buckets not used while also keeping buckets available likely to be used. Often 1 day (86400s) is appropriate.
Certain oversights in onboard data into Splunk impact both use-ability of data and performance review and resolve any issues identified by the Splunk Monitoring Console page Data Quality the most important indicators of concern are
_indextime - _time
)One common source of “latency” is events from offline endpoints such as windows laptops. Any endpoint that can spool locally for an undetermined period of time then forward old events should be routed to a index not used for normal streaming events. For example “oswinsec” is the normal index I use for Windows Security Events however for endpoint monitoring I use “oswinsecep”.
After the above activities are done, wait an hour before beginning this work. We should identify pre-mature bucket roll behavior that is buckets rolled from hot to warm regularly for not great reasons. The following search
index=_internal source="/opt/splunk/var/log/splunk/splunkd.log" component=HotDBManager evicting_count="*" | stats max(maxHotBuckets) values(count) as reason count by idx | sort -count
This search identifies buckets which are “high volume” and rolling due to lack of an available bucket to index a new event in correct relative order. For each index where the maxHotBuckets is less than 10 increase the value of maxHotBuckets in indexes.conf to no more than 10. For these indexes 10 is a safe value.