Preparation of a base infrastructure for high availability ingestion of syslog data with a default virtual server and configuration for test data on boarding. Reference technology specific on boarding procedures.
Requirement
Multiple critical log sources require a reliable syslog infrastructure. The following attributes must be present for the solution
Enterprise supported linux such as RHEL, OR Centos, or recent Ubuntu LTS
Syslog configuration which will not impact the logging of the host on which syslog is configured
External Load Balancing utilizing DNAT lacking available enterprise shared services NLB devices KEMP offers a free to use version of their product up to 20 Mbs suitable for many cases
Technical Environment
The following systems will be created utilizing physical or virtual systems. System specifications will vary due estimated load.
servers in n+1 configuration
Minimum 2 GB memory
Minimum 2 x 2.3 GHZ core
Mounts configure per enterprise standard with the following additions
/opt/splunk 40 GB XFS
/var/splunk-syslog 40 GB XFS
Dual interfaced load balancer configured for DNAT support.
Subnet with at minimum the number of unique syslog sources (technologies) additional space for growth is strongly advised
Subnet allocated for syslog servers
Solution Prepare the rsyslogd servers
The following procedure will be utilized to prepare the rsyslogd servers
Install the base operating system and harden according to enterprise standards
Provision and mount the application partitions /opt/splunk and /var/splunk-syslog according the estimates required for your environment.
Note 1 typical configuration utilize noatime on both mounts
Note 2 typical configuration utilizes no execute on the syslog mount
Create the following directories for modular configuration of rsyslogd
Create the Splunk master syslog-configuration /etc/rsyslog.d/splunk.conf
#
# Include all config files for splunk /etc/rsyslog.d/
#
$IncludeConfig /etc/rsyslog.d/splunk-0-rules/*.conf
$IncludeConfig /etc/rsyslog.d/splunk-1-inputs/*.conf
Create the catch all syslog collection source. /etc/rsyslog.d/splunk-1-inputs/default.conf
Deploy virtual load balancer to hypervisor with two virtual interfaces
#1 Enterprise LAN
#2 Private network for front end of syslog servers
Login to the load balancer web UI
Apply free or purchased license
Navigate to network setup
Set eth0 external ip
Set eth1 internal ip
Add the first virtual server (udp)
Navigate to Virtual Services –> Add New
set the virtual address
set port 514
set port name syslog-default-8100-udp
set protocol udp
Click Add this virtual service
Adjust virtual service settings
Force Layer 7
Transparency
set persistence mode source ip
set persistence time 6 min
set scheduling method lest connected
Use Server Address for NAT
Click Add new real server
Enter IP of syslog server 1
Enter port 8100
Add the first virtual server (tcp)
Navigate to Virtual Services –> Add New
set the virtual address
set port 514
set port name syslog-default-8100-tcp
set protocol tcp
Click Add this virtual service
Adjust virtual service settings
Service type Log Insight
Transparency
set scheduling method lest connected
TCP Connection only check port 8100
Click Add new real server
Enter IP of syslog server 1
Enter port 8100
Repeat the add virtual server process for additional resource servers
Update syslog server routing configuration
Update the default gateway of the syslog servers to utilize the NLB internal interface
Validation procedure
from a linux host utilize the following commands to validate the NLB and log servers are working together
logger -P 514 -T -n <vip_ip> "test TCP"
logger -P 514 -d -n <vip_ip> "test UDP"
verify the messages are logged in /var/splunk-syslog/default
Prepare Splunk Infrastructure for syslog
Follow procedure for deployment of the Universal Forwarder with deployment client ensure the client has has valid outputs and base configuration