Preparation of a base infrastructure for high availability ingestion of syslog data with a default virtual server and configuration for test data on boarding. Reference technology specific on boarding procedures.
Requirement
Multiple critical log sources require a reliable syslog infrastructure. The following attributes must be present for the solution
Enterprise supported linux such as RHEL, OR Centos
Syslog configuration which will not impact the logging of the host on which syslog is configured
External Load Balancing utilizing DNAT lacking available enterprise shared services NLB devices KEMP offers a free to use version of their product up to 20 Mbs suitable for many cases
Technical Environment
The following systems will be created utilizing physical or virtual systems. System specifications will vary due estimated load.
Centos 7.x (current) servers in n+1 configuration
Minimum 2 GB memory
Minimum 2 x 2.3 GHZ core
Mounts configure per enterprise standard with the following additions
/opt/splunk 40 GB XFS
/var/splunk-syslog 40 GB XFS
Dual interfaced load balancer configured for DNAT support.
Subnet with at minimum the number of unique syslog sources (technologies) additional space for growth is strongly advised
Subnet allocated for syslog servers
Solution Prepare the syslog-ng servers
The following procedure will be utilized to prepare the syslog-ng servers
Install the base operating system and harden according to enterprise standards
Provision and mount the application partitions /opt/splunk and /var/splunk-syslog according the estimates required for your environment.
Note 1 typical configuration utilize noatime on both mounts
Note 2 typical configuration utilizes no execute on the syslog moun
Enable the EPEL repository for RHEL/CENTOS as the source for syslog-ng installation
Create the Splunk master syslog-configuration /etc/syslog-ng/conf.d/splunk.conf
################################################################################
# SecKit syslog template based on the work of Vladimir
# Template from https://github.com/hire-vladimir/SA-syslog_collection/
################################################################################
################################################################################
#### Global config ####
options {
create-dirs(yes);
# Specific file/directory permissions can be set
# this is particularly needed, if Splunk UF is running as non-root
owner("splunk");
group("splunk");
dir-owner("splunk");
dir-group("splunk");
dir-perm(0755);
perm(0755);
time-reopen(10);
keep-hostname(yes);
log-msg-size(65536);
};
@include "/etc/syslog-ng/conf.d/splunk-0-source/*.conf"
@include "/etc/syslog-ng/conf.d/splunk-1-dest/*.conf"
@include "/etc/syslog-ng/conf.d/splunk-2-filter/*.conf"
@include "/etc/syslog-ng/conf.d/splunk-3-log/*.conf"
@include "/etc/syslog-ng/conf.d/splunk-4-simple/*.conf"
Create the catch all syslog collection source. /etc/syslog-ng/conf.d/splunk-4-simple/8100-default.conf
################################################################################
#### Enable listeners ####
source remote8100_default
{
udp(port(8100));
tcp(port(8100));
};
#### Log remote sources classification ####
destination d_default_syslog {
file("/var/splunk-syslog/default/$HOST.log");
};
# catch all, all data that did not meet above criteria will end up here
log {
source(remote8100_default);
destination(d_default_syslog);
flags(fallback);
};
Ensure splunk can read from the syslog folders. The paths should exist at this point due to the dedicated mount
semanage port -a -t syslogd_port_t -p tcp 8100
semanage port -a -t syslogd_port_t -p udp 8100
semanage fcontext -a -t var_log_t /var/splunk-syslog
restorecon -v '/var/splunk-syslog'
logger -d -P 8100 -n 127.0.0.1 -p 1 "test2"
cd /root
mkdir selinux
cd selinux
audit2allow -M syslog-ng-modified -l -i /var/log/audit/audit.log
#verify the file does not contain anything no related to syslog
vim syslog-ng-modified.te
semodule -i syslog-ng-modified.pp
Deploy virtual load balancer to hypervisor with two virtual interfaces
#1 Enterprise LAN
#2 Private network for front end of syslog servers
Login to the load balancer web UI
Apply free or purchased license
Navigate to network setup
Set eth0 external ip
Set eth1 internal ip
Add the first virtual server (udp)
Navigate to Virtual Services –> Add New
set the virtual address
set port 514
set port name syslog-default-8100-udp
set protocol udp
Click Add this virtual service
Adjust virtual service settings
Force Layer 7
Transparency
set persistence mode source ip
set persistence time 6 min
set scheduling method lest connected
Use Server Address for NAT
Click Add new real server
Enter IP of syslog server 1
Enter port 8100
Add the first virtual server (tcp)
Navigate to Virtual Services –> Add New
set the virtual address
set port 514
set port name syslog-default-8100-tcp
set protocol tcp
Click Add this virtual service
Adjust virtual service settings
Service type Log Insight
Transparency
set scheduling method lest connected
TCP Connection only check port 8100
Click Add new real server
Enter IP of syslog server 1
Enter port 8100
Repeat the add virtual server process for additional resource servers
Update syslog server routing configuration
Update the default gateway of the syslog servers to utilize the NLB internal interface
Validation procedure
from a linux host utilize the following commands to validate the NLB and log servers are working together
logger -P 514 -T -n <vip_ip> "test TCP"
logger -P 514 -d -n <vip_ip> "test UDP"
verify the messages are logged in /var/splunk-syslog/default
Prepare Splunk Infrastructure for syslog
Follow procedure for deployment of the Universal Forwarder with deployment client ensure the client has has valid outputs and base configuration