Ansible - Splunk Forwarder Deployment - Pt.1
This post will help you get up and running with Ansible with the end goal of deploying Splunk universal forwarders to both Windows and Linux.
Inspiration #
.conf 2015 talk - Splunk Configuration Management and Deployment with Ansible - Jose Hernandez / Sean Delaney
Prerequisites #
Ansible
Follow installation guidelines here:
Install dependencies for connecting to windows hosts
Linux Specific
- User who can ssh
- User who can sudo
Windows Specific
- Domain account who can access server via winrm, and install software
- A share location that your servers can pull down files from (using your domain account)
-
HTTP/AllowUnencrypted=True
winrm set winrm/config/service ‘@{AllowUnencrypted=“true”}’
or
- HTTPS/AllowUnencrypted=False
Install Environment #
File Structure
/etc/ansible/
ansible.cfg (ansible specific settings)
hosts (where you can store groups of hosts for ease of management)
group_vars/
windows.yml
playbooks/ (where you should store .yml playbooks)
SplunkUniversalForwarderInstallWindows.yml
SplunkUniversalForwarderInstallLinux.yml
splunk_binaries/ (where you should store installers)
splunkforwarder-6.3.0-aa7d4b1ccb80-x64-release.msi
splunkforwarder-6.3.0-aa7d4b1ccb80-linux-2.6-x86_64.rpm
roles/ (where you can group tasks by Linux/Windows)
universal_forwarder_linux/
tasks/
main.yml
forwarder.yml
universal_forwarder_windows/
files/
install-splunk.ps1
tasks/
main.yml
forwarder.yml
ansible.cfg - example
Set specifics depending on your environment. These are some of my settings.
[defaults]
inventory = /etc/ansible/hosts
remote_tmp = /tmp/hortonew
forks = 50
poll_interval = 5
sudo_user = root
host_key_checking = False
sudo_exe = sudo
timeout = 10
hosts - example
Here you can group hosts however you’d like. When you go to run an Ansible command, you’ll specify the group name. You can also target individual hosts, but that host must be included in this file.
[install-splunk-windows]
some-host-name-windows.at.my.domain.net
10.10.10.10
[install-splunk-linux]
192.168.8.4
some-host-name-linux.at.my.domain.net
WINDOWS #
Kerberos Config - /etc/krb5.conf #
[libdefaults]
default_realm = SOMETHING.YOURDOMAIN.COM
[realms]
SOMETHING.YOURDOMAIN.COM = {
kdc = your-kdc-server.something.yourdomain.com
default_domain = something.yourdomain.com
kpasswd_server = your-kdc-server.something.yourdomain.com
}
SOMETHING2.YOUROTHERDOMAIN2.COM = {
kdc = your-kdc-server2.something2.yourotherdomain2.com
default_domain = something2.yourotherdomain2.com
kpasswd_server = your-kdc-server2.something2.yourotherdomain2.com
}
SOMETHING3.YOUROTHERDOMAIN3.COM = {
kdc = your-kdc-server3.something3.yourotherdomain3.com
default_domain = something3.yourotherdomain3.com
kpasswd_server = your-kdc-server3.something3.yourotherdomain3.com
}
[domain_realm]
.something.yourdomain.com = SOMETHING.YOURDOMAIN.COM
.something2.yourotherdomain2.com = SOMETHING2.YOUROTHERDOMAIN2.COM
.something3.yourotherdomain3.com = SOMETHING3.YOUROTHERDOMAIN3.COM
Generate Kerberos Ticket #
kinit my.adminuser@SOMETHING.YOURDOMAIN.COM
Note: To see your tickets, you can use klist. To clear your tickets, you can use kdestroy.
group_vars/windows.yml
Settings to be used connecting to windows hosts via kerberos authentication
ansible_ssh_user: my.adminuser@SOMETHING.YOURDOMAIN.COM
ansible_ssh_pass:
ansible_ssh_port: 5985
ansible_connection: winrm
playbooks/SplunkUniversalForwarderInstallWindows.yml - example
Here you tell Ansible which hosts you want to target, which variables to pass through to a role, and which roles to target
---
- name: Windows x64 Universal Forwarder Install
hosts: install-splunk-windows
vars:
splunk_working_directory: 'c:\Temp\Splunk\'
splunk_remote_file_share: '\\server-name.yourdomain.com\c$\Distribute\'
splunk_site: 'CORPORATE'
splunk_install_path: 'c:\Program Files\SplunkUniversalForwarder\'
splunk_deployment_server: 'your-deployment-server.yourdomain.com'
splunk_deployment_server_port: '8089'
splunk_user: 'admin'
splunk_password: 'changeme'
splunk_uf_binary: 'splunkforwarder-6.3.0-aa7d4b1ccb80-x64-release.msi'
roles:
- universal_forwarder_windows
roles/universal_forwarder_windows/tasks/main.yml
You can include multiple files here. For this example, there is only one template we want to call.
---
- include: forwarder.yml
roles/universal_forwarder_windows/tasks/forwarder.yml
Here is where all the install actions happen for windows machines
---
- name: Checking if splunk is installed
tags: install
win_stat: path='{{splunk_install_path}}etc\splunk.version'
register: splunk_path
- name: splunk is installed
tags: install
debug: msg='splunk is already installed under {{splunk_install_path}}.'
when: splunk_path.stat.exists
- name: Assures {{splunk_working_directory}} exists
tags: install
win_file: path={{splunk_working_directory}} state=directory
when: splunk_path.stat.exists == False
- name: Checking if installer already copied over
tags: install
win_stat: path={{splunk_working_directory}}{{splunk_uf_binary}}
when: splunk_path.stat.exists == False
register: splunk_installer
- name: Installer on server
tags: install
debug: msg='installer already copied over'
when: (splunk_path.stat.exists == False) and (splunk_installer.stat.exists)
- name: Push package to server
tags: install
win_get_url:
url: '{{splunk_remote_file_share}}{{splunk_uf_binary}}'
dest: '{{splunk_working_directory}}{{splunk_uf_binary}}'
when: (splunk_path.stat.exists == False) and (splunk_installer.stat.exists == False)
- name: Install Splunk
tags: install
script: install-splunk.ps1 -site '{{splunk_site}}' -installer '{{splunk_working_directory}}{{splunk_uf_binary}}'
when: splunk_path.stat.exists == False
roles/universal_forwarder_windows/files/install-splunk.ps1
This is a custom script made for our environment. Modify as needed.
param (
[string]$site,
[string]$installer = "c:\Temp\Splunk\splunkforwarder-6.3.0-aa7d4b1ccb80-x64-release.msi",
[string]$log = "c:\Temp\Splunk\splunkinstall.log"
)
$splunk_install_file = $installer
switch ($site.ToUpper()) {
"CORPORATE" {$splunk_deployment_server = "10.10.10.10:8089"}
"SITE1" {$splunk_deployment_server = "10.10.10.11:8089"}
"SITE2" {$splunk_deployment_server = "10.10.10.12:8089"}
"SITE3" {$splunk_deployment_server = "10.10.10.13:8089"}
default {"-site parameter missing. Site list: corporate, site1, site2, site3"; exit}
}
& msiexec.exe /qn /Liwem! $log /i $splunk_install_file AGREETOLICENSE=Yes DEPLOYMENT_SERVER=`"$splunk_deployment_server`" INSTALL_SHORTCUT=0 /quiet
LINUX #
playbooks/SplunkUniversalForwarderInstallLinux.yml - example
---
- name: Linux Universal Forwarder Install
hosts: install-splunk-linux
remote_user: your-user-here
sudo: yes
vars:
splunk_working_directory: '/tmp/Splunk/'
splunk_deployment_server: 'your-deployment-server.yourdomain.com'
splunk_deployment_server_port: '8089'
splunk_user: 'admin'
splunk_password: 'changeme'
splunk_uf_binary_linux: 'splunkforwarder-6.3.0-aa7d4b1ccb80-linux-2.6-x86_64.rpm'
roles:
- universal_forwarder_linux
roles/universal_forwarder_linux/tasks/main.yml
You can include multiple files here. For this example, there is only one template we want to call.
---
- include: forwarder.yml
roles/universal_forwarder_linux/tasks/forwarder.yml
Here is where all the install actions happen for linux machines
---
- name: Checking if splunk is installed
tags: install
stat: path=/opt/splunkforwarder
register: splunk_path
- name: Checking if installer already copied over
tags: install
stat: path={{splunk_working_directory}}{{splunk_uf_binary_linux}}
when: splunk_path.stat.exists == false
register: splunk_installer
- name: splunk is installed
tags: install
debug: msg='splunk is already installed under /opt/splunkforwarder'
when: splunk_path.stat.exists == true
- name: Checking if deployment server set
tags: install
stat: path=/opt/splunkforwarder/etc/system/local/deploymentclient.conf
register: splunkds
- name: Splunk deployment server set
tags: install
debug: msg='Splunk Deployment Server is set'
when: splunkds.stat.exists == true
- name: Assures {{splunk_working_directory}} exists
tags: install
file: path={{splunk_working_directory}} state=directory
when: splunk_path.stat.exists == false
- name: Push RPM package to server
tags: install
become: yes
become_method: sudo
copy: src=splunk_binaries/{{splunk_uf_binary_linux}} dest={{splunk_working_directory}}{{splunk_uf_binary_linux}} owner=hortone group=hortone mode=0644
when: splunk_path.stat.exists == false and splunk_installer.stat.exists == false
- name: Install RPM
tags: install
become: yes
become_method: sudo
command: rpm -ivh {{splunk_working_directory}}{{splunk_uf_binary_linux}}
when: splunk_path.stat.exists == false
- name: CHOWN Directory
tags: install
become: yes
become_method: sudo
shell: chown -R splunk:splunk /opt/splunkforwarder
when: splunk_path.stat.exists == false
- name: Start splunk
tags: install
become_user: splunk
become_method: sudo
shell: /opt/splunkforwarder/bin/splunk start --accept-license
when: splunk_path.stat.exists == false
- name: Enable boot-start
tags: install
become: yes
become_method: sudo
shell: /opt/splunkforwarder/bin/splunk enable boot-start -user splunk
when: splunk_path.stat.exists == false
- name: Splunk set deploy-poll and Restart splunkd
tags: install
become_user: splunk
become_method: sudo
shell: /opt/splunkforwarder/bin/splunk set deploy-poll {{splunk_deployment_server}}:{{splunk_deployment_server_port}} -auth {{splunk_user}}:{{splunk_password}} & /opt/splunkforwarder/bin/splunk restart
when: splunkds.stat.exists == false
- name: Check if DS App for deploymentclient.conf has been pulled
tags: install
become_user: splunk
become_method: sudo
stat: path=/opt/splunkforwarder/etc/apps/deploymentclient_two/local/deploymentclient.conf
register: splunk_ds_settings
- name: Delete deployment client settings in etc/system/local
tags: install
become_user: splunk
become_method: sudo
shell: rm -f /opt/splunkforwarder/etc/system/local/deploymentclient.conf & /opt/splunkforwarder/bin/splunk restart
when: splunk_ds_settings.stat.exists == true