Ansible - Splunk Forwarder Deployment - Pt.1

This post will help you get up and running with Ansible with the end goal of deploying Splunk universal forwarders to both Windows and Linux.

 Inspiration

https://github.com/divious1

.conf 2015 talk - Splunk Configuration Management and Deployment with Ansible - Jose Hernandez / Sean Delaney

 Prerequisites

Ansible

Follow installation guidelines here:

http://docs.ansible.com/ansible/intro_installation.html

Install dependencies for connecting to windows hosts

http://docs.ansible.com/ansible/intro_windows.html

Linux Specific

Windows Specific

or

 Install Environment

File Structure

/etc/ansible/
 ansible.cfg  (ansible specific settings)
 hosts  (where you can store groups of hosts for ease of management)
 group_vars/
   windows.yml
 playbooks/  (where you should store .yml playbooks)
   SplunkUniversalForwarderInstallWindows.yml
   SplunkUniversalForwarderInstallLinux.yml

   splunk_binaries/  (where you should store installers)
     splunkforwarder-6.3.0-aa7d4b1ccb80-x64-release.msi
     splunkforwarder-6.3.0-aa7d4b1ccb80-linux-2.6-x86_64.rpm

  roles/  (where you can group tasks by Linux/Windows)
    universal_forwarder_linux/
      tasks/
        main.yml
        forwarder.yml
    universal_forwarder_windows/
      files/
        install-splunk.ps1
      tasks/
        main.yml
        forwarder.yml

ansible.cfg - example

Set specifics depending on your environment. These are some of my settings.

[defaults]
inventory      = /etc/ansible/hosts
remote_tmp     = /tmp/hortonew
forks          = 50
poll_interval  = 5
sudo_user      = root
host_key_checking = False
sudo_exe = sudo
timeout = 10

hosts - example

Here you can group hosts however you’d like. When you go to run an Ansible command, you’ll specify the group name. You can also target individual hosts, but that host must be included in this file.

[install-splunk-windows]
some-host-name-windows.at.my.domain.net
10.10.10.10

[install-splunk-linux]
192.168.8.4
some-host-name-linux.at.my.domain.net

 WINDOWS

 Kerberos Config - /etc/krb5.conf

[libdefaults]
default_realm = SOMETHING.YOURDOMAIN.COM

[realms]
  SOMETHING.YOURDOMAIN.COM = {
  kdc = your-kdc-server.something.yourdomain.com
  default_domain = something.yourdomain.com
  kpasswd_server = your-kdc-server.something.yourdomain.com
}
  SOMETHING2.YOUROTHERDOMAIN2.COM = {
  kdc = your-kdc-server2.something2.yourotherdomain2.com
  default_domain = something2.yourotherdomain2.com
  kpasswd_server = your-kdc-server2.something2.yourotherdomain2.com
}
  SOMETHING3.YOUROTHERDOMAIN3.COM = {
  kdc = your-kdc-server3.something3.yourotherdomain3.com
  default_domain = something3.yourotherdomain3.com
  kpasswd_server = your-kdc-server3.something3.yourotherdomain3.com
}

[domain_realm]
.something.yourdomain.com = SOMETHING.YOURDOMAIN.COM
.something2.yourotherdomain2.com = SOMETHING2.YOUROTHERDOMAIN2.COM
.something3.yourotherdomain3.com = SOMETHING3.YOUROTHERDOMAIN3.COM

 Generate Kerberos Ticket

kinit my.adminuser@SOMETHING.YOURDOMAIN.COM

Note: To see your tickets, you can use klist.  To clear your tickets, you can use kdestroy.

group_vars/windows.yml

Settings to be used connecting to windows hosts via kerberos authentication

ansible_ssh_user: my.adminuser@SOMETHING.YOURDOMAIN.COM
ansible_ssh_pass:
ansible_ssh_port: 5985
ansible_connection: winrm

playbooks/SplunkUniversalForwarderInstallWindows.yml - example

Here you tell Ansible which hosts you want to target, which variables to pass through to a role, and which roles to target

---
  - name: Windows x64 Universal Forwarder Install
    hosts: install-splunk-windows
    vars:
      splunk_working_directory: 'c:\Temp\Splunk\'
      splunk_remote_file_share: '\\server-name.yourdomain.com\c$\Distribute\'
      splunk_site: 'CORPORATE'
      splunk_install_path: 'c:\Program Files\SplunkUniversalForwarder\'
      splunk_deployment_server: 'your-deployment-server.yourdomain.com'
      splunk_deployment_server_port: '8089'
      splunk_user: 'admin'
      splunk_password: 'changeme'
      splunk_uf_binary: 'splunkforwarder-6.3.0-aa7d4b1ccb80-x64-release.msi'
    roles:
      - universal_forwarder_windows

roles/universal_forwarder_windows/tasks/main.yml

You can include multiple files here. For this example, there is only one template we want to call.

---
- include: forwarder.yml

roles/universal_forwarder_windows/tasks/forwarder.yml

Here is where all the install actions happen for windows machines

---
    - name: Checking if splunk is installed
      tags: install
      win_stat: path='{{splunk_install_path}}etc\splunk.version'
      register: splunk_path

    - name: splunk is installed
      tags: install
      debug: msg='splunk is already installed under {{splunk_install_path}}.'
      when: splunk_path.stat.exists

    - name: Assures {{splunk_working_directory}} exists
      tags: install
      win_file: path={{splunk_working_directory}} state=directory
      when: splunk_path.stat.exists == False

    - name: Checking if installer already copied over
      tags: install
      win_stat: path={{splunk_working_directory}}{{splunk_uf_binary}}
      when: splunk_path.stat.exists == False
      register: splunk_installer

    - name: Installer on server
      tags: install
      debug: msg='installer already copied over'
      when: (splunk_path.stat.exists == False) and (splunk_installer.stat.exists)

    - name: Push package to server
      tags: install
      win_get_url:
        url: '{{splunk_remote_file_share}}{{splunk_uf_binary}}'
        dest: '{{splunk_working_directory}}{{splunk_uf_binary}}'
      when: (splunk_path.stat.exists == False) and (splunk_installer.stat.exists == False)

    - name: Install Splunk
      tags: install
      script: install-splunk.ps1 -site '{{splunk_site}}' -installer '{{splunk_working_directory}}{{splunk_uf_binary}}'
      when: splunk_path.stat.exists == False

roles/universal_forwarder_windows/files/install-splunk.ps1

This is a custom script made for our environment. Modify as needed.

param (
    [string]$site,
    [string]$installer = "c:\Temp\Splunk\splunkforwarder-6.3.0-aa7d4b1ccb80-x64-release.msi",
    [string]$log = "c:\Temp\Splunk\splunkinstall.log"
)

$splunk_install_file = $installer

switch ($site.ToUpper()) {
    "CORPORATE" {$splunk_deployment_server = "10.10.10.10:8089"}
    "SITE1"     {$splunk_deployment_server = "10.10.10.11:8089"}
    "SITE2"     {$splunk_deployment_server = "10.10.10.12:8089"}
    "SITE3"     {$splunk_deployment_server = "10.10.10.13:8089"}
    default {"-site parameter missing.  Site list: corporate, site1, site2, site3"; exit}
}

& msiexec.exe /qn /Liwem! $log /i $splunk_install_file AGREETOLICENSE=Yes DEPLOYMENT_SERVER=`"$splunk_deployment_server`" INSTALL_SHORTCUT=0 /quiet

 LINUX

playbooks/SplunkUniversalForwarderInstallLinux.yml - example

---
  - name: Linux Universal Forwarder Install
    hosts: install-splunk-linux
    remote_user: your-user-here
    sudo: yes
    vars:
      splunk_working_directory: '/tmp/Splunk/'
      splunk_deployment_server: 'your-deployment-server.yourdomain.com'
      splunk_deployment_server_port: '8089'
      splunk_user: 'admin'
      splunk_password: 'changeme'
      splunk_uf_binary_linux: 'splunkforwarder-6.3.0-aa7d4b1ccb80-linux-2.6-x86_64.rpm'
    roles:
      - universal_forwarder_linux

roles/universal_forwarder_linux/tasks/main.yml

You can include multiple files here. For this example, there is only one template we want to call.

---
- include: forwarder.yml

roles/universal_forwarder_linux/tasks/forwarder.yml

Here is where all the install actions happen for linux machines

---
  - name: Checking if splunk is installed
    tags: install
    stat: path=/opt/splunkforwarder
    register: splunk_path

  - name: Checking if installer already copied over
    tags: install
    stat: path={{splunk_working_directory}}{{splunk_uf_binary_linux}}
    when: splunk_path.stat.exists == false
    register: splunk_installer

  - name: splunk is installed
    tags: install
    debug: msg='splunk is already installed under /opt/splunkforwarder'
    when: splunk_path.stat.exists == true

  - name: Checking if deployment server set
    tags: install
    stat: path=/opt/splunkforwarder/etc/system/local/deploymentclient.conf
    register: splunkds

  - name: Splunk deployment server set
    tags: install
    debug: msg='Splunk Deployment Server is set'
    when: splunkds.stat.exists == true

  - name: Assures {{splunk_working_directory}} exists
    tags: install
    file: path={{splunk_working_directory}} state=directory
    when: splunk_path.stat.exists == false

  - name: Push RPM package to server
    tags: install
    become: yes
    become_method: sudo
    copy: src=splunk_binaries/{{splunk_uf_binary_linux}} dest={{splunk_working_directory}}{{splunk_uf_binary_linux}} owner=hortone group=hortone mode=0644
    when: splunk_path.stat.exists == false and splunk_installer.stat.exists == false

  - name: Install RPM
    tags: install
    become: yes
    become_method: sudo
    command: rpm -ivh {{splunk_working_directory}}{{splunk_uf_binary_linux}}
    when: splunk_path.stat.exists == false

  - name: CHOWN Directory
    tags: install
    become: yes
    become_method: sudo
    shell: chown -R splunk:splunk /opt/splunkforwarder
    when: splunk_path.stat.exists == false

  - name: Start splunk
    tags: install
    become_user: splunk
    become_method: sudo
    shell: /opt/splunkforwarder/bin/splunk start --accept-license
    when: splunk_path.stat.exists == false

  - name: Enable boot-start
    tags: install
    become: yes
    become_method: sudo
    shell: /opt/splunkforwarder/bin/splunk enable boot-start -user splunk
    when: splunk_path.stat.exists == false

  - name: Splunk set deploy-poll and Restart splunkd
    tags: install
    become_user: splunk
    become_method: sudo
    shell: /opt/splunkforwarder/bin/splunk set deploy-poll {{splunk_deployment_server}}:{{splunk_deployment_server_port}} -auth {{splunk_user}}:{{splunk_password}} & /opt/splunkforwarder/bin/splunk restart
    when: splunkds.stat.exists == false

  - name: Check if DS App for deploymentclient.conf has been pulled
    tags: install
    become_user: splunk
    become_method: sudo
    stat: path=/opt/splunkforwarder/etc/apps/deploymentclient_two/local/deploymentclient.conf
    register: splunk_ds_settings

  - name: Delete deployment client settings in etc/system/local
    tags: install
    become_user: splunk
    become_method: sudo
    shell: rm -f /opt/splunkforwarder/etc/system/local/deploymentclient.conf & /opt/splunkforwarder/bin/splunk restart
    when: splunk_ds_settings.stat.exists == true
 
44
Kudos
 
44
Kudos

Now read this

Splunk Cisco_IPS app no longer pulls from IPS

Problem After an upgrade to Splunk 6, the Cisco_IPS app fails to download IPS logs. Troubleshooting Navigate to /var/log/splunk/sdee_get.log Events like the following show up Exception thrown in sdee.get(): URLError: <urlopen error... Continue →