And now a guest post from Justin Hall!!
Members of my incident investigation team are constantly on the move, traveling from customer to customer, gathering data, performing analysis, and writing reports. One of the tools we find ourselves commonly reaching for is a log analysis platform. Many of us are purists – âgive me grep and coffeeâ types – while others enjoy a more organized âbrowsingâ approach. Weâve found one solution that meets both of these needs – Splunk.
(Note, I am not a paid spokesmodel for Splunk, we do not resell their product or make any money from them. Weâre just big fans!)
I wanted to distribute a custom Splunk-based log analysis platform for our crew to use. I had a few key requirements in building this platform:
- It had to be portable. I wanted to build it and distribute it to our crew in a format that would be ready to use immediately, on any system in our fleet.
- It had to be powerful. I wanted it to meet as many of our log analysis needs as possible, in one package.
- It had to be secure. We need to make sure our customerâs data is protected while weâre investigating.
- It had to be free (as in beer). Our budget didnât have room for crazy expensive tools.
So I went with a virtual machine that could be distributed and re-used easily. The recipe I ended up with included Oracleâs VirtualBox, a free virtual computing platform; Ubuntu Linux Server, my personal favorite flavor of Linux; and Splunk 4.2 for Linux. Â Note, this article is not a âHow to use Splunkâ guide. If you are unfamiliar with Splunk, their documentationâs pretty good. Iâve found that just playing around with the tool is also a great learning experience!
Hereâs how I built the platform.
1. Install VirtualBox, if itâs not already installed. As of this writing, v4.0.4 is the stable release.
2. Configure host-only networking to use a private subnet between your PC and the virtual machines. We use host-only networking to transfer data between the VM and the host OS so that it cannot be intercepted outside of our machine. This feature is usually enabled by default – but you should verify the subnet youâre using, and the IP you have set on your host OSâs interface.
3. Create a new VM. I use two NICâs on the VM – the first interface is bridged to the LAN, and only up when I am patching. The second interface is the host-only network, which will receive an address on that network from VirtualBoxâs built-in DHCP server. Make sure to give your VM sufficient RAM – I use at least 1GB – and enough disk space for the largest log set you can imagine storing on the VMâs filesystem. I usually max it out at around 30GB.
4. Install Linux. During the installation, we enabled full-disk encryption – not just because you might store log data on the VMâs filesystem, but because Splunk will index the logs and those indexes also may contain sensitive data. I use a dummy key initially; when I distribute the VM, our investigators add their own decryption key and remove the dummy key, so that itâs unique to each userâs VM.
5. Youâll also want to install the SSH server, so that you can securely transfer logs to the VM over a network connection.
6. I have two users on the machine – a standard user for normal operation and administration, and for manual log analysis; and a Splunk service user. I call my first one ninja, but you can use whatever less awesome name you like. Then I add a Splunk user (called splunk), and give that user a home directory of /opt/splunk.
7. Once your OS is installed, configure the SSH server to disable root login, and if you donât want to use passwords every time you copy a file to the VM, add your public SSH key to ~/.ssh/authorized_keys (although you may not want to do so before distributing!).
8. Whatâs the first thing you do after installing an OS? Patch! I bring up my bridged adapter and sudo apt-get update; sudo apt-get upgrade once a week or so. Remember to shut eth0 down afterwards.
9. Install Splunk on the VM. Iâve found the easiest method is to download the .deb package from Splunk, SCP it to the VM and run dpkg -i. It will install by default in the splunk userâs home directory, /opt/splunk.
10. Add your ninja and splunk users to a âlogsâ group (I call mine âlogsâ). We will restrict the logs directory so only users in that group can write to it.
11. You can get your logs to the system in many ways – copy from mounted removable media, download from a webserver, direct syslog, etc. Personally I like to SCP them to a logs directory on the filesystem, like /logs. Give the logs group +rw permissions to this directory – this is important, because Splunk will need to read the logs from this directory, and weâre going to run Splunk as the splunk user. I use the ninja user to SCP logs to this directory, from my host OS, using the host-only network. You can remove other read/write permissions to the directory.
12. Fire up Splunk! Su to your Splunk user, make sure that user has +rw on the /opt/splunk/ directory tree, and run /opt/splunk/bin/splunk start. Youâll have to accept a EULA.
13. Open your browser on your host OS and hit the web interface (http://vm-host-only-address:8000) on the host-only network. Log in with the default credentials, which are listed at the login page.
14. Switch to Free license: go to Manage -> Licenses -> Change license group. The Free license has some features disabled (but probably nothing youâll miss with this setup) and can only index 1GB of logs per day. If you go over once, Splunk will nag you; if you go over 3 times in a rolling 30-day period, Splunk disables your search capability until you upgrade to a paid license. So be careful!
15. Configure Splunk to monitor the /logs directory. This will make Splunk read log data as soon as itâs written to /logs. In Splunk, go to Manager -> Data Inputs and open Files and Directories. Add a new file input and point Â it to the /logs directory on the filesystem. You can leave all the other options at the defaults. You also may want to disable the other inputs listed under Files and Directories – the defaults are just Splunk logfiles, but you might not want that data inadvertently showing up alongside your customerâs logs.
16. Test your setup â SCP a logfile to /logs and make sure it gets eaten by Splunk and is searchable. Watch Splunkâs main search page until events start showing up – the Events Indexed counter will increase. Run a search to verify the data is being processed properly.
17. You might want Splunk to start when you boot your VM. If so, enable Splunk boot-start with the splunk user: as root, run /opt/splunk/bin/splunk enable boot-start âuser splunk
18. Thatâs it! Youâre ready to analyze. Note, when you are done with an investigation, switch to the splunk user, and run /opt/splunk/bin/splunk clean eventdata to remove all of the indexed data and prepare a blank slate for future investigations.
Weâve found this platform extremely convenient for log analysis work. When we want to do full-scale analysis, we have Splunk; when we want to quickly grep | cut | sort | uniq, we have the command line. I handed the VM files – the VBOX config file, and the .VDI virtual disk it uses – to my investigation team, had them change the appropriate passwords, and they were ready to go.
We hope you find this recipe useful as well. Happy hunting!
Quick shameless plug: I am mentoring a session of SANSâ SEC504 – Hacker Techniques, Exploits & Incident Handling in Cincinnati, Ohio, starting May 11. You can find more information about the class, dates & times, and sign up here. Hope to see you there!