Effectively Using Splunk

(Scenario 1)

© 2019 Caendra Inc. | Hera for IHRP | Effectively Using Splunk (Scenario 1) 1
The organization you work for (Wayne Enterprises) is using Splunk as a SIEM solution to
enhance its intrusion detection capabilities. The SOC manager informed you that the
organization has been hit by an APT group. He tasked you with responding to this incident
by heavily utilizing Splunk and all the data that it ingested.

The data that Splunk has ingested consist of Windows event logs, Sysmon logs, Fortinet
next-generation firewall logs, Suricata logs, etc.

Note: This lab is based on the Boss Of The SOC (BOTS) v1 dataset released by Splunk.
Credits to Ryan Kovar, Dave Herrald and John Stoner for sharing the Splunk detection tips
this lab covers with the public, through this dataset.

The learning objective of this lab is to not only get familiar with Splunk’s architecture and
detection capabilities but also to learn effective Splunk search writing.

Specifically, you will learn how to use Splunk’s capabilities in order to:
 Have better visibility over a network
 Respond to incidents timely and effectively
 Proactively hunt for threats

Splunk’s creators describe it as a solution to aggregate, analyze and get answers from
machine data. Splunk can be used for Application Management, Operations Management,
Security & Compliance, etc.

© 2019 Caendra Inc. | Hera for IHRP | Effectively Using Splunk (Scenario 1) 2
When it comes to security, Splunk can be used as a log management solution but most
importantly as an analytics-driven SIEM. Splunk can fortify investigations of dynamic,
multi-step attacks with detailed visualizations and even enhance an organization’s
detection capabilities through User Behavior Analytics.

Splunk can literally ingest almost any data from almost any source, through both an agent-
less and a forwarder approach.

Splunk Architecture Overview:

© 2019 Caendra Inc. | Hera for IHRP | Effectively Using Splunk (Scenario 1) 3
Splunk’s architecture (at a high level) consists of the:
 Forwarder component
o Universal Forwarders collect data from remote sources and send them to one
or more Splunk Indexers. Universal Forwarders are separate downloads that can
be installed on any remote source, with little impact on network or host
performance.
o Heavy Forwarders also collect data from remote sources, but they are typically
used for heavy data aggregation tasks, from sources like firewalls or data
routing/filtering passing points. According to Splexicon, unlike other forwarder
types, heavy forwarders parse data before forwarding them and can route data
based on criteria such as source or type of event. They can also index data locally
while forwarding the data to another indexer. Heavy Forwarders are usually run
as “data collection nodes” for API/scripted data access, and they are only
compatible with Splunk Enterprise.
Note: HTTP Event Collectors (HECs) also exist to collect data directly from
applications, at-scale, through a token-based JSON or raw API way. Data are sent
directly to the Indexer level.

 Indexer component
The Indexer processes machine data, storing the results in indexes as events,
enabling fast search and analysis. As the indexer indexes data, it creates a number of
files organized in sets of directories by age. Each directory contains raw data
(compressed) and indexes (points to the raw data).

 Search Head component

The Search Head component allows users to use the Search language to search for
indexed data. It distributes user search requests to the Indexers and consolidates
the results as well as extracts field value pairs from the events to the user.
Knowledge Objects on the Search Heads can be created to extract additional fields
and transform the data without changing the underlying index data. It should be
noted that Search Heads also provide tools to enhance the search experience such as
reports, dashboards, and visualizations.

Splunk Apps and Technology Add-ons (TAs):

Splunk Apps are designed to address a wide variety of use cases and to extend the power
of Splunk. Essentially, they are collections of files containing data inputs, UI elements,
and/or knowledge objects. Splunk Apps also allow multiple workspaces for different use
cases/user roles to co-exist on a single Splunk instance. Ready-made apps are available on
Splunkbase (splunkbase.com).

© 2019 Caendra Inc. | Hera for IHRP | Effectively Using Splunk (Scenario 1) 4
Splunk Technology Add-ons abstract the collection methodology and they typically
include relevant field extractions (schema-on-the-fly). They also include relevant config
files (props/transforms) and ancillary scripts binaries.

You can think of a Splunk App as a complete solution, that typically uses one or more
Technology Add-ons.

Splunk Users and Roles:

Splunk users are assigned roles which determine their capabilities and data access. Out of
the box, there are three main roles:
 admin: This role has the most capabilities assigned to it.
 power: This role can edit all shared objects (saved searches, etc.) and alerts, tag
events, and other similar tasks.
 user: This role can create and edit its own saved searches, run searches, edit its own
preferences, create and edit event types, and other similar tasks.

Splunk’s Search & Reporting App:

You will spend most of your time inside Splunk’s Search & Reporting App.

© 2019 Caendra Inc. | Hera for IHRP | Effectively Using Splunk (Scenario 1) 5
Data Summary can provide you with hosts, sources or sourcetypes on separate tabs.

Finally, this is how Events will look like.

© 2019 Caendra Inc. | Hera for IHRP | Effectively Using Splunk (Scenario 1) 6
Splunk’s Search Processing Language (SPL):

According to Splunk, SPL combines the best capabilities of SQL with the Unix pipeline
syntax allowing you to:
 Access all data in its original format
 Optimize for time-series events
 Use the same language for visualizations

SPL provides over 140 commands that allow you to search, correlate, analyze and visualize
any data.

The below diagram represents a search, broken down to its syntax components.

Searches are made up of five basic components.

1. Search terms, where you specify what you are looking for. Search terms contain
keywords, phrases, Booleans, etc.
2. Commands, where you specify how you want to manipulate the results. For
example, create a chart, compute statistics, etc.
3. Functions, where you specify how exactly do you want to chart, compute or
evaluate the results.
4. Arguments, in case there are variables you want to apply to a function.
5. Clauses, where you specify how exactly do you want to group or rename the fields
in the results.

© 2019 Caendra Inc. | Hera for IHRP | Effectively Using Splunk (Scenario 1) 7
As you write searches, you will notice that some parts of the search string are automatically
colored. The color is based on the search syntax. Example:

Something else to consider while submitting searches is Splunk’s search modes.

There are three search modes:

 Fast Mode: Field discovery off for event searches. No event or field data for stats
searches.
 Smart Mode: Field discovery on for event searches. No event or field data for stats
searches.
 Verbose Mode: All event & field data.

It is recommended to start searching with Smart and then go from there.

We strongly suggest you spend time studying the Exploring Splunk e-book before
proceeding to the lab’s tasks. Especially Chapter 4, as that covers the most commonly
used search commands.

Various Search aspects are also nicely documented, in the following resource.
https://docs.splunk.com/Documentation/Splunk/7.2.4/Search/GetstartedwithSearch

 Splunk

© 2019 Caendra Inc. | Hera for IHRP | Effectively Using Splunk (Scenario 1) 8
  Incident Responder’s Subnet: 172.16.72.0/24

 Splunk: 172.16.72.100:8000

 Connection Type: Web-based

o Use a Chrome or Firefox browser to connect to Splunk's web interface
(http://172.16.72.100:8000), as follows
 Username: admin
 Password: els@nalyst

Throughout this lab, we will split attacker actions based on the Cyber Kill Chain.

Using Splunk’s capabilities, try to identify any reconnaissance activities performed by the
APT group. Your organization’s website is imreallynotbatman.com.

Hints:

 Focus on the stream:http sourcetype and identify the source IPs that are
responsible for the majority of the traffic. Then, validate your findings using the
suricata sourcetype.

 Move the investigation deeper by analyzing all important fields and sourcetypes

© 2019 Caendra Inc. | Hera for IHRP | Effectively Using Splunk (Scenario 1) 9
Using Open Source Intelligence (OSINT), try to identify any weaponization activities
performed by the APT group.

Hints:

 Identify any IP addresses tied to domains that are pre-staged to attack Wayne
Enterprises

 Try to understand the associations between IP addresses and domains among other
things

 Do the same as above to associate attacker emails with infrastructure on the

internet

Using OSINT, try to identify any delivery activities performed by the APT group.
Specifically, try to identify malware associated with the attacker infrastructure you have
previously uncovered.

Hints:

 Submit any attacker-related IP address to open sources such as ThreatMiner,

VirusTotal and Hybrid Analysis

Using Splunk’s capabilities, try to identify any exploitation activities performed by the APT
group.

Hints:

 Focus on the stream:http and iis sourcetypes and identify which of your servers is
the target as well as the Content Management System it uses

 Focus on the stream:http sourcetype and identify the source of a brute force attack

 Move the investigation deeper by analyzing all important fields and sourcetypes

© 2019 Caendra Inc. | Hera for IHRP | Effectively Using Splunk (Scenario 1) 10
Using Splunk’s capabilities, try to identify any installation activities performed by the APT
group.

Hints:

 Focus on the stream:http and suricata sourcetypes to identify any uploaded

executables

 Leverage Sysmon logs to identify additional information about any uploaded

executables

Using Splunk’s capabilities, try to identify any Command and Control (C2)-related activities
performed by the APT group.

Hints:

 Focus on the stream:http, fgt_utm, and stream:dns sourcetypes to identify any

domains acting as Command and Control.

© 2019 Caendra Inc. | Hera for IHRP | Effectively Using Splunk (Scenario 1) 11
SOLUTIONS

© 2019 Caendra Inc. | Hera for IHRP | Effectively Using Splunk (Scenario 1) 12
Below, you can find solutions for every task of this lab. Remember though, that you can
follow your own strategy, which may be different from the one explained in the following
lab.

Once you are logged into Splunk’s web management interface, click the Search &
Reporting application that resides on the Apps column on your left. You should see
something similar to the below.

In order to test if Splunk can successfully access the ingested/loaded data, first change the
time range picker to All time and then, submit the following search.

index="botsv1" earliest=0

© 2019 Caendra Inc. | Hera for IHRP | Effectively Using Splunk (Scenario 1) 13
You should see the number of events growing as time progresses.

Now that we know everything worked as expected, let’s identify any reconnaissance
activities against Wayne Enterprises. As a reminder, the organization’s website is
imreallynotbatman.com.

The first thing we should do is determine the sourcetypes to search. Specifically, we should
first determine the sourcetypes that are associated with imreallynotbatman.com. We can
do so by changing the time range picker to All time and submitting the following Splunk
search.

index=botsv1 imreallynotbatman.com

You should see something similar to the below.

© 2019 Caendra Inc. | Hera for IHRP | Effectively Using Splunk (Scenario 1) 14
To determine the sourcetypes, simply click on sourcetype (red rectangle above). You
should see the following.

Let’s also identify all source addresses. We can do so through the previous search, but this
time we will scroll down and click on the src field, as follows.

© 2019 Caendra Inc. | Hera for IHRP | Effectively Using Splunk (Scenario 1) 15
Since we are interested in identifying reconnaissance activities, it would be better to focus
on the stream:http sourcetype. (Stream is a free app for Splunk that collects wire data and
can focus on a number of different protocols including smtp, tcp, ip, http and so on.)

index=botsv1 imreallynotbatman.com sourcetype=stream:http

If we do so, the sources will be narrowed down to two, 40.80.148.42 and 23.22.63.114.
40.80.148.42 is associated with ~95% of the http traffic, so let’s focus on this one for the
time being.

An alternative way to identify all sources is the following.

index=botsv1 imreallynotbatman.com sourcetype=stream* | stats

count(src_ip) as Requests by src_ip | sort - Requests

© 2019 Caendra Inc. | Hera for IHRP | Effectively Using Splunk (Scenario 1) 16
So far, we can only assume that 40.80.148.42 was the IP from where the APT group
performed its reconnaissance/scanning activities. We can validate this finding, by checking
with Suricata, as follows.

index=botsv1 imreallynotbatman.com src=40.80.148.42

sourcetype=suricata

We see Suricata logs related to 40.80.148.42, but no signature field. We can see the
signatures by scrolling down, clicking on more fields and choosing signature. If we do so,
the signature field will be visible under the SELECTED FIELDS column.

From the Suricata signatures that were triggered, we can conclude that 40.80.148.42 was
actually scanning imreallynotbatman.com.

© 2019 Caendra Inc. | Hera for IHRP | Effectively Using Splunk (Scenario 1) 17
We are also interested in knowing our adversary’s level of sophistication. So the question
that arises is, did the APT group use known or sophisticated scanning techniques? Let’s
take a look at the submitted requests to answer that.

index=botsv1 src=40.80.148.42 sourcetype=stream:http

The detailed request information can be found inside the src_headers field.

The APT group utilized an instance of the reputable Acunetix vulnerability scanner.

© 2019 Caendra Inc. | Hera for IHRP | Effectively Using Splunk (Scenario 1) 18
We could have also identified the usage of this tool by looking for uncommon user agents.

We can easily identify which server was the target through the same search and the dest
field.

The target was obviously 192.168.250.70.

If we want to have a closer look at what has been requested by the APT group, we can do
that as follows.

index=botsv1 dest=192.168.250.70 sourcetype=stream:http

© 2019 Caendra Inc. | Hera for IHRP | Effectively Using Splunk (Scenario 1) 19
The URLs being requested can be found inside the uri field.

We are also interested in successful page loads. We can identify them, as follows.

index=botsv1 dest=192.168.250.70 sourcetype=stream:http status=200 |

stats count by uri | sort - count

© 2019 Caendra Inc. | Hera for IHRP | Effectively Using Splunk (Scenario 1) 20
We could have achieved similar results through the iis sourcetype, as follows. (This time
we are using a transformational search command called stats that will allow us to count
the number of events grouped by URI.)

index=botsv1 sourcetype=iis sc_status=200 | stats values(cs_uri_stem)

You may be wondering why we aren’t specifying 192.168.250.70. This is because if we

submit the below and check the host field, we will find only one host, we1149srv. This
host’s IP address is 192.168.250.70

index=botsv1 sourcetype=iis

© 2019 Caendra Inc. | Hera for IHRP | Effectively Using Splunk (Scenario 1) 21
Below are our findings so far.

© 2019 Caendra Inc. | Hera for IHRP | Effectively Using Splunk (Scenario 1) 22
At this point, we need to understand that Splunk is not panacea. During our investigations,
not every answer can be found within the SIEM. There will be times when we will need to
pivot from the SIEM to other internal or open sources to find answers.

We are interested in identifying domains that are pre-staged to attack Wayne Enterprises.

We gave the 40.80.148.42 IP address a good look through Splunk. Let’s do the same for
23.22.63.114 but through open sources since Splunk doesn’t contain too much
information about it.

If we go to an open source like http://www.robtex.com and submit the 23.22.63.114 IP,

we will come across the following.

As we can see, this IP has a number of other domain names associated with it. These
domain names are most probably phishing domains since their name is similar to the
organization we work for, Wayne Enterprises.

Open sources like https://threatcrowd.org and https://www.virustotal.com can provide us

with additional information.

© 2019 Caendra Inc. | Hera for IHRP | Effectively Using Splunk (Scenario 1) 23
For example, through threatcrowd.org, we identified additional domains associated with
the APT group we are dealing with by simply submitting the 23.22.63.114 IP.

Remember when we talked about whois information and how attackers leverage them for
targeted attacks? Well, let’s give attackers a taste of their own poison, by checking the
whois information of every associated domain.

While checking the whois information of wayncorpinc.com we come across the following.

© 2019 Caendra Inc. | Hera for IHRP | Effectively Using Splunk (Scenario 1) 24
We can then proceed to reverse email searches and possibly identify additional
infrastructure associated with the APT group. Find an example of a reverse email search
below.

https://www.threatcrowd.org/email.php?email=LILLIAN.ROSE@PO1S0N1VY.COM

Here are our findings so far:

© 2019 Caendra Inc. | Hera for IHRP | Effectively Using Splunk (Scenario 1) 25
We need to know as much as possible about this APT group’s TTPs and used malware, so
let’s dig deeper through open sources.

https://www.threatminer.org has a great capability of including related malware samples

when searching for information about an IP address. This is what we will come across
while searching for information about 23.22.63.114 on threatminer.org.

We can then submit these MD5 hashes to open sources like threatminer, VirusTotal or
hybrid-analysis.com to identify additional metadata about the sample(s).

© 2019 Caendra Inc. | Hera for IHRP | Effectively Using Splunk (Scenario 1) 26
The APT group may create mutants, so hashes may not prove useful. We should note down
that filename though in case they keep that.

Below are our findings so far.

© 2019 Caendra Inc. | Hera for IHRP | Effectively Using Splunk (Scenario 1) 27
It is about time we go back to Splunk to identify any exploitation activities. Let’s start by
identifying source IP addresses that are associated with the largest number of http events.
We can do that, as follows.

index=botsv1 sourcetype=stream:http dest="192.168.250.70"

http_method=POST

The src field contains what we are looking for. We specified that we are interested in POST
requests since logins are usually performed through POST requests (more on that in a bit).

Let’s take a look at those POST requests made by 40.80.148.42

index=botsv1 sourcetype=stream:http dest="192.168.250.70"

http_method=POST src="40.80.148.42"

The form_data field contains information that we want to check when dealing with POST
requests.

Nothing to justify successful exploitation activities. Let’s check 23.22.63.114.

index=botsv1 sourcetype=stream:http dest="192.168.250.70"

http_method=POST src="23.22.63.114"

© 2019 Caendra Inc. | Hera for IHRP | Effectively Using Splunk (Scenario 1) 28
It looks like 23.22.63.114 is brute forcing the web server’s authentication.

Let’s make sure, as follows.

index=botsv1 sourcetype=stream:http dest="192.168.250.70"

http_method=POST form_data=*username*passwd* | stats count by src

Indeed 23.22.63.114 performed a brute force attack against the web server’s
authentication.

© 2019 Caendra Inc. | Hera for IHRP | Effectively Using Splunk (Scenario 1) 29
We are quite interested in knowing if the brute force attack was successful. We can
determine that, as follows.

index=botsv1 sourcetype=stream:http form_data=*username*passwd*

dest_ip=192.168.250.70 | rex field=form_data
"passwd=(?<userpassword>\w+)" | stats count by userpassword | sort -
count

The search above extracts every user password and counts the times it has been
seen/used. If a password is seen more than one time, this probably means that attackers
got a hit and used the password again to log in. This is why we are sorting on count.

If we want to get an idea of the time of the compromise and the URI that was targeted, we
can do that as follows.

index=botsv1 sourcetype=stream:http form_data=*username*passwd*

dest_ip=192.168.250.70 src=40.80.148.42 | rex field=form_data
"passwd=(?<userpassword>\w+)"| search userpassword=* | table _time uri
userpassword

© 2019 Caendra Inc. | Hera for IHRP | Effectively Using Splunk (Scenario 1) 30
Finally, if we want to view the two successful logins we can do so, as follows.

index=botsv1 sourcetype=stream:http | rex field=form_data

"passwd=(?<userpassword>\w+)" | search userpassword=batman | table
_time userpassword src

Below are our findings so far.

© 2019 Caendra Inc. | Hera for IHRP | Effectively Using Splunk (Scenario 1) 31
As far as the installation phase of the cyber kill chain is concerned, we are mostly
interested in identifying any malware being uploaded.

We can identify that through various sourcetypes, specifically, stream:http and Suricata.

index=botsv1 sourcetype=stream:http dest="192.168.250.70" *.exe

The part_filename{} field contains the information we want to check. It won’t be visible by
default, so add it.

© 2019 Caendra Inc. | Hera for IHRP | Effectively Using Splunk (Scenario 1) 32
index=botsv1 sourcetype=suricata (dest=imreallynotbatman.com OR
dest="192.168.250.70") http.http_method=POST .exe

The fileinfo.filename field contains the information we want to check.

3791.exe must be the uploaded malware.

The source from where the file was uploaded can easily be identified, as follows.

index=botsv1 sourcetype=suricata dest_ip="192.168.250.70"

http.http_method=POST .exe

© 2019 Caendra Inc. | Hera for IHRP | Effectively Using Splunk (Scenario 1) 33
It would be great if we could also identify the hash of the uploaded file. But what
sourcetype should we use? Let’s find out, as follows.

index=botsv1 3791.exe

Sysmon is a good candidate since it logs information such as MD5, SHA1 and SHA256
hashes of files.

index=botsv1 3791.exe sourcetype=XmlWinEventLog:Microsoft-Windows-

Sysmon/Operational

The important fields, in this case, are Hashes, CommandLine and ParentCommandLine.
You will have to add the last two ones since they are not visible by default.

Before analyzing the results, let’s narrow things down a little bit.

index=botsv1 3791.exe sourcetype=XmlWinEventLog:Microsoft-Windows-

Sysmon/Operational EventCode=1

© 2019 Caendra Inc. | Hera for IHRP | Effectively Using Splunk (Scenario 1) 34
The search above includes EventCode 1 since this EventCode is related to process creation
events. Unfortunately, to get the MD5 hash of the uploaded file, we need to narrow things
down even further. Specifically, we will need to search for 3791.exe inside the command
line field, since this field captures the process starting.

So, the final search is the following.

index=botsv1 3791.exe CommandLine=3791.exe | stats values(md5)

Below are our findings so far.

© 2019 Caendra Inc. | Hera for IHRP | Effectively Using Splunk (Scenario 1) 35
As far as the Command and Control phase of the cyber kill chain is concerned, we are
mostly interested in identifying any domain used for command and control purposes.

We have a powerful ally inside Splunk to assist us in answering such questions. This ally is
the stream.dns sourcetype. Since we already know that 23.22.63.114 is of concern, we
can utilize Splunk and the stream.dns sourcetype to identify DNS events where
22.23.63.114 was the answer.

index=botsv1 answer=23.22.63.114 sourcetype=stream:dns | stats

values("name{}")

If you look carefully enough, you will identify that the

prankglassinebracket.jumpingcrab.com domain has been used by attackers to deface
the web server. Give it a try…

© 2019 Caendra Inc. | Hera for IHRP | Effectively Using Splunk (Scenario 1) 36
References:
1. https://www.slideshare.net/Splunk/splunk-data-onboarding-overview-splunk-
data-collection-architecture

2. https://www.splunk.com/en_us/training.html

© 2019 Caendra Inc. | Hera for IHRP | Effectively Using Splunk (Scenario 1) 37