Stairwell 101: Hunting for anomalies
I have clients asking me all the time, “How do YOU hunt?” It’s a very legitimate question, as there are numerous strategies one can take. You can:
- Follow an entire framework
- Leverage your threat intelligence feed
- Utilize YARA at scale to correlate disparate weak signals
- Take any number of SANS courses, such as: FOR508, FOR608, FOR572, FOR577
- Delay hunting, because “Without the right data, you cannot hunt”
Ok, so with the exception of that last one, these are all valid strategies for getting started with threat hunting. In fact, there are enough strategies that it’s easy to get Analysis Paralysis when evaluating what’s “best.”
I’ve hunted in EDR logs, network data, and file artifacts. In my mind, the only way to do hunting wrong is to not do it at all. The number one skill in hunting is curiosity, or “the desire to know,” so let’s talk about being curious with the data we have.
As a recent article by CrowdStrike points out, hunting is the practice where “Threat Hunters assume that adversaries are already in the system.” With our platform taking a “zero-trust” approach to all of your executables and code blobs, we have an advantage that allows you to get to know your data set.
In this article, we’ll create some rudimentary hunts meant to get new practitioners stretching their brains. I’ll do this on my platform (the Stairwell platform); think about how, or if, you can do it with yours.
IOC hunting
When MoveIT was making headlines, Huntress did a great investigation and gave us some nifty files and folders to look for. One such example was that this attack would drop a brand new .dll, App_Web_[RANDOM].dll. So do we pull and examine every such file, or can we take a holistic approach and look for the anomaly? In the name of efficiency, we want to take a look at the aggregate, so I searched for any file names containing App_Web. It becomes clear very quickly that one of these has a size that is very different from the others.
We can go further with the detail provided in that blog from Huntress and create even more useful queries. In Stairwell, we use CEL to create queries. You can write them manually or use our built-in query builder. Here are some of I created to help get started searching for IOCs:
object.path.contains("MOVEitTransfer\\wwwroot\\")object.path.contains("MOVEit") && object.path.contains("wwwroot")object.path.contains("Windows\\TEMP\\") && object.name.contains(".cmdline")object.name.contains("App_Web")object.name in ["human2.aspx", "guestaccess.aspx", "moveitisapi.aspx"]
IOC hunting has its weaknesses, though.It requires that someone has done the analysis, found the interesting bits, and reported on them. How do we take steps in defense when we don’t have an extensive teardown of an attack, when we have just the suspicion that there is something deeper?
Code scanning (aka YARA)
YARA, or Yet Another Ridiculous Acronym, is a tool that allows analysts to create rules to identify specific patterns within files. These rules can be used to identify malware, phishing emails, and other threats. YARA rules are based on regular expressions, which can be used to identify specific strings of text within a file. For example, a YARA rule might be created to identify files that contain a specific command that is commonly used by malware.
Similarly, when 3CX hit and with seemingly no knowledge from the company, the suspicion of a supply-chain attack was immediately front of mind. While defenders are doing everything they can to block hashes/hostnames/IPs, our Threat Research team took us a step further and wrote a rule for the compromised DLL:
rule Compromised_DLL_3CX
{
meta:
author = "Daniel Mayer ([email protected])"
description = "A rule for detecting a malicious DLL included in legitimate 3CX installers"
version = "1.0"
date = "2023-03-29"
reference1="https://twitter.com/DanielStepanic/status/1641115302246207489" reference2="https://twitter.com/DanielStepanic/status/1641115302246207489" sample1="aa4e398b3bd8645016d8090ffc77d15f926a8e69258642191deb4e68688ff973"
strings:
$s_tutma = "__tutma"
$s_tutmc = "__tutmc"
$s_manifest = "manifest" wide
$s_crypto = "Software\\Microsoft\\Cryptography" wide
$s_guid = "MachineGuid"
$s_github = "raw.githubusercontent.com" wide
$url = "https://raw.githubusercontent.com/IconStorages/images" wide
$lcg_chunk_1 = {
41 81 C0 87 D6 12 00 // add r8d, 12D687h
02 C8 // add cl, al
49 C1 E9 20 // shr r9, 20h
41 88 4B 03 // mov [r11+3], cl
4D 03 D1 // add r10, r9
8B C8 // mov ecx, eax
45 8B CA // mov r9d, r10d
C1 E1 05 // shl ecx, 5
33 C1 // xor eax, ecx
41 69 D0 7D 50 BF 12 // imul edx, r8d, 12BF507Dh
8B C8 // mov ecx, eax
C1 E9 07 // shr ecx, 7
33 C1 // xor eax, ecx
8B C8 // mov ecx, eax
C1 E1 16 // shl ecx, 16h
}
condition:
filesize < 500KB and ( all of ($s*) or ( 3 of ( $s* ) and $lcg_chunk_1 ) or $url )
}
Mandiant’s reporting helped us go even further up the Pyramid of Pain to find the tooling that was being used by the actors. As we found out that the original compromise was that of X_TRADER upstream, it became more critical to be able to scan for any other binaries that may have received the same compromised version. Rules that look out for the VEILEDSIGNAL backdoor can be leveraged to hunt and find other victims.
Again, this requires some kind of external trigger – some known attack that we can study. We had the early discussions on 3CX’s own forums or Crowdstrike’s Reddit reporting to guide us. But while it can feel like there is a new report every day, there is every need to pull upon a defender’s knowledge and experience to find those that we might call “unknown unknowns.”
Hypothesis hunting
This is a broad category that traditionally asks defenders to know their environment and their top prioritized threats. Then, they need to round up any TTPs (maybe using ATT&CK) from those threats and use any data they have available. After, they form a hypothesis about how they would catch that adversary using that behavior in their environment. That’s structured threat hunting, and it’s great, but we essentially covered it in the MoveIT example.
There truly is nothing more intimidating than an empty search bar, and that’s where unstructured hunting (a type of hypothesis hunting) lives. It’s based on knowing what is normal in systems and networks and asking yourself, “What does anomalous look like to us?” It doesn’t always work, and that is an opportunity to assess what happened. Are we collecting the right data? Did we understand what we’re looking for correctly? Do we want to seed our logs with some emulated data? Remember, these are hunts, not hard detections – but they also flex the creative side of your brain, and that’s always a good thing.
Part of what I do is getting to know what’s normal for each customer, so these likely won’t all apply to you. Still, let’s look at some hypotheses and how we’d hunt them.
First, story time. Inspiration for unstructured hunts/hypotheses can come from anywhere, like this first one from my daughter. She’s 3, and sometimes, I have her home with me while I work. All she needs is food and some toys, and she is good on her own – she’s seriously the best co-worker. We were onboarding an IR customer, and I had made my daughter a sandwich and put her at the table. A few minutes later, I heard playing noises, so I checked on her; no food, no mess, no way she ate.
“Baby girl, where is your sandwich?”
“In the playhouse” (shows me she’s placed it in her doll house)
“We don’t put food in doll houses. That’s not. Where. Food. Goes…” (runs off to hunt)
Hypothesis: Files and folders have a certain structure and hierarchy that SHOULD be respected. The logs folder is for logs; the “Pictures” folder should contain photos.
Hunt: Show all executables in the Pictures folder.
Hypothesis: We are an in-person company, and we don’t perform business activities over the weekend.
Hunt: Display all new executables between Friday night and Monday morning. If the offices are closed with nobody working, this should return a null.
Hypothesis: Many of the executables on your user’s computers are common, everyday applications.
Hunt: Give me every program that is rare and that my platform thinks is worth looking at. (This is one of our saved searches that all of our customers can access with one click.)
Hypothesis: We have a policy against allowing users to install browser extensions, so there shouldn’t be any browser extensions downloaded.
Hunt: Show me if anything exists in the known extensions folders. If this is your policy, this query should’ve returned nothing.
Here are some additional tips for hypothesis hunting:
- Start with simple hypotheses and then build on them as you learn more about your environment and your top prioritized threats.
- Be creative and think outside the box when developing hypotheses.
- Use as many data sources as you have access to for testing your hypotheses, such as security logs, network traffic, file artifacts, and user behavior data.
Be patient and persistent. Hypothesis hunting can take time, but it can be a very effective and rewarding way to identify threats.
Wrapping up
In this blog post, we discussed the basics of threat hunting, some steps involved, and the tools and techniques that can be used. We also provided some tips for getting started with threat hunting. Hunting doesn’t always lead to detection, but it can always teach you about your specific environment.
If you are not already doing so, we encourage you to start threat hunting as part of your cybersecurity program. By proactively searching for malicious activity, you can help to protect your organization from harm. If you are interested in learning more about threat hunting, there are a number of resources available online (linked above).
