Malware Analysis
Foundations & Behavioural

Every analysis decision — what to look for, which tool to use next, how to prioritise findings — flows from understanding what kind of malware you are looking at. A dropper behaves differently from an implant. A ransomware payload behaves differently from a stealer. The taxonomy is not academic classification for its own sake; it is the map that guides the analysis.

The Malware Taxonomy

The Infection Chain

Real-world malware rarely operates as a single binary. Modern attacks use a staged delivery chain where each stage is purpose-built, small, and limited in what it does. Understanding the chain tells you where in it the sample you're analysing sits — and what to look for next.

The Malware Ecosystem

Understanding who writes malware and why shapes how you approach analysis. A sample from a sophisticated APT group will have anti-analysis features, custom encryption, and living-off-the-land techniques that a commodity crimeware tool won't bother with. A ransomware-as-a-service affiliate using a purchased builder produces something structurally different from a custom implant developed over months by a nation-state team.

• Crimeware / eCrime — financially motivated. Commodity tooling, RaaS (Ransomware-as-a-Service), MaaS (Malware-as-a-Service). Broad targeting, high volume, moderate sophistication. Examples: LockBit, Emotet, RedLine Stealer.
• APT (Advanced Persistent Threat) — nation-state or state-sponsored. Custom tooling, targeted victims, high sophistication, long-term persistence. Examples: Cobalt Strike customised variants, SUNBURST (SolarWinds), WannaCry (attributed to Lazarus Group).
• Hacktivism — ideologically motivated. Often simpler tooling, DDoS-heavy, opportunistic targeting. Wipers sometimes used for destruction rather than financial gain.
• Commodity vs Custom — commodity malware is purchased or rented; custom malware is purpose-built. Commodity samples have existing detection signatures and threat intel; custom samples require full analysis.

How Samples Reach Analysts

• Incident response — extracted from a compromised endpoint during an active engagement. Highest context (you know the victim, the timeline, the infection vector).
• VirusTotal / MalwareBazaar — community-submitted samples. Good for finding commodity malware; limited context about original deployment.
• Threat intelligence feeds — commercial (Recorded Future, CrowdStrike Intelligence) and open (MISP communities, abuse.ch). Often include context about campaign and attribution.
• Sandboxes — any.run, Hybrid Analysis, Joe Sandbox — sometimes expose unpacked or deobfuscated variants that don't appear in other feeds.
• Honeypots — passive capture of samples actively scanning or exploiting internet-facing systems. Useful for seeing what is actively weaponised in the wild.

Malware analysis on a production machine is not a calculated risk — it is an error. The consequences range from personal data theft to network-wide compromise. The analysis environment must be isolated, reversible, and configured to surface behaviour rather than hide it. Getting this right before touching a single sample is what makes everything else in this module safe to do.

Why VMs and Why Snapshots

A virtual machine provides three essential properties for malware analysis. First, isolation — the guest OS is separated from the host OS and from the network. Second, reversibility — a snapshot taken before analysis can be restored in seconds, returning the environment to a clean state regardless of what the malware did. Third, observability — VMs can be paused, their memory dumped, and their network traffic captured at the hypervisor layer without the guest OS knowing.

FlareVM — The Windows Analysis Toolkit

FlareVM (FLARE = FireEye/Mandiant Advanced Reverse Engineering team) is an automated PowerShell installer that turns a clean Windows installation into a fully equipped malware analysis workstation. It installs and configures tools across every analysis category.

Network Isolation Options

Network configuration is the most consequential setup decision. Different analysis scenarios require different network postures:

Safe File Handling

Several conventions prevent accidental execution of samples outside the analysis environment:

• Password-protected archives — the universal convention is password infected. All malware samples shared in the community are archived with this password. Windows will not auto-execute a file inside a password-protected ZIP.
• The .mal extension — renaming sample.exe to sample.exe.mal prevents accidental double-click execution on Windows. Rename back inside the analysis VM before executing.
• Disable autorun — in the analysis VM, disable Windows AutoPlay/AutoRun for all drive types. A USB drive or mounted ISO with a malicious autorun.inf file should not execute automatically.
• Shared folders with caution — the shared folder between host and guest VM is a potential escape path. Keep it read-only for the guest, transfer samples in one direction only (host to guest), and disable it when not actively transferring files.

Anti-Analysis at Setup Time

Many malware families detect that they are running in a virtual machine and alter their behaviour — running benign code, sleeping for extended periods, or terminating. Before taking the clean baseline snapshot, configure the VM to be less fingerprint-able:

• Set a realistic VM hostname (not "DESKTOP-FLAREVM" or "SANDBOX") — common sandbox hostnames are blocklisted by malware
• Create a realistic user account name and populate recent documents/browser history to simulate a real user environment
• Change the screen resolution to a realistic size (1920×1080) — some malware checks for very small sandbox resolutions
• Ensure the VM has more than 2 CPU cores and 4GB RAM — resource thresholds are a common sandbox detection technique
• Install VMware Tools or VirtualBox Guest Additions but be aware that their presence can be detected — some analysts deliberately remove or rename these components

The first sixty seconds with a new sample determines how you spend the next sixty minutes. Triage is not shallow analysis — it is structured, efficient analysis aimed at answering one question as quickly as possible: what kind of problem is this, and what analysis approach does it require? Commodity malware with existing detection may need only IOC extraction; a novel sample needs full behavioural and static analysis.

File Type Identification

Never trust the file extension. Malware is routinely distributed with misleading extensions — a PDF icon and a .pdf.exe filename, a .doc file that is actually a PE binary, a .jpg that is shellcode. File type is determined by the magic bytes at the start of the file, not the extension.

Hashing and VirusTotal Workflow

Reading Sandbox Reports Critically

Automated sandbox reports are extremely useful for rapid triage — but they require critical reading. The sandbox can only report what it observed during its analysis window (typically 60–120 seconds), which may not represent the malware's full behaviour. Several factors cause sandboxes to produce incomplete reports:

• Sleep bombs — the malware calls Sleep(600000) (10 minutes) before doing anything. The sandbox times out. The report shows nothing. The malware is not benign.
• VM detection — the malware detects it is running in a sandbox and executes benign code only. The report shows nothing suspicious. The malware is not benign.
• User interaction requirements — the malware checks for mouse movement, window focus, or specific user actions before proceeding. The sandbox environment does not simulate these. The report is incomplete.
• Network dependency — the malware requires a response from its C2 before proceeding. If the sandbox cannot simulate the response, the execution chain stops early.

The Triage Decision Tree

Almost every piece of Windows malware is a PE (Portable Executable) file — an EXE, DLL, SYS, or one of several other variants. Every static analysis technique in the following chapters operates on the PE structure. Understanding what each field means and where it lives in the file is what makes pestudio's output intelligible rather than a list of numbers.

PE Structure Overview

The Section Table — What Each Section Tells You

The Import Address Table — Capability from DLL Imports

The Import Address Table (IAT) lists every external function the binary calls, organised by DLL. This single data structure is often the most informative piece of static analysis — the imports are a capability manifest. Before running a single instruction, you can make an educated assessment of what the malware can do based purely on what it imports.

Strings extraction is the highest-value-per-effort technique in static malware analysis. A single run of the right tool against an unpacked sample can surface C2 domains, file paths, registry keys, mutex names, PDB paths, encryption keys, and error messages that define what the malware does before a single instruction executes. The challenge is that modern malware increasingly obfuscates its strings — and the standard strings tool misses the obfuscated ones.

strings vs FLOSS

The standard strings utility extracts sequences of printable ASCII or Unicode characters of a minimum length. This finds plaintext strings embedded directly in the binary. It misses strings that are built on the stack character by character (stack strings), strings that are XOR-encoded, strings that are base64-encoded, and strings that are concatenated in small pieces. FLOSS (FLARE Obfuscated String Solver) addresses all of these.

What Strings Reveal — The Intelligence Categories

Reading Obfuscated Strings — What FLOSS Reconstructs

Stack strings appear as isolated single characters in the raw binary. They are invisible to strings but FLOSS executes the relevant code fragments in an emulator and captures the fully-formed string after the stack-building loop completes. The hex dump below shows what the raw binary looks like for a stack-built API name — each byte loaded individually with a MOV instruction — versus the same string stored plaintext.

Building the Capability Hypothesis

The output of strings analysis and import analysis should be combined into a capability hypothesis before moving to dynamic analysis. This hypothesis directs what to watch for in ProcMon and Wireshark — if your static analysis suggests persistence via Run key and C2 via HTTP, you know exactly which ProcMon filters and Wireshark display filters to configure.

Most malware distributed in the wild is packed or obfuscated. Packing serves two purposes: it reduces file size (the original purpose, now irrelevant) and it defeats signature-based detection and static analysis (the current purpose). Understanding what packing looks like — in entropy measurements, in section characteristics, in the import table — tells you when your static analysis is working on a stub rather than the real code, and what to do about it.

Shannon Entropy

Shannon entropy measures the information density (randomness) of data on a scale from 0 to 8 bits per byte. Plaintext English text has entropy around 3.5–4.5. Compiled code has entropy around 5.5–6.5. Compressed or encrypted data has entropy approaching 7.5–8.0 — because compression and encryption both produce output that is nearly indistinguishable from random bytes.

Packer Detection with Detect-It-Easy

Manual UPX Unpacking

UPX is by far the most common packer encountered in commodity malware. It is trivial to unpack — the UPX tool itself can unpack a UPX-packed binary in a single command. Custom and commercial packers require the dynamic unpacking techniques covered in Book 2.

Dynamic API Resolution — Hiding Imports

Even without packing, malware can hide its true capabilities by resolving API functions at runtime rather than declaring them in the import table. Two common techniques: using GetProcAddress to look up functions by name at runtime, and API hashing — looking up functions by a computed hash of their name so even the string "CreateRemoteThread" doesn't appear in the binary.

YARA is the universal pattern-matching language for malware analysis. A YARA rule encodes what you know about a malware sample — its strings, byte patterns, structural characteristics — into a portable, shareable, deployable detection artefact. Writing YARA rules is the bridge between analysis and detection: the findings from static analysis become the rules that hunt for the same malware across an entire environment.

Basic Rule Syntax

The PE Module — Structure-Aware Rules

Testing and False Positive Management

Community Rule Sets

• YARA-Forge — automatically compiled community rules from multiple sources, deduplicated and tested. Best starting point for operational deployment.
• Neo23x0 (Florian Roth) — extensive, actively maintained rule sets covering hundreds of malware families. High quality and widely used.
• Mandiant/FLARE rules — from the same team that wrote FLOSS. Technically rigorous rules focused on advanced threats.
• abuse.ch / YARAhub — community-contributed rules, often tied to current threat intelligence.

Dynamic analysis begins the moment you execute the sample. From that point, three things are happening simultaneously: the malware is doing what it was designed to do, your monitoring tools are recording every action, and the capability hypothesis you built from static analysis is being confirmed or revised. The skill is not in running the tools — it is in filtering the noise and recognising the signal.

Process Monitor — The Event Stream

Process Monitor (ProcMon) captures every file system, registry, process, and network event on the system in real time. A single minute of execution on a Windows machine produces thousands of events — the vast majority from legitimate system processes. The workflow is filter first, analyse second.

Key Persistence Registry Locations

The following registry locations are the most commonly used by malware for persistence. Every one of these should appear in your ProcMon filters and your Autoruns review:

Process Explorer — Live Injection Detection

Autoruns — What Persists Across Reboots

Autoruns from Sysinternals enumerates every autostart location on the system — every place where something is configured to run automatically. After executing malware, run Autoruns and look for new entries added since the clean baseline. Entries highlighted in yellow indicate the backing file is missing (the malware may have already deleted its dropper). Entries highlighted in red indicate the backing file is not signed.

Network traffic analysis during dynamic analysis captures the malware's communication behaviour — what it connects to, what protocol it uses, what it sends, what it expects to receive. This produces the network IOCs that defenders can deploy immediately: C2 domains and IPs for firewall block lists, HTTP patterns for proxy rules, JA3 hashes for IDS signatures. It also reveals the C2 protocol structure, which becomes the foundation for more targeted detection.

Setting Up Network Simulation

INetSim (Internet Services Simulator) or FakeNet-NG runs on the REMnux VM and simulates internet services — DNS, HTTP, HTTPS, SMTP, FTP, IRC. When the analysis VM's DNS is pointed at REMnux, all DNS queries resolve to the REMnux IP, and all connections to simulated services receive plausible fake responses. This allows loaders that require a C2 response to continue executing past the initial check-in.

Reading the C2 Check-In in Wireshark

Inspecting the Raw Beacon Packet

When a malware beacon uses a custom binary protocol over raw TCP rather than HTTP, Wireshark shows the raw bytes. Understanding the packet structure from a hex view is the first step toward writing a protocol parser or detection rule.

Identifying C2 Protocols

Writing a Suricata Rule from Traffic Analysis

Memory analysis is the technique that finds what disk analysis misses. Fileless malware executes entirely in memory with no persistent file on disk. Injected code lives in the memory of a legitimate process — it appears nowhere in the file system. Encryption keys, decrypted configuration, and plaintext C2 credentials exist in memory during execution and nowhere else. Volatility provides a framework for interrogating a memory image to surface all of these.

Memory Acquisition in the Analysis VM

Core Volatility 3 Plugins

malfind — The Injection Detection Plugin

The malfind plugin finds memory regions that are: executable, not backed by a file on disk, and contain what appears to be executable code (PE header or shellcode). This is the primary signature of process injection — the injected code lives in allocated memory with no corresponding file.

Finding Encryption Keys and Strings in Memory

Process injection is the technique of running code within the address space of another, legitimate process. It is used to hide malware behind trusted process names, bypass application whitelisting, inherit the privileges and network access of the host process, and evade process-based detection. Each injection technique leaves a characteristic footprint — understanding that footprint is what lets you identify which technique was used from a memory image alone.

Classic DLL Injection

The most common and well-understood injection technique. The attacker allocates memory in the target process, writes the path to their malicious DLL, and creates a remote thread that calls LoadLibrary with that path. Windows loads the DLL into the target process as if it were legitimate.

Reflective DLL Injection

Reflective DLL injection improves on classic injection by eliminating the need to write a file to disk. The malicious DLL contains a custom loader (the reflective loader) that maps the DLL into memory from a byte array, resolves its own imports, and calls its entry point — all without calling LoadLibrary. The DLL never appears on disk and doesn't appear in the standard module list.

Process Hollowing

Process hollowing creates a legitimate process in a suspended state, unmaps its original executable image from memory, and maps the malicious image in its place. The process name, PID, and path all appear legitimate — the content is entirely replaced.

Analysis findings become intelligence when they are structured, contextualised, and shared. An IP address in a Wireshark capture is a data point. That same IP address formatted as a STIX indicator, linked to a malware family, a campaign, and a threat actor, with a confidence score and a decay date, and shared via TAXII to every SIEM in your sector — that is threat intelligence. This chapter covers the transformation.

The IOC Taxonomy

IOC Extraction Workflow

Infrastructure Pivoting

A single C2 domain or IP is a starting point, not an endpoint. Infrastructure analysis — finding related domains, certificates, and IPs operated by the same threat actor — dramatically expands the IOC set and provides early warning of future campaigns.

MITRE ATT&CK is the common language between malware analysis and detection engineering. When an analyst finds that a sample modifies the Run registry key, they can tag that finding as T1547.001. When a detection engineer asks which techniques are covered by current rules, they reference the same taxonomy. ATT&CK makes the conversation between analysts, detection engineers, and defenders possible without ambiguity about what behaviour is being described.

Mapping Sample Findings to ATT&CK

Writing Sigma Rules from Behavioural Findings

Sigma rules translate behavioural observations — what ProcMon saw, what event IDs fired, what Autoruns found — into portable SIEM detection rules. A Sigma rule written for Splunk can be converted to Sentinel KQL, Elastic EQL, or QRadar AQL by the Sigma converter.

Detection Coverage Assessment

Once a sample's ATT&CK techniques are mapped, you can assess which of those techniques your current detection stack covers. This answers the question: "If this malware ran in our environment today, which stages would we detect, and which would be blind spots?"

At scale, manual analysis of every sample is impossible. A mid-sized organisation's email gateway may block hundreds of suspicious attachments per day. Sandboxes provide automated dynamic analysis — executing samples and reporting behaviour without an analyst manually operating the tools. Understanding how sandboxes work, where they fail, and how to make them more effective is the skill that makes automated analysis operationally useful rather than a false confidence generator.

Sandbox Architectures

Sandbox Evasion — Why Reports Are Incomplete

Malware Clustering — Finding Variants at Scale

The preceding fourteen chapters covered individual techniques in isolation. This final chapter applies them together — showing what each family type looks like when you run the full Book 1 analysis pipeline, and how those findings flow directly into incident response actions. It also closes the loop on the career path: where malware analysis fits in security operations, and how to build the skill through structured practice.

Ransomware Triage — What Book 1 Finds

Stealer — What Book 1 Finds

The Analysis-to-Containment Loop

The value of malware analysis in an IR context is measured by the speed of the loop from finding to containment action. Analysis findings drive five categories of IR action:

• Network isolation — C2 domains and IPs block at firewall and proxy immediately; DNS sinkholing for domains the C2 might rotate to
• Endpoint hunting — YARA scans across all endpoints for the sample's static indicators; Sigma rules deployed to SIEM for behavioural indicators; Event ID hunting for specific actions the malware performs
• Containment — isolate affected endpoints identified by hunting; disable persistence mechanisms found in analysis (Run keys, services, scheduled tasks)
• Recovery — remove malware components at the paths identified in analysis; verify removal with YARA scan; restore from backup where data was destroyed
• Lessons learned — detection gaps identified in the ATT&CK coverage assessment become the next sprint's detection engineering work

Malware Analyst Career Path

Practice Resources

• MalwareBazaar (abuse.ch) — free, searchable database of malware samples. Filter by tag, file type, or family. The primary free source for real samples.
• any.run (free tier) — interactive sandbox — you control the mouse and keyboard inside the analysis environment. Excellent for samples requiring user interaction.
• theZoo (GitHub) — curated repository of live malware samples for educational use. Handle with care — these are real, functional malware binaries.
• Malware Traffic Analysis (malware-traffic-analysis.net) — exercises in the form of PCAP files and questions. Excellent for network analysis and IOC extraction practice.
• FLARE-VM malware analysis challenges — Mandiant's periodic challenge samples released publicly after competitions.
• Blue Team Labs Online / CyberDefenders — structured labs that include malware analysis scenarios with guided questions and scoring.