Before
- Read instructions line by line
- Trust decompiler variable names
- Treat missing functions as suspicious
Module 5 of 17
Assembly and Compiler Literacy
Read compiler output as evidence and recover a harmless program's control flow without guessing from syntax alone.
Start here
Learning objectives
- Recognize instructions, registers, stack frames, branches, calls, and return values in a small source-built utility.
- Explain how calling conventions connect function arguments, return values, and preserved state.
- Distinguish source behavior from compiler artifacts introduced by optimization, runtime startup, and library code.
- Annotate a control-flow graph with evidence, confidence, and unresolved questions.
After
- Map functions and basic blocks first
- Validate types through data use
- Explain compiler transformations with build context
Learn the Minimum Useful Machine Model
Reverse engineering becomes manageable when you stop treating assembly as a wall of instructions. A processor changes state: registers, memory, flags, and instruction position. Functions receive values according to a platform calling convention, perform calculations and memory access, call other functions, branch on conditions, and return a value.
Start with the few questions that support analysis. Where do arguments arrive? Which value is returned? Which registers must a function preserve? Which memory address comes from a local variable, a global, or a caller-provided pointer? Which comparison controls a branch? You do not need to memorize the entire instruction set before answering these questions.
The course uses only a transparent harmless utility that reads instructor-authored text from its fixed course directory and writes a summary there. It has no network feature, arbitrary target path, plugin system, privilege request, persistence, credential access, destructive behavior, or hidden capability.
Follow Data and Control Flow
Data flow tracks where a value comes from, how it changes, and where it is used. Control flow tracks which instruction or basic block may execute next. Mark function boundaries, calls, comparisons, conditional branches, loops, and returns. Then annotate important values at those points.
A basic block is a straight-line sequence with one entry and a transfer at the end. A control-flow graph connects these blocks. For a harmless parser, one branch might represent “input line is empty” and another “record is accepted.” Name a block after evidence supports its role, not after the first guess produced by a decompiler.
Pay special attention to bounds, lengths, signedness, and error paths because these often differ between source intent and machine behavior. In this course, the goal is understanding and secure review, not discovering or exploiting a vulnerability.
Use Calling Conventions as a Map
A calling convention defines how functions exchange arguments and return values and which registers a callee must preserve. It depends on architecture, operating system, application binary interface, compiler, and sometimes function attributes. Identify the correct convention before assigning meaning to a register.
Observe the call site and the called function together. Track argument preparation, stack alignment, indirect calls, return-value checks, and cleanup. Library calls can reveal broad intent, but names may be missing or wrapped. A familiar function name remains only one observation; confirm how its inputs and outputs are used.
Recognize Compiler Artifacts
Compilers transform source. Optimization can inline functions, remove variables, fold constants, reorder independent operations, reuse registers, replace loops with equivalent patterns, or remove branches that can never run. Runtime startup and safety features can add code that the developer did not write directly.
Compile the same harmless source with debug and release settings and compare the output. The difference teaches an important limit: a decompiler reconstructs a plausible high-level representation, not the original source. Variable names, comments, types, and exact control structures may be gone.
Build metadata also matters. Record compiler family and version, target architecture, optimization level, debug-symbol policy, and linked runtime. Without this context, a normal toolchain pattern may look suspicious.
Validate Every High-Level Claim
Use multiple views: source, compiler-generated assembly for the harmless utility, disassembly, symbols, control-flow graph, and observed synthetic test output. When a decompiler suggests a type or condition, confirm it against argument use, instruction width, branches, and known inputs.
Keep an annotation table with address or block ID, observation, interpretation, evidence, confidence, and open question. This prevents a guessed function name from becoming an unchallenged fact later in the report. Stop if the compiled hash or observed scope differs from the lab manifest.
Suggested Study Plan (8 hours)
The published duration includes active practice, not video playback alone. Complete each block with the course-owned evidence and retain the stated deliverable so another reviewer can reproduce your reasoning.
| Study block | Time | Required evidence |
|---|---|---|
| Guided lesson and primary-source review | 1h 30m | Annotated notes that separate observations, hypotheses, limits, and version-sensitive facts. |
| Worked evidence walkthroughs | 1h 15m | Reproduce the lesson's tables or decision flow and challenge at least two assumptions. |
| Independent practice rounds | 1h 15m | Apply the method to two alternate records in the sanitized evidence pack and compare the conclusions. |
| Required lab | 3 hours | A reproducible build record, debug-versus-release comparison, annotated control-flow graph, two data-flow traces, confidence notes, and a completed teardown record. |
| Knowledge check and review | 1 hour | Answer the evidence check, review the rubric, and record one production follow-up. |
Evidence Check
Question: The release build has no function matching a small helper visible in the source and debug build. Is this evidence that the release binary was altered?
Reveal the defensive reasoning
Not by itself. The optimizer may have inlined the helper, removed it, or transformed its control flow. Compare the compiler settings, call sites, resulting behavior, build provenance, and trusted-build hash before concluding that the artifact differs unexpectedly.
Primary References
Use these primary sources for the current standard or tool behavior. The course records framework versions so mappings can be reviewed when upstream guidance changes.
Real world
Where this shows up
- Confirming how an input-validation branch appears in a production build.
- Comparing a signed release binary with a reproducible trusted build.
- Explaining why a decompiler view differs from source during incident triage.
Production notes
Keep these close
- Record architecture, ABI, compiler, optimization, symbol, and runtime context with every interpretation.
- Keep source-to-binary provenance so analysts can compare trusted builds efficiently.
- Use isolated tooling even for parsing and disassembly because malformed artifacts can target analysis software.
- Review only the code regions needed to answer the authorised question before expanding scope.
Common mistakes
What usually breaks
- Assigning meaning to a register without identifying the architecture and calling convention.
- Assuming decompiled variable names and types are original or authoritative.
- Treating inlining, stripped symbols, or optimization as evidence of concealment.
- Following every instruction instead of first identifying functions and basic blocks.
Security risks
Threats to watch
- Incorrect type or calling-convention assumptions can reverse the meaning of a condition.
- Analysis tools process complex input and should be patched and isolated.
- Running a build outside the fixed behavior contract can widen the lab scope.
- A trusted source file does not prove an output binary unless build provenance is verified.
Tradeoffs
Design choices you should be able to defend
Debug build with symbols
Pros
- Easier mapping to source
- Clearer function and variable context
Cons
- Does not match production optimization
- May expose implementation details if distributed
Optimized release build
Pros
- Represents production artifact
- Shows real compiler transformations
Cons
- Harder to read
- Types and boundaries may be ambiguous
Think like an engineer
Questions to answer before shipping
- What evidence identifies the calling convention and function boundary?
- Which source construct disappeared because of optimization, and where is its behavior now represented?
- Which high-level type is inferred rather than proven by instruction width and use?
- Can the trusted build be reproduced and compared without executing the artifact?
Key terms
Vocabulary used in this module
Instruction
An architecture-defined operation that changes processor or memory state.
Calling convention
Rules for passing arguments, returning values, preserving registers, and maintaining stack layout across a function call.
Basic block
A straight-line instruction sequence with one entry and a control transfer at its end.
Control-flow graph
A graph whose nodes are basic blocks and whose edges represent possible execution paths.
Inlining
A compiler optimization that replaces a function call with an adapted copy of the called function's body.
Exercises
Practice inside the lesson
Recover the Control Flow of a Harmless Utility
Compile a transparent course utility inside an isolated VM and connect selected source constructs to debug and optimized machine code.
- Verify the supplied harmless source and build-instruction hashes, then read the complete behavior contract before compiling.
- Confirm the disposable VM has no network adapter, shared storage, host credentials, or access outside the fixed course directory.
- Build debug and release variants using the pinned compiler and fixed commands; record compiler version, options, and output hashes.
- Use compiler-generated assembly and a non-executing disassembler to identify function boundaries, arguments, branches, calls, loops, and return values.
- Draw a control-flow graph for the supplied parsing function and trace two instructor-authored input records through it.
- Run only the declared harmless test inside the VM, compare its fixed-directory summary with the predicted paths, export notes only, and restore the clean snapshot.
Expected evidence
A reproducible build record, debug-versus-release comparison, annotated control-flow graph, two data-flow traces, confidence notes, and a completed teardown record.
Assessment criteria
- Build provenance, compiler version, options, and output hashes are recorded.
- The control-flow graph correctly distinguishes calls, conditions, loops, and returns.
- Decompiler or disassembler interpretations are supported by specific instructions and data flow.
- Debug and optimized differences are explained without treating compiler transformation as malicious behavior.
- Execution stays inside the declared behavior contract and teardown is completed.
Course-owned resources
Teardown
- Export only the expected notes or synthetic logs.
- Power off the VM.
- Restore the clean snapshot and verify network isolation.
Recap
Key takeaways
- Assembly literacy begins with state, data flow, control flow, and calling conventions.
- Architecture and ABI context determine how registers and stack locations should be interpreted.
- Optimized output may look very different from source while preserving behavior.
- A decompiler provides a hypothesis, not the original program.
- Annotations should preserve evidence, confidence, and open questions.
Related resources