feat(scanner): add function prologue helper

[tkhquang]

Summary

Adds Scanner::is_likely_function_prologue(addr) for filtering scan poison after a cascade resolves. Cheap first-byte blacklist (0x00, 0xCC, 0xC2, 0xC3) gated by Memory::is_readable, with a null-address short-circuit. Accepts 0xE9 / 0xEB / 0xFF 0x25 so nested-hook scenarios (a sibling mod has already overwritten the prologue with a JMP trampoline) still resolve.

Test plan

• ScannerPrologueTest (8 cases) green: null, zero byte, int3 pad, both bare-RET forms, push rbp, patched JMP, PAGE_NOACCESS
• Full local suite green (1083/1083 on MinGW debug)
• CI green on MinGW + MSVC + ASan/UBSan presets

Docs

README scanner bullet, docs/misc/aob-signatures.md 7.3 and troubleshooting row updated to point at the helper. Also wrapped the previously-loose "Config Hot-Reload" section in a <details> block so it matches the other feature panels.

Summary by CodeRabbit

• New Features

  • Added prologue-validation helper to scanner API for enhanced scanning accuracy.

• Documentation

  • Updated AOB Scanner guidance with prologue-detection methodology.
  • Reorganized Config hot-reload section into collapsible documentation blocks.

• Tests

  • Added test coverage for prologue-detection functionality across multiple memory scenarios.

[coderabbitai]

📝 Walkthrough

Walkthrough

This PR introduces Scanner::is_likely_function_prologue(addr), a new public helper that filters AOB scan results by checking if an address contains a likely x86-64 function prologue byte. The function validates memory readability, rejects known non-prologue opcodes (0x00, 0xCC, 0xC2, 0xC3), and accepts valid prologues and already-hooked prologue patterns. Includes full header declaration, source implementation, comprehensive test coverage, and documentation updates.

Changes

Scanner Prologue Detection Helper

Layer / File(s)	Summary
Header declaration and contract include/DetourModKit/scanner.hpp	Public is_likely_function_prologue(std::uintptr_t addr) noexcept declared with Doxygen documentation covering readability checks, blacklisted opcodes, null-address handling, and unreadable-memory behavior.
Source implementation src/scanner.cpp	Function short-circuits to false for null/unreadable addresses; otherwise reads the first byte and returns false only for the opcode blacklist (0x00, 0xCC, 0xC2, 0xC3).
Test suite and fixtures tests/test_scanner.cpp	New ScannerPrologueTest group verifies false for null addresses, non-prologue bytes (padding, ret instructions), and inaccessible memory; verifies true for canonical prologue bytes (push rbp) and patched JMP indicators (0xE9).
Documentation and usage guidance README.md, docs/misc/aob-signatures.md	README adds prologue heuristic to AOB Scanner feature list and restructures Config hot-reload into a collapsible <details> block; AOB signatures guide updated to reference the new helper for prologue validation and crash-on-call troubleshooting.

🎯 2 (Simple) | ⏱️ ~12 minutes

Possibly Related PRs

• tkhquang/DetourModKit#69: Both PRs modify prologue-related Scanner detection heuristics and tests/test_scanner.cpp test coverage.

🚥 Pre-merge checks | ✅ 4

✅ Passed checks (4 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title 'feat(scanner): add function prologue helper' directly and clearly describes the main change: adding a new Scanner helper function for detecting function prologues.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[coderabbitai]

🧹 Nitpick comments (1)

tests/test_scanner.cpp (1)

2127-2204: ⚡ Quick win

Add explicit tests for 0xEB and 0xFF 0x25 accepted prologue forms.

The helper docs promise acceptance for short/indirect JMP-shaped starts, but this suite currently only pins the 0xE9 case. Adding both cases will prevent silent regressions in that documented contract.

Proposed test additions

+TEST(ScannerPrologueTest, PatchedJmpEbReturnsTrue)
+{
+    ExecBuffer buf(0x1000);
+    ASSERT_NE(buf.base, nullptr);
+    std::memset(buf.base, 0xCC, buf.size);
+    buf.base[0x100] = 0xEB;
+    EXPECT_TRUE(Scanner::is_likely_function_prologue(
+        reinterpret_cast<std::uintptr_t>(buf.base + 0x100)));
+}
+
+TEST(ScannerPrologueTest, PatchedJmpFf25ReturnsTrue)
+{
+    ExecBuffer buf(0x1000);
+    ASSERT_NE(buf.base, nullptr);
+    std::memset(buf.base, 0xCC, buf.size);
+    buf.base[0x100] = 0xFF;
+    buf.base[0x101] = 0x25;
+    EXPECT_TRUE(Scanner::is_likely_function_prologue(
+        reinterpret_cast<std::uintptr_t>(buf.base + 0x100)));
+}

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@tests/test_scanner.cpp` around lines 2127 - 2204, Add two new unit tests
mirroring the existing PatchedJmpE9ReturnsTrue case to cover short JMP 0xEB and
indirect JMP opcode pair 0xFF 0x25: create tests (e.g., PatchedJmpEBReturnsTrue
and PatchedJmpFF25ReturnsTrue) that allocate an ExecBuffer, fill with 0xCC,
write 0xEB at offset 0x100 for the short-jmp case and write 0xFF then 0x25 at
offset 0x100 for the indirect-jmp case, and assert
EXPECT_TRUE(Scanner::is_likely_function_prologue(reinterpret_cast<std::uintptr_t>(buf.base
+ 0x100))); to ensure the Scanner::is_likely_function_prologue behavior for
these documented prologue forms is covered.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@tests/test_scanner.cpp`:
- Around line 2127-2204: Add two new unit tests mirroring the existing
PatchedJmpE9ReturnsTrue case to cover short JMP 0xEB and indirect JMP opcode
pair 0xFF 0x25: create tests (e.g., PatchedJmpEBReturnsTrue and
PatchedJmpFF25ReturnsTrue) that allocate an ExecBuffer, fill with 0xCC, write
0xEB at offset 0x100 for the short-jmp case and write 0xFF then 0x25 at offset
0x100 for the indirect-jmp case, and assert
EXPECT_TRUE(Scanner::is_likely_function_prologue(reinterpret_cast<std::uintptr_t>(buf.base
+ 0x100))); to ensure the Scanner::is_likely_function_prologue behavior for
these documented prologue forms is covered.

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 6c755ee9-54f2-40f5-8f5f-a9a8ae6a7249

📥 Commits

Reviewing files that changed from the base of the PR and between 20c39de and a798784.

📒 Files selected for processing (5)

• README.md
• docs/misc/aob-signatures.md
• include/DetourModKit/scanner.hpp
• src/scanner.cpp
• tests/test_scanner.cpp