Skip to content

Repository files navigation

 ░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░
░ ░
░ ███████╗████████╗ █████╗ ██████╗ ██████╗ ██╗ ██╗ ░
░ ██╔════╝╚══██╔══╝██╔══██╗██╔══██╗██╔══██╗╚██╗ ██╔╝ ░
░ ███████╗ ██║ ███████║██████╔╝██████╔╝ ╚████╔╝ ░
░ ╚════██║ ██║ ██╔══██║██╔══██╗██╔══██╗ ╚██╔╝ ░
░ ███████║ ██║ ██║ ██║██║ ██║██║ ██║ ██║ ░
░ ╚══════╝ ╚═╝ ╚═╝ ╚═╝╚═╝ ╚═╝╚═╝ ╚═╝ ╚═╝ ░
░ N O T E ░
░ ░
░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░

Cybernetic Knowledge Architecture System

Transform raw academic chaos into structured, exam-ready study guides — powered by Gemma 3 running locally on Apple Silicon.

PythonMLXGemma 3Rich TUITestsLicense


📋 Table of Contents


🌟 What Is StarryNote?

StarryNote is a local-first, AI-powered knowledge synthesis engine that transforms raw study materials — lecture notes, code files, PDFs, screenshots — into professional-grade, structured study guides with zero cloud dependency.

Unlike generic summarizers, StarryNote acts as a Knowledge Architect: it doesn't just restate your input — it synthesizes original code examples, mathematical proofs, Mermaid diagrams, and exam questions that explain the source material at a deeper level.

The Philosophy:Your notes are fragments. StarryNote turns them into architecture.

Why StarryNote?

ProblemStarryNote's Solution
Notes are scattered across formatsUniversal MIME scanner processes any file type
AI summaries are surface-levelKnowledge Architect prompt forces synthesis > summary
Cloud AI raises privacy concernsRuns 100% locally on Apple Silicon via MLX
Output varies wildlyMaster Template enforces consistent, exam-ready output
No way to self-assessMetacognitive Calibration with confidence meters
LLM output has rendering bugsTriple-layer PostProcessor auto-fixes every output

✨ Key Features

🧠 AI Engine

  • Gemma 3 4B-IT running natively on Metal GPU
  • Multimodal: processes text, images, and PDFs
  • OCR fallback for scanned/image-based PDFs
  • Knowledge Architect prompt with 5 core directives
  • 8,192 token budget for complete 10-section output

🛡️ Post-Processing Pipeline

  • MermaidFixer: Auto-injects cyberpunk classDef, removes semicolons, replaces forbidden diagram types
  • OutputCleaner: Strips leaked AI instructions and unfilled placeholders
  • OutputValidator: Checks all 10 sections, Mermaid diagrams, exam questions
  • Triple-layer defense guarantees clean output

📜 10-Section Master Template

  • Executive Summary · Concept Register
  • Cyberpunk Mermaid diagrams (auto-styled)
  • 3-tier exam questions (Apply → Analyze → Synthesize)
  • Quick Reference Card · Metacognitive Calibration

🔍 Universal Scanner

  • DFS directory traversal with directory pruning
  • MIME-based detection (not file extensions)
  • Auto-skips .venv, __pycache__, .git, etc.
  • ScanResult with file stats and error tracking

🖥️ Cyberpunk Terminal UI

  • Large ASCII hero banner in neon purple
  • 4-phase pipeline with animated spinners
  • Resource discovery table with MIME icons
  • Knowledge Density star rating (✦ to ✦✦✦✦✦)

🧪 382 Unit Tests

  • 12 test files covering every module
  • 50+ MIME types classified and routing-tested
  • Edge cases: symlinks, empty files, Unicode, large content
  • Realistic dirty LLM output simulation
  • Full traceability matrix (75 requirements → 382 tests)

🏗️ System Architecture

graph TD
classDef default fill:#1a1a1a,stroke:#bc13fe,stroke-width:2px,color:#00f3ff
classDef highlight fill:#2a0a3a,stroke:#00f3ff,stroke-width:2px,color:#bc13fe
classDef input fill:#1a1a1a,stroke:#ff6ec7,stroke-width:2px,color:#ff6ec7
classDef output fill:#1a1a1a,stroke:#39ff14,stroke-width:2px,color:#39ff14
A["📂 Raw Study Materials"]:::input --> B["🔍 StarryScanner<br/>MIME Detection · DFS Walk"]
B --> C{"File Type Router"}
C -->|"image/*"| D["🖼️ Image Analyzer<br/>PIL · Multimodal"]
C -->|"application/pdf"| E["📄 PDF Analyzer<br/>PyMuPDF · OCR"]
C -->|"text/*"| F["📝 Text Analyzer<br/>UTF-8 Read"]
D --> G["🧠 Gemma 3 Engine"]:::highlight
E --> G
F --> G
G --> H["📐 PromptBuilder<br/>System Rules + Template"]:::highlight
H --> I["🛡️ PostProcessor<br/>Mermaid Fix · Clean · Validate"]:::highlight
I --> J["💾 StarryFormatter<br/>Instructions/ Output"]
J --> K["📘 Study Guides"]:::output
Loading

Module Dependency Graph

graph LR
classDef default fill:#1a1a1a,stroke:#bc13fe,stroke-width:2px,color:#00f3ff
classDef highlight fill:#2a0a3a,stroke:#00f3ff,stroke-width:2px,color:#bc13fe
main[main.py] --> engine[StarryEngine]
main --> scanner[StarryScanner]
main --> formatter[StarryFormatter]
engine --> tl[TemplateLoader]:::highlight
engine --> pb[PromptBuilder]:::highlight
engine --> pp[PostProcessor]:::highlight
formatter --> pp
pp --> mf[MermaidFixer]
pp --> oc[OutputCleaner]
pp --> ov[OutputValidator]
Loading

Data Flow

sequenceDiagram
participant U as 👤 User
participant M as main.py<br/>TUI Hub
participant S as StarryScanner
participant E as StarryEngine
participant PB as PromptBuilder
participant G as Gemma 3<br/>MLX Metal
participant PP as PostProcessor
participant F as StarryFormatter
U->>M: python main.py
M->>E: Initialize (load model)
E->>G: Load weights into Unified Memory
G-->>E: Model ready
M->>S: scan(cwd)
S-->>M: ScanResult{resources, stats}
loop For each resource
M->>E: process_resource(resource)
E->>PB: build(template, content)
PB-->>E: Complete prompt
E->>G: stream_generate(prompt)
G-->>E: Raw Markdown
E->>PP: PostProcessor.process(raw)
PP-->>E: Clean Markdown
E-->>M: guide_content
M->>F: save_guide(path, content)
F->>PP: PostProcessor.process(content)
F-->>M: output_path
end
M-->>U: Mission Report + Constellation
Loading

📁 Project Structure

StarryNote/
├── main.py # 🖥️ TUI entry point (4-phase pipeline)
├── requirements.txt # 📦 Python dependencies
├── README.md # 📖 You are here
├── .gitignore # 🚫 Git exclusion rules
│
├── src/ # ⚙️ Core engine modules (6 files, 10 classes)
│ ├── __init__.py # Package initializer
│ ├── scanner.py # 🔍 UniversalResource + ScanResult + StarryScanner
│ ├── template_loader.py # 📐 Template I/O, cleaning, and compaction
│ ├── prompt_builder.py # 🤖 Knowledge Architect prompt construction
│ ├── model_engine.py # 🧠 MimeClassifier + TextExtractor + StarryEngine
│ ├── postprocessor.py # 🛡️ MermaidFixer + OutputCleaner + OutputValidator
│ └── formatter.py # 💾 Post-process + save to Instructions/
│
├── templates/ # 📐 AI output templates
│ └── master_template.md # 📜 10-section study guide scaffold
│
├── tests/ # 🧪 Test suite (382 tests across 12 files)
│ ├── __init__.py # Package initializer
│ ├── test_engine.py # 🔬 StarryEngine prompt + routing tests (22)
│ ├── test_file_types.py # 🔬 MimeClassifier + TextExtractor + routing (92)
│ ├── test_postprocessor.py # 🔬 MermaidFixer + Cleaner + Validator (27)
│ ├── test_prompt_builder.py # 🔬 PromptBuilder rules tests (14)
│ ├── test_template_loader.py # 🔬 TemplateLoader I/O tests (14)
│ ├── test_template.py # 🔬 Master template structure tests (33)
│ ├── test_formatter.py # 🔬 Formatter + post-processing tests (15)
│ ├── test_scanner.py # 🔬 Scanner + ScanResult tests (22)
│ ├── test_edge_cases.py # 🔬 Cross-module edge cases (19)
│ ├── test_tui.py # 🔬 TUI utility + animation tests (112)
│ ├── test_model.py # 🔬 GPU + metal validation (1, requires GPU)
│ ├── test_universal_scanner.py # 🔬 Integration smoke test (1)
│ └── sample_note.txt # 📝 Test fixture
│
├── docs/ # 📚 Documentation
│ ├── TestLog.md # 📋 Complete test execution log
│ ├── TraceabilityMatrix.md # 🔗 Requirements → Code → Tests mapping
│ └── FunctionExplanations.md # 📖 Detailed function documentation
│
├── .github/ # 🤖 CI/CD
│ └── workflows/
│ └── main.yml # ▶️ GitHub Actions: pytest on push/PR
│
├── models/ # 🗄️ MLX model weights (auto-downloaded, gitignored)
└── Instructions/ # 📘 Generated study guides (created at runtime)

⚡ Prerequisites

RequirementMinimumRecommended
macOS13.0 (Ventura)14.0+ (Sonoma)
ChipApple M1Apple M3 / M4
RAM8 GB Unified16 GB+ Unified
Python3.113.12+
Disk~5 GB (model weights)10 GB+
libmagicRequiredbrew install libmagic

⚠️Apple Silicon Required. StarryNote uses MLX, Apple's Metal-optimized ML framework. It will not run on Intel Macs or Linux/Windows without modifying the engine.


🚀 Installation

1. Clone the Repository

git clone https://github.com/NikanEidi/StarryNote.git
cd StarryNote

2. Install System Dependencies

# libmagic is required for MIME type detection
brew install libmagic

3. Create & Activate Virtual Environment

python3 -m venv .venv
source .venv/bin/activate

4. Install Python Dependencies

pip install -r requirements.txt

5. Verify GPU Access

python -c "import mlx.core as mx; print(f'Metal GPU: {mx.metal.is_available()}')"# Expected output: Metal GPU: True

💡 First Run Note: Gemma 3 weights (~5 GB) are downloaded automatically from Hugging Face on the first execution. Subsequent runs load from cache.


🎯 Usage

Basic Usage

Navigate to any directory containing study materials, then run:

cd /path/to/your/study/materials
python /path/to/StarryNote/main.py

Or from the StarryNote directory itself:

python main.py

What Happens

  1. ⚡ Phase 1 — Neural Initialization: Loads Gemma 3 into Apple Silicon's unified memory
  2. 🔍 Phase 2 — Deep Scan: DFS traversal discovering all files via MIME detection
  3. 🧠 Phase 3 — Knowledge Synthesis: Processes each file through the Knowledge Architect pipeline
  4. 📊 Phase 4 — Mission Report: Displays results table with timing and density ratings

Output

Study guides are saved to an Instructions/ folder in the current working directory:

Instructions/
├── lecture_notes_StudyGuide.md
├── algorithm_code_StudyGuide.md
└── exam_review_StudyGuide.md

Every saved guide is automatically post-processed — Mermaid diagrams are fixed, leaked instructions are stripped, and output is validated.


🔬 Pipeline Deep Dive

The Scanner (src/scanner.py)

graph LR
classDef default fill:#1a1a1a,stroke:#bc13fe,stroke-width:2px,color:#00f3ff
A["os.walk()"] --> B["python-magic<br/>MIME detection"]
B --> C{"Classify"}
C -->|"image/jpeg"| D["🖼️ UniversalResource"]
C -->|"application/pdf"| E["📄 UniversalResource"]
C -->|"text/x-python"| F["🐍 UniversalResource"]
C -->|"text/plain"| G["📝 UniversalResource"]
Loading

The StarryScanner uses libmagic to read binary headers and determine the true MIME type. Each file is packaged into a UniversalResource dataclass:

@dataclassclassUniversalResource:
file_path: str# Absolute path to the filemime_type: str# e.g., 'image/jpeg', 'application/pdf'raw_data: Any# Path reference for downstream processingsize_bytes: int=0# File size in bytes

The enhanced scan() method returns a ScanResult with full statistics:

result=scanner.scan("/path/to/notes")
print(f"Found {result.count} files, {result.total_bytes} bytes")
print(f"Skipped {result.skipped_count}, Errors: {result.error_count}")

The Engine (src/model_engine.py)

The engine routes each UniversalResource through the appropriate analyzer:

MIME TypeAnalyzerStrategy
image/*_analyze_image()PIL → RGB conversion → multimodal prompt
application/pdf_analyze_pdf()PyMuPDF text extraction → OCR fallback if <100 chars
text/*_analyze_text()Direct content injection into prompt

All three analyzers run PostProcessor.process() on the raw output before returning.

The Formatter (src/formatter.py)

  • Creates Instructions/ directory at the current working directory
  • Generates filenames: {original_name}_StudyGuide.md
  • Automatically post-processes every guide before saving (Mermaid fixing, instruction stripping)
  • Provides validate_guide() for checking structural completeness of saved files

📜 The Master Template

Every generated study guide follows a strict 10-section structure:

graph TD
classDef default fill:#1a1a1a,stroke:#bc13fe,stroke-width:2px,color:#00f3ff
classDef highlight fill:#1a1a1a,stroke:#39ff14,stroke-width:2px,color:#39ff14
A["I. Executive Summary"] --> B["II. Core Concepts"]
B --> C["III. Visual Knowledge Graph"]
C --> D["IV. Technical Deep Dive"]
D --> E["V. Annotated Glossary"]
E --> F["VI. Exam Preparation"]
F --> G["VII. Knowledge Connections"]
G --> H["VIII. Quick Reference Card"]:::highlight
H --> I["IX. Metacognitive Calibration"]:::highlight
I --> J["X. Source Archive"]
Loading

Section Breakdown

#SectionPurposeUnique Feature
IExecutive SummaryAbstract + Central Thesis + Applied ContextForces non-obvious insight extraction
IICore ConceptsConcept Register table + Comparative AnalysisRequires specific "Common Pitfall" per concept
IIIVisual Knowledge GraphAuto-generated Mermaid diagramCyberpunk styling: #bc13fe stroke, #00f3ff text
IVTechnical Deep DiveCode (CS) / LaTeX (Math) / Source Analysis (Humanities)Auto-selects block type by subject classification
VAnnotated GlossaryDomain terms with etymology & related termsRequires linguistic root for scientific terms
VIExam Preparation3-tier questions: Application → Analysis → SynthesisCollapsible answers with reasoning chains
VIIKnowledge ConnectionsDependencies, next topics, cross-domain linksMaps learning pathways
VIIIQuick Reference CardCondensed cheat sheet: takeaways + formulas + trapsPre-exam checklist
IXMetacognitive CalibrationConfidence Meter (🔴🟡🟢🔵) per conceptPersonalized study prescriptions
XSource ArchiveVerbatim original input (read-only)Audit trail for review

🛡️ Post-Processing Pipeline

StarryNote uses a triple-layer defense to guarantee clean output regardless of what the LLM generates:

graph LR
classDef default fill:#1a1a1a,stroke:#bc13fe,stroke-width:2px,color:#00f3ff
classDef highlight fill:#2a0a3a,stroke:#00f3ff,stroke-width:2px,color:#bc13fe
A["Raw LLM Output"] --> B["OutputCleaner<br/>Strip leaked instructions"]:::highlight
B --> C["MermaidFixer<br/>Fix diagrams + inject classDef"]:::highlight
C --> D["OutputValidator<br/>Check sections + warnings"]:::highlight
D --> E["Clean Study Guide"]
Loading

Layer 1: PromptBuilder (Prevention)

All rules are baked into the system prompt — the model is instructed to generate clean output from the start.

Layer 2: PostProcessor (Correction)

Even if the LLM ignores the rules, PostProcessor.process() auto-fixes the output:

FixerWhat It Does
OutputCleanerStrips <!-- AI INSTRUCTION -->, [[AI INSTRUCTION]], **RULES:**, unfilled {{PLACEHOLDERS}}
MermaidFixerReplaces sequenceDiagram/mindmap/classDiagramgraph TD, injects cyberpunk classDef, removes ; and inline style
OutputValidatorLogs warnings for missing sections, missing mermaid, missing exam questions

Layer 3: Formatter (Final Gate)

StarryFormatter.save_guide() runs the full PostProcessor pipeline again before writing to disk — the final safety net.


🤖 Knowledge Architect Prompt

The AI follows 4 Core Directives defined in src/prompt_builder.py:

DirectiveRule
AUTHORSHIPSet Author to "S T A R R Y N O T E"
SYNTHESIS > SUMMARYCreate original examples, proofs, and diagrams — don't just repeat the input
FORMATTINGFollow the Master Template exactly, generate ALL 10 sections
ACADEMIC TONEScholarly, precise, no conversational filler

Plus section-specific rules for each of the 10 sections, Mermaid rules with exact classDef values, and explicit output rules forbidding HTML comments and instruction markers.


🖥️ Terminal UI

StarryNote's TUI is built with Rich and follows a 4-phase pipeline design:

Phase Layout

PhaseNameVisual Elements
⚡ 1Neural InitializationAnimated spinner while loading Gemma 3 into unified memory
🔍 2Deep ScanResource table with MIME icons (🐍🖼📄📝📦), file sizes
🧠 3Knowledge SynthesisProgress bar per file + overall, elapsed time, density rating
📊 4Mission ReportResults table, summary panel, constellation footer

Knowledge Density Rating

Measures AI amplification — how much original content the AI generated relative to the input size:

RatingRatioMeaning
< 1×Minimal expansion
✦✦1–2×Moderate synthesis
✦✦✦3–4×Strong synthesis
✦✦✦✦5–7×Deep synthesis
✦✦✦✦✦8×+Maximum amplification

🧪 Testing

Run All Tests

source .venv/bin/activate
pytest tests/ -v

Test Summary

FileTestsWhat It Covers
test_engine.py22Engine prompt building, MIME routing, token budget
test_file_types.py92MimeClassifier (50+ MIME types), TextExtractor (all readers), routing (24 formats)
test_postprocessor.py27MermaidFixer, OutputCleaner, OutputValidator, pipeline
test_prompt_builder.py24All rules, Mermaid classDef, structural rules, table format rules
test_template_loader.py14Template I/O, clean, compact, recovery mode
test_template.py33Master template structure, sections, placeholders
test_formatter.py15Save, naming, UTF-8, post-processing integration
test_scanner.py22Resources, ScanResult, filtering, errors
test_edge_cases.py19Symlinks, Unicode, nested dirs, realistic dirty output
test_tui.py112Icons, sizing, density, starfield, glitch, matrix rain, waveform, orbital, neon pulse, gradient bar, design system
test_model.py1GPU validation (requires Apple Silicon)
test_universal_scanner.py1Integration smoke test
TOTAL382100% pass rate

CI/CD

GitHub Actions runs pytest tests/ on every push to main/master and on pull requests. See .github/workflows/main.yml.

⚠️Note:test_model.py requires Apple Silicon with Metal GPU — it will skip in CI (Ubuntu runner).


📚 Documentation

DocumentPathDescription
Test Logdocs/TestLog.mdComplete test execution results with all 196 tests
Traceability Matrixdocs/TraceabilityMatrix.mdMaps 53 requirements → implementations → 196 tests
Function Explanationsdocs/FunctionExplanations.mdDetailed documentation of every class and method

⚙️ Configuration

Model Selection

Change the model in src/model_engine.py:

engine=StarryEngine(model_path="google/gemma-3-4b-it") # Defaultengine=StarryEngine(model_path="google/gemma-3-12b-it") # Larger (needs 16GB+ RAM)

Max Token Output

Adjust MAX_TOKENS in src/model_engine.py:

MAX_TOKENS=8192# Default — full 10-section guideMAX_TOKENS=12000# Longer, more detailed guides

Skip Patterns

Customize skip patterns in src/scanner.py:

scanner=StarryScanner(skip_patterns={
"Instructions", ".venv", "__pycache__", ".git",
".DS_Store", ".idea", "node_modules",
})

🤝 Contributing

  1. Fork the repository
  2. Create a feature branch: git checkout -b feature/my-feature
  3. Commit with clear messages: git commit -m "feat: add X"
  4. Push to your fork: git push origin feature/my-feature
  5. Open a Pull Request

Code Style

black src/ main.py tests/

Test Before Pushing

pytest tests/ -v
# All 382 tests should pass

📊 Tech Stack

graph LR
classDef default fill:#1a1a1a,stroke:#bc13fe,stroke-width:2px,color:#00f3ff
subgraph "AI Layer"
A["Gemma 3 4B-IT"] --> B["MLX Framework"]
B --> C["Metal GPU"]
end
subgraph "Processing Layer"
D["python-magic"] --> E["StarryScanner"]
F["PyMuPDF"] --> G["PDF Analyzer"]
H["Pillow"] --> I["Image Analyzer"]
end
subgraph "Safety Layer"
J["MermaidFixer"] --> K["PostProcessor"]
L["OutputCleaner"] --> K
M["OutputValidator"] --> K
end
subgraph "Presentation Layer"
N["Rich"] --> O["Cyberpunk TUI"]
P["Master Template"] --> Q["Markdown Output"]
end
E --> A
G --> A
I --> A
A --> P
A --> K
K --> Q
Loading

🏗️ Module Architecture

ModuleClassesResponsibility
scanner.pyUniversalResource, ScanResult, StarryScannerDFS file discovery, MIME detection, skip filtering, stats
template_loader.pyTemplateLoaderTemplate I/O, cleaning, compaction, recovery mode
prompt_builder.pyPromptBuilderSystem prompt with all rules (single source of truth)
model_engine.pyMimeClassifier, TextExtractor, StarryEngineMIME classification, universal file reading, LLM orchestration
postprocessor.pyMermaidFixer, OutputCleaner, OutputValidator, PostProcessorOutput sanitization pipeline
formatter.pyStarryFormatterPost-process + save to disk + validation

 ─────────────────────────────────────────────────────────────────────────────
S T A R R Y N O T E · Knowledge Architecture System · v2.1
Gemma 3 · Apple Silicon · MLX · 382 Tests · 12 Classes
Structured for clarity. Engineered for mastery. Calibrated for you.
─────────────────────────────────────────────────────────────────────────────

Made with ✦ by Nikan Eidi

About

A local-first, AI-powered knowledge architecture system that transforms raw study materials into professional-grade study guides using Gemma 3 on Apple Silicon.

Topics

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages