FileFlux

.NET document processing library for RAG systems

Overview

FileFlux is a .NET library that transforms various document formats into optimized chunks for RAG (Retrieval-Augmented Generation) systems. Built on high-performance Rust FFI libraries for document parsing.

Key Features

• 5-Stage Stateful Pipeline: Extract → Rule-Refine → LLM-Refine → Chunk → Enrich
• Native Document Readers: Rust FFI-based readers (Unpdf, Undoc, Unhwp) for 2-5x faster processing. Binaries are NuGet-pinned for reproducibility; runtime self-update from GitHub releases is opt-in (off by default — set UndocNativeLoader.AutoUpdateEnabled = true / UnhwpNativeLoader.AutoUpdateEnabled = true or the FILEFLUX_NATIVE_AUTOUPDATE=1 environment variable)
• Multiple Document Formats: PDF, DOCX, XLSX, PPTX, HWP, HWPX, Markdown, HTML, TXT, JSON, CSV
• Flexible Chunking Strategies: Auto, Smart, Intelligent, Semantic, Paragraph, FixedSize, Hierarchical, PageLevel
• Interface-Driven AI: Define AI service interfaces, implement with your preferred provider
• Document Graph: Inter-chunk relationship tracking with sequential, hierarchical, and semantic edges
• Structural Metadata: HeadingPath, page numbers, ContextDependency scores for enhanced RAG
• Language Detection: Automatic language detection using NTextCat
• IEnrichedChunk Interface: Standardized interface for RAG system integration
• Metadata Enrichment: AI-powered metadata extraction with caching and fallback
• Extensible Architecture: Interface-based design for easy customization
• Async Processing: Streaming and parallel processing for large documents

Installation

Full RAG Pipeline

dotnet add package FileFlux

Extraction Only (Minimal Dependencies)

dotnet add package FileFlux.Core

Package Comparison:

Feature	FileFlux.Core	FileFlux
Document Readers (PDF, DOCX, etc.)	✅	✅
Core Interfaces & Models	✅	✅
AI Service Interfaces	✅	✅
Chunking Strategies	❌	✅
FluxCurator & FluxImprover	❌	✅
DocumentProcessor	❌	✅
Use Case	Custom chunking	Full RAG pipeline

Quick Start

Basic Usage

usingFileFlux;usingMicrosoft.Extensions.DependencyInjection;varservices=newServiceCollection();// Optional: Register AI services for advanced features// services.AddScoped<IDocumentAnalysisService, YourLLMService>();// Register FileFlux services (no logger required)services.AddFileFlux();varprovider=services.BuildServiceProvider();varprocessor=provider.GetRequiredService<IDocumentProcessor>();// Process documentvarchunks=awaitprocessor.ProcessAsync("document.pdf");foreach(varchunkinchunks){Console.WriteLine($"Chunk {chunk.Index}: {chunk.Content}");}

Clean chunk content. Before a chunk is surfaced, FileFlux removes all HTML comments (<!-- ... -->) from chunk.Content. This covers the internal structural markers FileFlux emits and consumes during boundary detection (<!-- HEADING_START:H2 -->, <!-- TABLE_START -->, <!-- DOCUMENT_IMAGES_START -->, etc.), so consumers no longer need their own marker-removal step. Note that any HTML comment authored in your source document is also stripped from chunk content. When a chunk begins with a heading marker, its level (1-6) is preserved in chunk.Props[ChunkPropsKeys.HierarchyHeadingLevel].

Markdown link reference definitions ([label]: https://…) are likewise excluded from chunk content — they are document metadata, not body text. The referencing link keeps its display text and resolved target inline, so no content is lost.

Known limitation: if a chunker splits a marker across a chunk boundary (e.g. <!-- HEADING_ST | ART:H1 -->), neither half matches and both leak — identical to a downstream <!--.*?--> regex.

Streaming Processing

awaitforeach(varresultinprocessor.ProcessStreamAsync("document.pdf")){if(result.IsSuccess&&result.Result!=null){foreach(varchunkinresult.Result){Console.WriteLine($"Chunk {chunk.Index}: {chunk.Content.Length} chars");}}}

Chunking Options

varoptions=newChunkingOptions{Strategy="Auto",// Automatic strategy selectionMaxChunkSize=512,// Maximum chunk sizeOverlapSize=64// Overlap between chunks};varchunks=awaitprocessor.ProcessAsync("document.pdf",options);

Stateful Pipeline (v0.9.0+)

The new stateful pipeline provides explicit control over each processing stage:

usingFileFlux;usingFileFlux.Infrastructure.Factories;// Create processor via factoryvarfactory=provider.GetRequiredService<IDocumentProcessorFactory>();usingvarprocessor=factory.Create("document.pdf");// Execute stages explicitlyawaitprocessor.ExtractAsync();// Stage 1: Raw content extractionawaitprocessor.RefineAsync();// Stage 2: Rule-based text cleaningawaitprocessor.LlmRefineAsync();// Stage 3: LLM-powered refinement (optional)awaitprocessor.ChunkAsync();// Stage 4: Content chunkingawaitprocessor.EnrichAsync();// Stage 5: LLM-powered enrichment (optional)// Access results at each stageConsole.WriteLine($"State: {processor.State}");Console.WriteLine($"Raw text length: {processor.Result.Raw?.Text.Length}");Console.WriteLine($"Sections found: {processor.Result.Refined?.Sections.Count}");Console.WriteLine($"Chunks created: {processor.Result.Chunks?.Count}");// Or run full pipeline at onceawaitprocessor.ProcessAsync(newProcessingOptions{IncludeEnrich=true,Enrich=newEnrichOptions{BuildGraph=true}});// Access the document graphif(processor.Result.Graph!=null){Console.WriteLine($"Graph nodes: {processor.Result.Graph.NodeCount}");Console.WriteLine($"Graph edges: {processor.Result.Graph.EdgeCount}");}

Pipeline Stages:

Stage	Interface	AI	Description
Extract	IDocumentReader	❌	Raw content extraction from files
Rule-Refine	IDocumentRefiner	❌	Text cleaning, normalization, structure analysis
LLM-Refine	ILlmRefiner	✅	AI-powered noise removal, sentence restoration
Chunk	IChunkerFactory	Optional	Content segmentation with various strategies
Enrich	IDocumentEnricher	✅	LLM-powered summaries, keywords, contextual text

Metadata Enrichment

varoptions=newChunkingOptions{Strategy="Auto",MaxChunkSize=512,CustomProperties=newDictionary<string,object>{["enableMetadataEnrichment"]=true,["metadataSchema"]=MetadataSchema.General}};varchunks=awaitprocessor.ProcessAsync("document.pdf",options);// Access enriched metadataforeach(varchunkinchunks){varkeywords=chunk.Metadata.CustomProperties.GetValueOrDefault("enriched_keywords");vardescription=chunk.Metadata.CustomProperties.GetValueOrDefault("enriched_description");vardocumentType=chunk.Metadata.CustomProperties.GetValueOrDefault("enriched_documentType");varlanguage=chunk.Metadata.CustomProperties.GetValueOrDefault("enriched_language");}

AI Service Interfaces

FileFlux defines AI service interfaces - consumer applications provide implementations.

Available Interfaces

Interface	Purpose	Example Implementations
IDocumentAnalysisService	Text generation, intelligent chunking	OpenAI, Anthropic, LMSupply
IImageToTextService	Image captioning, OCR	OpenAI Vision, LMSupply Captioner/OCR
IEmbeddingService	Embedding generation	OpenAI, LMSupply Embedder

Example: Custom AI Provider

usingFileFlux;usingMicrosoft.Extensions.DependencyInjection;varservices=newServiceCollection();// Implement your own AI serviceservices.AddScoped<IDocumentAnalysisService,YourOpenAIService>();services.AddScoped<IImageToTextService,YourVisionService>();services.AddScoped<IEmbeddingService,YourEmbeddingService>();// Register FileFluxservices.AddFileFlux();varprovider=services.BuildServiceProvider();varprocessor=provider.GetRequiredService<IDocumentProcessor>();

Local AI with LMSupply (CLI Example)

For local AI processing without external API calls, see LMSupply. The FileFlux CLI demonstrates LMSupply integration:

// Example from FileFlux.CLI - local AI processingvarlmSupplyOptions=newLMSupplyOptions{UseGpuAcceleration=true,EmbeddingModel="default",GeneratorModel="microsoft/Phi-4-mini-instruct-onnx"};// Create LMSupply service implementationsvarembedder=awaitLMSupplyEmbedderService.CreateAsync(lmSupplyOptions);vargenerator=awaitLMSupplyGeneratorService.CreateAsync(lmSupplyOptions);// Register as AI service implementationsservices.AddSingleton<IEmbeddingService>(embedder);services.AddSingleton<IDocumentAnalysisService>(generator);services.AddFileFlux();

Note: LMSupply is not a direct dependency of FileFlux. Consumer applications that need local AI should reference LMSupply packages directly.

Supported Document Formats

Format	Extension	Reader	Features
PDF	.pdf	Unpdf (Rust FFI)	Text, tables, image extraction
Word	.docx	Undoc (Rust FFI)	Style and structure preservation
Excel	.xlsx	Undoc (Rust FFI)	Multi-sheet and table structure
Excel (legacy)	.xls	Built-in (ExcelDataReader)	BIFF binary workbooks; per-sheet markdown tables; CP949 (EUC-KR) fallback for codepage-less BIFF5/7

Mislabelled workbooks (since 0.17.0) — the two Excel readers route on the container's magic bytes rather than the declared extension, in both directions: a compound-file (.xls) workbook named .xlsx extracts through the legacy reader, and an OOXML package named .xls extracts through the OOXML one. RawContent.File.Extension reports the container that was actually parsed, not the name the file arrived under. Content that is neither container fails with extraction_failure_reason=container_mismatch instead of the ZIP parser's "could not find EOCD", which reads as corruption when the file is simply not a workbook. | PowerPoint | .pptx | Undoc (Rust FFI) | Slide and notes extraction | | HWP | .hwp, .hwpx | Unhwp (Rust FFI) | Native Korean document support | | Markdown | .md | Built-in | Structure preservation | | HTML | .html, .htm | Built-in | Web content extraction | | CSV/TSV | .csv, .tsv | Built-in (CsvHelper) | Header-aware markdown table serialization; UTF-8/BOM + CP949 (EUC-KR) fallback decoding | | Text | .txt, .json | Built-in | Basic text processing |

Known Limitations

PDF Processing

• Vector Graphics Tables: Tables created with drawing primitives (lines/rectangles) instead of text layout may not be detected. These are rendered as images in most PDF viewers.
• Complex Multi-column Layouts: Documents with intricate multi-column arrangements may have suboptimal text ordering.
• Scanned Documents: OCR is not included; scanned PDFs require pre-processing with external OCR tools. When a PDF parses but yields no text at all, the reader returns empty content with Hints["extraction_failure_reason"] set to "no_text_layer" (image-only/scanned — pages draw images without a readable text layer, via Unpdf page introspection) or "blank_page" (no text or image content at all), plus an explanatory warning — so consumers can classify these distinctly from parse errors.
• Partial Extraction: When whole-document extraction fails, FileFlux automatically falls back to per-page extraction. Pages that cannot be extracted are skipped and recorded in RawContent.Errors. RawContent.Status is set to ProcessingStatus.Partial when some pages fail, allowing RAG pipelines to use the successfully extracted content rather than losing the entire document.
• Parse Failures: extraction_failure_reason above covers documents that parse but yield no text; extraction_error_kind covers documents that fail to parse, naming Unpdf's structured failure classification (PdfParse, UnknownFormat, Encrypted, Corrupted, Io, MissingObject, …) so consumers can classify without matching on message prose. On partial extraction it arrives as Hints["extraction_error_kind"], listing every distinct kind seen across the skipped pages ("PdfParse+MissingObject"). When extraction fails entirely there is no RawContent to carry hints, so the same value is embedded in the thrown DocumentProcessingException.Message as a extraction_error_kind=<kind> token:

try{varcontent=awaitreader.ExtractAsync("scan.pdf");if(content.Hints.TryGetValue("extraction_error_kind",outvarkind))logger.LogWarning("Some pages unreadable: {Kind}",kind);// e.g. "PdfParse"}catch(DocumentProcessingExceptionex){// ex.Message contains "... [extraction_error_kind=Corrupted]"logger.LogError(ex,"PDF could not be parsed");}

A value from a newer native build passes through as its number rather than being dropped, so unknown kinds stay reportable. Kinds numbered 100 and above are raised at the native library's interop boundary rather than by the document, so a failure carrying one is a library-side problem to report upstream, not a defect in the file — the thrown message says so instead of filing it under parse failure.

• Incomplete Extraction: A damaged PDF does not always fail. If part of its page tree cannot be read, the parser recovers the rest and extraction succeeds over a shorter document. FileFlux flags that with Hints["pages_incomplete"] = true and RawContent.Status = ProcessingStatus.Partial, plus a warning — so a page that never arrived is not indexed as a page that never existed. Hints["declared_page_count"] carries the count the document declares, to compare against the extracted page_count. The flag is deliberately a boolean and never a loss figure: one unresolved page-tree node can cost a single page or a whole subtree, so the number of lost pages is not knowable.

varcontent=awaitreader.ExtractAsync("damaged.pdf");if(content.Hints.ContainsKey("pages_incomplete")){// declared_page_count is absent when the document's own declaration was unreadable —// itself a damage signal, so the flag can be set without a number to compare against.content.Hints.TryGetValue("declared_page_count",outvardeclared);logger.LogWarning("Indexing an incomplete document: declared {Declared}, extracted {Extracted}",declared??"unknown",content.Hints["page_count"]);}

ReadAsync (stage 0) carries the same signal in DocumentProps, with ReadResult.Status set to Partial — that stage reports the page count, so it is where a short page set most easily passes for a whole document.

Table Extraction

FileFlux uses layout-based table detection with confidence scoring:

• Tables with confidence score ≥ 0.5 are converted to Markdown format
• Low-confidence tables fall back to plain text to prevent garbled output
• Table quality metrics are exposed via StructuralHints for consumer applications

Document-Specific Notes

• Excel: Very large worksheets (>100K rows) may impact memory usage
• PowerPoint: Embedded objects are extracted as placeholder text
• HTML: JavaScript-rendered content is not supported

Chunking Strategies

Strategy	Output characteristics	Prerequisites
Auto (default)	Resolved to a concrete strategy by content analysis: short text → Sentence, 4+ paragraphs → Paragraph, sentence-structured → Sentence, otherwise Token. The resolved strategy is logged and recorded in each chunk's Strategy.	—
Sentence	Sentence-boundary chunks, language-aware	—
Paragraph	Paragraph-boundary chunks; best for Markdown/blogs; oversized paragraphs fall back to sentence splits	—
Token	Token-budget chunks for unstructured text	—
Hierarchical	Heading-structure-aware chunks	—
Semantic	Embedding-similarity boundaries	Requires an IEmbedder registered beforeAddFileFlux() — otherwise chunker creation throws ArgumentException

Structural metadata: every ProcessAsync/ChunkAsync chunk carries Location.StartChar/EndChar (offsets into the refined text), Location.HeadingPath/Section (hierarchical heading context, e.g. Root Title > Sub Section), and Props["HierarchyPath"]. Location.StartPage/EndPage are currently populated only on the legacy batch path for PDF page ranges.

AI Service Integration

FileFlux defines interfaces while implementation is up to the consumer application.

// Optional: Register AI services for advanced features// - IDocumentAnalysisService: For intelligent chunking and metadata enrichment// - IImageToTextService: For multimodal document processingservices.AddScoped<IDocumentAnalysisService,YourLLMService>();services.AddScoped<IImageToTextService,YourVisionService>();// Register FileFlux services (works without AI services too)services.AddFileFlux();

Note: Logger registration is optional. FileFlux uses NullLogger internally if no logger is provided.

For AI service implementation examples, see the samples/ directory.

Advanced Features

🤖 AI Integration (Optional)

FileFlux defines interfaces - YOU implement them with your preferred AI provider.

// Register your AI service implementationservices.AddScoped<IDocumentAnalysisService,YourAIService>();services.AddFileFlux();

Features enabled with AI services:

• Intelligent structure analysis for optimal chunking
• Semantic content summarization
• AI-powered quality assessment
• Q&A benchmark generation for RAG testing

📖 See Tutorial for AI service implementation examples.

📊 Quality Analysis

Evaluate and optimize chunking quality for RAG systems:

varanalyzer=serviceProvider.GetRequiredService<IDocumentQualityAnalyzer>();// Analyze document qualityvarreport=awaitanalyzer.AnalyzeQualityAsync("document.pdf");Console.WriteLine($"Quality Score: {report.OverallQualityScore:P2}");// Generate Q&A benchmark for RAG testingvarbenchmark=awaitanalyzer.GenerateQABenchmarkAsync("document.pdf",questionCount:20);// Compare different chunking strategiesvarstrategies=new[]{"Intelligent","Semantic","Smart"};varcomparison=awaitanalyzer.BenchmarkChunkingAsync("document.pdf",strategies);

📖 See Architecture for quality analysis details.

🔧 Dependency Injection

FileFlux works with or without AI services:

// Minimal setup (no AI)services.AddFileFlux();// With AI serviceservices.AddScoped<IDocumentAnalysisService,YourAIService>();services.AddFileFlux();// Environment-specific configurationif(Environment.IsDevelopment())services.AddScoped<IDocumentAnalysisService,MockTextCompletionService>();elseservices.AddScoped<IDocumentAnalysisService,ProductionAIService>();services.AddFileFlux();

📖 See Tutorial for more DI patterns and examples.

Documentation

• Tutorial - Detailed usage guide and examples
• Architecture - System design and pipeline documentation
• Changelog - Version history and release notes

Project Structure

FileFlux/
├── src/
│ ├── FileFlux.Core/ # Extraction only (zero AI dependencies)
│ │ ├── Contracts/ # IDocumentProcessor, ProcessingResult
│ │ ├── Core/ # IDocumentRefiner, IDocumentEnricher
│ │ └── Domain/ # DocumentGraph, RefinedContent, StructuredElement
│ └── FileFlux/ # Full RAG pipeline (interface-driven)
│ └── Infrastructure/ # StatefulDocumentProcessor, DocumentRefiner, DocumentEnricher
├── cli/ # CLI with LMSupply integration (published: `dotnet tool install -g FileFlux.CLI`)
│ └── FileFlux.CLI/
│ └── Services/LMSupply/ # LMSupply service implementations
├── tests/
│ └── FileFlux.Tests/ # Test suite (343+ tests)
└── samples/
└── FileFlux.SampleApp/ # Usage examples

Contributing

1. Create and discuss an issue
2. Work on a feature branch
3. Add/modify tests
4. Submit a pull request

License

MIT License - See LICENSE file

Support

• Issue Reports: GitHub Issues
• Feature Requests: GitHub Discussions