xtotext - AI-Ready Document Conversion System

A powerful document conversion system that transforms any document into AI-friendly text formats, designed for seamless integration with Large Language Models and AI workflows, with document storage and management capabilities.

Features

• Universal Document Conversion: Transform documents (PDF, DOCX, LaTeX, Markdown, etc.) into AI-optimized text formats
• AI-Ready Output: Structured text output specifically formatted for LLM consumption and analysis
• Document Storage: Securely store and manage documents in Azure Data Lake Storage Gen2
• Permission-Based Access: Fine-grained user permissions for document access and conversion
• LaTeX to PDF Conversion: Convert LaTeX documents to PDF format with error handling
• Repository Integration: Batch convert entire project documentation into single AI-friendly files
• Smart Context Preservation: Maintain document structure and relationships for better AI understanding
• API-First Design: RESTful API designed for AI tool integration and automation workflows
• Proper Package Structure: Follows Python packaging best practices
• Type Hints: Full type annotation support
• CLI Interface: Easy-to-use command-line tool
• Modular Design: Separate core, utils, and workflow modules

Project Structure

xtox/
├── xtox/ # Main package
│ ├── __init__.py # Package initialization
│ ├── core/ # Core conversion functionality
│ │ ├── __init__.py
│ │ ├── document_converter.py # Main converter class
│ │ ├── markdown_to_latex.py # Markdown to LaTeX conversion
│ │ └── latex_to_pdf.py # LaTeX to PDF conversion
│ ├── utils/ # Utility functions
│ │ ├── __init__.py
│ │ └── image_handler.py
│ ├── workflows/ # High-level workflows
│ │ ├── __init__.py
│ │ └── md_to_pdf.py
│ ├── cli/ # Command-line interface
│ │ ├── __init__.py
│ │ └── main.py
│ ├── api/ # API routes
│ ├── backend/ # Backend services
│ ├── azure-functions/ # Azure Functions
│ └── frontend/ # React frontend
├── tests/ # Test files
├── infra/ # Infrastructure code
├── setup.py # Package setup
├── pyproject.toml # Modern Python project config
├── requirements.txt # Dependencies
├── Makefile # Development tasks
└── README.md # This file

Architecture

This project consists of:

1. Backend: Azure Functions for serverless document processing, storage, retrieval, and conversion
2. Frontend: React application with AI-focused conversion interfaces
3. Storage: Azure Data Lake Storage Gen2 for document and converted text storage
4. Database: MongoDB for metadata, conversion history, and AI context mapping
5. AI Conversion Engine: Specialized pipeline for creating LLM-optimized text outputs
6. Repository Processor: Batch conversion system for entire codebases and documentation sets

Prerequisites

• Azure subscription (for cloud deployment)
• Azure CLI or PowerShell Az module
• Node.js 14+
• MongoDB
• Python 3.9+
• AI conversion dependencies (transformers, tiktoken, etc.)

Installation

Development Installation

# Clone the repository
git clone <repository-url>cd xtox
# Install in development mode with all dependencies
make setup
# Or manually:
pip install -e ".[dev,azure,api]"

Production Installation

pip install -e .

Usage

Command Line Interface

# Convert Markdown to PDF
xtotext input.md -o output_dir -r 2
# Convert LaTeX to PDF
xtotext document.tex -o output_dir
# Show help
xtotext --help

Python API

fromxtox.coreimportDocumentConverter# Initialize converterconverter=DocumentConverter(output_dir="./output")
# Convert Markdown to PDFresult=converter.markdown_to_pdf(
"document.md", refinement_level=2
)
# Convert LaTeX to PDFpdf_path=converter.latex_to_pdf("document.tex")

Using Workflows

fromxtox.workflowsimportprocess_markdown_to_pdfresult=process_markdown_to_pdf(
"document.md",
output_dir="./output",
refinement_level=1
)

API Endpoints

AI-Focused Conversion

• POST /api/ai/convert - Convert document for AI consumption
• POST /api/ai/repository - Process entire repository for AI
• GET /api/ai/context/{id} - Get document with full context
• POST /api/ai/optimize - Optimize existing text for specific AI models

Document Management

• POST /api/documents/upload - Upload a document
• GET /api/documents - List documents available to the user
• GET /api/documents/{id} - Get document metadata
• GET /api/documents/{id}/download - Download document
• POST /api/documents/{id}/permissions - Update document permissions
• DELETE /api/documents/{id} - Delete a document

Document Conversion

• POST /api/convert - Convert LaTeX to PDF
• GET /api/conversion/{id} - Get conversion result
• GET /api/download/{id} - Download converted PDF
• GET /api/documents/{id}/ai-text - Get AI-optimized text output
• POST /api/batch/repository - Repository-wide batch conversion

Usage Examples

Single Document for AI

importrequests# Convert document for AI consumptionwithopen('technical_doc.pdf', 'rb') asf:
response=requests.post(
'https://yourfunctionapp.azurewebsites.net/api/ai/convert',
files={'file': f},
json={
'target_model': 'gpt-4',
'preserve_structure': True,
'include_metadata': True
},
headers={'Authorization': 'Bearer your_token'}
)
ai_ready_text=response.json()['ai_text']

Repository-Wide AI Conversion

# Convert entire project documentation for AIresponse=requests.post(
'https://yourfunctionapp.azurewebsites.net/api/ai/repository',
json={
'repository_path': '/path/to/project',
'include_code': True,
'include_docs': True,
'target_model': 'claude-3',
'output_format': 'contextual'
},
headers={'Authorization': 'Bearer your_token'}
)
# Get single AI-friendly file representing entire projectproject_context=response.json()['consolidated_text']

Development

Setup Development Environment

make dev-install

Run Tests

make test
make test-cov # with coverage

Code Quality

make lint # Run linting
make format # Format code

Build Package

make build

Local Setup with Azure Functions Core Tools

1. Install Azure Functions Core Tools:

   npm install -g azure-functions-core-tools@4

2. Run Functions locally:

   cd azure-functions
   func start

3. AI Model Integration:

   # Install AI optimization tools
   pip install tiktoken transformers sentence-transformers
   # Configure model-specific tokenizers
   python scripts/setup_ai_models.py

AI Integration Features

LLM Optimization

# config/ai_optimization.yamlai_optimization:
token_limits:
gpt-4: 8192claude-3: 100000gpt-3.5: 4096formatting:
preserve_code_blocks: trueadd_context_headers: trueinclude_file_paths: truemaintain_hierarchy: truechunking:
strategy: "semantic"# or "fixed", "adaptive"overlap_tokens: 200respect_boundaries: true

Repository Processing

# config/repository.yamlrepository_processing:
include_patterns:
- "*.md"
- "*.rst" - "*.txt"
- "README*"
- "docs/**"exclude_patterns:
- "node_modules/**"
- ".git/**"
- "*.log"
- "build/**"ai_enhancements:
add_file_context: truepreserve_directory_structure: trueinclude_git_info: false

Recent Improvements

Security Enhancements

• ✅ JWT secret key management via environment variables and Azure Key Vault
• ✅ Removed mock authentication bypass
• ✅ CORS origin restrictions
• ✅ File path sanitization to prevent path traversal attacks
• ✅ Input validation for all endpoints

Performance Optimizations

• ✅ Database connection pooling
• ✅ Rate limiting middleware
• ✅ Database indexes for faster queries
• ✅ Centralized file validation

UI/UX Improvements

• ✅ Accessibility components (ARIA labels, keyboard navigation)
• ✅ Design token integration
• ✅ Responsive design support
• ✅ Error handling improvements

Contributing

1. Fork the repository
2. Create a feature branch focused on AI optimization
3. Add tests for AI-specific functionality
4. Ensure compatibility with major LLM providers
5. Submit a pull request

Documentation

• API Documentation - Complete API reference
• Architecture - System architecture and design
• Deployment Guide - Deployment instructions
• Contributing - Contribution guidelines
• Testing Guide - Testing documentation
• Design System - Design tokens and components

Environment Variables

See .env.example for all configuration options.

Required for Production:

• JWT_SECRET_KEY - Minimum 32 characters
• MONGO_URL - MongoDB connection string
• ALLOWED_ORIGINS - Comma-separated frontend URLs
• ENVIRONMENT - Set to production
• ALLOW_MOCK_AUTH - Set to false

License

MIT License - see LICENSE file for details