LargeFilesManager

Overview

LargeFilesManager is a modern .NET 8 solution designed to efficiently manage, process, and transfer large files in enterprise environments. The architecture emphasizes scalability, maintainability, and testability, leveraging best practices and industry standards.

Solution include 2 applications:

1. File Generator (LFM.FileGenerator): Creates large text files efficiently.

2. File Sorter (LFM.FileParser): Sorts very large files deterministically without loading the whole file into memory.

Both apps are designed to process large files with parallelism, streaming I/O, and consistent progress reporting.

File Generator:

• Choose output folder, file name, target size (B/KB/MB/GB).
• Start generation; progress bar and logs indicate status.
  
  

File Sorter:

• Choose input file and output file path.
• Start sorting; the app splits, sorts parts, and merges to output.
• Final output is sorted by text then number, matching the generator’s semantics.

---

Architecture Style

• Layered Architecture: The solution is organized into distinct layers (Presentation, Application, Domain, Infrastructure, and Testing), promoting separation of concerns and easy extensibility.
• Dependency Injection: Utilizes .NET’s built-in DI for loose coupling and testability.
• Repository & Unit of Work Patterns: Abstracts data access and transaction management.
• CQRS (Command Query Responsibility Segregation): Segregates read and write operations for performance and clarity.
• Asynchronous Programming: Uses async/await for scalable I/O operations.

---

Project Structure

LargeFilesManager/
│
├── LargeFilesManager.sln
├── src/
│   ├── LargeFilesManager.WebApi/        # ASP.NET Core Web API (Presentation Layer)
│   ├── LargeFilesManager.Application/   # Application Layer (CQRS, Services)
│   ├── LargeFilesManager.Domain/        # Domain Layer (Entities, Interfaces)
│   ├── LargeFilesManager.Infrastructure/# Infrastructure Layer (Data, File Storage, External Integrations)
│
├── tests/
│   ├── LargeFilesManager.UnitTests/     # Unit Tests (xUnit/NUnit/MSTest)
│   ├── LargeFilesManager.IntegrationTests/ # Integration Tests
│
└── README.md

---

Technical Stack

• .NET 8 (C#).
• WPF.
• Entity Framework Core (for data access).
• SQL Server (default, can be swapped for other providers).
• xUnit/NUnit/MSTest (for testing).
• Swagger/OpenAPI (for API documentation).
• AutoMapper (object mapping).
• Serilog (logging).
• Third-Party Libraries:
  • • AutoMapper.Extensions.Microsoft.DependencyInjection.
  • • Microsoft.EntityFrameworkCore.SqlServer.
  • • Microsoft.EntityFrameworkCore.InMemory (for tests).

---

Patterns & Practices

• Repository Pattern: Encapsulates data access logic.
• Unit of Work Pattern: Manages transactions.
• CQRS: Separates command and query responsibilities.
• Dependency Injection: Promotes testability and modularity.
• Configuration via appsettings.json: Centralized configuration management.
• Logging & Monitoring: Serilog for structured logging.

---

File Format

Lines generated and processed follow the template: .

Example:

1. Apple
2. Apple
3. Banana is yellow
4. Cherry is the best
5. Something something something

---

File Generator

Purpose: Generates large text files quickly, splitting work across part files and merging them into a single output.

Key characteristics:

1. Streaming I/O with FileStream + StreamWriter.
2. Parallel part generation based on processor count.
3. Deterministic line structure; words-only generation for text.
4. Accurate, thread-safe progress tracking.
5. Explicit UTF-8 (no BOM) encoding for consistency.

---

Generation steps:

1. Initialization

• Reset progress panel state: ProgressMinValue, ProgressMaxValue, ProgressValue, ProgressStatus.
• Compute target file size in bytes using ByteHelper.ConvertToBytes(fileSizeType, fileSize).
• Determine buffer size from BufferFileWriteSize with a safe minimum (4 KB).
• Select degree of parallelism based on ProcessorCount.
• Compute sizePerFile = targetSize / parallelParts.
• Delete any existing final file and stale part files.

2. Information added:

3. Parallel part file creation

• For i in [0..parallelParts):
• Compute part file name: .part_{i+1}. when parallel parts > 1.
• Open a write stream: FileStream(partPath, Create) + StreamWriter(UTF8 no BOM).
• Loop until part reaches sizePerFile:
• Generate [text] using words-only generator:
• Build alphabetic words separated by single spaces; total length <= maxLineLength.
• Increment the global LineNumber atomically.
• Write line with template: "{LineNumber}. {text}".
• Update progress by actual bytes written: encoding byte count of line + newline.
• If the next similar write would exceed sizePerFile, write one last line to approximate target and exit.

4. Merge part files into final file

• Open the final output stream once.
• Read each part file line-by-line in parallel.
• Serialize writes to the final writer via a lock.
• Update progress using accurate byte counts on each merged line.
• Log merge completion, delete part files, and mark process complete.

5. Process completed state

6. Click "Reset Form" button to start new generation process.
   
   

Error handling and logging

• All major operations (generation, merging, file deletion) log progress and errors using Serilog.
• Exceptions during merge do not delete part files, enabling retry.

Encoding and counting

• Use UTF-8 without BOM for both writer and reader.
• Calculate progress by encoding-aware byte counts (line + newline) to avoid drift.

---

File Sorter

Purpose: Sorts very large files according to the format’s semantics, using external sorting:

• Sort by the text portion alphabetically (Ordinal).
• When texts are equal, sort by the numeric prefix ascending.
  
  

Key characteristics:

• Split-then-merge pipeline (external sorting).
• Streaming I/O, blocking queues, and parallel consumers.
• Stable, duplicate-preserving k-way merge.
• Accurate, thread-safe progress updates.
• Explicit UTF-8 (no BOM) encoding throughout.

---

Sorting steps:

1. Initialization

• Reset progress panel state and status.
• Inspect input file size for progress tracking.
• Compute target part size in bytes using MaxPartFileSizeMegaBytes.
• Derive bounded capacity for the internal BlockingCollection to regulate memory/flow.
• Start consumer tasks (writers) based on TotalConsumerTasks.

2. Information added:

3. Producer: read and split

• Open the input file with StreamReader (UTF-8 no BOM).
• Read line-by-line, calculating byte size per line (text + newline).
• Parse each line to ParsedLine:
• NumericPrefix: integer portion before .
• Text: substring after .
• OriginalLine: original text line
• Accumulate lines until the current part reaches targetPartFileSizeBytes.
• Sort the current part in-memory using ParsedLineComparer:
• Compare by Text (Ordinal).
• If equal, compare by NumericPrefix.
• Add the sorted lines (OriginalLine) to the blocking collection as a PartQueue.
• Repeat until EOF; flush remaining lines as the last part.

4. Consumers: write sorted parts

• Each consumer:
• Dequeues PartQueue items.
• Writes sorted lines to a temporary part file in a unique temp directory.
• Updates progress by the resulting part file length.
• Records the part file path and clears queue memory.

5. Merge sorted part files

• Initialize readers for each part (UTF-8 no BOM).
• Build a candidate map: SortedDictionary<ParsedLine, Queue> keyed by the next line from each part (parsed).
• Queue maintains file paths producing identical parsed lines; this preserves duplicates.
• Open the final output writer (UTF-8 no BOM).
• While candidates exist:
• Pop the smallest parsed line (Text first, then NumericPrefix).
• Write the OriginalLine to output.
• Update progress with accurate bytes written.
• Dequeue the producing file path and read its next line:
• If available, reinsert into candidates; otherwise, that path is exhausted.
• Dispose readers, report completion, and delete temp part files.

6. Process completed state

7. Click "Reset Form" button to start new generation process.

---

Error handling and logging

• Both split and merge stages update ProgressStatus and ProgressValue.
• Exceptions log context and are allowed to propagate to global handlers (App-level safety net).

---

Configuration, Logging, and Localization

• AppSettings (LFM.Core.AppSettings):
• BufferFileWriteSize (KB), TotalConsumerTasks, MaxPartFileSizeMegaBytes, and UI resources.
• Logging:
• Serilog configured in AppStartupHelper.ConfigureLogging(), with console + file sinks.
• Localization:
• StringLocalizer provides UI strings and messages.

---

Progress and UI

• BaseService (LFM.Core.Services.BaseService) manages:
• ProgressMinValue, ProgressMaxValue, ProgressValue, ProgressStatus.
• Dispatcher timer tick to display elapsed time.
• Thread-safe locks for progress and shared counters (e.g., LineNumberLock).

---

Best Practices Implemented

• Streaming I/O with explicit buffer sizes.
• Accurate encoding-aware byte counting for progress.
• Deterministic formatting and sorting semantics.
• Parallelism used where safe (part generation; part writing; merge reads).
• Thread-safe updates via locks around shared state.
• Resource cleanup via using and guarded deletes.
• Error logging with actionable context; failures avoid destructive cleanup when retry is possible.

---

Solution Startup Process

Prerequisites

• Visual Studio 2022+ (with .NET 8 SDK)
• SQL Server (local or remote)

Step-by-Step Deployment

1. Clone the Repository

 git clone https://github.com/DenisKovalyonokSamples/LargeFilesManager.git

2. Open Solution

   • Launch Visual Studio.
   • Open LargeFilesManager.sln.

3. Configure Database

   • Update appsettings.json in LargeFilesManager.WebApi with your SQL Server connection string.

4. Restore NuGet Packages

   • Visual Studio will auto-restore packages on build.

5. Build Solution

   • Right-click the solution > Build.

6. Apply Migrations

   • Open Package Manager Console.
   • Select LargeFilesManager.Infrastructure as default project.
   • Run:

  Update-Database

7. Run the Web API
   • Set LargeFilesManager.WebApi as startup project.
   • Press F5 or click "Start Debugging".

---

Testing

Unit Tests

• Located in LargeFilesManager.UnitTests.
• Covers core business logic, domain entities, and application services.
• Uses in-memory database for isolation.

Running Tests Locally

1. Open Test Explorer

   • Visual Studio: View > Test Explorer.

2. Run All Tests

   • Click "Run All" in Test Explorer.

3. Command Line (Optional)

   • Navigate to solution directory.
   • Run:

   dotnet test

---

References

• .NET Documentation
• Entity Framework Core
• Serilog

---

Contribution & Code Quality

• Follows SOLID principles and clean code practices.
• All code changes require passing tests and code review.

---

License

This project is licensed under the MIT License.