FlatFile

FlatFile is a library to work with flat files (work up-to 100 times faster then FileHelpers)

Modernization status

• 🚨 v2 breaking change: dropped legacy .NET Framework targets (net35-net48) and old build pipeline.
• ✅ Modernized runtime support to .NET 8 only via SDK-style projects.
• ✅ CI now builds modern projects with dotnet build on GitHub Actions.

Modern .NET support

Active projects:

• src/FlatFile.Core.Modern
• src/FlatFile.Core.Attributes.Modern
• src/FlatFile.Delimited.Modern
• src/FlatFile.FixedLength.Modern
• src/FlatFile.Delimited.Attributes.Modern
• src/FlatFile.FixedLength.Attributes.Modern

All of them target net8.0 and carry package/assembly version 2.0.0.

NuGet publishing from GitHub

When changes are merged to master, GitHub Actions can publish v2 packages automatically using .github/workflows/publish-nuget.yml.

Required repository secret:

• NUGET_API_KEY: NuGet.org API key with push permission for FlatFile packages.

The publish workflow packs all *.Modern projects and pushes resulting .nupkg files to NuGet (--skip-duplicate).

Installing FlatFile

Installing all packages

You should install FlatFile with NuGet:

Install-Package FlatFile

Installing FlatFile.Delimited

You should install FlatFile.Delimited with NuGet:

Install-Package FlatFile.Delimited

Add attribute-mapping extensions

You should install FlatFile.Delimited.Attributes with NuGet:

Install-Package FlatFile.Delimited.Attributes

Installing FlatFile.FixedLength

You should install FlatFile.FixedLength with NuGet:

Install-Package FlatFile.FixedLength

Add attribute-mapping extensions

You should install FlatFile.FixedLength.Attributes with NuGet:

Install-Package FlatFile.FixedLength.Attributes

This commands from Package Manager Console will download and install FlatFile and all required dependencies.

Benchmarks

Simple write

Name	Milliseconds	Percent
FileHelperEngine.WriteStream	5175	11266.8%
FlatFileEngine.Write	45	100%

Simple read

Name	Milliseconds	Percent
FileHelperEngine.ReadStream	7636	2764.4%
FlatFileEngine.Read	276	100%

Big (100000 entities) write

Name	Milliseconds	Percent
FileHelperEngine.WriteStream	17246	838.4%
FlatFileEngine.Write	2057	100%

Big (100000 entities) write with reflection magic

Name	Milliseconds	Percent
FileHelperEngine.WriteStream	17778	1052.5%
FlatFileEngine.Write	1689	100%

FlatFile vs CsvHelper

Write all records with class mapping

Name	Milliseconds	Percent
CsvWriter.WriteRecords	26578	7988.8%
FlatFileEngine.Write	332	100%

Read all records with class mapping

Name	Milliseconds	Percent
CsvWriter.ReadRecords	18795	3190.5%
FlatFileEngine.Read	589	100%

Usage

Class mapping

DelimitedLayout

publicsealedclassDelimitedSampleRecordLayout:DelimitedLayout<FixedSampleRecord>{publicDelimitedSampleRecordLayout(){this.WithDelimiter(";").WithQuote("\"").WithMember(x =>x.Cuit).WithMember(x =>x.Nombre).WithMember(x =>x.Actividad, c =>c.WithName("AnotherName"));}}

FixedLayout

publicsealedclassFixedSampleRecordLayout:FixedLayout<FixedSampleRecord>{publicFixedSampleRecordLayout(){this.WithMember(x =>x.Cuit, c =>c.WithLength(11)).WithMember(x =>x.Nombre, c =>c.WithLength(160)).WithMember(x =>x.Actividad, c =>c.WithLength(6));}}

Run-time mapping

DelimitedLayout

publicclassLayoutFactory{publicIDelimitedLayout<TestObject>GetLayout(){IDelimitedLayout<TestObject>layout=newDelimitedLayout<TestObject>().WithDelimiter(";").WithQuote("\"").WithMember(o =>o.Id).WithMember(o =>o.Description).WithMember(o =>o.NullableInt, set =>set.AllowNull("=Null"));returnlayout;}}

FixedLayout

publicclassLayoutFactory{publicIFixedLayout<TestObject>GetLayout(){IFixedLayout<TestObject>layout=newFixedLayout<TestObject>().WithMember(o =>o.Id, set =>set.WithLength(5).WithLeftPadding('0')).WithMember(o =>o.Description, set =>set.WithLength(25).WithRightPadding(' ')).WithMember(o =>o.NullableInt, set =>set.WithLength(5).AllowNull("=Null").WithLeftPadding('0'));returnlayout;}// you can also register a StringNormalizer function to convert input into the FixedLengthLineBuilder// to a string compatible with the specifications for your target File. //// Note that the StringNormalizer function is only used when creating/building files. Not when reading/parsing files.//// example:publicIFixedLayout<TestObject>GetLayout(){IFixedLayout<TestObject>layout=newFixedLayout<TestObject>().WithMember(o =>o.Description, set =>set.WithLength(25).WithRightPadding(' ').WithStringNormalizer((input)=>{// the normalization to FormD splits accented letters in accents+letters,// the rest aftet that removes those accents (and other non-spacing characters) from the ouput// So unicode L'été becomes L'etereturnnewstring(input.Normalize(System.Text.NormalizationForm.FormD).ToCharArray().Where(c =>CharUnicodeInfo.GetUnicodeCategory(c)!=UnicodeCategory.NonSpacingMark).ToArray());}))
return layout;}

Attribute mapping

Delimited

usingFlatFile.Delimited.Attributes;[DelimitedFile(Delimiter=";",Quotes="\"")]publicclassTestObject{[DelimitedField(1)]publicintId{get;set;}[DelimitedField(2)]publicstringDescription{get;set;}[DelimitedField(3,NullValue="=Null")]publicint?NullableInt{get;set;}}

Fixed

usingFlatFile.FixedLength;usingFlatFile.FixedLength.Attributes;[FixedLengthFile]publicclassTestObject{[FixedLengthField(1,5,PaddingChar='0')]publicintId{get;set;}[FixedLengthField(2,25,PaddingChar=' ',Padding=Padding.Right)]publicstringDescription{get;set;}[FixedLengthField(2,5,PaddingChar='0',NullValue="=Null")]publicint?NullableInt{get;set;}}

Read from stream

With layout

varlayout=newFixedSampleRecordLayout();varfactory=newFixedLengthFileEngineFactory();using(varstream=newMemoryStream(Encoding.UTF8.GetBytes(FixedFileSample))){varflatFile=factory.GetEngine(layout);varrecords=flatFile.Read<FixedSampleRecord>(stream).ToArray();}

With attribute-mapping

varfactory=newFixedLengthFileEngineFactory();using(varstream=newMemoryStream(Encoding.UTF8.GetBytes(FixedFileSample))){varflatFile=factory.GetEngine<FixedSampleRecord>();varrecords=flatFile.Read<FixedSampleRecord>(stream).ToArray();}

With multiple fixed record types

varfactory=newFixedLengthFileEngineFactory();using(varstream=newMemoryStream(Encoding.UTF8.GetBytes(FixedFileSample))){// If using attribute mapping, pass an array of record types// rather than layout instancesvarlayouts=newILayoutDescriptor<IFixedFieldSettingsContainer>[]{newHeaderRecordLayout(),newDetailRecordLayout(),newTrailerRecordLayout()};varflatFile=factory.GetEngine(layouts,
line =>{// For each line, return the proper record type.// The mapping for this line will be loaded based on that type.// In this simple example, the first character determines the// record type.if(String.IsNullOrEmpty(line)||line.Length<1)returnnull;switch(line[0]){case'H':returntypeof(HeaderRecord);case'D':returntypeof(DetailRecord);case'T':returntypeof(TrailerRecord);}returnnull;});flatFile.Read(stream);varheader=flatFile.GetRecords<HeaderRecord>().FirstOrDefault();varrecords=flatFile.GetRecords<DetailRecord>();vartrailer=flatFile.GetRecords<TrailerRecord>().FirstOrDefault();}

Write to stream

With layout

varsampleRecords=GetRecords();varlayout=newFixedSampleRecordLayout();varfactory=newFixedLengthFileEngineFactory();using(varstream=newMemoryStream()){varflatFile=factory.GetEngine(layout);flatFile.Write<FixedSampleRecord>(stream,sampleRecords);}

With attribute-mapping

varsampleRecords=GetRecords();varfactory=newFixedLengthFileEngineFactory();using(varstream=newMemoryStream()){varflatFile=factory.GetEngine<FixedSampleRecord>();flatFile.Write<FixedSampleRecord>(stream,sampleRecords);}