Use Case

Many external integrations make use of specific text based file formats delivered via protocols other than http. The data integration service makes these types of legacy integrations simple and based on modern c# programming concepts such as automatic serialization/deserialization and easy access to common transport protocols.

The consistent interface provided by the library is useful to create a single paradigm for communicating with legacy APIs that often used fixed width or csv file drops on remote servers.

Concepts

Any data integration implementation is based on four distinct interchangeable components, namely a serializer, a transport, various pre-processors and type converters.

Serializer

Any serializer can be built for the data integration service, whether it be CSV, flat files or even Morse code, as long as the concrete implements IIntegrationDataSerializer. The serializer takes class objects and converts them to a MemoryStream for the transport, or reads a MemoryStream from a transport and converts data back to strongly typed C# objects.

Transport

The transport reads data from an external source or writes it back to an external source. Any transport only has to inherit from IIntegrationTransport and implement the required methods. Any kind of transport can be defined from FTP to scp to local files to archives.

It all comes together

You only require a serializer and a transport to get data to and from an external integration party. The serializer takes objects and converts them to a MemoryStream and the given transport takes that data and sends it over a network, or to a local file system or wherever you wish. When data is read from the third party the transport passes the raw data to the serializer for conversion back to C# objects.

Simple Example

Send data to a local file in CSV format.

vara=newIntegrator();varplanets=newList<Planet>(){newPlanet(){Name="Mercury",DistanceFromSun=57.91,OrderFromSun=1},newPlanet(){Name="Venus",DistanceFromSun=108.2,OrderFromSun=2},newPlanet(){Name="Earth",DistanceFromSun=149.6,OrderFromSun=3},newPlanet(){Name="Mars",DistanceFromSun=227.9,OrderFromSun=4},newPlanet(){Name="Jupiter",DistanceFromSun=778.5,OrderFromSun=5},newPlanet(){Name="Saturn",DistanceFromSun=1429,OrderFromSun=6},newPlanet(){Name="Uranus",DistanceFromSun=2877,OrderFromSun=7},newPlanet(){Name="Neptune",DistanceFromSun=4498,OrderFromSun=8},};varcsvSerializer=newCsvSerializer{Delimiter="|",HasHeaderRecord=true};vartransport=newLocalFileTransport{FilePath=$"C:\\temp\\testplanets.csv"};varbuild=newIntegrator().SetSerializer(csvSerializer).AddListData(planets);varresult=a.SendData(build,transport);

Read data back from a CSV file on the local filesystem

vara=newIntegrator();varcsvSerializer=newCsvSerializer{Delimiter="|",HasHeaderRecord=true};vartransport=newLocalFileTransport{FilePath=$"C:\\temp\\testplanets.csv"};varplanets=newList<Planet>();varnb=newInputBuilder().SetData(builtFile).SetSerializer(csvSerializer).ReadAll(planets);a.ReceiveData(nb,transport);// planets will now be filled with data from testplanets.

Using the builder for more complex files

Some external parties have files that contain different types of records in the same files. Such files usually have a header, body with multiple records and a footer record, but this may vary according to the integration. The input and Output builders of the data integration service can cater for such complex files by allowing a definition of the file format to be given to a builder to create or read such files.

Below is an example of how to create a file with a header, body items and a footer using the OutputBuilder

Our example classes

publicclassPlanet{publicstringRecordType{get;set;}="PLANET";publicstringName{get;set;}publicintOrderFromSun{get;set;}/// <summary>/// In millions of kilometers/// </summary>publicdoubleDistanceFromSun{get;set;}}publicclassStellarSystem{publicstringRecordType{get;set;}="STELLARSYSTEM";publicstringStarType{get;set;}publicboolIsBinarySystem{get;set;}publicstringName{get;set;}publicdoubleRadius{get;set;}}publicclassGalaxy{publicstringRecordType{get;set;}="GALAXY";publicstringName{get;set;}="Milky Way";publicstringType{get;set;}="Spiral";publicstringAddress{get;set;}="Local Group";}

Using the output builder

vara=newModules.DataIntegration.Service.DataIntegrationService();varstar=newStellarSystem(){IsBinarySystem=false,Name="Sol",StarType="Yellow Dwarf",Radius=695.700};varplanets=newList<Planet>(){newPlanet(){Name="Mercury",DistanceFromSun=57.91,OrderFromSun=1},newPlanet(){Name="V#e#n#u#s",DistanceFromSun=108.2,OrderFromSun=2},newPlanet(){Name="_êarth_",DistanceFromSun=149.6,OrderFromSun=3},newPlanet(){Name="((Mars))",DistanceFromSun=227.9,OrderFromSun=4},newPlanet(){Name="Jupitër",DistanceFromSun=778.5,OrderFromSun=5},newPlanet(){Name="Sa____turn",DistanceFromSun=1429,OrderFromSun=6},newPlanet(){Name="<Uranus>",DistanceFromSun=2877,OrderFromSun=7},newPlanet(){Name="Nep~~~~~~~tune!!!%",DistanceFromSun=4498,OrderFromSun=8},};varcsvSerializer=newCsvSerializer{Delimiter="|",HasHeaderRecord=false,};vartransport=newLocalFileTransport{FilePath=$"C:\\temp\\broken.csv"};varbuild=newIntegrator().SetSerializer(csvSerializer).AddPreProcessor(newDiacriticRemover()).AddPreProcessor(newRegexRemover(@"[^a-zA-Z0-9/\.\-+&><=*,;'\(\)$]+")).AddData(star).AddListData(planets).AddData(newGalaxy(){Name="Alpha Centauri"});varresult=a.SendData(build,transport);

Data is added sequentially to the output, but data is only actually read from the input variables when the .build() method is called on the output builder object. In this case, an overload of SendData() will call build and create the output for sending via the transport.

Below is an example of using the InputBuilder. In this example data is read into C# objects by specifying which records in a file match which type with a Discriminator object

Our input file:

STELLARSYSTEM|YellowDwarf|False|Sol|695.7
PLANET|Mercury|1|57.91
PLANET|Venus|2|108.2
PLANET|earth|3|149.6
PLANET|((Mars))|4|227.9
PLANET|Jupiter|5|778.5
PLANET|Saturn|6|1429
PLANET|<Uranus>|7|2877
PLANET|Neptune|8|4498
GALAXY|AlphaCentauri|Spiral|LocalGroup

Code Example:

varstar=newStellarSystem();varplanets=newList<Planet>();vargalaxy=newGalaxy();varcsvSerializer=newCsvSerializer{Delimiter="|",HasHeaderRecord=false,};varbuiltFile=BuilderTest().Build();// input data is faked. Would normally come through a transportvarnb=newInputBuilder().SetData(builtFile).SetSerializer(csvSerializer).ReadOnce<StellarSystem,FirstFieldDiscriminator<string>>(system =>star=system, discriminator =>discriminator.Value=="STELLARSYSTEM").ReadMany<Planet,FirstFieldDiscriminator<string>>(planets, discriminator =>discriminator.Value=="PLANET").ReadOnce<Galaxy,FirstFieldDiscriminator<string>>(galaxy1 =>galaxy=galaxy1, discriminator =>discriminator.Value=="GALAXY");nb.Build();

Calls to ReadOnce() specify which records are only expected once in a file, and a lambda is passed in to assign the value to an output system. Calls to ReadMany() specify record types that will occur multiple times in a file, and will be pushed into the specified List<T>.