DataFrame.Extensions

<<<<<<< HEAD An open-source set of extensions to enhance the capabilities of the DataFrame class in Microsoft.Data.Analysis.

Overview

DataFrame.Extensions provides a collection of utility methods and extension functions designed to streamline common operations and improve the usability of DataFrames in .NET applications.

Features

• Extension methods for DataFrame manipulation
• Enhanced querying and filtering capabilities
• Simplified data transformation workflows

Requirements

• .NET framework compatible with Microsoft.Data.Analysis
• System, System.Collections.Generic, System.Linq

Installation

Add the DataFrameExtensions.cs file to your project and reference Microsoft.Data.Analysis.

Usage

usingMicrosoft.Data.Analysis;usingYourNamespace;// Update with appropriate namespace// Use the provided extension methods on DataFrame instances=======[![CI/CD](https://github.com/dimension-zero/Dimension.Data.Extensions.DataFrame/actions/workflows/ci.yml/badge.svg)](https://github.com/dimension-zero/Dimension.Data.Extensions.DataFrame/actions/workflows/ci.yml)[![NuGet](https://img.shields.io/nuget/v/Dimension.DataFrame.Extensions.svg)](https://www.nuget.org/packages/Dimension.DataFrame.Extensions/)[![License:MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)Acomprehensive set of extension methods for `Microsoft.Data.Analysis.DataFrame` that provides **pandas-like functionality**for.NET data science and numerical computing.
## Features
-**Arithmetic Operations**-Element-wise Plus, Minus, Times, Divide
-**Calculations**-Diff, Apply, Pow operations
-**Cumulative Operations**-Running sums and absolute sums
-**Rolling Windows**-Moving averages and custom rolling calculations
-**Statistical Methods**-Mean, Median, StdDev, Variance, Min, Max, Sum, Count, Quantile, Describe
-**Mathematical Functions**-Abs, Log, Log10, Exp, Sqrt, Sin, Cos, Round
-**Filtering**-Predicate-based and index-based filtering
-**Column Management**-Selection, existence checking, type-safe retrieval
-**Null/NaN Handling**-Drop rows with missing data
-**Shift Operations**-Lag/lead column values
-**I/O Operations**-Pretty printing and RFC 4180 compliant CSV export
-**Syntactic Sugar**-Method chaining with fluent API
-**Multi-targeting**-Supports.NET 6.0,7.0, and 8.0
## Installation
### NuGet Package Manager

Install-Package Dimension.DataFrame.Extensions

### .NET CLI

dotnet add package Dimension.DataFrame.Extensions

### PackageReference
```xml
<PackageReference Include="Dimension.DataFrame.Extensions" Version="1.1.0" />

Quick Start

usingDimension.DataFrame.Extensions;usingMicrosoft.Data.Analysis;// Create a DataFramevarprices=newPrimitiveDataFrameColumn<double>("Price",new[]{100.0,105.0,103.0,108.0,110.0});varvolumes=newPrimitiveDataFrameColumn<int>("Volume",new[]{1000,1500,1200,1800,2000});vardf=newDataFrame(prices,volumes);// Calculate price differencesvarpriceDiff=prices.Diff<double>();priceDiff.AddTo(df,"PriceChange");// Calculate rolling average (3-period)varrollingAvg=prices.Rolling(3, values =>values.Average(v =>v!.Value));rollingAvg.AddTo(df,"MA_3");// Print the DataFramedf.Print();

Usage Examples

Arithmetic Operations

varcol1=newPrimitiveDataFrameColumn<int>("A",new[]{1,2,3,4,5});varcol2=newPrimitiveDataFrameColumn<int>("B",new[]{10,20,30,40,50});// Additionvarsum=col1.Plus(col2);// [11, 22, 33, 44, 55]// Subtractionvardiff=col1.Minus(col2);// [-9, -18, -27, -36, -45]// Multiplicationvarproduct=col1.Times(col2);// [10, 40, 90, 160, 250]// Divisionvarquotient=col2.Divide(col1,"Quotient");// [10.0, 10.0, 10.0, 10.0, 10.0]

Cumulative Operations

vardata=newPrimitiveDataFrameColumn<int>("Data",new[]{1,2,3,4,5});// Cumulative sumvarcumSum=data.Cumulate();// [1, 3, 6, 10, 15]// Cumulative absolute sumvarnegData=newPrimitiveDataFrameColumn<int>("NegData",new[]{-1,2,-3,4,-5});varcumAbsSum=negData.CumulateAbs();// [1, 3, 6, 10, 15]

Shift Operations (Lag/Lead)

varprices=newPrimitiveDataFrameColumn<double>("Price",new[]{100.0,105.0,103.0,108.0});// Lag by 1 period (shift forward)varlag1=prices.Shift(1);// [null, 100.0, 105.0, 103.0]// Lead by 1 period (shift backward)varlead1=prices.Shift(-1);// [105.0, 103.0, 108.0, null]// Custom fill valuevarlagWithFill=prices.Shift(1,0.0);// [0.0, 100.0, 105.0, 103.0]

Rolling Window Calculations

vardata=newPrimitiveDataFrameColumn<double>("Data",new[]{1.0,2.0,3.0,4.0,5.0});// Rolling sumvarrollingSum=data.Rolling(3, values =>values.Sum(v =>v!.Value));// [null, null, 6.0, 9.0, 12.0]// Rolling averagevarrollingAvg=data.Rolling(3, values =>values.Average(v =>v!.Value));// [null, null, 2.0, 3.0, 4.0]// Rolling maximumvarrollingMax=data.Rolling(3, values =>values.Max(v =>v!.Value));// [null, null, 3.0, 4.0, 5.0]

Apply Custom Functions

vardata=newPrimitiveDataFrameColumn<int>("Data",new[]{1,2,3,4,5});// Square all valuesvarsquared=data.Apply(x =>x*x,"Squared");// [1, 4, 9, 16, 25]// Apply custom transformationvartransformed=data.Apply(x =>x*2+1,"Transformed");// [3, 5, 7, 9, 11]

Filtering

vardf=newDataFrame(newPrimitiveDataFrameColumn<int>("A",new[]{1,2,3,4,5}),newPrimitiveDataFrameColumn<double>("B",new[]{1.5,2.5,3.5,4.5,5.5}));// Filter by predicatevarfiltered=df.Filter<int>("A", value =>value>3);// Returns DataFrame with rows where A > 3// Filter by row indicesvarsubset=df.Filter(new[]{0,2,4});// Returns rows at indices 0, 2, and 4

Null and NaN Handling

vardf=newDataFrame(newPrimitiveDataFrameColumn<int>("A",newint?[]{1,null,3,4}),newPrimitiveDataFrameColumn<double>("B",new[]{1.0,2.0,double.NaN,4.0}));// Drop rows with nullsvarnoNulls=df.DropNulls();// Rows 0 and 3 remain// Drop rows with NaN valuesvarnoNaNs=df.DropNAs();// Rows 0, 1, and 3 remain// Drop rows with either nulls or NaNsvarclean=df.DropNullsOrNAs();// Only rows 0 and 3 remain

Method Chaining (Fluent API)

vardf=newDataFrame();varcol1=newPrimitiveDataFrameColumn<int>("A",new[]{1,2,3});varcol2=newPrimitiveDataFrameColumn<int>("B",new[]{10,20,30});// Chain operations togethercol1.Plus(col2).Pow(2).WithName<int>("Sum_Squared").AddTo(df);// df now contains column "Sum_Squared" with values [121, 484, 1089]

Column Operations

vardf=newDataFrame(newPrimitiveDataFrameColumn<int>("A",new[]{1,2,3}),newPrimitiveDataFrameColumn<int>("B",new[]{10,20,30}),newPrimitiveDataFrameColumn<int>("C",new[]{100,200,300}));// Select specific columnsvarsubset=df.SelectColumns("A","C");// Check if column existsboolhasColumn=df.ColumnExists("B");// true// Try to get column with type safetyif(df.TryGetColumn<int>("A",outvarcolumnA)){// Use columnA}

I/O Operations

vardf=newDataFrame(newPrimitiveDataFrameColumn<int>("ID",new[]{1,2,3}),newPrimitiveDataFrameColumn<string>("Name",new[]{"Alice","Bob","Charlie"}),newPrimitiveDataFrameColumn<double>("Score",new[]{95.5,87.3,92.1}));// Print to debug output (aligned columns)df.Print(numRows:10,numberFormat:"F2");// Save to CSVdf.SaveToCsv("output.csv",sep:",",includeHeader:true);

Statistical Methods

vardata=newPrimitiveDataFrameColumn<double>("Data",new[]{1.5,2.3,3.7,4.2,5.8,6.1,7.9,8.4,9.2,10.5});// Calculate meanvarmean=data.Mean();// 5.96// Calculate medianvarmedian=data.Median();// 5.95// Calculate standard deviationvarstdDev=data.StdDev();// Sample std dev// Calculate variancevarvariance=data.Variance();// Sample variance// Get min and maxvarmin=data.Min();// 1.5varmax=data.Max();// 10.5// Calculate sumvarsum=data.Sum();// 59.6// Get count of non-null valuesvarcount=data.Count();// 10// Calculate specific quantile (e.g., 75th percentile)varq75=data.Quantile(0.75);// Get comprehensive statisticsvarstats=data.Describe();// Returns: (Count, Mean, StdDev, Min, Q25, Median, Q75, Max)Console.WriteLine($"Count: {stats.Count}, Mean: {stats.Mean}, Median: {stats.Median}");

Mathematical Functions

vardata=newPrimitiveDataFrameColumn<double>("Data",new[]{-2.5,-1.0,0.0,1.0,2.5});// Absolute valuevarabsValues=data.Abs();// [2.5, 1.0, 0.0, 1.0, 2.5]// Natural logarithmvarpositiveData=newPrimitiveDataFrameColumn<double>("Positive",new[]{1.0,2.718,7.389});varlogValues=positiveData.Log();// [0.0, 1.0, 2.0]// Base-10 logarithmvarlog10Values=positiveData.Log10();// Logarithm with custom basevarlog2Values=positiveData.Log(2);// Log base 2// Exponential (e^x)varexpData=newPrimitiveDataFrameColumn<double>("Exp",new[]{0.0,1.0,2.0});varexpValues=expData.Exp();// [1.0, 2.718, 7.389]// Square rootvarsqrtData=newPrimitiveDataFrameColumn<double>("SqrtData",new[]{0.0,1.0,4.0,9.0,16.0});varsqrtValues=sqrtData.Sqrt();// [0.0, 1.0, 2.0, 3.0, 4.0]// Trigonometric functionsvarangles=newPrimitiveDataFrameColumn<double>("Angles",new[]{0.0,Math.PI/2,Math.PI});varsineValues=angles.Sin();varcosineValues=angles.Cos();// Roundingvardecimals=newPrimitiveDataFrameColumn<double>("Decimals",new[]{1.234,5.678,9.999});varrounded=decimals.Round(2);// [1.23, 5.68, 10.0]varroundedInt=decimals.Round();// [1.0, 6.0, 10.0]

Requirements

• .NET 6.0, 7.0, or 8.0
• Microsoft.Data.Analysis 0.21.1 or later
• MathNet.Numerics 5.0.0 or later

Null Handling

Different operations handle null values in different ways:

Arithmetic Operations (Plus, Minus, Times, Divide)

• Null values are treated as default(T) (typically 0 for numeric types)
• Example: 1 + null = 1 + 0 = 1

Statistical Operations (Mean, Median, StdDev, Variance, etc.)

• Null values are skipped and excluded from calculations
• Example: Mean([1, null, 3]) = (1 + 3) / 2 = 2.0

Shift Operations

• Null values are preserved in their new positions
• Fill values can be specified for positions vacated by the shift

Rolling Window Operations

• Null values are skipped within each window
• The operation is applied only to non-null values

Filtering Operations

• DropNulls() - Removes rows containing null values
• DropNAs() - Removes rows containing NaN values (for float/double)
• DropNullsOrNAs() - Removes rows containing either nulls or NaNs

Contributing

Contributions are welcome! Please feel free to submit a Pull Request.

1. Fork the repository
2. Create your feature branch (git checkout -b feature/AmazingFeature)
3. Commit your changes (git commit -m 'Add some AmazingFeature')
4. Push to the branch (git push origin feature/AmazingFeature)
5. Open a Pull Request

Testing

dotnet test

Performance Benchmarks

Run performance benchmarks to compare operations:

cd Dimension.DataFrame.Extensions.Benchmarks
dotnet run -c Release

Run specific benchmarks:

# Run only arithmetic benchmarks
dotnet run -c Release -- --filter *ArithmeticBenchmarks*# Run only statistics benchmarks
dotnet run -c Release -- --filter *StatisticsBenchmarks*# Export results to HTML and JSON
dotnet run -c Release -- --exporters json,html

Benchmark categories:

• ArithmeticBenchmarks - Plus, Minus, Times, Divide performance
• StatisticsBenchmarks - Mean, Median, StdDev, Variance, Describe performance
• MathBenchmarks - Abs, Log, Exp, Sqrt, trigonometric functions
• RollingWindowBenchmarks - Rolling window operations with various sizes

Building from Source

git clone https://github.com/dimension-zero/Dimension.Data.Extensions.DataFrame.git
cd Dimension.Data.Extensions.DataFrame
dotnet build

Creating NuGet Package

dotnet pack --configuration Release
>>>>>>> 8c6160fa77adf5c233ef7ac5350a310532bd0c0d

License

<<<<<<< HEAD Copyright (c) 2024 Harrow Ventures Limited (HVL)

This project is licensed under the MIT License. See the LICENSE file for details.

Contributing

Contributions are welcome. Please ensure all changes maintain compatibility with the existing API and follow the established code style.

This project is licensed under the MIT License - see the LICENSE file for details.

Authors

Dimension Technologies

Acknowledgments

• Built on top of Microsoft.Data.Analysis
• Inspired by pandas for Python
• Uses MathNet.Numerics for numerical operations

Support

For issues, questions, or contributions, please visit the GitHub repository.

---

Issued under the MIT Licence by Dimension Technologies.

8c6160fa77adf5c233ef7ac5350a310532bd0c0d