Implement Include/ThenInclude as adapter

[devbased]

Closes#187.

Summary

Query should be built by ORM (EfCore and EF6 in our case). This work uses data from IncludeExpressionInfo to call public API of specified libraries instead of building query manually.

Part with EF6 is not done yet

[fiseni]

Hey @ardalis,

If you don't have any comments here, I'll merge this. In the end, we decided to be an opt-in feature, so there is no high risk of implementing it.

[fiseni]

Hey @devbased,

The tests are passing and everything seems working fine with caching enabled. But, I'm somewhat uncomfortable with benchmark results. We're doing additional work on top of EF, and we just can't be faster than EF. What we're missing here?

When you compile the lambda for the proxy delegates, perhaps it compiles recursively the inner LambdaExpression too (the query expression)? We're passing the compiled version? 🤔 That would not be good.

Benchmark results for 1.000.000 iterations:

Method	Mean	Error	StdDev	Gen 0	Allocated
EFIncludeExpression	4.600 s	0.0849 s	0.1441 s	502000.0000	3 GB
SpecIncludeExpression	5.619 s	0.1122 s	0.1292 s	585000.0000	3 GB
SpecIncludeExpressionCached	3.642 s	0.0724 s	0.1015 s	415000.0000	2 GB

[fiseni]

Ok, the issue was in the benchmark tests. For the EF direct approach, the expressions were created on each iteration, while the spec versions have the expression stored as a state. Once we fix this, then we get much more accurate and reasonable results. Actually, now we can see how much slower we are compared with direct EF usage 😞

Benchmark results for 100.000 iterations:

Method	Mean	Error	StdDev	Gen 0	Allocated
EFIncludeExpression	273.9 ms	1.10 ms	0.86 ms	31500.0000	189 MB
SpecIncludeExpression	642.3 ms	7.26 ms	6.79 ms	69000.0000	417 MB
SpecIncludeExpressionCached	377.7 ms	7.44 ms	6.96 ms	44000.0000	266 MB

The test code:

namespaceSpecification.Benchmark{[MemoryDiagnoser]publicclassIncludeBenchmark{privatereadonlyintmax=100000;privatereadonlySpecificationEvaluatorevaluator=SpecificationEvaluator.Default;privatereadonlySpecificationEvaluatorevaluatorCached=SpecificationEvaluator.Cached;privatereadonlySpecification<Store>specInclude=newStoreIncludeProductsSpec();privatereadonlyIQueryable<Store>Stores;publicIncludeBenchmark(){Stores=newBenchmarkDbContext().Stores.AsQueryable();}[Benchmark]publicvoidEFIncludeExpression(){Expression<Func<Store,IEnumerable<Product>>>IncludeExpr= x =>x.Products;Expression<Func<Product,CustomFields>>ThenIncludeExpr= x =>x.CustomFields;for(inti=0;i<max;i++){_=Stores.Include(IncludeExpr).ThenInclude(ThenIncludeExpr);}}[Benchmark]publicvoidSpecIncludeExpression(){for(inti=0;i<max;i++){_=evaluator.GetQuery(Stores,specInclude);}}[Benchmark]publicvoidSpecIncludeExpressionCached(){for(inti=0;i<max;i++){_=evaluatorCached.GetQuery(Stores,specInclude);}}}publicclassBenchmarkDbContext:DbContext{publicDbSet<Store>Stores{get;set;}protectedoverridevoidOnConfiguring(DbContextOptionsBuilderoptionsBuilder){base.OnConfiguring(optionsBuilder);optionsBuilder.UseSqlServer("Server=(localdb)\\mssqllocaldb;Integrated Security=SSPI;Initial Catalog=SpecificationEFTestsDB;ConnectRetryCount=0");}}publicclassStoreIncludeProductsSpec:Specification<Store>{publicStoreIncludeProductsSpec(){Query.Include(x =>x.Products).ThenInclude(x =>x.CustomFields);}}publicclassStore{publicintId{get;set;}publicIEnumerable<Product>Products{get;set;}}publicclassProduct{publicintId{get;set;}publicCustomFieldsCustomFields{get;set;}}publicclassCustomFields{publicintId{get;set;}publicstringCustomText1{get;set;}publicstringCustomText2{get;set;}}}

[devbased]

Closes#187.

Summary

Query should be built by ORM (EfCore and EF6 in our case). This work uses data from IncludeExpressionInfo to call public API of specified libraries instead of building query manually.

Part with EF6 is not done yet

[fiseni]

Hey @ardalis,

If you don't have any comments here, I'll merge this. In the end, we decided to be an opt-in feature, so there is no high risk of implementing it.

[fiseni]

Hey @devbased,

The tests are passing and everything seems working fine with caching enabled. But, I'm somewhat uncomfortable with benchmark results. We're doing additional work on top of EF, and we just can't be faster than EF. What we're missing here?

When you compile the lambda for the proxy delegates, perhaps it compiles recursively the inner LambdaExpression too (the query expression)? We're passing the compiled version? 🤔 That would not be good.

Benchmark results for 1.000.000 iterations:

Method	Mean	Error	StdDev	Gen 0	Allocated
EFIncludeExpression	4.600 s	0.0849 s	0.1441 s	502000.0000	3 GB
SpecIncludeExpression	5.619 s	0.1122 s	0.1292 s	585000.0000	3 GB
SpecIncludeExpressionCached	3.642 s	0.0724 s	0.1015 s	415000.0000	2 GB

[fiseni]

Ok, the issue was in the benchmark tests. For the EF direct approach, the expressions were created on each iteration, while the spec versions have the expression stored as a state. Once we fix this, then we get much more accurate and reasonable results. Actually, now we can see how much slower we are compared with direct EF usage 😞

Benchmark results for 100.000 iterations:

Method	Mean	Error	StdDev	Gen 0	Allocated
EFIncludeExpression	273.9 ms	1.10 ms	0.86 ms	31500.0000	189 MB
SpecIncludeExpression	642.3 ms	7.26 ms	6.79 ms	69000.0000	417 MB
SpecIncludeExpressionCached	377.7 ms	7.44 ms	6.96 ms	44000.0000	266 MB

The test code:

namespaceSpecification.Benchmark{[MemoryDiagnoser]publicclassIncludeBenchmark{privatereadonlyintmax=100000;privatereadonlySpecificationEvaluatorevaluator=SpecificationEvaluator.Default;privatereadonlySpecificationEvaluatorevaluatorCached=SpecificationEvaluator.Cached;privatereadonlySpecification<Store>specInclude=newStoreIncludeProductsSpec();privatereadonlyIQueryable<Store>Stores;publicIncludeBenchmark(){Stores=newBenchmarkDbContext().Stores.AsQueryable();}[Benchmark]publicvoidEFIncludeExpression(){Expression<Func<Store,IEnumerable<Product>>>IncludeExpr= x =>x.Products;Expression<Func<Product,CustomFields>>ThenIncludeExpr= x =>x.CustomFields;for(inti=0;i<max;i++){_=Stores.Include(IncludeExpr).ThenInclude(ThenIncludeExpr);}}[Benchmark]publicvoidSpecIncludeExpression(){for(inti=0;i<max;i++){_=evaluator.GetQuery(Stores,specInclude);}}[Benchmark]publicvoidSpecIncludeExpressionCached(){for(inti=0;i<max;i++){_=evaluatorCached.GetQuery(Stores,specInclude);}}}publicclassBenchmarkDbContext:DbContext{publicDbSet<Store>Stores{get;set;}protectedoverridevoidOnConfiguring(DbContextOptionsBuilderoptionsBuilder){base.OnConfiguring(optionsBuilder);optionsBuilder.UseSqlServer("Server=(localdb)\\mssqllocaldb;Integrated Security=SSPI;Initial Catalog=SpecificationEFTestsDB;ConnectRetryCount=0");}}publicclassStoreIncludeProductsSpec:Specification<Store>{publicStoreIncludeProductsSpec(){Query.Include(x =>x.Products).ThenInclude(x =>x.CustomFields);}}publicclassStore{publicintId{get;set;}publicIEnumerable<Product>Products{get;set;}}publicclassProduct{publicintId{get;set;}publicCustomFieldsCustomFields{get;set;}}publicclassCustomFields{publicintId{get;set;}publicstringCustomText1{get;set;}publicstringCustomText2{get;set;}}}

[devbased]

Closes#187.

Summary

Query should be built by ORM (EfCore and EF6 in our case). This work uses data from IncludeExpressionInfo to call public API of specified libraries instead of building query manually.

Part with EF6 is not done yet

[fiseni]

Hey @ardalis,

If you don't have any comments here, I'll merge this. In the end, we decided to be an opt-in feature, so there is no high risk of implementing it.

[fiseni]

Hey @devbased,

The tests are passing and everything seems working fine with caching enabled. But, I'm somewhat uncomfortable with benchmark results. We're doing additional work on top of EF, and we just can't be faster than EF. What we're missing here?

When you compile the lambda for the proxy delegates, perhaps it compiles recursively the inner LambdaExpression too (the query expression)? We're passing the compiled version? 🤔 That would not be good.

Benchmark results for 1.000.000 iterations:

Method	Mean	Error	StdDev	Gen 0	Allocated
EFIncludeExpression	4.600 s	0.0849 s	0.1441 s	502000.0000	3 GB
SpecIncludeExpression	5.619 s	0.1122 s	0.1292 s	585000.0000	3 GB
SpecIncludeExpressionCached	3.642 s	0.0724 s	0.1015 s	415000.0000	2 GB

[fiseni]

Ok, the issue was in the benchmark tests. For the EF direct approach, the expressions were created on each iteration, while the spec versions have the expression stored as a state. Once we fix this, then we get much more accurate and reasonable results. Actually, now we can see how much slower we are compared with direct EF usage 😞

Benchmark results for 100.000 iterations:

Method	Mean	Error	StdDev	Gen 0	Allocated
EFIncludeExpression	273.9 ms	1.10 ms	0.86 ms	31500.0000	189 MB
SpecIncludeExpression	642.3 ms	7.26 ms	6.79 ms	69000.0000	417 MB
SpecIncludeExpressionCached	377.7 ms	7.44 ms	6.96 ms	44000.0000	266 MB

The test code:

namespaceSpecification.Benchmark{[MemoryDiagnoser]publicclassIncludeBenchmark{privatereadonlyintmax=100000;privatereadonlySpecificationEvaluatorevaluator=SpecificationEvaluator.Default;privatereadonlySpecificationEvaluatorevaluatorCached=SpecificationEvaluator.Cached;privatereadonlySpecification<Store>specInclude=newStoreIncludeProductsSpec();privatereadonlyIQueryable<Store>Stores;publicIncludeBenchmark(){Stores=newBenchmarkDbContext().Stores.AsQueryable();}[Benchmark]publicvoidEFIncludeExpression(){Expression<Func<Store,IEnumerable<Product>>>IncludeExpr= x =>x.Products;Expression<Func<Product,CustomFields>>ThenIncludeExpr= x =>x.CustomFields;for(inti=0;i<max;i++){_=Stores.Include(IncludeExpr).ThenInclude(ThenIncludeExpr);}}[Benchmark]publicvoidSpecIncludeExpression(){for(inti=0;i<max;i++){_=evaluator.GetQuery(Stores,specInclude);}}[Benchmark]publicvoidSpecIncludeExpressionCached(){for(inti=0;i<max;i++){_=evaluatorCached.GetQuery(Stores,specInclude);}}}publicclassBenchmarkDbContext:DbContext{publicDbSet<Store>Stores{get;set;}protectedoverridevoidOnConfiguring(DbContextOptionsBuilderoptionsBuilder){base.OnConfiguring(optionsBuilder);optionsBuilder.UseSqlServer("Server=(localdb)\\mssqllocaldb;Integrated Security=SSPI;Initial Catalog=SpecificationEFTestsDB;ConnectRetryCount=0");}}publicclassStoreIncludeProductsSpec:Specification<Store>{publicStoreIncludeProductsSpec(){Query.Include(x =>x.Products).ThenInclude(x =>x.CustomFields);}}publicclassStore{publicintId{get;set;}publicIEnumerable<Product>Products{get;set;}}publicclassProduct{publicintId{get;set;}publicCustomFieldsCustomFields{get;set;}}publicclassCustomFields{publicintId{get;set;}publicstringCustomText1{get;set;}publicstringCustomText2{get;set;}}}

[devbased]

Closes#187.

Summary

Query should be built by ORM (EfCore and EF6 in our case). This work uses data from IncludeExpressionInfo to call public API of specified libraries instead of building query manually.

Part with EF6 is not done yet

[fiseni]

Hey @ardalis,

If you don't have any comments here, I'll merge this. In the end, we decided to be an opt-in feature, so there is no high risk of implementing it.

[fiseni]

Hey @devbased,

The tests are passing and everything seems working fine with caching enabled. But, I'm somewhat uncomfortable with benchmark results. We're doing additional work on top of EF, and we just can't be faster than EF. What we're missing here?

When you compile the lambda for the proxy delegates, perhaps it compiles recursively the inner LambdaExpression too (the query expression)? We're passing the compiled version? 🤔 That would not be good.

Benchmark results for 1.000.000 iterations:

Method	Mean	Error	StdDev	Gen 0	Allocated
EFIncludeExpression	4.600 s	0.0849 s	0.1441 s	502000.0000	3 GB
SpecIncludeExpression	5.619 s	0.1122 s	0.1292 s	585000.0000	3 GB
SpecIncludeExpressionCached	3.642 s	0.0724 s	0.1015 s	415000.0000	2 GB

[fiseni]

Ok, the issue was in the benchmark tests. For the EF direct approach, the expressions were created on each iteration, while the spec versions have the expression stored as a state. Once we fix this, then we get much more accurate and reasonable results. Actually, now we can see how much slower we are compared with direct EF usage 😞

Benchmark results for 100.000 iterations:

Method	Mean	Error	StdDev	Gen 0	Allocated
EFIncludeExpression	273.9 ms	1.10 ms	0.86 ms	31500.0000	189 MB
SpecIncludeExpression	642.3 ms	7.26 ms	6.79 ms	69000.0000	417 MB
SpecIncludeExpressionCached	377.7 ms	7.44 ms	6.96 ms	44000.0000	266 MB

The test code:

namespaceSpecification.Benchmark{[MemoryDiagnoser]publicclassIncludeBenchmark{privatereadonlyintmax=100000;privatereadonlySpecificationEvaluatorevaluator=SpecificationEvaluator.Default;privatereadonlySpecificationEvaluatorevaluatorCached=SpecificationEvaluator.Cached;privatereadonlySpecification<Store>specInclude=newStoreIncludeProductsSpec();privatereadonlyIQueryable<Store>Stores;publicIncludeBenchmark(){Stores=newBenchmarkDbContext().Stores.AsQueryable();}[Benchmark]publicvoidEFIncludeExpression(){Expression<Func<Store,IEnumerable<Product>>>IncludeExpr= x =>x.Products;Expression<Func<Product,CustomFields>>ThenIncludeExpr= x =>x.CustomFields;for(inti=0;i<max;i++){_=Stores.Include(IncludeExpr).ThenInclude(ThenIncludeExpr);}}[Benchmark]publicvoidSpecIncludeExpression(){for(inti=0;i<max;i++){_=evaluator.GetQuery(Stores,specInclude);}}[Benchmark]publicvoidSpecIncludeExpressionCached(){for(inti=0;i<max;i++){_=evaluatorCached.GetQuery(Stores,specInclude);}}}publicclassBenchmarkDbContext:DbContext{publicDbSet<Store>Stores{get;set;}protectedoverridevoidOnConfiguring(DbContextOptionsBuilderoptionsBuilder){base.OnConfiguring(optionsBuilder);optionsBuilder.UseSqlServer("Server=(localdb)\\mssqllocaldb;Integrated Security=SSPI;Initial Catalog=SpecificationEFTestsDB;ConnectRetryCount=0");}}publicclassStoreIncludeProductsSpec:Specification<Store>{publicStoreIncludeProductsSpec(){Query.Include(x =>x.Products).ThenInclude(x =>x.CustomFields);}}publicclassStore{publicintId{get;set;}publicIEnumerable<Product>Products{get;set;}}publicclassProduct{publicintId{get;set;}publicCustomFieldsCustomFields{get;set;}}publicclassCustomFields{publicintId{get;set;}publicstringCustomText1{get;set;}publicstringCustomText2{get;set;}}}

[devbased]

Closes#187.

Summary

Query should be built by ORM (EfCore and EF6 in our case). This work uses data from IncludeExpressionInfo to call public API of specified libraries instead of building query manually.

Part with EF6 is not done yet

[fiseni]

Hey @ardalis,

If you don't have any comments here, I'll merge this. In the end, we decided to be an opt-in feature, so there is no high risk of implementing it.

[fiseni]

Hey @devbased,

The tests are passing and everything seems working fine with caching enabled. But, I'm somewhat uncomfortable with benchmark results. We're doing additional work on top of EF, and we just can't be faster than EF. What we're missing here?

When you compile the lambda for the proxy delegates, perhaps it compiles recursively the inner LambdaExpression too (the query expression)? We're passing the compiled version? 🤔 That would not be good.

Benchmark results for 1.000.000 iterations:

Method	Mean	Error	StdDev	Gen 0	Allocated
EFIncludeExpression	4.600 s	0.0849 s	0.1441 s	502000.0000	3 GB
SpecIncludeExpression	5.619 s	0.1122 s	0.1292 s	585000.0000	3 GB
SpecIncludeExpressionCached	3.642 s	0.0724 s	0.1015 s	415000.0000	2 GB

[fiseni]

Ok, the issue was in the benchmark tests. For the EF direct approach, the expressions were created on each iteration, while the spec versions have the expression stored as a state. Once we fix this, then we get much more accurate and reasonable results. Actually, now we can see how much slower we are compared with direct EF usage 😞

Benchmark results for 100.000 iterations:

Method	Mean	Error	StdDev	Gen 0	Allocated
EFIncludeExpression	273.9 ms	1.10 ms	0.86 ms	31500.0000	189 MB
SpecIncludeExpression	642.3 ms	7.26 ms	6.79 ms	69000.0000	417 MB
SpecIncludeExpressionCached	377.7 ms	7.44 ms	6.96 ms	44000.0000	266 MB

The test code:

namespaceSpecification.Benchmark{[MemoryDiagnoser]publicclassIncludeBenchmark{privatereadonlyintmax=100000;privatereadonlySpecificationEvaluatorevaluator=SpecificationEvaluator.Default;privatereadonlySpecificationEvaluatorevaluatorCached=SpecificationEvaluator.Cached;privatereadonlySpecification<Store>specInclude=newStoreIncludeProductsSpec();privatereadonlyIQueryable<Store>Stores;publicIncludeBenchmark(){Stores=newBenchmarkDbContext().Stores.AsQueryable();}[Benchmark]publicvoidEFIncludeExpression(){Expression<Func<Store,IEnumerable<Product>>>IncludeExpr= x =>x.Products;Expression<Func<Product,CustomFields>>ThenIncludeExpr= x =>x.CustomFields;for(inti=0;i<max;i++){_=Stores.Include(IncludeExpr).ThenInclude(ThenIncludeExpr);}}[Benchmark]publicvoidSpecIncludeExpression(){for(inti=0;i<max;i++){_=evaluator.GetQuery(Stores,specInclude);}}[Benchmark]publicvoidSpecIncludeExpressionCached(){for(inti=0;i<max;i++){_=evaluatorCached.GetQuery(Stores,specInclude);}}}publicclassBenchmarkDbContext:DbContext{publicDbSet<Store>Stores{get;set;}protectedoverridevoidOnConfiguring(DbContextOptionsBuilderoptionsBuilder){base.OnConfiguring(optionsBuilder);optionsBuilder.UseSqlServer("Server=(localdb)\\mssqllocaldb;Integrated Security=SSPI;Initial Catalog=SpecificationEFTestsDB;ConnectRetryCount=0");}}publicclassStoreIncludeProductsSpec:Specification<Store>{publicStoreIncludeProductsSpec(){Query.Include(x =>x.Products).ThenInclude(x =>x.CustomFields);}}publicclassStore{publicintId{get;set;}publicIEnumerable<Product>Products{get;set;}}publicclassProduct{publicintId{get;set;}publicCustomFieldsCustomFields{get;set;}}publicclassCustomFields{publicintId{get;set;}publicstringCustomText1{get;set;}publicstringCustomText2{get;set;}}}

[devbased]

Closes#187.

Summary

Query should be built by ORM (EfCore and EF6 in our case). This work uses data from IncludeExpressionInfo to call public API of specified libraries instead of building query manually.

Part with EF6 is not done yet

[fiseni]

Hey @ardalis,

If you don't have any comments here, I'll merge this. In the end, we decided to be an opt-in feature, so there is no high risk of implementing it.

[fiseni]

Hey @devbased,

The tests are passing and everything seems working fine with caching enabled. But, I'm somewhat uncomfortable with benchmark results. We're doing additional work on top of EF, and we just can't be faster than EF. What we're missing here?

When you compile the lambda for the proxy delegates, perhaps it compiles recursively the inner LambdaExpression too (the query expression)? We're passing the compiled version? 🤔 That would not be good.

Benchmark results for 1.000.000 iterations:

Method	Mean	Error	StdDev	Gen 0	Allocated
EFIncludeExpression	4.600 s	0.0849 s	0.1441 s	502000.0000	3 GB
SpecIncludeExpression	5.619 s	0.1122 s	0.1292 s	585000.0000	3 GB
SpecIncludeExpressionCached	3.642 s	0.0724 s	0.1015 s	415000.0000	2 GB

[fiseni]

Ok, the issue was in the benchmark tests. For the EF direct approach, the expressions were created on each iteration, while the spec versions have the expression stored as a state. Once we fix this, then we get much more accurate and reasonable results. Actually, now we can see how much slower we are compared with direct EF usage 😞

Benchmark results for 100.000 iterations:

Method	Mean	Error	StdDev	Gen 0	Allocated
EFIncludeExpression	273.9 ms	1.10 ms	0.86 ms	31500.0000	189 MB
SpecIncludeExpression	642.3 ms	7.26 ms	6.79 ms	69000.0000	417 MB
SpecIncludeExpressionCached	377.7 ms	7.44 ms	6.96 ms	44000.0000	266 MB

The test code:

namespaceSpecification.Benchmark{[MemoryDiagnoser]publicclassIncludeBenchmark{privatereadonlyintmax=100000;privatereadonlySpecificationEvaluatorevaluator=SpecificationEvaluator.Default;privatereadonlySpecificationEvaluatorevaluatorCached=SpecificationEvaluator.Cached;privatereadonlySpecification<Store>specInclude=newStoreIncludeProductsSpec();privatereadonlyIQueryable<Store>Stores;publicIncludeBenchmark(){Stores=newBenchmarkDbContext().Stores.AsQueryable();}[Benchmark]publicvoidEFIncludeExpression(){Expression<Func<Store,IEnumerable<Product>>>IncludeExpr= x =>x.Products;Expression<Func<Product,CustomFields>>ThenIncludeExpr= x =>x.CustomFields;for(inti=0;i<max;i++){_=Stores.Include(IncludeExpr).ThenInclude(ThenIncludeExpr);}}[Benchmark]publicvoidSpecIncludeExpression(){for(inti=0;i<max;i++){_=evaluator.GetQuery(Stores,specInclude);}}[Benchmark]publicvoidSpecIncludeExpressionCached(){for(inti=0;i<max;i++){_=evaluatorCached.GetQuery(Stores,specInclude);}}}publicclassBenchmarkDbContext:DbContext{publicDbSet<Store>Stores{get;set;}protectedoverridevoidOnConfiguring(DbContextOptionsBuilderoptionsBuilder){base.OnConfiguring(optionsBuilder);optionsBuilder.UseSqlServer("Server=(localdb)\\mssqllocaldb;Integrated Security=SSPI;Initial Catalog=SpecificationEFTestsDB;ConnectRetryCount=0");}}publicclassStoreIncludeProductsSpec:Specification<Store>{publicStoreIncludeProductsSpec(){Query.Include(x =>x.Products).ThenInclude(x =>x.CustomFields);}}publicclassStore{publicintId{get;set;}publicIEnumerable<Product>Products{get;set;}}publicclassProduct{publicintId{get;set;}publicCustomFieldsCustomFields{get;set;}}publicclassCustomFields{publicintId{get;set;}publicstringCustomText1{get;set;}publicstringCustomText2{get;set;}}}

[devbased]

Closes#187.

Summary

Query should be built by ORM (EfCore and EF6 in our case). This work uses data from IncludeExpressionInfo to call public API of specified libraries instead of building query manually.

Part with EF6 is not done yet

[fiseni]

Hey @ardalis,

If you don't have any comments here, I'll merge this. In the end, we decided to be an opt-in feature, so there is no high risk of implementing it.

[fiseni]

Hey @devbased,

The tests are passing and everything seems working fine with caching enabled. But, I'm somewhat uncomfortable with benchmark results. We're doing additional work on top of EF, and we just can't be faster than EF. What we're missing here?

When you compile the lambda for the proxy delegates, perhaps it compiles recursively the inner LambdaExpression too (the query expression)? We're passing the compiled version? 🤔 That would not be good.

Benchmark results for 1.000.000 iterations:

Method	Mean	Error	StdDev	Gen 0	Allocated
EFIncludeExpression	4.600 s	0.0849 s	0.1441 s	502000.0000	3 GB
SpecIncludeExpression	5.619 s	0.1122 s	0.1292 s	585000.0000	3 GB
SpecIncludeExpressionCached	3.642 s	0.0724 s	0.1015 s	415000.0000	2 GB

[fiseni]

Ok, the issue was in the benchmark tests. For the EF direct approach, the expressions were created on each iteration, while the spec versions have the expression stored as a state. Once we fix this, then we get much more accurate and reasonable results. Actually, now we can see how much slower we are compared with direct EF usage 😞

Benchmark results for 100.000 iterations:

Method	Mean	Error	StdDev	Gen 0	Allocated
EFIncludeExpression	273.9 ms	1.10 ms	0.86 ms	31500.0000	189 MB
SpecIncludeExpression	642.3 ms	7.26 ms	6.79 ms	69000.0000	417 MB
SpecIncludeExpressionCached	377.7 ms	7.44 ms	6.96 ms	44000.0000	266 MB

The test code:

namespaceSpecification.Benchmark{[MemoryDiagnoser]publicclassIncludeBenchmark{privatereadonlyintmax=100000;privatereadonlySpecificationEvaluatorevaluator=SpecificationEvaluator.Default;privatereadonlySpecificationEvaluatorevaluatorCached=SpecificationEvaluator.Cached;privatereadonlySpecification<Store>specInclude=newStoreIncludeProductsSpec();privatereadonlyIQueryable<Store>Stores;publicIncludeBenchmark(){Stores=newBenchmarkDbContext().Stores.AsQueryable();}[Benchmark]publicvoidEFIncludeExpression(){Expression<Func<Store,IEnumerable<Product>>>IncludeExpr= x =>x.Products;Expression<Func<Product,CustomFields>>ThenIncludeExpr= x =>x.CustomFields;for(inti=0;i<max;i++){_=Stores.Include(IncludeExpr).ThenInclude(ThenIncludeExpr);}}[Benchmark]publicvoidSpecIncludeExpression(){for(inti=0;i<max;i++){_=evaluator.GetQuery(Stores,specInclude);}}[Benchmark]publicvoidSpecIncludeExpressionCached(){for(inti=0;i<max;i++){_=evaluatorCached.GetQuery(Stores,specInclude);}}}publicclassBenchmarkDbContext:DbContext{publicDbSet<Store>Stores{get;set;}protectedoverridevoidOnConfiguring(DbContextOptionsBuilderoptionsBuilder){base.OnConfiguring(optionsBuilder);optionsBuilder.UseSqlServer("Server=(localdb)\\mssqllocaldb;Integrated Security=SSPI;Initial Catalog=SpecificationEFTestsDB;ConnectRetryCount=0");}}publicclassStoreIncludeProductsSpec:Specification<Store>{publicStoreIncludeProductsSpec(){Query.Include(x =>x.Products).ThenInclude(x =>x.CustomFields);}}publicclassStore{publicintId{get;set;}publicIEnumerable<Product>Products{get;set;}}publicclassProduct{publicintId{get;set;}publicCustomFieldsCustomFields{get;set;}}publicclassCustomFields{publicintId{get;set;}publicstringCustomText1{get;set;}publicstringCustomText2{get;set;}}}

[devbased]

Closes#187.

Summary

Query should be built by ORM (EfCore and EF6 in our case). This work uses data from IncludeExpressionInfo to call public API of specified libraries instead of building query manually.

Part with EF6 is not done yet

[fiseni]

Hey @ardalis,

If you don't have any comments here, I'll merge this. In the end, we decided to be an opt-in feature, so there is no high risk of implementing it.

[fiseni]

Hey @devbased,

The tests are passing and everything seems working fine with caching enabled. But, I'm somewhat uncomfortable with benchmark results. We're doing additional work on top of EF, and we just can't be faster than EF. What we're missing here?

When you compile the lambda for the proxy delegates, perhaps it compiles recursively the inner LambdaExpression too (the query expression)? We're passing the compiled version? 🤔 That would not be good.

Benchmark results for 1.000.000 iterations:

Method	Mean	Error	StdDev	Gen 0	Allocated
EFIncludeExpression	4.600 s	0.0849 s	0.1441 s	502000.0000	3 GB
SpecIncludeExpression	5.619 s	0.1122 s	0.1292 s	585000.0000	3 GB
SpecIncludeExpressionCached	3.642 s	0.0724 s	0.1015 s	415000.0000	2 GB

[fiseni]

Ok, the issue was in the benchmark tests. For the EF direct approach, the expressions were created on each iteration, while the spec versions have the expression stored as a state. Once we fix this, then we get much more accurate and reasonable results. Actually, now we can see how much slower we are compared with direct EF usage 😞

Benchmark results for 100.000 iterations:

Method	Mean	Error	StdDev	Gen 0	Allocated
EFIncludeExpression	273.9 ms	1.10 ms	0.86 ms	31500.0000	189 MB
SpecIncludeExpression	642.3 ms	7.26 ms	6.79 ms	69000.0000	417 MB
SpecIncludeExpressionCached	377.7 ms	7.44 ms	6.96 ms	44000.0000	266 MB

The test code:

namespaceSpecification.Benchmark{[MemoryDiagnoser]publicclassIncludeBenchmark{privatereadonlyintmax=100000;privatereadonlySpecificationEvaluatorevaluator=SpecificationEvaluator.Default;privatereadonlySpecificationEvaluatorevaluatorCached=SpecificationEvaluator.Cached;privatereadonlySpecification<Store>specInclude=newStoreIncludeProductsSpec();privatereadonlyIQueryable<Store>Stores;publicIncludeBenchmark(){Stores=newBenchmarkDbContext().Stores.AsQueryable();}[Benchmark]publicvoidEFIncludeExpression(){Expression<Func<Store,IEnumerable<Product>>>IncludeExpr= x =>x.Products;Expression<Func<Product,CustomFields>>ThenIncludeExpr= x =>x.CustomFields;for(inti=0;i<max;i++){_=Stores.Include(IncludeExpr).ThenInclude(ThenIncludeExpr);}}[Benchmark]publicvoidSpecIncludeExpression(){for(inti=0;i<max;i++){_=evaluator.GetQuery(Stores,specInclude);}}[Benchmark]publicvoidSpecIncludeExpressionCached(){for(inti=0;i<max;i++){_=evaluatorCached.GetQuery(Stores,specInclude);}}}publicclassBenchmarkDbContext:DbContext{publicDbSet<Store>Stores{get;set;}protectedoverridevoidOnConfiguring(DbContextOptionsBuilderoptionsBuilder){base.OnConfiguring(optionsBuilder);optionsBuilder.UseSqlServer("Server=(localdb)\\mssqllocaldb;Integrated Security=SSPI;Initial Catalog=SpecificationEFTestsDB;ConnectRetryCount=0");}}publicclassStoreIncludeProductsSpec:Specification<Store>{publicStoreIncludeProductsSpec(){Query.Include(x =>x.Products).ThenInclude(x =>x.CustomFields);}}publicclassStore{publicintId{get;set;}publicIEnumerable<Product>Products{get;set;}}publicclassProduct{publicintId{get;set;}publicCustomFieldsCustomFields{get;set;}}publicclassCustomFields{publicintId{get;set;}publicstringCustomText1{get;set;}publicstringCustomText2{get;set;}}}