Skip to content

[tests] option to include/exclude categories from Test APKs#1024

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
jonpryor merged 1 commit into from
Nov 14, 2017
Merged

[tests] option to include/exclude categories from Test APKs #1024

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 14 additions & 0 deletions Documentation/DevelopmentTips.md
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -178,6 +178,20 @@ For example:
$ tools/scripts/xabuild /t:DeployTestApks tests/locales/Xamarin.Android.Locale-Tests/Xamarin.Android.Locale-Tests.csproj
$ tools/scripts/xabuild /t:RunTestApks tests/locales/Xamarin.Android.Locale-Tests/Xamarin.Android.Locale-Tests.csproj

## Running `.apk` Projects with Include/Exclude

For example, to exclude tests that use the internet (`InetAccess` category):

$ make run-apk-tests EXCLUDECATEGORIES=InetAccess

On Windows:

$ msbuild Xamarin.Android.sln /t:RunApkTests /p:ExcludeCategories=InetAccess

`INCLUDECATEGORIES` and `IncludeCategories` function in the same fashion.

To specify multiple categories, delimit each category with a `:` character. The `:` delimiter works well with both `MSBuild` properties and `make` variables.

### Running A Single Test Fixture

A single NUnit *Test Fixture* -- a class with the `[TestFixture]`
Expand Down
5 changes: 5 additions & 0 deletions build-tools/scripts/TestApks.targets
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -152,12 +152,17 @@

<Target Name="RunTestApks"
Condition=" '@(TestApk)' != '' ">
<PropertyGroup>
<_IncludeCategories Condition=" '$(IncludeCategories)' != '' ">include=$(IncludeCategories)</_IncludeCategories>
<_ExcludeCategories Condition=" '$(ExcludeCategories)' != '' ">exclude=$(ExcludeCategories)</_ExcludeCategories>
</PropertyGroup>
<RunInstrumentationTests
Condition=" '%(TestApk.InstrumentationType)' != ''"
AdbTarget="$(_AdbTarget)"
AdbOptions="$(AdbOptions)"
Component="%(TestApk.Package)/%(TestApk.InstrumentationType)"
NUnit2TestResultsFile="%(TestApk.ResultsPath)"
InstrumentationArguments="$(_IncludeCategories);$(_ExcludeCategories)"

@jonathanpeppersjonathanpeppersNov 14, 2017•
edited
Loading

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, this is better: it lets you specify include/exclude at the same time.

My example that worked:

make run-apk-tests INCLUDECATEGORIES=InetAccess EXCLUDECATEGORIES=Bologna

Of course no category is named Bologna, but the command in the log was:

Tool /Users/jonathanpeppers/android-toolchain/sdk/platform-tools/adb execution started with arguments: shell am instrument -e "include" "InetAccess" -e "exclude" "Bologna" -w "Mono.Android_Tests/xamarin.android.runtimetests.TestInstrumentation"

TestFixture="$(TestFixture)"
ToolExe="$(AdbToolExe)"
ToolPath="$(AdbToolPath)">
Expand Down
2 changes: 1 addition & 1 deletion src/Mono.Android/Test/System.Net/ProxyTest.cs
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -6,7 +6,7 @@

namespace System.NetTests {

[TestFixture]
[TestFixture, Category ("InetAccess")]
public class ProxyTest {

// https://bugzilla.xamarin.com/show_bug.cgi?id=14968
Expand Down
2 changes: 1 addition & 1 deletion src/Mono.Android/Test/System.Net/SslTest.cs
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -10,7 +10,7 @@

namespace System.NetTests {

[TestFixture]
[TestFixture, Category ("InetAccess")]
public class SslTest {

// https://xamarin.desk.com/agent/case/35534
Expand Down
4 changes: 2 additions & 2 deletions src/Mono.Android/Test/Xamarin.Android.Net/AndroidClientHandlerTests.cs
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -39,10 +39,10 @@
using Android.OS;

namespace Xamarin.Android.NetTests {

[Category("InetAccess")]
public abstract class HttpClientHandlerTestBase
{
protected abstract HttpClientHandler CreateHandler ();
protected abstract HttpClientHandler CreateHandler ();

class Proxy : IWebProxy
{
Expand Down
2 changes: 1 addition & 1 deletion src/Mono.Android/Test/Xamarin.Android.Net/HttpClientIntegrationTests.cs
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -38,7 +38,7 @@
using System.IO;

namespace Xamarin.Android.NetTests {

[Category ("InetAccess")]
public abstract class HttpClientIntegrationTestBase
{
protected abstract HttpClientHandler CreateHandler ();
Expand Down
4 changes: 2 additions & 2 deletions src/Xamarin.Android.NUnitLite/Gui/Activities/TestSuiteActivity.cs
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -103,12 +103,12 @@ protected override void OnResume ()

protected virtual IEnumerable <string> GetIncludedCategories ()
{
return null;
yield break;
}

protected virtual IEnumerable <string> GetExcludedCategories ()
{
return null;
yield break;
}

// Subclasses can override this method to update the test filtering that the runner will use.
Expand Down
3 changes: 2 additions & 1 deletion src/Xamarin.Android.NUnitLite/Gui/AndroidRunner.cs
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -389,7 +389,8 @@ static void ChainCategoryFilter (IEnumerable <string> categories, bool negate, r
gotCategories = true;
}

chain = new AndFilter (chain, negate ? (TestFilter)new NotFilter (filter) : (TestFilter)filter);
if (gotCategories)
chain = new AndFilter (chain, negate ? (TestFilter)new NotFilter (filter) : (TestFilter)filter);
}

if (!gotCategories)
Expand Down
14 changes: 12 additions & 2 deletions src/Xamarin.Android.NUnitLite/Gui/Instrumentations/TestSuiteInstrumentation.cs
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -146,12 +146,22 @@ protected void AddTest (Assembly assembly)

protected virtual IEnumerable <string> GetIncludedCategories ()
{
return null;
string include = arguments?.GetString ("include");
if (!string.IsNullOrEmpty (include)) {
foreach (var category in include.Split (':')) {
yield return category;
}
}
}

protected virtual IEnumerable <string> GetExcludedCategories ()
{
return null;

@jonathanpeppersjonathanpeppersNov 13, 2017

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Instead of returning null, these will be the equivalent of yield break

@jonpryorjonpryorNov 14, 2017

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

TestSuiteActivity should also be updated to not return null.

string exclude = arguments?.GetString ("exclude");
if (!string.IsNullOrEmpty (exclude)) {
foreach (var category in exclude.Split (':')) {
yield return category;
}
}
}

protected virtual void UpdateFilter ()
Expand Down
7 changes: 6 additions & 1 deletion tests/Xamarin.Android.Bcl-Tests/TestInstrumentation.cs
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -29,7 +29,12 @@ protected override void AddTests ()

protected override IEnumerable<string> GetExcludedCategories ()
{
return App.GetExcludedCategories ();
foreach (var category in base.GetExcludedCategories ()) {
yield return category;
}
foreach (var category in App.GetExcludedCategories ()) {
yield return category;
}
}

protected override void UpdateFilter ()
Expand Down