[Xamarin.Android.Build.Tests] Add support for testing the UI by dellis1972 · Pull Request #2855 · dotnet/android

dellis1972 · 2019-03-20T13:16:51Z

Context #2608

This commit adds support for testing that the
app launches on device AND can be interacted
with. It does this by making use of a few features
of adb.

First is adb input. This allow us to send taps and
clicks to the device. It simulates user input. So
we can use this to say click a button.

Next up is the uiautomator this allows use to get
the complete ui layout for the active activity.
Using

exec-out uiautomator dump /dev/tty

we can send this data to the console. As it is
xml we can then parse it looking for our ui element
we want. We can then get the bounds of the control
and pass that information to adb input tap.

This commit also adds support for testing that
the timezone set on a device is correctly picked
up by mono.

And finally we add support for testing that the
debugger can actaully connect to the application.
These tests are only run if the commerial parts
of the product are available.

Seeing as all of these new tests REQUIRE the
emulator or device they have been split out into
a new MSBuildDeviceIntegration project. This
is built as part of the main solution. However
it is run as part of the make run-apk-tests
call. This is so we can be sure the emulator is
running.

In future any MSBuild related tests that REQUIRE
a device should be placed in this new project.

jonpryor · 2019-04-03T14:19:34Z

Conceptual question: for this to work, there needs to be an Android device available on which to run the test and manipulate the UI.

Currently, unless I've forgotten something, Xamarin.Android.Build.Tests.dll doesn't consistently execute in an environment in which the emulator is running; the emulator is currently created as part of the RunApkTests target, which is run in parallel with the RunNUnitTests target. There is no guaranteed reliability that the emulator will -- or will not! -- be running when the local NUnit tests are run.

Perhaps we're just really lucky so far?

jonpryor · 2019-04-11T01:38:38Z

Ouch: https://jenkins.mono-project.com/job/xamarin-android-pr-pipeline-release/517/testReport/Xamarin.Android.Build.Tests.DeploymentTest/CheckTimeZoneInfoIsCorrect/__Australia_NSW_____Release/

System.InvalidOperationException : Cannot process request because the process (87383) has exited.
at (wrapper managed-to-native) System.Reflection.MonoMethod.InternalInvoke(System.Reflection.MonoMethod,object,object[],System.Exception&)
at System.Reflection.MonoMethod.Invoke (System.Object obj, System.Reflection.BindingFlags invokeAttr, System.Reflection.Binder binder, System.Object[] parameters, System.Globalization.CultureInfo culture) [0x0006a] in <6d83221bea054562936f2f38bd41deeb>:0

dellis1972 · 2019-04-11T12:10:12Z

Test failures are as follows

Test Result (5 failures / -1)
SslTest.Mono.Android_Tests, System.NetTests.SslTest.SslWithinTasksShouldWork / Release
SslTest.Mono.Android_TestsMultiDex, System.NetTests.SslTest.SslWithinTasksShouldWork / Release
SslTest.Mono.Android_TestsAppBundle, System.NetTests.SslTest.SslWithinTasksShouldWork / Release
AndroidClientHandlerTests.Mono.Android_Tests, Xamarin.Android.NetTests.AndroidClientHandlerTests.Sanity_Tls_1_2_Url_WithMonoClientHandlerFails / Release
AndroidClientHandlerTests.Mono.Android_TestsAppBundle, Xamarin.Android.NetTests.AndroidClientHandlerTests.Sanity_Tls_1_2_Url_WithMonoClientHandlerFails / Release

dellis1972 · 2019-04-11T12:11:40Z

Ouch: https://jenkins.mono-project.com/job/xamarin-android-pr-pipeline-release/517/testReport/Xamarin.Android.Build.Tests.DeploymentTest/CheckTimeZoneInfoIsCorrect/__Australia_NSW_____Release/

System.InvalidOperationException : Cannot process request because the process (87383) has exited.
at (wrapper managed-to-native) System.Reflection.MonoMethod.InternalInvoke(System.Reflection.MonoMethod,object,object[],System.Exception&)
at System.Reflection.MonoMethod.Invoke (System.Object obj, System.Reflection.BindingFlags invokeAttr, System.Reflection.Binder binder, System.Object[] parameters, System.Globalization.CultureInfo culture) [0x0006a] in <6d83221bea054562936f2f38bd41deeb>:0

That is a very very weird crash. Maybe its from the nunit runner... Its an actual test crash.

This commit adds support for testing that the app launches on device AND can be interacted with. It does this by making use of a few features of `adb`. First is `adb input`. This allow us to send taps and clicks to the device. It simulates user input. So we can use this to say click a button. Next up is the `uiautomator` this allows use to get the complete ui layout for the active activity. Using exec-out uiautomator dump /dev/tty we can send this data to the console. As it is xml we can then parse it looking for our ui element we want. We can then get the bounds of the control and pass that information to `adb input tap`. This commit also adds support for testing that the timezone set on a device is correctly picked up by mono. And finally we add support for testing that the debugger can actaully connect to the application. These tests are only run if the commerial parts of the product are available. Seeing as all of these new tests REQUIRE the emulator or device they have been split out into a new `MSBuildDeviceIntegration` project. This is built as part of the main solution. However it is run as part of the `make run-apk-tests` call. This is so we can be sure the emulator is running. In future any MSBuild related tests that REQUIRE a device should be placed in this new project.

dellis1972 · 2019-04-15T20:09:36Z

Not sure why this test failed https://jenkins.mono-project.com/job/xamarin-android-pr-pipeline-release/569/testReport/junit/Mono/Android_TestsMultiDex/Possible_Crash___Release/

It does not seem to be related to this PR.

dellis1972 · 2019-04-15T20:10:46Z

Commercial build failure is

 EXEC : error : option --boot-class-path not allowed with target 11

looks like that java version problem

jonpryor · 2019-04-16T00:51:47Z

@dellis1972 asked:

Not sure why this test failed: https://jenkins.mono-project.com/job/xamarin-android-pr-pipeline-release/569/testReport/junit/Mono/Android_TestsMultiDex/Possible_Crash___Release/

The problem is described in commit a972624:

We have been seeing adb crash during RunApkTests:

If you download xa-test-results-v9.2.99.163_Darwin-x86_64_pr_07dd6aee-Release.zip, then bin/TestRelease/msbuild-20190415T070436-Target-RunApkTests.binlog contains:

Task "RunInstrumentationTests"
Task Parameter:AdbTarget=-s emulator-5570
Task Parameter:LogLevel=Verbose
Task Parameter:PackageName=Mono.Android_TestsMultiDex
Task Parameter:Component=Mono.Android_TestsMultiDex/xamarin.android.runtimetests.TestInstrumentation
Task Parameter:NUnit2TestResultsFile=…bin/TestRelease/TestResult-Mono.Android_TestsMultiDex.xml
Task Parameter:LogcatFilename=…/bin/TestRelease/logcat-Release-Aot-Mono.Android_TestsMultiDex.txt
Task Parameter:ToolExe=adb
Task Parameter:ToolPath=/Users/builder/android-toolchain/sdk/platform-tools
Task Parameter:Timeout=3600000
Executing: /Users/builder/android-toolchain/sdk/platform-tools/adb -s emulator-5570 shell am instrument -e "loglevel Verbose" -w "Mono.Android_TestsMultiDex/xamarin.android.runtimetests.TestInstrumentation"
Executing: /Users/builder/android-toolchain/sdk/platform-tools/adb -s emulator-5570 logcat -v threadtime -d
appending stdout to file: …/bin/TestRelease/logcat-Release-Aot-Mono.Android_TestsMultiDex.txt
/Users/builder/jenkins/workspace/xamarin-android-pr-pipeline-release/xamarin-android/build-tools/scripts/TestApks.targets(226,5): warning : * daemon not running; starting now at tcp:5037

i.e. adb is "not running" when it most certainly should have already been running, i.e. it crashed.

Looks like bumping to platform-tools 28.0.2 didn't entirely fix things. :-(

See also 1b53d15

jonathanpeppers · 2019-04-16T15:52:40Z

Ah! we merged this and it breaks the Windows build...

external\debugger-libs\Mono.Debugging.Soft\SoftDebuggerSession.cs(2606,30): Error CS0234: The type or namespace name 'Unix' does not exist in the namespace 'Mono' (are you missing an assembly reference?)
Process 'msbuild.exe' exited with code '1'.

Commit 66d0165 [broke the macOS build][0]: …/xamarin-android/tests/MSBuildDeviceIntegration/MSBuildDeviceIntegration.targets(18,5): error MSB3030: Could not copy the file "../../bin/TestRelease/Xamarin.Android.Cecil.Mdb.dll" because it was not found. This was probably "overlooked" because the builds for PR #2980 didn't include bf58a88/PR #2855. Update the paths used for `Xamarin.Android.Cecil*.dll` so that `MSBuildDeviceIntegration.targets` can run without error. [0]: https://jenkins.mono-project.com/view/Xamarin.Android/job/xamarin-android/463/

dellis1972 added the do-not-merge PR should not be merged. label Mar 20, 2019

dellis1972 force-pushed the addondevicetesting branch 3 times, most recently from 83d7e15 to 3da30ecCompare April 1, 2019 09:01

dellis1972 removed the do-not-merge PR should not be merged. label Apr 1, 2019

dellis1972 force-pushed the addondevicetesting branch from f13fd74 to efdfb57Compare April 2, 2019 10:01

dellis1972 marked this pull request as ready for review April 2, 2019 10:51

dellis1972 requested a review from jonathanpeppers as a code owner April 2, 2019 10:51

jonathanpeppers reviewed Apr 2, 2019
View reviewed changes

jonpryor reviewed Apr 2, 2019
View reviewed changes

Comment threadsrc/Xamarin.Android.Build.Tasks/Tests/Xamarin.Android.Build.Tests/DeploymentTest.cs Outdated

jonpryor reviewed Apr 2, 2019
View reviewed changes

Comment threadsrc/Xamarin.Android.Build.Tasks/Tests/Xamarin.Android.Build.Tests/DeploymentTest.cs Outdated

jonpryor reviewed Apr 2, 2019
View reviewed changes

Comment threadsrc/Xamarin.Android.Build.Tasks/Tests/Xamarin.Android.Build.Tests/Utilities/BaseTest.cs Outdated

dellis1972 force-pushed the addondevicetesting branch from d8857ba to 2044732Compare April 3, 2019 10:25

dellis1972 force-pushed the addondevicetesting branch 3 times, most recently from b7b1dd7 to bc6ba03Compare April 9, 2019 12:26

jonathanpeppers reviewed Apr 9, 2019
View reviewed changes

Comment threadsrc/Xamarin.Android.Build.Tasks/Tests/Xamarin.Android.Build.Tests/Utilities/DeviceTest.cs Outdated

dellis1972 force-pushed the addondevicetesting branch 4 times, most recently from 0bca494 to 5e6d395Compare April 10, 2019 13:21

jonpryor reviewed Apr 10, 2019
View reviewed changes

Comment thread.gitmodules Outdated

jonpryor reviewed Apr 10, 2019
View reviewed changes

Comment threadsrc/Xamarin.Android.Build.Tasks/Tests/Xamarin.Android.Build.Tests/Utilities/BaseTest.cs Outdated

dellis1972 force-pushed the addondevicetesting branch 2 times, most recently from 5a05ee3 to 953ebc5Compare April 10, 2019 19:42

jonpryor reviewed Apr 11, 2019
View reviewed changes

Comment threadsrc/Xamarin.Android.Build.Tasks/Tests/Xamarin.Android.Build.Tests/Utilities/DeviceTest.cs Outdated

jonpryor reviewed Apr 11, 2019
View reviewed changes

Comment threadsrc/Xamarin.Android.Build.Tasks/Tests/Xamarin.Android.Build.Tests/Utilities/DeviceTest.cs

jonpryor reviewed Apr 11, 2019
View reviewed changes

Comment threadtests/MSBuildDeviceIntegration/MSBuildDeviceIntegration.targets Outdated

jonpryor reviewed Apr 11, 2019
View reviewed changes

Comment threadtests/MSBuildDeviceIntegration/MSBuildDeviceIntegration.targets Outdated

dellis1972 force-pushed the addondevicetesting branch from 953ebc5 to 13e6f67Compare April 11, 2019 08:49

dellis1972 force-pushed the addondevicetesting branch from a87c8db to acfd213Compare April 11, 2019 21:12

dellis1972 force-pushed the addondevicetesting branch from acfd213 to fb81f6aCompare April 15, 2019 08:21

Removed redundant Install in PackagingTest
b73e115

jonpryor added 4 commits April 15, 2019 20:54

$(ManagedRuntime) use should include $(ManagedRuntimeArgs)
da662e0
See also 1b53d15

Include branch information
85886e5

Fix indentation
bd00aa5

Fix indentation
2b1938d

jonpryor merged commit bf58a88 into dotnet:masterApr 16, 2019

github-actionsBot locked and limited conversation to collaborators Jan 31, 2024

Uh oh!

Conversation

dellis1972 commented Mar 20, 2019• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jonpryor commented Apr 3, 2019

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jonpryor commented Apr 11, 2019

Uh oh!

dellis1972 commented Apr 11, 2019

Uh oh!

dellis1972 commented Apr 11, 2019

Uh oh!

dellis1972 commented Apr 15, 2019

Uh oh!

dellis1972 commented Apr 15, 2019

Uh oh!

jonpryor commented Apr 16, 2019

Uh oh!

jonathanpeppers commented Apr 16, 2019

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

dellis1972 commented Mar 20, 2019•
edited
Loading