[ci] Upload screenshots, logs, and Xcode test results for drive and integration_test runs by jmagman · Pull Request #7430 · flutter/packages

jmagman · 2024-08-16T17:34:33Z

Native Xcode tests will output a helpful "xcresult" package on failure containing logs, screenshots, and screen recordings. Zip and upload these results when tests fail.
Pass flutter test --debug-logs-dir flag to upload logs like Upload DerivedData logs in CI flutter#142643.
Pass flutter drive --screenshot flag to upload screenshots on timeout like Take drive screenshot on test failure before app is stopped flutter#96973.

Example of failing Xcode analyzer build has the zipped xcresult attached as a log.

The unzipped xcresult looks like this in Xcode:

A failing "native test" step build:

Fixesflutter/flutter#144795

Pre-launch Checklist

If you need help, consider asking for advice on the #hackers-new channel on Discord.

stuartmorgan-g · 2024-10-08T19:40:57Z

From triage: What's the status of this? Is it still something we plan to deploy? It would definitely be useful for debugging!

jmagman · 2024-10-24T23:58:18Z

From triage: What's the status of this? Is it still something we plan to deploy? It would definitely be useful for debugging!

Sorry, I lost track of this. I need to page back in why the tests are failing.

This reverts commit d329678b5be29c3c03ac0d3e9dc18d3bc0d8d154.

jmagman · 2024-10-29T17:04:48Z

@stuartmorgan this is ready to review, when you get a chance.

stuartmorgan-g

LGTM

Is the FLUTTER_LOGS_DIR support low-level enough that it already applies to this repo, or do we need recipe changes too?

stuartmorgan-g · 2024-10-31T14:22:39Z

+ Directory? screenshotDirectory;
+ if (logsDirectory != null) {
+ screenshotDirectory = logsDirectory.childDirectory('$exampleName-drive');
+ }


Directory? screenshotDirectory = logsDirectory?.childDirectory(...);

Oops, I copied code that was written before null safety and clumsily migrated (likely by me).

stuartmorgan-g · 2024-10-31T14:26:52Z

 for (final File target in targets) {
+ Directory? screenshotDirectory;
+ if (logsDirectory != null) {
+ screenshotDirectory = logsDirectory.childDirectory('$exampleName-drive');


exampleName is a relative posix path; IIRC childDirectory will interpret that as a path and actually make intervening directories. Is that what we want? If not we should replace / with _ or something before this.

stuartmorgan-g · 2024-10-31T14:30:24Z

 required List<File> testFiles,
 }) async {
 final String enableExperiment = getStringArg(kEnableExperiment);
+ final Directory? logsDirectory = testFiles.isNotEmpty ? ciLogsDirectory(platform, testFiles.first.fileSystem) : null;


Why is this gated on testFiles not being empty? (I'd have to check but I wouldn't think this would ever be called with an empty test list.)

I think I was just guarding the .first, I'll remove since I think you're right it's never called with an empty list.

jmagman · 2024-11-04T19:51:12Z

Is the FLUTTER_LOGS_DIR support low-level enough that it already applies to this repo, or do we need recipe changes too?

It already applies as of https://flutter-review.googlesource.com/c/recipes/+/39703

You can see it in the env:
https://logs.chromium.org/logs/flutter/buildbucket/cr-buildbucket/8732173142577412545/+/u/Run_package_tests/xcode_analyze_deprecation/l_execution_details

…ve and integration_test runs (flutter/packages#7430)

flutter/packages@7219431...bb5a258 2024-11-06 magder@google.com [ci] Upload screenshots, logs, and Xcode test results for drive and integration_test runs (flutter/packages#7430) 2024-11-05 737941+loic-sharma@users.noreply.github.com Remove use_modular_headers! from Podfiles (flutter/packages#7796) 2024-11-05 30872003+misos1@users.noreply.github.com [camera_avfoundation] enable more than 30 fps (flutter/packages#7394) 2024-11-05 engine-flutter-autoroll@skia.org Roll Flutter from 8591d0c to 29d40f7 (25 revisions) (flutter/packages#8027) 2024-11-05 stuartmorgan@google.com [ci] Add vector_graphics and flutter_svg to autolabeler (flutter/packages#8025) 2024-11-05 alex@mariuti.com [vector_graphics_compiler] wasm compatibility (flutter/packages#8021) If this roll has caused a breakage, revert this CL and stop the roller using the controls here: https://autoroll.skia.org/r/flutter-packages-flutter-autoroll Please CC flutter-ecosystem@google.com on the revert to ensure that a human is aware of the problem. To file a bug in Flutter: https://github.com/flutter/flutter/issues/new/choose To report a problem with the AutoRoller itself, please file a bug: https://issues.skia.org/issues/new?component=1389291&template=1850622 Documentation for the AutoRoller is here: https://skia.googlesource.com/buildbot/+doc/main/autoroll/README.md

* main: (1187 commits) [various] Update example app minSdkVersions (flutter#8035) [go_router] Activate leak testing (flutter#7546) [in_app_purchase_storekit] Add restore purchases and receipts (flutter#7964) [interactive_media_ads] Adds remaining methods for internal wrapper of the Android native `BaseManager` (flutter#7943) [google_sign_in/google_identity_services] Clear-up documentation of callbacks in various APIs and uses of those APIs (flutter#8029) [flutter_svg] wasm compatibility (flutter#8014) Applied Gradle Plugins Declaratively for Multiple Plugin Example Apps (Part 2) (flutter#8019) Roll Flutter from 29d40f7 to 73546b3 (20 revisions) (flutter#8028) [ci] Upload screenshots, logs, and Xcode test results for drive and integration_test runs (flutter#7430) Remove use_modular_headers! from Podfiles (flutter#7796) [camera_avfoundation] enable more than 30 fps (flutter#7394) Roll Flutter from 8591d0c to 29d40f7 (25 revisions) (flutter#8027) [ci] Add vector_graphics and flutter_svg to autolabeler (flutter#8025) [vector_graphics_compiler] wasm compatibility (flutter#8021) [vector_graphics*] Relax dependency constraints of vector_graphics, vector_graphics_codec, vector_graphics_compiler, flutter_svg (flutter#8018) [various] Add `missing_code_block_language_in_doc_comment` lint to flutter/packages. (flutter#6473) [various] Update example apps to Kotlin 1.9.0 (flutter#7998) [go_router] add current state getter (flutter#7651) Applied Gradle Plugins Declaratively for Multiple Plugin Example Apps (flutter#7968) Roll Flutter from f86b777 to 8591d0c (16 revisions) (flutter#8015) ... # Conflicts: # packages/quick_actions/quick_actions/CHANGELOG.md # packages/quick_actions/quick_actions_ios/CHANGELOG.md # packages/quick_actions/quick_actions_platform_interface/CHANGELOG.md

…ntegration_test runs (flutter#7430) 1. Native Xcode tests will output a helpful "xcresult" package on failure containing logs, screenshots, and screen recordings. Zip and upload these results when tests fail. 2. Pass `flutter test --debug-logs-dir` flag to upload logs like flutter/flutter#142643. 3. Pass `flutter drive --screenshot` flag to upload screenshots on timeout like flutter/flutter#96973. Example of [failing Xcode analyzer build](https://ci.chromium.org/ui/p/flutter/builders/try/Mac_arm64%20ios_platform_tests_shard_5%20master/17374/overview) has the [zipped xcresult](https://storage.googleapis.com/flutter_logs/flutter/ff98c32e-18ca-4ad4-a910-9db1d7f7e4b0/xcode%20analyze/ff98c32e-18ca-4ad4-a910-9db1d7f7e4b0/xcodebuild-2024-10-25T09:56:46.440913.zip) attached as a log. ![Screenshot 2024-10-25 at 10 10 36â�¯AM](https://github.com/user-attachments/assets/dd7ae9bc-6161-4381-8a4f-f10b8c981801) The unzipped xcresult looks like this in Xcode: ![Screenshot 2024-10-25 at 10 11 55â�¯AM](https://github.com/user-attachments/assets/d4dd8420-f272-459c-9785-ab0c03887a74) A [failing "native test" step build](https://ci.chromium.org/ui/p/flutter/builders/try/Mac_arm64%20macos_platform_tests%20master%20-%20packages/17315/overview): ![Screenshot 2024-10-25 at 10 19 55â�¯AM](https://github.com/user-attachments/assets/76a86a15-2150-482a-8b15-e3e7ac90485e) Fixesflutter/flutter#144795

jmagman marked this pull request as ready for review October 25, 2024 16:11

jmagman marked this pull request as draft October 25, 2024 16:11

jmagman force-pushed the screenshot-drive branch from 2a21d04 to d329678Compare October 25, 2024 16:48

github-actionsBot added platform-ios platform-macos p: video_player labels Oct 25, 2024

jmagman added 3 commits October 25, 2024 10:22

Add screenshot functionality to drive
f39f734

test failures
8ac52ff

Revert "test failures"
54003a8
This reverts commit d329678b5be29c3c03ac0d3e9dc18d3bc0d8d154.

jmagman force-pushed the screenshot-drive branch from d329678 to 54003a8Compare October 25, 2024 17:23

github-actionsBot removed platform-ios platform-macos p: video_player labels Oct 25, 2024

jmagman changed the title ~~Upload screenshots and Xcode test results for drive and integration_test runs~~[ci] Upload screenshots and Xcode test results for drive and integration_test runsOct 25, 2024

jmagman self-assigned this Oct 25, 2024

jmagman marked this pull request as ready for review October 25, 2024 17:24

jmagman changed the title ~~[ci] Upload screenshots and Xcode test results for drive and integration_test runs~~[ci] Upload screenshots, logs, and Xcode test results for drive and integration_test runsOct 25, 2024

jmagman requested a review from stuartmorgan-g October 25, 2024 21:10

stuartmorgan-g approved these changes Oct 31, 2024
View reviewed changes

jmagman added 5 commits November 4, 2024 21:34

Review edits
937765c

path separator
7e6e02c

deletesync
885a407

Merge branch 'main' into screenshot-drive
5719da2

New formatter
a45f634

jmagman added the autosubmit Merge PR when tree becomes green via auto submit App label Nov 6, 2024

auto-submitBot merged commit bb5a258 into flutter:mainNov 6, 2024

jmagman deleted the screenshot-drive branch November 6, 2024 06:58

engine-flutter-autoroll mentioned this pull request Nov 6, 2024
Roll Packages from 721943194945 to bb5a25815cae (6 revisions) flutter/flutter#158267
Merged

engine-flutter-autoroll added a commit to engine-flutter-autoroll/flutter that referenced this pull request Nov 6, 2024
bb5a258[ci] Upload screenshots, logs, and Xcode test results for dri…
16cad07
…ve and integration_test runs (flutter/packages#7430)

Navigation Menu

Uh oh!

[ci] Upload screenshots, logs, and Xcode test results for drive and integration_test runs - #7430

[ci] Upload screenshots, logs, and Xcode test results for drive and integration_test runs#7430
auto-submit[bot] merged 8 commits into
flutter:mainfrom
jmagman:screenshot-drive

jmagman commented Aug 16, 2024•
edited
Loading

Uh oh!

stuartmorgan-g commented Oct 8, 2024

Uh oh!

jmagman commented Oct 24, 2024

Uh oh!

jmagman commented Oct 29, 2024

Uh oh!

stuartmorgan-g left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

stuartmorgan-gOct 31, 2024

Uh oh!

jmagmanNov 5, 2024

Uh oh!

stuartmorgan-gOct 31, 2024

Uh oh!

jmagmanNov 6, 2024

Uh oh!

stuartmorgan-gOct 31, 2024

Uh oh!

jmagmanNov 5, 2024

Uh oh!

jmagman commented Nov 4, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

jmagman commented Aug 16, 2024• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Pre-launch Checklist

Uh oh!

stuartmorgan-g commented Oct 8, 2024

Uh oh!

jmagman commented Oct 24, 2024

Uh oh!

jmagman commented Oct 29, 2024

Uh oh!

stuartmorgan-g left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

stuartmorgan-gOct 31, 2024

Choose a reason for hiding this comment

Uh oh!

jmagmanNov 5, 2024

Choose a reason for hiding this comment

Uh oh!

stuartmorgan-gOct 31, 2024

Choose a reason for hiding this comment

Uh oh!

jmagmanNov 6, 2024

Choose a reason for hiding this comment

Uh oh!

stuartmorgan-gOct 31, 2024

Choose a reason for hiding this comment

Uh oh!

jmagmanNov 5, 2024

Choose a reason for hiding this comment

Uh oh!

jmagman commented Nov 4, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

jmagman commented Aug 16, 2024•
edited
Loading