Added imitation game benchmarks

[gaaclarke]

relevant issue: flutter/flutter#47561

imitation_game is a platform for performing automated tests to compare the
performance of different UI frameworks. For example, how much memory does the
same app written in Flutter and UIKit take?

This is an MVP implementation that has one test that checks the startup time of UIKit
versus Flutter when rendering one image.

[gaaclarke]

@liyuqian Here is the draft if you want to take a peek. The technical work is done. I have to make the output pretty and write up some README files and I'll be done with the PR.

[liyuqian]

Having each implementation specify an IP.txt file somewhere and let imitation_game.dart find it recursively and modify its content is clever! I was just wondering how to circumvent the restriction that Flutter has to read from assets, and who knows if other UI frameworks couldn't have more strange constraints.

I didn't have time to dive into too many technical details but most part of smiley example look good.

My biggest concern is on how performance metrics are measured. Leaving it for each implementation seems to be not very scalable and hard to prove the fairness. I think we need to create a single measurement tool for a test and all implementations should share it. For the first frame metric, here's what I can think of:

1. imitation_game.dart calls ios/smiley/[implementation]/install_app.sh to install the app and do all necessary initializations.
2. imitation_game.dart calls ios/smiley/measure/first_frame_begin.sh to start recording iOS screen video either by triggering an iOS app to do it, or using some host side tools.
3. imitation_game.dart calls ios/smiley/[implementation]/run_test.sh. Such script first opens a black screen app to provide reference frames for the video recording. Then the script switches to home screen and opens the smiley app.
4. imitation_game.dart calls ios/smiley/measure/first_frame_end.sh to end the iOS screen video recording, compute the time between the last black screen frame and the first smiley frame, and use that time as the first frame metric.

The video recording may use up some CPU/GPU resources, but it's the same for all UI frameworks and implementations so the result should still be fair. And the measure/memory_begin.sh and measure/memory_end.sh could be much simpler using instruments or adb without screen recording.

I've also left some other high-level comments in README.md.

[liyuqian]

We could use a more general compare multiple UI frameworks than compare Flutter to alternative platforms. I can imagine that even if someone is only interested in comparing React Native and native Android/iOS, this package could still be helpful.

[gaaclarke]

done

[liyuqian]

A local mode with a mobile device and a host computer is definitely needed for development. Additionally, I think it's nice to also have a cloud mode where the tests can run from something like Firebase test lab.

[liyuqian]

It would be nice for README.md to designate an implementation as the imitation goal so all other implementations should be compared to it for validation. When differences are found, other implementations should always try to match the imitation goal.

[liyuqian]

One more thing I just recalled today from checking this doc: the test README.md should also specify

• the ideal device D*. D* is usually very fast/powerful that eliminates any user-perceivable performance differences. D* is used for imitation test: test if different UI implementations are distinguishable by users on D*. This ensures different implementations are implementing the same UIs.
• the target device D. D is usually limited in power that will surface performance differences.

For example, we can probably use Pixel 4 as D* and Moto G4 as D for Android tests.

[liyuqian]

Since we allow one UI framework to have multiple implementations (and we'll choose the best one to represent the UI framework), I think we should have one more level of subdirectories under flutter, uikit. For example

 ├─ smiley/
│ ├─ README.md
│ ├─ flutter/
│ │ ├─ liyuqian/
│ │ | └─ run_ios.sh
│ │ ├─ gaaclarke/
│ │ | └─ run_ios.sh

[gaaclarke]

I don't know how well that is going to scale. The tests take a long time to execute since compiling and launching on a device takes so long. Seems like we should have the best implementation per platform. If there were 2 different approaches it might be better to name the implementation accordingly (ex flutter_with_fast_library).

[liyuqian]

Yes, we should only maintain the best implementation and the imitation goal implementation in the long run, and only use this extra level for a few new implementations to be uploaded and tested temporarily. They'll either be eliminated or replace the old best implementation. I also agree that the name should be more informative. Sorry for my bad naming examples 😄

[liyuqian]

There may need to be another OS subdirectory under tests like

tests/ios/smiley/...
tests/android/smiley/...

This may introduce some duplicate code if a UI framework is cross-platform (e.g., Flutter), or maybe we can symlink to make tests/ios/smiley/flutter and tests/android/smiley/flutter the same. But some UI framework only works for a single OS (e.g., uikit doesn't make sense for Android). It's also very likely that run_ios.sh and the measurement tool/code would be OS specific.

[gaaclarke]

Yep, I think there will be an opportunity for sharing more between tests. I think those things will become apparent when we have more implementations and they should come organically from that.

[liyuqian]

1. Let rename run_ios.sh to run_test.sh as we can put ios in the upper level directory name. Similarly, there could also be .../android/.../run_test.sh, .../web/.../run_test.sh.
2. We should have both run_test.sh and install_app.sh. The install_app.sh would install the app on the target OS so users can freely interact with the UI to see if they look and function the same on a powerful device. The run_test.sh is for automated testing often on a slow device where no human interaction would be needed or allowed.
   • If there's only run_test.sh that does the automated test, then one could potentially just submit an app that replays a recorded video of the test as the implementation.
   • The install_app.sh can also be used for setup and other costly operations that we don't want to measure the performance. For example, regarding the first frame performance, install_app.sh could be used to build, install, and set up everything, while run_test.sh would simply launch the app and see when the first frame is presented on screen.

[liyuqian]

BTW, how do you think about changing run_test.sh to run_test.dart so the script can be run from Linux/Windows/Mac? See the discussion here.

[liyuqian]

I feel that the measurement code/tool should not live inside the implementation directory. There should only be a single measurement code/tool that's under the test (e.g., smiley) directory, and all implementations must share it to measure the performance. Otherwise, it's hard to prove that different implementations are measuring the same thing.

This might be critical when a UI framework that no one is familiar enters the arena. Say if someone invented a new framework called Glutter, we probably don't have the expertise and time to vet its measuring code and tools.

[gaaclarke]

We just have to rely on humans to read the code. Someone reading the code can say if it is measuring the same thing. Certain things are only measurable on device, like time since the process was launched. From the host computer we can just measure the time between when we asked it to launch which isn't as accurate.

If we find there is a way to share measurement code that can be factored out. Nothing about the current design disallows it.

[liyuqian]

What do multiple results correspond to? Are they for different implementations such as some_result_name for smiley/flutter/liyuqian and some_result_name2 for smiley/flutter/gaaclarke?

[gaaclarke]

No, it's a way to measure 2 different things for the same execution. For example, the startup test could measure memory as well as startup time.

[liyuqian]

I still feel that it's better to put the device-to-host reporting logic inside the test (e.g., smiley) so different target OSes and tests could choose reporting mechanisms differently. For example, for Android memory reporting, it's probably easier to just use adb on the host side so no HTTP communication is required from the device side. If multiple tests do want to use the HTTP reporting, then they could all import the same HTTP reporting module?

[gaaclarke]

There is value in having the reporting mechanism the same across all implementations. HTTP is the most universal mechanism that should be easy for users. Once we have a few tests we can probably just wrap it up in a library so when you implement a test you just call reportResult(result); and you don't even care how it's getting reported.

[liyuqian]

I like just calling reportResult without having to worry about HTTP. It feels better to me if we don't put this HTTP communication as a contract or API of the imitation game so we'll have the freedom to change.

[gaaclarke]

Thanks yuquian for your thoughts. I responded inline below:

Having each implementation specify an IP.txt file somewhere and let imitation_game.dart find it recursively and modify its content is clever! I was just wondering how to circumvent the restriction that Flutter has to read from assets, and who knows if other UI frameworks couldn't have more strange constraints.

I don't understand the concern. What is the problem with reading from assets? Reading from a text file that ships with an app should be standard and easy.

I didn't have time to dive into too many technical details but most part of smiley example look good.

My biggest concern is on how performance metrics are measured. Leaving it for each implementation seems to be not very scalable and hard to prove the fairness. I think we need to create a single measurement tool for a test and all implementations should share it. For the first frame metric, here's what I can think of:

Fairness is subjective. For example, is it fair to measure a garbage collected language versus a non garbage collected language in terms of performance? It will be up to us to judge if something is representative. When the invitation is open to submit a revision, eventually things will become more and more fair if people find fault with something and address it.

1. imitation_game.dart calls ios/smiley/[implementation]/install_app.sh to install the app and do all necessary initializations.
2. imitation_game.dart calls ios/smiley/measure/first_frame_begin.sh to start recording iOS screen video either by triggering an iOS app to do it, or using some host side tools.
3. imitation_game.dart calls ios/smiley/[implementation]/run_test.sh. Such script first opens a black screen app to provide reference frames for the video recording. Then the script switches to home screen and opens the smiley app.
4. imitation_game.dart calls ios/smiley/measure/first_frame_end.sh to end the iOS screen video recording, compute the time between the last black screen frame and the first smiley frame, and use that time as the first frame metric.

The video recording may use up some CPU/GPU resources, but it's the same for all UI frameworks and implementations so the result should still be fair. And the measure/memory_begin.sh and measure/memory_end.sh could be much simpler using instruments or adb without screen recording.

I've also left some other high-level comments in README.md.

Ideas about video recording are interesting but they are out of scope for the MVP. Their ROI is questionable. There is a lot of value to be had with comparing numerical results.

The point of the startup time test was to get a measurement of how fast it is to startup an app. For platforms like Unity this is a big concern. I wanted to show we were comparable to native speeds. The only way to measure this is on device. If we just measure a frame it isn't capturing everything.

[liyuqian]

Having each implementation specify an IP.txt file somewhere and let imitation_game.dart find it recursively and modify its content is clever! I was just wondering how to circumvent the restriction that Flutter has to read from assets, and who knows if other UI frameworks couldn't have more strange constraints.

I don't understand the concern. What is the problem with reading from assets? Reading from a text file that ships with an app should be standard and easy.

There's no concern. I was just trying to make some compliments about your way of circumventing possibly different asset directories enforced by different frameworks 😄

It seems that there's a lot to discuss about how to measure the performance metrics. I just scheduled a meeting this Friday to go over that.

[xster]

I'd list out the currently existing scenarios tested and a brief description of the scenarios

[gaaclarke]

That information should exist in the readme file for each test. I don't want it duplicated here.

[xster]

memory is self explanatory. Maybe smiley needs a more descriptive name? :D

[gaaclarke]

"Smiley" is just referencing the name in the example above. It could have been anything. What it is testing is ill-relevant to the point. =)

[xster]

your tree above didn't list out the various platform implementations. Is it a partial tree for illustration? Maybe add that to the description.

[gaaclarke]

Added <flutter project files> and '` as placeholders for those concepts.

[xster]

looks like this directory isn't in this PR. Is this readme right?

[gaaclarke]

It's just an example of the layout

[gaaclarke]

explicitly called it an example

[xster]

while we're here, should we add a cumulative cpu+gpu cost test? (in a different pr as needed)

[xster]

add a code comment for what this is for?

[gaaclarke]

done

[xster]

ditto, add code comment for what this is for?

[gaaclarke]

done

[xster]

this is probably right. Perhaps double check with Trace in observatory and assert that this callback indeed lines up time wise with the raster thread actually finishing drawing to the gpu?

[xster]

maybe add linux later if we end up using it?

[gaaclarke]

Sure, I don't know how easy it is to add later, removed.

[xster]

ditto for macos

[gaaclarke]

ditto

[xster]

I'm assuming this doesn't run right? Either edit it or remove it.

[gaaclarke]

done

[xster]

ditto for web

[gaaclarke]

done

[xster]

can't you use flutter tool for this? Maybe add a comment for the reasoning here or in the top readme

[gaaclarke]

Nope, we didn't make it that generic to be used to launch an arbitrary app.

[xster]

change headers

[gaaclarke]

done

[gaaclarke]

I had a chat with @liyuqian offline. It would be nice to also have an android example that uses adb to poll memory usage as an example of how to measure the results on the host computer.

(ex adb shell dumpsys meminfo 'your apps package name')

[gaaclarke]

Leaving leftover discussions for future PRs.