Uh oh!
There was an error while loading. Please reload this page.
[QNN] Change default rounding to UPWARD. - #4131
Merged
Merged
Conversation
tmoreau89
left a comment
Contributor
There was a problem hiding this comment.
Thank you for the thorough evaluation @anijain2305 ! Interesting results in terms of accuracy/performance tradeoffs. Let's just check with @ZihengJiang whether this would affect the quantization nightly runs.
vinx13
approved these changes
Oct 15, 2019
FrozenGene
approved these changes
Oct 16, 2019
zhenhuaw-me
approved these changes
Oct 16, 2019
zhenhuaw-me
left a comment
Contributor
There was a problem hiding this comment.
Impressive! Thanks for the solid data! It's very interesting that the rounding policy impacts performance so much.
zhiics
commented
Oct 16, 2019
Member
Thanks everyone. This is now merged. |
zhiics
commented
Oct 16, 2019
Member
@tmoreau89 Sorry. I didn't see your comment above. Should I revert? |
ContributorAuthor
This is QNN stuff. So, it would not affect automatic quantization nightly runs. So, this should be fine. |
tmoreau89
commented
Oct 16, 2019
Contributor
@zhiics no problem, no need to revert |
anijain2305 added a commit
to anijain2305/tvm
that referenced
this pull request
Oct 17, 2019
wweic pushed a commit
to neo-ai/tvm
that referenced
this pull request
Oct 18, 2019
petrex pushed a commit
to petrex/tvm
that referenced
this pull request
Oct 29, 2019
* master: (51 commits) [QNN][TFLite] Parsing QNN Add op. Adding MobilenetV2. (apache#4142) [CI] Pin NNPack pthreadtools version (apache#4152) Fix typo (apache#4144) [Relay][Frontend][TF] Add tensor array ops (apache#3798) [relay][vm] Separate VM runtime with executable (apache#4100) [PATCH] Fix undefined __floatdihf in libtvmruntime.so on aarch64. (apache#4119) [DOCKER] Pin torchvision==0.4.1 (apache#4140) [TOPI][x86] Cascade lake support. (apache#4123) [Relay] Improve build error when no lowered funcs are produced (apache#4132) [RUNTIME] Refactor object python FFI to new protocol. (apache#4128) Update PULL_REQUEST_TEMPLATE.md Adding support for dequantizing from int32 to float32. (apache#4130) [Relay][Training] Add and fix gradients (apache#4126) [QNN] Change default rouning to UPWARD. (apache#4131) Fix infer type of kernel in dense. (apache#4125) [Relay][AlterOpLayout] NHWC to NCHWc pad operator. (apache#4103) [ARITH] Fix lowering of floormod(x, y) != 0 (apache#4127) [RFC][RUNTIME] Introduce new object protocol. (apache#4115) [Relay][Topi] Disable conv NHWC pack int8. (apache#4038) Update task_cpp_unittest.sh ...
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
6 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR is to change the default rounding from TONEAREST to UPWARD. QNN ops are converted to a sequence of Relay operator while compilation. TONEAREST rounding leads to many more relay operators as shown here
https://github.com/dmlc/tvm/blob/11a3a777ec78ad81ce12a940d07521870f546ad1/src/relay/qnn/util.cc#L113-L124
This PR makes a case for the default rounding option to be UPWARD.
I ran all TFLite hosted Inception Quantized models on a Cascade Lake machine with these two rounding options. I am giving the comparison numbers that show that UPWARD rounding is overall better alternative.
Accuracy Comparison
Accuracy is same regardless of the rounding choice. Tested over 50k images with pre-processing.
Latency Comparison
Because TONEAREST leads to many more operators, the observed latency is higher.
Compilation Time Comparison
ToNearest has very high compilation time. Though not included, it blows up drastically for mobilenet V1.
Memory Footprint
Though I don't have numbers for that yet. @shoubhik also pointed out that memory footprint for the params increased with TONEAREST. This is mostly because TONEAREST introduces some more constants in the network, and they are in
int64.Background
Background about QNN rounding choices
https://github.com/dmlc/tvm/blob/11a3a777ec78ad81ce12a940d07521870f546ad1/include/tvm/relay/qnn/attrs.h#L52-L61
The point is that TONEAREST and UPWARD differ at only negative mid values (like -1.5, -2.5 and so on). This difference is invisible in end-to-end accuracy.
What does other frameworks do?
Every toolchain has their own favorite rounding choice. There is no widely chosen in-production standard AFAIK.
Note that this PR is only changing the default value, one can still choose any rounding option if one wants. This is for TVM user, who does not want to worry about different roundings.
@tqchen@jackwish@vinx13@FrozenGene@u99127@zhiics@yzhliu