Filter backtrace by bjorn3 · Pull Request #48702 · rust-lang/rust

bjorn3 · 2018-03-03T17:01:00Z

cc #47824

rust-highfive · 2018-03-03T17:01:10Z

r? @BurntSushi

(rust_highfive has picked a reviewer for you, use r? to override)

bjorn3 · 2018-03-16T15:31:20Z

ping @BurntSushi

BurntSushi · 2018-03-16T15:34:28Z

r? @alexcrichton

alexcrichton

Thanks for the PR and sorry for the delay! Dealing with filtering backtraces is a pretty tricky issue though so I'd want to be sure that we're being careful here. This all started in #38165 and some refinements were proposed in #40264 but during that discussion (and while the original PR was baking on nightly) it was found out that we were too aggressive about pruning backtraces, consequently leading to #41364 which removed almost all filtering.

There's a lot of comments on #40264 but I think the general consensus is that we're not shooting for 100% clean backtraces here but rather "mostly clean" backtraces by default. In that sense we're mostly looking to simply prune the top frames related to panic runtime gunk and the bottom frames that are related to thread start and whatnot (only showing user-written Rust code in the middle in theory).

To that end I think the logic here is a bit too aggressive, filtering out frames in the middle by accident possibly. Additionally searching for strings like panic or sys is pretty general, so I think we'd want to refine those as well.

Can you also gist some before/after of what the backtraces look like?

alexcrichton · 2018-03-16T16:11:52Z

 /src/libstd_unicode/UnicodeData.txt
 /stage[0-9]+/
-/target
+target/


Ah unfortunately this has the unintended side effect of ignoring src/librustc_back/target, so perhaps these changes could be left out for now?

alexcrichton · 2018-03-16T16:13:10Z

+ for (index, frame, is_on_filter_edge) in filtered_frames {
+ // Don't use ANSI escape codes on windows, because most terminals on it don't support them
+ if is_on_filter_edge && cfg!(not(windows)) {
+ write!(w, "\x1B[2m")?;


Dealing with ansi codes can be sort of a minefield unfortunately. This isn't handling terminals on Unix, for example, that don't support ansi codes. It also doesn't handle cases on Windows where ansi codes are supported (like MSYS and newer Windows terminals I think).
I'm not entirely certain what this ansi code is doing, but could it be left out for now?

This ansi code makes the line darker, as it is less important.

Ok, in that case can this be omitted for now? I'm not sure that libstd is really prepared to accurately place ANSI codes everywhere

alexcrichton · 2018-03-16T16:13:50Z

+/// If the bool is true the frame is on the edge between showing and not showing.
+fn filter_frames<'a>(frames: &'a [Frame],
+ context: &BacktraceContext,
+ format: PrintFormat) -> Vec<(usize, &'a Frame, bool)>


Backtraces are currently generated in some sensitive contexts that often work best if we avoid allocation as much as possible, so could a callback or a different scheme be used instead of a vector?

Yes, but it is harder, due to having to know if the previous or next frame have to be filtered out.

Ok, I think this'll be a requirement for landing though

bjorn3 · 2018-03-17T09:49:25Z

To that end I think the logic here is a bit too aggressive, filtering out frames in the middle by accident possibly.

I want to extend this PR, so that you could set custom filters (eg. don't show the internal details of the rustc query system).

Additionally searching for strings like panic or sys is pretty general, so I think we'd want to refine those as well.

I can make it search a prefix instead.

bjorn3 · 2018-03-17T10:04:13Z

Please ignore the branch name, I was in a different rust checkout.

alexcrichton · 2018-03-19T21:49:04Z

I want to extend this PR, so that you could set custom filters (eg. don't show the internal details of the rustc query system).

I think that's something that we probably don't want in std itself, but perhaps an external crate which configures the global panic hook?

bjorn3 · 2018-03-20T07:12:40Z

I think that's something that we probably don't want in std itself, but perhaps an external crate which configures the global panic hook?

No problem

emilyalbini · 2018-03-26T14:15:57Z

Ping from triage @alexcrichton! The author pushed new commits, can you review them?

@bjorn3, the build is failing because you didn't add the license header on the test. Can you add that?

tidy error: /checkout/src/test/run-make/filter_backtrace/main.rs: incorrect license

shepmaster · 2018-04-07T19:50:18Z

Ping from triage, @alexcrichton !

alexcrichton · 2018-04-07T21:50:52Z

ah I was still waiting on https://github.com/rust-lang/rust/pull/48702/files#r175140985 to be addressed

bjorn3 · 2018-04-08T08:39:28Z

Sorry, forgot about that

TimNN · 2018-04-08T09:26:20Z

Your PR failed on Travis. Through arcane magic we have determined that the following fragments from the build log may contain information about the problem.

Click to expand the log.

Resolving deltas: 100% (601193/601193), completed with 4719 local objects.
---
[00:00:42] configure: rust.quiet-tests := True
---
[00:04:35] 144 | let _ = resolve_symname(*frame, |symname| {
[00:04:35] | ^^^^^^ expected struct `sys_common::backtrace::Frame`, found tuple
[00:04:35] |
[00:04:35] = note: expected type `sys_common::backtrace::Frame`
[00:04:35] found type `(usize, &sys_common::backtrace::Frame)`
[00:04:35]
[00:04:35] error[E0599]: no method named `peek` found for type `core::iter::TakeWhile<core::iter::Peekable<core::iter::Enumerate<core::slice::Iter<'_, sys_common::backtrace::Frame>>>, [closure@libstd/sys_common/backtrace.rs:142:21: 154:10 context:_]>` in the current scope
[00:04:35] --> libstd/sys_common/backtrace.rs:162:14
---
[00:04:35] process didn't exit successfully: `/checkout/obj/build/bootstrap/debug/rustc --crate-name std libstd/lib.rs --color always --error-format json --crate-type dylib --crate-type rlib --emit=dep-info,link -C prefer-dynamic -C opt-level=2 --cfg feature="alloc_jemalloc" --cfg feature="backtrace" --cfg feature="jemalloc" --cfg feature="panic-unwind" --cfg feature="panic_unwind" -C metadata=e23f5ab1b35c3cbb -C extra-filename=-e23f5ab1b35c3cbb --out-dir /checkout/obj/build/x86_64-unknown-linux-gnu/stage0-std/x86_64-unknown-linux-gnu/release/deps --target x86_64-unknown-linux-gnu -L dependency=/checkout/obj/build/x86_64-unknown-linux-gnu/stage0-std/x86_64-unknown-linux-gnu/release/deps -L dependency=/checkout/obj/build/x86_64-unknown-linux-gnu/stage0-std/release/deps --extern unwind=/checkout/obj/build/x86_64-unknown-linux-gnu/stage0-std/x86_64-unknown-linux-gnu/release/deps/libunwind-ea5547b02114e2c4.rlib --extern panic_abort=/checkout/obj/build/x86_64-unknown-linux-gnu/stage0-std/x86_64-unknown-linux-gnu/release/deps/libpanic_abort-709ac4dccd81a82f.rlib --extern core=/checkout/obj/build/x86_64-unknown-linux-gnu/stage0-std/x86_64-unknown-linux-gnu/release/deps/libcore-1b8789e893adb899.rlib --extern rustc_msan=/checkout/obj/build/x86_64-unknown-linux-gnu/stage0-std/x86_64-unknown-linux-gnu/release/deps/librustc_msan-6f7b4911da25e333.rlib --extern rustc_lsan=/checkout/obj/build/x86_64-unknown-linux-gnu/stage0-std/x86_64-unknown-linux-gnu/release/deps/librustc_lsan-059700c621bf940b.rlib --extern std_unicode=/checkout/obj/build/x86_64-unknown-linux-gnu/stage0-std/x86_64-unknown-linux-gnu/release/deps/libstd_unicode-efb916052e2ce91d.rlib --extern compiler_builtins=/checkout/obj/build/x86_64-unknown-linux-gnu/stage0-std/x86_64-unknown-linux-gnu/release/deps/libcompiler_builtins-a2d7f090df844ebf.rlib --extern rustc_asan=/checkout/obj/build/x86_64-unknown-linux-gnu/stage0-std/x86_64-unknown-linux-gnu/release/deps/librustc_asan-9b401d4f1df5ac3a.rlib --extern rustc_tsan=/checkout/obj/build/x86_64-unknown-linux-gnu/stage0-std/x86_64-unknown-linux-gnu/release/deps/librustc_tsan-71f5363a0f32984a.rlib --extern alloc_system=/checkout/obj/build/x86_64-unknown-linux-gnu/stage0-std/x86_64-unknown-linux-gnu/release/deps/liballoc_system-1f93143c583ac549.rlib --extern alloc=/checkout/obj/build/x86_64-unknown-linux-gnu/stage0-std/x86_64-unknown-linux-gnu/release/deps/liballoc-6efa3f327e315680.rlib --extern alloc_jemalloc=/checkout/obj/build/x86_64-unknown-linux-gnu/stage0-std/x86_64-unknown-linux-gnu/release/deps/liballoc_jemalloc-209d69cbee1ef856.rlib --extern panic_unwind=/checkout/obj/build/x86_64-unknown-linux-gnu/stage0-std/x86_64-unknown-linux-gnu/release/deps/libpanic_unwind-6843ee90cbd31c10.rlib --extern libc=/checkout/obj/build/x86_64-unknown-linux-gnu/stage0-std/x86_64-unknown-linux-gnu/release/deps/liblibc-e53cce9e670d242c.rlib -L native=/checkout/obj/build/x86_64-unknown-linux-gnu/native/libbacktrace/.libs -l static=backtrace -l dl -l rt -l pthread -L native=/checkout/obj/build/x86_64-unknown-linux-gnu/stage0-std/x86_64-unknown-linux-gnu/release/build/compiler_builtins-6789d0ad544f7553/out -L native=/checkout/obj/build/x86_64-unknown-linux-gnu/native/jemalloc/lib` (exit code: 101)
[00:04:35] command did not execute successfully: "/checkout/obj/build/x86_64-unknown-linux-gnu/stage0/bin/cargo" "build" "--target" "x86_64-unknown-linux-gnu" "--release" "--locked" "--color" "always" "--features" "panic-unwind jemalloc backtrace" "--manifest-path" "/checkout/src/libstd/Cargo.toml" "--message-format" "json"
[00:04:35] expected success, got: exit code: 101
[00:04:35] thread 'main' panicked at 'cargo must succeed', bootstrap/compile.rs:1085:9
[00:04:35] note: Run with `RUST_BACKTRACE=1` for a backtrace.
[00:04:35] travis_fold:end:stage0-std
[00:04:35] travis_time:end:stage0-std:start=1523179519619591257,finish=1523179575945681380,duration=56326090123
[00:04:35] failed to run: /checkout/obj/build/bootstrap/debug/bootstrap test src/tools/tidy
[00:04:35] Build completed unsuccessfully in 0:00:57
[00:04:35] make: *** [tidy] Error 1
[00:04:35] Makefile:79: recipe for target 'tidy' failed
---
$ ls -lat $HOME/Library/Logs/DiagnosticReports/
ls: cannot access /home/travis/Library/Logs/DiagnosticReports/: No such file or directory
travis_time:end:059091d8:start=1523179576443242280,finish=1523179576449101644,duration=5859364
travis_fold:end:after_failure.2
travis_fold:start:after_failure.3
travis_time:start:000832c5
$ find $HOME/Library/Logs/DiagnosticReports -type f -name '*.crash' -not -name '*.stage2-*.crash' -not -name 'com.apple.CoreSimulator.CoreSimulatorService-*.crash' -exec printf travis_fold":start:crashlog\n\033[31;1m%s\033[0m\n" {} \; -exec head -750 {} \; -exec echo travis_fold":"end:crashlog \; || true
find: `/home/travis/Library/Logs/DiagnosticReports': No such file or directory
travis_time:end:000832c5:start=1523179576454164755,finish=1523179576459553499,duration=5388744
travis_fold:end:after_failure.3
travis_fold:start:after_failure.4
travis_time:start:0251be3f
$ dmesg | grep -i kill
[ 10.469385] init: failsafe main process (1093) killed by TERM signal

I'm a bot! I can only do what humans tell me to, so if this was not helpful or you have suggestions for improvements, please ping or otherwise contact @TimNN.

Zoxc · 2018-04-09T21:41:26Z

I don't like this approach, I much prefer my more robust and cleaner solution in #45637 which was rejected by the libs team for not being robust enough.

emilyalbini · 2018-04-16T10:04:34Z

Ping from triage @alexcrichton! The author addressed your comment!

bjorn3 · 2018-04-16T10:12:47Z

@pietroalbini I have only partly addressed the comment, there is still a heap alloc. I haven't got time to fix it completely

bjorn3 · 2018-04-16T18:15:47Z

Addressed the comment now. Ready for review @alexcrichton

rust-highfive · 2018-04-18T13:51:40Z

Your PR failed on Travis (raw log). Through arcane magic we have determined that the following fragments from the build log may contain information about the problem.

Click to expand the log.

Resolving deltas: 100% (601494/601494), completed with 4717 local objects.
---
[00:00:46] configure: rust.quiet-tests := True
---
[00:41:35] .................................................................................i..................
[00:41:41] ........................i...........................................................................
---
[00:42:21] .....................i...........................................................................i..
[00:42:27] ....................................................................................................
[00:42:34] ...........ii.......................................................................................
[00:42:42] ............................................................................................i.......
---
[00:43:16] .............................................i......................................................
---
[00:46:54] ..............................i.....................................................................
[00:47:08] ...............................................................i....................................
[00:47:22] .................................................i..................................................
[00:47:40] ....................................................................................................
[00:47:59] ....................................................................................................
[00:48:19] ....................................................................................................
[00:48:41] .......i............................................................................................
[00:49:13] ...i..............................................................................................test [run-pass] run-pass/mir_heavy_promoted.rs has been running for over 60 seconds
[00:49:14] ..
[00:49:43] ....................................................................................................
[00:50:11] .....................................................................ii.............................
[00:50:59] ................................i....................................................i.ii...........
[00:51:03] ........test [run-pass] run-pass/saturating-float-casts.rs has been running for over 60 seconds
[00:51:37] .....................................................................................iiiiiii
---
[00:53:49] ....................i............................................................ii.iii.............
[00:53:56] ....................................................................................................
[00:54:03] .........i..............................i...........................................................
[00:54:10] ....................................................................................................
[00:54:16] ...........i........................................................................................
[00:54:24] ....................................................................................................
[00:54:33] ....................................................................................................
[00:54:43] ....................................................................................................
[00:54:52] ....................................................................................................
[00:55:04] ....................................................................................................
[00:55:12] ....i...............................................................................................
[00:55:20] ........i..ii.......................................................................................
[00:55:30] ....................................................................................................
[00:55:39] ....................................................................................................
[00:55:48] ..........................................................................i.........................
[00:55:58] ....................i...............................................................................
---
[00:56:30] ............................i.......................................................................
[00:56:31] ....................................................................i...............................
[00:56:32] ...............i.......................................................
---
[00:56:46] .............i........................
---
[00:57:14] i...i..ii....i.............ii.........iii......i..i...i...ii..i..i..ii.....
---
[00:57:16] i.......i......................i.......
---
[00:57:49] iiii.......i..i........i..i.i.............i..........iiii...........i...i..........ii.i.i.......ii..
[00:57:50] ....ii...
---
[01:04:39] ..................i.................................................................................
[01:06:10] ........i...........................................................................................
---
[01:07:58] .....................................i..............................................................
[01:08:16] ....................................................................................................
[01:08:34] ..........................................i.........................................................
---
[01:09:57] ..........................................................ii........................................
---
[01:11:12] .....................i..............................................................................
---
[01:16:04] ii..................................................................................................
[01:16:23] ....................................................................................................
[01:16:38] .......................iii......i......i...i......i.................................................
[01:16:46] ....................................................................................................
[01:17:00] .............................................iiii........ii.........................................
[01:17:10] ....................................................................................................
[01:17:25] ............................................................................................i.......
[01:17:47] ....................................................................................................
[01:17:57] ....................................................................................................
[01:18:06] ..iiii...............................................
---
[01:28:04] make[1]: Entering directory '/checkout/src/test/run-make/filter_backtrace'
[01:28:04] /checkout/obj/build/x86_64-unknown-linux-gnu/stage2/bin/rustc main.rs -o /checkout/obj/build/x86_64-unknown-linux-gnu/test/run-make/filter_backtrace.stage2-x86_64-unknown-linux-gnu/main
[01:28:04] # with short backtrace
[01:28:04] RUST_BACKTRACE=1 /checkout/obj/build/x86_64-unknown-linux-gnu/test/run-make/filter_backtrace.stage2-x86_64-unknown-linux-gnu/main 2> /checkout/obj/build/x86_64-unknown-linux-gnu/test/run-make/filter_backtrace.stage2-x86_64-unknown-linux-gnu/short_bt.stderr || exit 0 && exit 1
[01:28:04] cat /checkout/obj/build/x86_64-unknown-linux-gnu/test/run-make/filter_backtrace.stage2-x86_64-unknown-linux-gnu/short_bt.stderr # FIXME: remove this debug cat
[01:28:04] thread 'main' panicked at 'called `Option::unwrap()` on a `None` value', libcore/option.rs:335:21
[01:28:04] note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
[01:28:04] stack backtrace:
[01:28:04] 8: core::panicking::panic
[01:28:04] 9: <core::option::Option<T>>::unwrap
[01:28:04] 10: main::main
[01:28:04] 11: std::rt::lang_start::{{closure}}
[01:28:04] 14: std::panicking::try
[01:28:04] 15: std::panic::catch_unwind
[01:28:04] 16: std::rt::lang_start_internal
[01:28:04] 18: main
[01:28:04] 19: __libc_start_main
[01:28:04] 20: _start
[01:28:04] "panicked at" < /checkout/obj/build/x86_64-unknown-linux-gnu/test/run-make/filter_backtrace.stage2-x86_64-unknown-linux-gnu/short_bt.stderr
[01:28:04] Makefile:6: recipe for target 'all' failed
[01:28:04] make[1]: Leaving directory '/checkout/src/test/run-make/filter_backtrace'
[01:28:04]
[01:28:04] ------------------------------------------
[01:28:04] stderr:
[01:28:04] ------------------------------------------
[01:28:04] /bin/sh: 1: panicked at: not found
[01:28:04] make[1]: *** [all] Error 127
[01:28:04]
[01:28:04] ------------------------------------------
[01:28:04]
[01:28:04] thread '[run-make] run-make/filter_backtrace' panicked at 'explicit panic', tools/compiletest/src/runtest.rs:2930:9
---
[01:28:04] command did not execute successfully: "/checkout/obj/build/x86_64-unknown-linux-gnu/stage0-tools-bin/compiletest" "--compile-lib-path" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage2/lib" "--run-lib-path" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage2/lib/rustlib/x86_64-unknown-linux-gnu/lib" "--rustc-path" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage2/bin/rustc" "--src-base" "/checkout/src/test/run-make" "--build-base" "/checkout/obj/build/x86_64-unknown-linux-gnu/test/run-make" "--stage-id" "stage2-x86_64-unknown-linux-gnu" "--mode" "run-make" "--target" "x86_64-unknown-linux-gnu" "--host" "x86_64-unknown-linux-gnu" "--llvm-filecheck" "/usr/lib/llvm-3.9/bin/FileCheck" "--host-rustcflags" "-Crpath -O -Zunstable-options " "--target-rustcflags" "-Crpath -O -Zunstable-options -Lnative=/checkout/obj/build/x86_64-unknown-linux-gnu/native/rust-test-helpers" "--docck-python" "/usr/bin/python2.7" "--lldb-python" "/usr/bin/python2.7" "--gdb" "/usr/bin/gdb" "--quiet" "--llvm-version" "3.9.1\n" "--system-llvm" "--cc" "" "--cxx" "" "--cflags" "" "--llvm-components" "" "--llvm-cxxflags" "" "--adb-path" "adb" "--adb-test-dir" "/data/tmp/work" "--android-cross-path" "" "--color" "always"
[01:28:04] expected success, got: exit code: 101
[01:28:04]
[01:28:04]
[01:28:04] failed to run: /checkout/obj/build/bootstrap/debug/bootstrap test
[01:28:04] Build completed unsuccessfully in 0:47:40
[01:28:04] make: *** [check] Error 1
[01:28:04] Makefile:58: recipe for target 'check' failed
---
$ ls -lat $HOME/Library/Logs/DiagnosticReports/
ls: cannot access /home/travis/Library/Logs/DiagnosticReports/: No such file or directory
travis_time:end:0bfac9d4:start=1524059494436186313,finish=1524059494444654772,duration=8468459
travis_fold:end:after_failure.2
travis_fold:start:after_failure.3
travis_time:start:19f84472
$ find $HOME/Library/Logs/DiagnosticReports -type f -name '*.crash' -not -name '*.stage2-*.crash' -not -name 'com.apple.CoreSimulator.CoreSimulatorService-*.crash' -exec printf travis_fold":start:crashlog\n\033[31;1m%s\033[0m\n" {} \; -exec head -750 {} \; -exec echo travis_fold":"end:crashlog \; || true
find: `/home/travis/Library/Logs/DiagnosticReports': No such file or directory
travis_time:end:19f84472:start=1524059494451860731,finish=1524059494459703482,duration=7842751
travis_fold:end:after_failure.3
travis_fold:start:after_failure.4
travis_time:start:181bcef0
$ dmesg | grep -i kill
[ 9.852829] init: failsafe main process (1092) killed by TERM signal

I'm a bot! I can only do what humans tell me to, so if this was not helpful or you have suggestions for improvements, please ping or otherwise contact @TimNN. (Feature Requests)

alexcrichton · 2018-04-24T15:10:54Z

Ah sorry for taking awhile to get back to this @bjorn3. I had forgotten about #47824 but I do agree with @Zoxc that it's best solution for now as it invovles far less guesswork and will hopefully be more robust over time.

The reopening at #49825 only covers stack frames below the original "main frame" (aka binaries and tests). I think that there's possible work to be done after that lands to filter out the panic infrastructure in libstd (aka everything after the entry point to panics). Would you be interested in taking that on?

bjorn3 · 2018-04-24T17:23:50Z

Would you be interested in taking that on?

yes

Filter backtrace
6d49756

rust-highfive assigned BurntSushi Mar 3, 2018

rust-highfive added the S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. label Mar 3, 2018

rust-highfive assigned alexcrichton and unassigned BurntSushi Mar 16, 2018

alexcrichton reviewed Mar 16, 2018
View reviewed changes

Wip
6ccf28a

bjorn3 added 2 commits March 20, 2018 08:17

Don't emit ansi codes
199b485

s/begins_with/starts_with
8e4a951

bjorn3 added 3 commits March 26, 2018 18:57

Add license
a9c5a95

Silence warning
2a74f37

Use RUST_BACKTRACE=full in run-pass/backtrace.rs
1edc781

shepmaster added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Apr 7, 2018

A bit more iterator like
7663b93

emilyalbini added T-compiler Relevant to the compiler team, which will review and decide on the PR/issue. S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Apr 16, 2018

emilyalbini removed the S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. label Apr 16, 2018

emilyalbini added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Apr 16, 2018

Use an iterator instead of a vec
0c4e2d2

This comment has been minimized.
Sign in to view

Remove now stable feature gate
d87fd39

This comment has been minimized.
Sign in to view

Add debug cat to test
313dfd5

bjorn3 closed this Apr 24, 2018

bjorn3 deleted the filter_backtrace branch March 30, 2021 17:58

Uh oh!

Uh oh!

Conversation

bjorn3 commented Mar 3, 2018

Uh oh!

rust-highfive commented Mar 3, 2018

Uh oh!

bjorn3 commented Mar 16, 2018

Uh oh!

BurntSushi commented Mar 16, 2018

Uh oh!

alexcrichton left a comment

Choose a reason for hiding this comment

Uh oh!

alexcrichtonMar 16, 2018

Choose a reason for hiding this comment

Uh oh!

alexcrichtonMar 16, 2018

Choose a reason for hiding this comment

Uh oh!

bjorn3Mar 17, 2018

Choose a reason for hiding this comment

Uh oh!

alexcrichtonMar 19, 2018

Choose a reason for hiding this comment

Uh oh!

alexcrichtonMar 16, 2018

Choose a reason for hiding this comment

Uh oh!

bjorn3Mar 17, 2018

Choose a reason for hiding this comment

Uh oh!

alexcrichtonMar 19, 2018

Choose a reason for hiding this comment

Uh oh!

bjorn3 commented Mar 17, 2018

Uh oh!

bjorn3 commented Mar 17, 2018

Uh oh!

alexcrichton commented Mar 19, 2018

Uh oh!

bjorn3 commented Mar 20, 2018

Uh oh!

emilyalbini commented Mar 26, 2018

Uh oh!

shepmaster commented Apr 7, 2018

Uh oh!

alexcrichton commented Apr 7, 2018

Uh oh!

bjorn3 commented Apr 8, 2018

Uh oh!

TimNN commented Apr 8, 2018

Uh oh!

Zoxc commented Apr 9, 2018

Uh oh!

emilyalbini commented Apr 16, 2018

Uh oh!

bjorn3 commented Apr 16, 2018

Uh oh!

bjorn3 commented Apr 16, 2018

Uh oh!

This comment has been minimized.

This comment has been minimized.

rust-highfive commented Apr 18, 2018

Uh oh!

alexcrichton commented Apr 24, 2018

Uh oh!

bjorn3 commented Apr 24, 2018

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants