Turn on using LLVM threadsafely

[alexcrichton]

@catamorphism, this re-enables threadsafe rustpkg tests, @brson this will fail unless the bots have LLVM rebuilt, so this is a good indicator of whether that happened or not.

[catamorphism]

Ok -- I'm remote so I don't think I can get into the bots right now. @brson , can you?

[brson]

@catamorphism yes, looking into it now

[brson]

I'm being asked not to do this because it will slow down the queue. I instead scheduled a build on auto with clean-llvm set.

[alexcrichton]

Just retried, and it turned out that auto-linux-64-nopt wasn't cleaned, or as @thestinger pointed out it was more likely running on a different slave (one of the other slaves was cleaned).

@brson, this may have to wait until @graydon comes back to manually clean all workspaces on all slaves for all builders.

[alexcrichton]

Hmm... upon further inspection, this may need some refinement. When running make check-stage2-rusti, the actual testing portion of rusti takes ~6min. Upon profiling, it turns out that ~5min is spent on contention of one LLVM lock (llvm::PassRegistry::getPassInfo). The odd part is that this isn't even running with optimizations, so rustc isn't adding any extra passes. The passes are all added by the MCJIT that we're using (it requires them). The other odd part is that the function itself isn't slow at all, it's just called a huge number of times.

When using RUST_THREADS=1, the testing time takes ~40s which is much more reasonable. This would be a lot smaller if rustc compiled small programs much more quickly (most time is spent deflating/decoding).

Upon further investigation I changed the Mutex to a RWMutex with the appropriate read/write locks and the multithreaded testing went from 6min to 20s. Long story short, it looks like that patch is a very good idea for us to include in LLVM (and perhaps even easy to push upstream?)

Edit: I sent a patch to LLVM's mailing list: http://lists.cs.uiuc.edu/pipermail/llvm-commits/Week-of-Mon-20130624/179287.html

[alexcrichton]

With newer profiles (after my LLVM change), there's still an enormous amount of time spent in contention on that rwlock, so it would probably be useful to have something like #6512 to control the level of concurrency in tests. Although this also brings up the question of why that mutex is a global mutex and not within the PassRegistry object itself...

[alexcrichton]

@catamorphism, @brson, I've updated this using graydon's new strategy for invalidating LLVM builds, so it should be good for another go.