[SPARK-16473][MLLIB] Fix BisectingKMeans Algorithm failing in edge case by imatiach-msft · Pull Request #16355 · apache/spark

imatiach-msft · 2016-12-20T18:36:52Z

[SPARK-16473][MLLIB] Fix BisectingKMeans Algorithm failing in edge case where no children exist in updateAssignments

What changes were proposed in this pull request?

Fix a bug in which BisectingKMeans fails with error:
java.util.NoSuchElementException: key not found: 166
at scala.collection.MapLike$class.default(MapLike.scala:228)
at scala.collection.AbstractMap.default(Map.scala:58)
at scala.collection.MapLike$class.apply(MapLike.scala:141)
at scala.collection.AbstractMap.apply(Map.scala:58)
at org.apache.spark.mllib.clustering.BisectingKMeans$$anonfun$org$apache$spark$mllib$clustering$BisectingKMeans$$updateAssignments$1$$anonfun$2.apply$mcDJ$sp(BisectingKMeans.scala:338)
at org.apache.spark.mllib.clustering.BisectingKMeans$$anonfun$org$apache$spark$mllib$clustering$BisectingKMeans$$updateAssignments$1$$anonfun$2.apply(BisectingKMeans.scala:337)
at org.apache.spark.mllib.clustering.BisectingKMeans$$anonfun$org$apache$spark$mllib$clustering$BisectingKMeans$$updateAssignments$1$$anonfun$2.apply(BisectingKMeans.scala:337)
at scala.collection.TraversableOnce$$anonfun$minBy$1.apply(TraversableOnce.scala:231)
at scala.collection.LinearSeqOptimized$class.foldLeft(LinearSeqOptimized.scala:111)
at scala.collection.immutable.List.foldLeft(List.scala:84)
at scala.collection.LinearSeqOptimized$class.reduceLeft(LinearSeqOptimized.scala:125)
at scala.collection.immutable.List.reduceLeft(List.scala:84)
at scala.collection.TraversableOnce$class.minBy(TraversableOnce.scala:231)
at scala.collection.AbstractTraversable.minBy(Traversable.scala:105)
at org.apache.spark.mllib.clustering.BisectingKMeans$$anonfun$org$apache$spark$mllib$clustering$BisectingKMeans$$updateAssignments$1.apply(BisectingKMeans.scala:337)
at org.apache.spark.mllib.clustering.BisectingKMeans$$anonfun$org$apache$spark$mllib$clustering$BisectingKMeans$$updateAssignments$1.apply(BisectingKMeans.scala:334)
at scala.collection.Iterator$$anon$11.next(Iterator.scala:328)
at scala.collection.Iterator$$anon$14.hasNext(Iterator.scala:389)

How was this patch tested?

The dataset was run against the code change to verify that the code works. I will try to add unit tests to the code.

(Please explain how this patch was tested. E.g. unit tests, integration tests, manual tests)
(If this patch involves UI changes, please attach a screenshot; otherwise, remove this)

Please review http://spark.apache.org/contributing.html before opening a pull request.

…se where no children exist in updateAssignments

wangmiao1981 · 2016-12-21T06:37:42Z

Jenkins, test this please.

wangmiao1981 · 2016-12-21T06:39:26Z

@imatiach-msft Can you add a test case?

alokob · 2016-12-21T12:43:19Z

@imatiach-msft , thanks for creating pull request and committing change which I have shared , I will try to share some sample dataset for this issue.

srowen · 2016-12-21T15:30:50Z

 val children = Seq(leftChildIndex(index), rightChildIndex(index))
- val selected = children.minBy { child =>
- KMeans.fastSquaredDistance(newClusterCenters(child), v)
+ if (children.length > 0) {


Nit: you could write children.nonEmpty, but, when would it not have length 2? it's initialized in the line above.

I can see following possibility to get java.util.NoSuchElementException in the original code.
leftChildIndex(index) , this methods returns index2 and rightChildIndex(index) returns index2+1. If the Map object newClusterCenters does not have entries belonging to leftChildIndex(index) or rightChildIndex(index) .
Here we need to have this check newClusterCenters.contains(child) , to avoid this issue.

I added a filter on whether new cluster centers contains the children. Would you be able to send me a sample dataset so I can validate the fix? Thank you!

imatiach-msft · 2016-12-21T16:31:30Z

Good point. It looks like we should be checking if the map contains the child or not. However, I'm not sure if that is the correct solution either. I need a repro dataset from the bug reporter.

srowen · 2016-12-21T16:48:57Z

That makes more sense as a fix, yes. Sounds like there is still a to-do to verify the fix. If it's possible to write a simple unit test to cover it, all the better.

imatiach-msft · 2016-12-22T15:16:59Z

Yep, there is still a TODO to verify the fix. I'm waiting for the dataset from Alok to reproduce the issue:
https://issues.apache.org/jira/browse/SPARK-16473

alokob · 2016-12-23T11:55:11Z

You can get sample vectors at this location https://github.com/alokob/SparkClusteringDataSet/SampleVectors.txt.

Also while executing bisecting K-Means , we have set following configuration settings ,

BisectingKMeans.setK(100);
BisectingKMeans.setMinDivisibleClusterSize(4);
BisectingKMeans.setMaxIter(4);

Please let me know if any additional details are needed.

alokob · 2016-12-26T05:29:59Z

@imatiach-msft Did you find the dataset suitable. Is anything else needed from my side?

imatiach-msft · 2016-12-26T16:57:00Z

Hi Alok!
Sorry I was away for holiday break. I will try to reproduce the failure.
Thank you, Ilya

alokob · 2016-12-28T09:49:01Z

Thats ok , enjoy Xmas.
Please keep me posted if you find that issue is not resolved.

imatiach-msft · 2016-12-28T16:06:56Z

I have very good news :). I was not only able to repro the issue with your dataset, but I was also able to verify that with the suggested fix the algorithm does not fail (adding the val newClusterChildren = children.filter(newClusterCenters.contains(_)) fixed the issue).
The error I saw was:
Job aborted due to stage failure: Task 0 in stage 53.0 failed 1 times, most recent failure: Lost task 0.0 in stage 53.0 (TID 105, localhost, executor driver): java.util.NoSuchElementException: key not found: 162

I need to figure out how to actually add the test case to spark though - my understanding is that checking in dataset files is not allowed? My test code (not cleaned up yet) was:

import org.apache.spark.mllib.linalg.{
Vector => OldVector,
Vectors => OldVectors,
DenseVector => OldDenseVector,
SparseVector => OldSparseVector}

import org.apache.spark.ml.linalg.{
Vector => NewVector,
Vectors => NewVectors,
DenseVector => NewDenseVector,
SparseVector => NewSparseVector
}

@transient var loadedDataset: Dataset[_] = _

override def beforeAll(): Unit = {
super.beforeAll()
dataset = KMeansSuite.generateKMeansData(spark, 50, 3, k)
loadedDataset = spark.createDataFrame(sc.textFile("/home/ilya/SampleVectors.txt")
.map(line => Row({ OldVectors.parse(line.substring(1, line.length - 1)) match {
case sv: OldSparseVector => NewVectors.sparse(sv.size, sv.indices, sv.values)
case dv: OldDenseVector => NewVectors.dense(dv.values)
} })),
StructType(Seq(StructField("features", SQLDataTypes.VectorType))))
}
.....
test("SPARK-16473: Verify Bisecting K-Means does not fail in edge case where no children exist") {
val bkm = new BisectingKMeans().setK(100).setMinDivisibleClusterSize(4).setMaxIter(4)

assert(bkm.getK === 100)
assert(bkm.getFeaturesCol === "features")
assert(bkm.getPredictionCol === "prediction")
assert(bkm.getMaxIter === 4)
assert(bkm.getMinDivisibleClusterSize === 4)
val model = bkm.fit(loadedDataset)
assert(model.hasSummary)

}

jkbradley · 2016-12-28T17:55:36Z

ok to test

alokob · 2016-12-28T18:12:17Z

Nice to know that , codefix I suggested is working. Its really nice to contribute in spark.

SparkQA · 2016-12-28T18:56:19Z

Test build #70679 has finished for PR 16355 at commit 55ab179.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

…c sparse data which can generate the exception user encountered.

imatiach-msft · 2016-12-28T21:36:11Z

I've updated with a new commit. I was able to reproduce the issue by generating a synthetic sparse dataset similar to the one Alok sent me, in accordance with the test-style of spark test methods. Hence, I was able to verify the fix. Please review the new code changes and let me know who else needs to review in order for the changes to be committed.

imatiach-msft · 2016-12-28T21:37:04Z

Jenkins, retest this please

SparkQA · 2016-12-28T21:39:50Z

Test build #70682 has finished for PR 16355 at commit 75192a7.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

imatiach-msft · 2016-12-28T23:37:53Z

Jenkins, retest this please

SparkQA · 2016-12-29T00:41:51Z

Test build #70688 has finished for PR 16355 at commit 14c7ce3.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

imatiach-msft · 2016-12-29T14:33:10Z

@jkbradley @srowen any comments on the changes? Thank you!

jkbradley · 2016-12-29T22:27:21Z

@yu-iskw Pinging on this since you wrote bisecting k-means originally. Do you have time to take a look? Thanks!

imatiach-msft · 2016-12-29T22:48:11Z

the only problem I see is that with this code we generate k-1 clusters instead of k, but it states in the algorithm documentation that it is not guaranteed to generate k clusters, it could be fewer if the leaf clusters are not divisible (see spark/mllib/src/main/scala/org/apache/spark/mllib/clustering/BisectingKMeans.scala):

Iteratively it finds divisible clusters on the bottom level and bisects each of them using
k-means, until there are k leaf clusters in total or no leaf clusters are divisible.

It seems in the dataset Alok gave, one of the clusters which was assumed to be divisible and was divided ended up generating two clusters, one which contained all the points and the other none, which is what created the error (his cluster 162, child of 81, was empty, but cluster 163 was non-empty after reassignment).

My intuition is that it should be possible for the algorithm to account for failed splits and then try to split in a different leaf node, but that change seems non-trivial to me.

jkbradley

Thanks for the updates! I agree we shouldn't try to retry failed splits in this PR.

jkbradley · 2017-01-06T19:11:06Z

 override def beforeAll(): Unit = {
 super.beforeAll()
 dataset = KMeansSuite.generateKMeansData(spark, 50, 3, k)
+ sparseDataset = KMeansSuite.generateSparseData(spark, 100, 1000, k, 42)


Does the test really need to be this large? It takes ~1 sec which is long-ish for a unit test.

I can only repro the issue with very sparse data where number of columns is around 1k. I was able to reduce the number of rows to be only 10 however. I hope that is a small enough dataset.

jkbradley · 2017-01-06T19:11:59Z

+ val bkm = new BisectingKMeans().setK(k).setMinDivisibleClusterSize(4).setMaxIter(4)
+
+ assert(bkm.getK === k)
+ assert(bkm.getFeaturesCol === "features")


There's no need to test featuresCol, predictionCol, and other things which aren't relevant to this unit test. I'd simplify it.

done, removed

jkbradley · 2017-01-06T19:12:03Z

+ assert(bkm.getPredictionCol === "prediction")
+ assert(bkm.getMaxIter === 4)
+ assert(bkm.getMinDivisibleClusterSize === 4)
+ // Verify fit does not fail on very sparse data


It's not clear to me that this unit test actually tests the issue fixed in this PR. Is there a good way to see why it would? If not, then it would be great to write a tiny dataset by hand which would trigger the failure.

I added this check to verify:
// Verify we hit the edge case
assert(numClusters < k && numClusters > 1)
the issue only occurs for very sparse data, but it occurs very consistently (almost all very sparse data that I generate can trigger the error)

jkbradley · 2017-01-06T19:12:05Z

 spark.createDataFrame(rdd)
 }

+ def generateSparseData(spark: SparkSession, rows: Int, dim: Int, k: Int, seed: Int): DataFrame = {


k is never used

modified the method to use it

jkbradley

Just 2 small comments left. Thanks!

jkbradley · 2017-01-13T02:28:09Z

+ "one cluster is empty after split") {
+ val bkm = new BisectingKMeans().setK(k).setMinDivisibleClusterSize(4).setMaxIter(4)
+
+ assert(bkm.getK === k)


Don't bother testing getK, getMaxIter, or getMinDivisibleClusterSize
The setters should be tested elsewhere, so you may assume they work here.

done, removed

jkbradley · 2017-01-13T02:28:15Z

+ val nnz = random.nextInt(dim)
+ val rdd = sc.parallelize(1 to rows)
+ .map(i => Vectors.sparse(dim, random.shuffle(0 to dim - 1).slice(0, nnz).sorted.toArray,
+ Array.fill(nnz)(random.nextInt(k).toDouble)))


I don't understand this use of k. The feature value can be any random number. I'd remove k.

done, removed k

…ation

imatiach-msft · 2017-01-13T04:54:41Z

@jkbradley thanks, I've updated the code based on your latest comments - I removed k and the verification for the setters.

SparkQA · 2017-01-13T05:58:46Z

Test build #71289 has finished for PR 16355 at commit 138ab34.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

imatiach-msft · 2017-01-17T15:33:16Z

ping @jkbradley would you be able to take another look at the bisecting K-Means fix?

jkbradley · 2017-01-17T22:11:24Z

test this please

jkbradley · 2017-01-17T22:38:09Z


+ test("SPARK-16473: Verify Bisecting K-Means does not fail in edge case where" +
+ "one cluster is empty after split") {
+ val bkm = new BisectingKMeans().setK(k).setMinDivisibleClusterSize(4).setMaxIter(4)


Please set the seed too since this test seems at risk of flakiness

done, added seed

jkbradley · 2017-01-17T22:51:42Z

I was about to say this is ready, but I do think we should add the seed. Other than that, this should be ready!

imatiach-msft · 2017-01-17T23:10:57Z

@jkbradley done, added seed. Thanks!

SparkQA · 2017-01-17T23:15:58Z

Test build #71533 has finished for PR 16355 at commit 138ab34.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2017-01-18T00:36:04Z

Test build #71538 has finished for PR 16355 at commit ba00ad0.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2017-01-18T01:14:42Z

Test build #3538 has finished for PR 16355 at commit ba00ad0.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

imatiach-msft · 2017-01-18T02:36:53Z

ping @jkbradley would you be able to take another look at the bisecting kmeans model? I've updated with the random seed as requested.

imatiach-msft · 2017-01-18T18:18:35Z

ping @jkbradley would you be able to take another look at the bisecting kmeans model? I've updated with the random seed as requested, and the build succeeded. Thank you!

imatiach-msft · 2017-01-20T14:53:17Z

ping @jkbradley would you be able to take another look at the bisecting kmeans model? Thanks!

imatiach-msft · 2017-01-23T16:23:05Z

ping @jkbradley would you be able to take another look at the bisecting kmeans model? Thanks!

jkbradley · 2017-01-23T20:30:21Z

LGTM
Thanks!
Will merge after fresh tests

SparkQA · 2017-01-23T21:32:00Z

Test build #3548 has finished for PR 16355 at commit ba00ad0.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

jkbradley · 2017-01-23T21:33:57Z

Merging with master. Will try to backport to branch-2.1 as well.
Thanks!

jkbradley · 2017-01-23T21:48:56Z

I was able to check out this commit and test it with branch-2.1, but now I can't get the merge script to merge it for branch-2.1. @srowen would you mind trying? Thanks!

vanzin · 2017-01-23T21:50:01Z

but now I can't get the merge script to merge it for branch-2.1

I just had some issues with that too. But manually merging (git cherry-pick + git push) seems to still work, so maybe try that.

srowen · 2017-01-24T11:20:36Z

It's an apache-github sync issue:

https://github.com/apache/spark/commits/branch-2.1
is missing the latest commit from
https://git-wip-us.apache.org/repos/asf?p=spark.git;a=shortlog;h=refs/heads/branch-2.1

I'll cherry-pick onto apache/branch-2.1 and push as that might also kick the sync to try again.

[SPARK-16473][MLLIB] Fix BisectingKMeans Algorithm failing in edge case where no children exist in updateAssignments ## What changes were proposed in this pull request? Fix a bug in which BisectingKMeans fails with error: java.util.NoSuchElementException: key not found: 166 at scala.collection.MapLike$class.default(MapLike.scala:228) at scala.collection.AbstractMap.default(Map.scala:58) at scala.collection.MapLike$class.apply(MapLike.scala:141) at scala.collection.AbstractMap.apply(Map.scala:58) at org.apache.spark.mllib.clustering.BisectingKMeans$$anonfun$org$apache$spark$mllib$clustering$BisectingKMeans$$updateAssignments$1$$anonfun$2.apply$mcDJ$sp(BisectingKMeans.scala:338) at org.apache.spark.mllib.clustering.BisectingKMeans$$anonfun$org$apache$spark$mllib$clustering$BisectingKMeans$$updateAssignments$1$$anonfun$2.apply(BisectingKMeans.scala:337) at org.apache.spark.mllib.clustering.BisectingKMeans$$anonfun$org$apache$spark$mllib$clustering$BisectingKMeans$$updateAssignments$1$$anonfun$2.apply(BisectingKMeans.scala:337) at scala.collection.TraversableOnce$$anonfun$minBy$1.apply(TraversableOnce.scala:231) at scala.collection.LinearSeqOptimized$class.foldLeft(LinearSeqOptimized.scala:111) at scala.collection.immutable.List.foldLeft(List.scala:84) at scala.collection.LinearSeqOptimized$class.reduceLeft(LinearSeqOptimized.scala:125) at scala.collection.immutable.List.reduceLeft(List.scala:84) at scala.collection.TraversableOnce$class.minBy(TraversableOnce.scala:231) at scala.collection.AbstractTraversable.minBy(Traversable.scala:105) at org.apache.spark.mllib.clustering.BisectingKMeans$$anonfun$org$apache$spark$mllib$clustering$BisectingKMeans$$updateAssignments$1.apply(BisectingKMeans.scala:337) at org.apache.spark.mllib.clustering.BisectingKMeans$$anonfun$org$apache$spark$mllib$clustering$BisectingKMeans$$updateAssignments$1.apply(BisectingKMeans.scala:334) at scala.collection.Iterator$$anon$11.next(Iterator.scala:328) at scala.collection.Iterator$$anon$14.hasNext(Iterator.scala:389) ## How was this patch tested? The dataset was run against the code change to verify that the code works. I will try to add unit tests to the code. (Please explain how this patch was tested. E.g. unit tests, integration tests, manual tests) (If this patch involves UI changes, please attach a screenshot; otherwise, remove this) Please review http://spark.apache.org/contributing.html before opening a pull request. Author: Ilya Matiach <ilmat@microsoft.com> Closes#16355 from imatiach-msft/ilmat/fix-kmeans.

srowen · 2017-01-24T11:30:17Z

Done, and it synced now. Merged to master/2.1

jkbradley · 2017-01-24T18:17:22Z

Oh OK! Thanks @srowen

[SPARK-16473][MLLIB] Fix BisectingKMeans Algorithm failing in edge case where no children exist in updateAssignments ## What changes were proposed in this pull request? Fix a bug in which BisectingKMeans fails with error: java.util.NoSuchElementException: key not found: 166 at scala.collection.MapLike$class.default(MapLike.scala:228) at scala.collection.AbstractMap.default(Map.scala:58) at scala.collection.MapLike$class.apply(MapLike.scala:141) at scala.collection.AbstractMap.apply(Map.scala:58) at org.apache.spark.mllib.clustering.BisectingKMeans$$anonfun$org$apache$spark$mllib$clustering$BisectingKMeans$$updateAssignments$1$$anonfun$2.apply$mcDJ$sp(BisectingKMeans.scala:338) at org.apache.spark.mllib.clustering.BisectingKMeans$$anonfun$org$apache$spark$mllib$clustering$BisectingKMeans$$updateAssignments$1$$anonfun$2.apply(BisectingKMeans.scala:337) at org.apache.spark.mllib.clustering.BisectingKMeans$$anonfun$org$apache$spark$mllib$clustering$BisectingKMeans$$updateAssignments$1$$anonfun$2.apply(BisectingKMeans.scala:337) at scala.collection.TraversableOnce$$anonfun$minBy$1.apply(TraversableOnce.scala:231) at scala.collection.LinearSeqOptimized$class.foldLeft(LinearSeqOptimized.scala:111) at scala.collection.immutable.List.foldLeft(List.scala:84) at scala.collection.LinearSeqOptimized$class.reduceLeft(LinearSeqOptimized.scala:125) at scala.collection.immutable.List.reduceLeft(List.scala:84) at scala.collection.TraversableOnce$class.minBy(TraversableOnce.scala:231) at scala.collection.AbstractTraversable.minBy(Traversable.scala:105) at org.apache.spark.mllib.clustering.BisectingKMeans$$anonfun$org$apache$spark$mllib$clustering$BisectingKMeans$$updateAssignments$1.apply(BisectingKMeans.scala:337) at org.apache.spark.mllib.clustering.BisectingKMeans$$anonfun$org$apache$spark$mllib$clustering$BisectingKMeans$$updateAssignments$1.apply(BisectingKMeans.scala:334) at scala.collection.Iterator$$anon$11.next(Iterator.scala:328) at scala.collection.Iterator$$anon$14.hasNext(Iterator.scala:389) ## How was this patch tested? The dataset was run against the code change to verify that the code works. I will try to add unit tests to the code. (Please explain how this patch was tested. E.g. unit tests, integration tests, manual tests) (If this patch involves UI changes, please attach a screenshot; otherwise, remove this) Please review http://spark.apache.org/contributing.html before opening a pull request. Author: Ilya Matiach <ilmat@microsoft.com> Closesapache#16355 from imatiach-msft/ilmat/fix-kmeans.

[SPARK-16473][MLLIB] Fix BisectingKMeans Algorithm failing in edge ca…
f77904d
…se where no children exist in updateAssignments

srowen reviewed Dec 21, 2016
View reviewed changes

Updated to validate children in newClusterCenters based on comments
55ab179

Adding test case as requested in review. Test case generates syntheti…
75192a7
…c sparse data which can generate the exception user encountered.

Fixing scala style and test case
14c7ce3

srowen approved these changes Jan 6, 2017
View reviewed changes

jkbradley reviewed Jan 6, 2017
View reviewed changes

Updated code based on comments in review
a8cf2f0

jkbradley reviewed Jan 13, 2017
View reviewed changes

Updated based on latest comments - removed k, removed setters verific…
138ab34
…ation

jkbradley reviewed Jan 17, 2017
View reviewed changes

Added seed based on comment
ba00ad0

srowen approved these changes Jan 23, 2017
View reviewed changes

asfgit closed this in 5b258b8Jan 23, 2017

Uh oh!

Conversation

imatiach-msft commented Dec 20, 2016

What changes were proposed in this pull request?

How was this patch tested?

Uh oh!

wangmiao1981 commented Dec 21, 2016

Uh oh!

wangmiao1981 commented Dec 21, 2016

Uh oh!

alokob commented Dec 21, 2016

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

imatiach-msft commented Dec 21, 2016

Uh oh!

srowen commented Dec 21, 2016

Uh oh!

imatiach-msft commented Dec 22, 2016

Uh oh!

alokob commented Dec 23, 2016

Uh oh!

alokob commented Dec 26, 2016

Uh oh!

imatiach-msft commented Dec 26, 2016

Uh oh!

alokob commented Dec 28, 2016

Uh oh!

imatiach-msft commented Dec 28, 2016

Uh oh!

jkbradley commented Dec 28, 2016

Uh oh!

alokob commented Dec 28, 2016

Uh oh!

SparkQA commented Dec 28, 2016

Uh oh!

imatiach-msft commented Dec 28, 2016

Uh oh!

imatiach-msft commented Dec 28, 2016

Uh oh!

SparkQA commented Dec 28, 2016

Uh oh!

imatiach-msft commented Dec 28, 2016

Uh oh!

SparkQA commented Dec 29, 2016

Uh oh!

imatiach-msft commented Dec 29, 2016

Uh oh!

jkbradley commented Dec 29, 2016

Uh oh!

imatiach-msft commented Dec 29, 2016• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jkbradley left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jkbradley left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

imatiach-msft commented Dec 29, 2016•
edited
Loading