[SPARK-19662][SCHEDULER][TEST] Add Fair Scheduler Unit Test coverage for different build cases

[erenavsarogullari]

What changes were proposed in this pull request?

Fair Scheduler can be built via one of the following options:

• By setting a spark.scheduler.allocation.file property,
• By setting fairscheduler.xml into classpath.

These options are checked in order and fair-scheduler is built via first found option. If invalid path is found, FileNotFoundException will be expected.

This PR aims unit test coverage of these use cases and a minor documentation change has been added for second option(fairscheduler.xml into classpath) to inform the users.

Also, this PR was related with #16813 and has been created separately to keep patch content as isolated and to help the reviewers.

How was this patch tested?

Added new Unit Tests.

[erenavsarogullari]

cc @kayousterhout@markhamstra@squito

[kayousterhout]

Jenkins this is ok to test

[kayousterhout]

@erenavsarogullari there are currently a bunch of higher priority outstanding scheduler PRs, so I'm guessing it will take a while for anyone to get a chance to review this, just so you know.

[SparkQA]

Test build #73367 has finished for PR 16992 at commit 0b38e69.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[jiangxb1987]

Could you please bring this up-to-date? @erenavsarogullari

[SparkQA]

Test build #79854 has finished for PR 16992 at commit 2e5afe4.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[erenavsarogullari]

Hi @kayousterhout@markhamstra@squito,
This PR is ready to review.
Thanks in advance for all feedbacks.

[squito]

@erenavsarogullari sorry this has taken forever to review! lgtm, just a super tiny nit on wording in the docs.

[squito]

super nit:

... and either putting a file named fairscheduler.xml on the classpath, or setting spark.scheduler.allocation.file ...

[erenavsarogullari]

Addressed.

[squito]

been a while so lets run tests again just to check:
Jenkins, test this please

[SparkQA]

Test build #3893 has finished for PR 16992 at commit 2e5afe4.

• This patch passes all tests.
• This patch does not merge cleanly.
• This patch adds no public classes.

[erenavsarogullari]

Hi @squito,

Thanks for the review this patch. It is ready to re-review / merge.

[SparkQA]

Test build #81168 has finished for PR 16992 at commit 77cfb03.

• This patch fails SparkR unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #81169 has finished for PR 16992 at commit 3d6c80b.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[squito]

merged to master

thanks @erenavsarogullari , sorry again for the delays