Let pyspark execute files even when IPYTHON=1

[bouk]

With the release of IPython 2.0 having the PYTHONSTARTUP environment variable and passing a file at the same time works as expected. So, we can just pass the commandline args straight to IPython.

Also remove IPYTHON_OPTS, as you can now just pass options to IPython directly

This puts the behavior of IPYTHON=1 in line with the regular Python interpreter

Note that this won't work correctly when IPython version < 2.0

[AmplabJenkins]

Merged build triggered.

[AmplabJenkins]

Merged build started.

[AmplabJenkins]

Merged build finished. All automated tests passed.

[AmplabJenkins]

All automated tests passed.
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/14393/

[mateiz]

Because this won't work on IPython < 2, we unfortunately can't merge it. You should just file an issue in https://issues.apache.org/jira/browse/SPARK and we will revisit it when IPython 2 has much broader adoption. Unfortunately for a while lots of people will probably still be on 1.

[JoshRosen]

If you can figure out a way to retain backwards-compatibility with IPython < 2, I'd be happy to merge this. Maybe you can do something like parsing ipython --version and change behavior based on that?