[SPARK-17631] [SQL] Add HttpStreamSink for structured streaming.

[zhangxinyu1]

What changes were proposed in this pull request?

Add a class HttpStreamSink for structured streaming. This class extends StreamSinkProvider and DataSourceRegister. Streaming query results can be sinked to http server if we configure DataStreamWrite with .format("http").option("url", yourHttpUrl).
e.g.

val query = counts.writeStream
.outputMode("append")
.format("http")
.option("url", "yourHttpUrl")
.start()

How was this patch tested?

Use HttpStreamSinkSuite to test

[AmplabJenkins]

Can one of the admins verify this patch?

[zhangxinyu1]

@marmbrus
I propose this feature mainly because we require output streaming query results via http api. One of use cases is real-time alarm. It's essential to send alarm messages in real time via Http Api, when we use structured streaming to analyze logs and find some Exceptions in logs.
What do you think about it?

[marmbrus]

Thanks for working on this, it does seem like it could be useful. I'm not sure if this should go into Spark or into a separate package. It really depends on how many people want this feature.

Regardless, a few high level comments on this PR:

• Check out the contributing to Spark guide. Patches need to have tests and follow the style guide.
• I would not define a new HttpDataFormat interface. Instead I would mandate that the input is a single string column (similar to what we do for df.write.text). Users can use all of the existing DataFrame/Dataset operations to convert their data into a string.
• It would be good to write up a short design on JIRA and debate there. A few things that I can think of off the top of my head:
  • should we support https too?
  • do we need to set any headers (i.e. maybe the batch id?)
• We'd also need to add docs for this feature.

[zhangxinyu1]

@marmbrus
Thanks for all your sugguestion! I have done the following things:

• Write a short design on JIRA
• I replace HttpDataFormat with the input which only has one string column. I like this idea very much! thanks again.
• Complete HttpStreamSinkSuite to test. It's my first time to write this, could you please help me test it?