[Spark load][Fe 3/5] Fe create job

[wyb]

1. Users create spark load job through MySQL client.
   Spark load interface #3010 (comment)

LOAD LABEL db_name.label_name (
DATA INFILE ("/tmp/file1") INTO TABLE table_name, ...
)
WITH RESOURCE resource_name
[(key1 = value1, ...)]
[PROPERTIES (key2 = value2, ... )]

The spark configurations in load stmt can override the existing configuration in the resource for temporary use.

2. Fe analyzes LoadStmt and creates SparkLoadJob in LoadManager.

3. Abstract a base class BulkLoadJob that contains shared code between BrokerLoadJob and SparkLoadJob.

4. Users cancel spark load job through MySQL client.

CANCEL LOAD WHERE LABEL ='label0'

#3433

[morningman]

I think brokerDesc should be in the subclass of BulkLoadJob.
Although currently both broker load and spark load need a broker, but for spark load, it may not be required in future.

[wyb]

After discussion with @morningman, I will improve this later, including persistence with json

[morningman]

this property is hard to understand and is coupled with the detail implementation of the global dict.
How about changing it to a more abstract nouns?

[wyb]

This is for temporary use. I am investigating load from hive table, and i will update it recently.

[morningman]

Save the error msg somewhere for user to get?

[wyb]

I think it’s not necessary, because clear job is just trying to kill etl job as much as possible.

[morningman]

LGTM

[morningman]

Did you just removed this grammar?

[wyb]

hadoop uses opt_system, opt_cluster is no longer used