[SPARK-16438] Add Asynchronous Actions documentation by phalodi · Pull Request #14104 · apache/spark

phalodi · 2016-07-08T09:23:07Z

What changes were proposed in this pull request?

Add Asynchronous Actions documentation inside action of programming guide

How was this patch tested?

check the documentation indentation and formatting with md preview.

phalodi · 2016-07-08T09:36:18Z

@AmplabJenkins I dont know i just create the issue on JIRA and make changes and make pull request for merge , I think it is valid pull request because in real world all application need non blocking execution. So if its is valid then merge it.

srowen · 2016-07-08T11:07:03Z

@phalodi it's a bot, not a person.

srowen · 2016-07-08T11:07:07Z

Jenkins test this please

SparkQA · 2016-07-08T11:30:51Z

Test build #61980 has finished for PR 14104 at commit aba737f.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2016-07-08T11:31:34Z

Test build #61979 has finished for PR 14104 at commit aba737f.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

rxin · 2016-07-08T23:08:23Z

Can you paste a screenshot of the doc?

phalodi · 2016-07-09T08:55:43Z

@rxin Below are screen shot of Asynchronous Actions inside actions and screen shot of index of programming guide.

srowen · 2016-07-10T11:33:35Z

 </table>

+#### Asynchronous Actions
+Spark provide asynchronous actions to execute two or more actions concurrently, these actions are execute asynchronously without blocking each other. 


"without blocking each other": actions never block each other anyway. They block the calling thread.

srowen · 2016-07-10T11:34:15Z

Mostly this just duplicates the non-async documentation. I think it's worth mentioning the existence of the async methods with a pointer to the API docs, but, I don't think this table adds value.

phalodi · 2016-07-10T17:15:29Z

@srowen yeah you are right its not blocking calling thread but they execute sequentially right but in async action its return future so its running on different threads soo not run sequentially soo its fully non blocking example if i have action in that we have some transformation like map n inside map i have blocking code then how its work its block or not?

phalodi · 2016-07-10T17:35:09Z

@srowen yeah i agree with you its same like non async actions but yeah we should list it because its more useful in real life application when single application running multiple jobs of different users. so we should list it so user will more familiar with it and also know which are the async actions. i think its helpful for users still if you think its not a valid point then just tell me i change pull request and add just reference for async actions but i want user should know because when i first time see it i m surprise that in whole documentation no where mention about async actions.

srowen · 2016-07-10T18:09:07Z

I fixed my comment above, sorry. They do block the calling thread, but actions triggered in different threads on one RDD do not block each other. The first sentence isn't correct. My suggestion is to omit the table, not all of the text you added.

phalodi · 2016-07-10T18:46:26Z

@srowen ok no problem so what you suggest should i remove the table and just add the line below the actions table and give link to scala and java docs? And the line "Spark provide asynchronous actions to execute two or more actions concurrently, these actions are execute asynchronously without blocking each other." is not correct so should i add that to "return future to non-block calling thread?" even in above statement my intention is also talk about calling thread because when we call action it will always in calling thread so calling thread will block but when we use async actions it will not block so we are targeted here non-blocking for calling thread because actions are called in calling thread. so i make changes and push it again as u suggest just add single line at end of actions and give link for scala and java doc api. Its cool?

phalodi · 2016-07-10T19:05:24Z

@srowen i make changes as you suggest i remove the table and add a single below actions table and also change statement to be more clear about non blocking.

phalodi · 2016-07-10T19:39:46Z

@srowen @rxin Please review it and give your comments.

srowen · 2016-07-11T08:55:49Z

 </tr>
 </table>

+Spark provide asynchronous actions to execute two or more actions concurrently, these actions are execute asynchronously without blocking calling thread. Refer to the RDD API doc for asynchronous actions ([Scala](api/scala/index.html#org.apache.spark.rdd.AsyncRDDActions),[Java](api/java/org/apache/spark/rdd/AsyncRDDActions.html))


I still don't think this is accurate. Spark can execute actions concurrently without this API. This merely makes the call non-blocking for the caller. It really adds very little beyond calling the normal API and wrapping in a Future (though it's more useful for the Java API where that's harder). Hence why I thought it was unofficially deprecated.

phalodi · 2016-07-11T09:37:21Z

@srowen so what do you think we should not add this api reference in document?

phalodi · 2016-07-11T09:39:37Z

@srowen i know its nothing major just call normal functions in future but still naive user first time who learn scala and spark dont know what are future and all so at least we should add reference of it. if you suggest i simple add reference link without adding any statement of blocking and non blocking. so atleast user know there is some api like this.

srowen · 2016-07-11T09:39:58Z

Although I don't think it's essential to mention this API (it's in the API doc) it wouldn't hurt to point to it here. The mention should be accurate though. I think it's correct to say that the Spark RDD API also exposes asynchronous versions of some actions like foreach, which return a handle immediately to the caller which can be used to wait for its completion.

phalodi · 2016-07-11T09:47:45Z

@srowen sure i will make appropriate changes and push it again

phalodi · 2016-07-11T10:02:12Z

@srowen now its perfect i think at last from table to one line but yeah you are right it good to just mention that spark provide it and not duplicate table of actions.

phalodi · 2016-07-11T11:23:03Z

@srowen @rxin

srowen · 2016-07-11T11:27:02Z

 </tr>
 </table>

+Spark RDD API also exposes asynchronous versions of some actions like foreach, collect, count etc. which return a handle immediately to the caller which can be used to wait for its completion.


I have a few edits to suggest so let me just propose some text directly:
The Spark RDD API also exposes asynchronous versions of some actions, like foreachAsync for foreach, which immediately return a FutureAction to the caller instead of blocking on completion of the action. This can be used to manage or wait for the asynchronous execution of the action.

phalodi · 2016-07-11T11:33:17Z

@srowen Thanks man i know i am not giving correct language but my intention is also same i hope its final commit :) if its done can you merge it.

srowen · 2016-07-11T11:38:33Z

 </tr>
 </table>

+The Spark RDD API also exposes asynchronous versions of some actions, like foreachAsync for foreach, which immediately return a FutureAction to the caller instead of blocking on completion of the action. This can be used to manage or wait for the asynchronous execution of the action.


Last nit pick: back-tick-quote the code nouns like I did.

phalodi · 2016-07-11T11:52:13Z

@srowen I put back-tick-quote for foreach, foreachAsync and FutureAction.

phalodi · 2016-07-11T12:17:19Z

@srowen Below is final screen shot as you suggested i hope its looks good please review it and merge it.

phalodi · 2016-07-12T09:34:28Z

@srowen Is this correct?? if it is then please merge it otherwise give comment if needed.

srowen · 2016-07-12T11:41:39Z

@phalodi no need to ping. We don't merge things immediately, especially when non-essential. It gives more time for review.

srowen · 2016-07-13T09:43:18Z

Jenkins retest this please

SparkQA · 2016-07-13T10:13:31Z

Test build #62231 has finished for PR 14104 at commit 845a123.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

## What changes were proposed in this pull request? Add Asynchronous Actions documentation inside action of programming guide ## How was this patch tested? check the documentation indentation and formatting with md preview. Author: sandy <phalodi@gmail.com> Closes#14104 from phalodi/SPARK-16438. (cherry picked from commit bf107f1) Signed-off-by: Sean Owen <sowen@cloudera.com>

Add Asynchronous Actions documentation
aba737f

phalodi changed the title ~~Add Asynchronous Actions documentation~~[SPARK-16438]Add Asynchronous Actions documentationJul 8, 2016

phalodi changed the title ~~[SPARK-16438]Add Asynchronous Actions documentation~~[SPARK-16438] Add Asynchronous Actions documentationJul 8, 2016

srowen reviewed Jul 10, 2016
View reviewed changes

fix statement to more clear about blocking statement
4afcec8

remove some statement related to table
81d1f71

srowen reviewed Jul 11, 2016
View reviewed changes

make changes in programming guide doc for asynchronous actions
db09b2c

srowen reviewed Jul 11, 2016
View reviewed changes

add explanation about async actions
2321d64

srowen reviewed Jul 11, 2016
View reviewed changes

add back-tick-quote for code noun
845a123

phalodi mentioned this pull request Jul 11, 2016
[Spark-16479] Add Example for asynchronous action #14135
Closed

asfgit closed this in bf107f1Jul 13, 2016

Uh oh!

Conversation

phalodi commented Jul 8, 2016

What changes were proposed in this pull request?

How was this patch tested?

Uh oh!

phalodi commented Jul 8, 2016

Uh oh!

srowen commented Jul 8, 2016

Uh oh!

srowen commented Jul 8, 2016

Uh oh!

SparkQA commented Jul 8, 2016

Uh oh!

SparkQA commented Jul 8, 2016

Uh oh!

rxin commented Jul 8, 2016

Uh oh!

phalodi commented Jul 9, 2016

Uh oh!

srowenJul 10, 2016• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

srowen commented Jul 10, 2016

Uh oh!

phalodi commented Jul 10, 2016

Uh oh!

phalodi commented Jul 10, 2016

Uh oh!

srowen commented Jul 10, 2016

Uh oh!

phalodi commented Jul 10, 2016

Uh oh!

phalodi commented Jul 10, 2016

Uh oh!

phalodi commented Jul 10, 2016

Uh oh!

srowenJul 11, 2016

Choose a reason for hiding this comment

Uh oh!

phalodi commented Jul 11, 2016

Uh oh!

phalodi commented Jul 11, 2016

Uh oh!

srowen commented Jul 11, 2016

Uh oh!

phalodi commented Jul 11, 2016

Uh oh!

phalodi commented Jul 11, 2016

Uh oh!

phalodi commented Jul 11, 2016

Uh oh!

srowenJul 11, 2016

Choose a reason for hiding this comment

Uh oh!

phalodi commented Jul 11, 2016

Uh oh!

srowenJul 11, 2016

Choose a reason for hiding this comment

Uh oh!

phalodi commented Jul 11, 2016

Uh oh!

phalodi commented Jul 11, 2016

Uh oh!

phalodi commented Jul 12, 2016• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

srowen commented Jul 12, 2016

Uh oh!

srowen commented Jul 13, 2016

Uh oh!

SparkQA commented Jul 13, 2016

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

srowenJul 10, 2016•
edited
Loading

phalodi commented Jul 12, 2016•
edited
Loading