STORM-1345: UpdateTopology API and implementation.

[Parth-Brahmbhatt]

No unit tests added. Update will allow to update config, jar (if some dependency is updated without the topology changing in any way), or the topology change it self.

[unsleepy22]

I have one question, since all worker must be restarted, what's the point of 'update'?
I suggest there should be at least two kinds of 'update':

1. update application's config dynamically without restarting the topology, which is what JStorm has done.
2. update topology's config, which requires restarting.

[revans2]

@Parth-Brahmbhatt how far along is this patch with testing/work? I see a lot of commented out code in the pull request and I am having a little bit of difficulty understanding why. There is also at least one big TODO in the code. Could you please either remove the unneeded code or explain what the next steps are to use it and to deal with the TODO?

Also could you give a brief overview of the design of this. From looking at the code, including the commented out sections, it appears that the intentions is to upload a new jar, config, and/or topology layout. Then rebalance the topology to have it rescheduled. The supervisors also now look at a new topology version field to see if they should kill/restart the worker or not. Is that correct?

The blob store feature moves the conf, topology, and topology-jar into the blob store which also supports doing dynamic updates within the supervisor. I think it would be a lot cleaner if we could try and leverage this functionality.

I also agree with @unsleepy22 that there are several types of updates, and going off of your TODO in the code I think there are several different ways we want to potentially update a topology. There are changes that require a restart of the worker and ones that do not (but may require changes to the system/user code to allow this). There are also changes that should be done as a rolling upgrade, one worker at a time, to allow for minimal impact, but there are others that are so big that you are killing off the topology and just restarting it again, but keeping the ID for some reason.

I am fine with this being a first step where we only support a single type of update, which is the most generic of them. You can change anything and we will restart the entire topology as quickly as possible. But I would like to understand you view about where this feature ultimately ends up.

[Parth-Brahmbhatt]

@revans2@unsleepy22 I did not know Jstorm has it and hasn't had a chance to look at the dist cache.

My goal was to allow users to update topology or config or jar and support workers being restarted in a rolling fashion, When the topology it self is changed we can still do rolling restart and that is the part where I have a TODO that I need to finish.

I thought all of those required workers to bounce, I am not sure which parts can work without bouncing the worker.

The code I have commented is about making update a state just like rebalance/kill/activate and I am still not convinced that it needs to be a state of its own.

I am fine with waiting till we are done with JStorm integration as long as JStorm achieves the same goals. If others think that this is an important enough feature that we should include in the release before we do a feature lockdown, I can upmerge with dist cache and use it as @revans2 suggested.