STORM-997: Add proxy user functionality for storm hdfs connector.

[priyank5485]

No description provided.

[harshach]

@priyank5485 we need to add this trident as well.

[priyank5485]

@harshach thanks for the review. Yes, I will do it. I realized that it needs to be added for trident after creating the pull request.

[priyank5485]

@Parth-Brahmbhatt@ptgoetz Can you please review this?

[revans2]

@priyank5485 and others I am not totally sure that this is a good idea. If HDFS is running secure and Storm is running insecure, with this feature enabled you have now disabled security in HDFS, so anyone who can submit a topology (a.k.a everyone) can pretend to be any user in HDFS. To me I would much rather see the feature that allows this to work with an arbitrary UGI. Or better have a keytab installed on the worker nodes then you just need a config to point to that keytab and the principal you want to use out of it.

I am -0 on this change. I am not going to block it, but I would prefer to see it done a different way.

[priyank5485]

@revans2 I think what you mentioned above would not happen. I mean the part where you say that storm running in unsecured mode and hdfs in secured mode will disable security. Please correct me if i am wrong. I am new to this. But the way i imagine it will work is that the proxy user functionality code(which is doAs part) will be executed only after the user is logged in to HDFS. In secured mode user will already be validated using keytab. Above that hdfs needs config entries in core-site.xml which is needed for the proxy user functionality to work. Please look at the configurations section at http://hadoop.apache.org/docs/r2.7.1/hadoop-project-dist/hadoop-common/Superusers.html
hdfs in that case would make sure that if the logged in user is not authorized to impersonate as some other user based on that config it will throw an exception. Again, there is a good chance i might be missing something here. Please correct me if i am wrong.

[revans2]

@priyank5485 You are correct that the name-node needs to be configured to explicitly allow super users and/or groups and optionally restrict those to specific hosts. My issue is not with the code but with how it is intended to be used which is why I am -0. I will not block this going in, but I will not support it either. From the description of STORM-997

In a non-secure environment, Storm HDFS component that provides interaction with HDFS from storm currently does that as the user storm with which the worker process had been started. We want to allow the component to interact with hdfs as the user provided instead of user running the worker process.

To me that sounds like I have an "guest" account on my computer that everyone logs in to. Some of the users don't want to be "guest" so instead of giving them all a separate login and password. I will grant "guest" sudo privileges so they can become any user.

The issue is the chain of attach. If storm is "non-secure" anyone with access to the nimbus thrift port can run arbitrary code on the storm cluster. If you configure HDFS so the "storm" user can become anyone on HDFS anyone with access to the nimbus thrift port now has complete access to all of HDFS. You have effectively made HDFS as secure as your storm cluster which by definition is "non-secure".

To really make this correct you either want to make storm secure (give each user their own login and password to your computer, and have them stop using "guest") or at a minimum severely restrict what the storm user can do on HDFS. To do that I would think you could make AutoTGT work with an insecure storm cluster. This would not be that hard, we just have to let it look someplace else for the TGT, instead of relying on the StormClient section of the jaas.conf. Or update AutoHDFS to work in insecure mode and have some configuration to restrict which delegation tokens it will fetch for users of this cluster. To be sure that works you have to be sure "nimbus" is a different user from "storm", or at least HDFS will only allow super user access from nimbus.

This is still not great because anyone with access to ZK could now steal either the TGT or the delegation tokens, but you have only made some very specific HDFS accounts "non-secure" instead of all of HDFS.