ORC-2102: [C++] Remove HDFS support

[dongjoon-hyun]

We deprecated C++ HDFS Support to unblock protobuf upgrade

• ORC-1669: [C++] Deprecate HDFS support #1885

This issue aims to remove HDFS support code from Apache ORC repository.

To address

• ORC-1662: [C++] Upgrade protobuf to 3.21.12 #1857
• the conflict of protobuf version between apache orc and tensorflow #2134

[h-vetinari]

We deprecated C++ HDFS Support to unblock protobuf upgrade

What issues are there with C++ on windows? That's one of the major languages (perhaps even the principal one) of protobuf, so bugs there are fixed quickly.

I looked in #1885, but the issues noted in cea0629 (e.g. an extra dependency on boost when updating libhdfspp) don't seem to be a good enough reason to drop a major storage technology completely?

[wgtmac]

The main reason is to reduce maintenance burden. We don't want to introduce extra dependencies (especially boost).

[h-vetinari]

It seems strange to me to remove a core component over maintenance concerns (e.g. many projects manage successfully with boost as a dependency), but ultimately this is a decision of the maintainers here. 🤷

[dongjoon-hyun]

It seems strange to me to remove a core component over maintenance concerns (e.g. many projects manage successfully with boost as a dependency), but ultimately this is a decision of the maintainers here. 🤷

Every ASF projects are maintained by independently by the contributors voluntarily. Why don't you make a contribution to support it, @h-vetinari ?

[h-vetinari]

Why don't you make a contribution to support it

#1882, #2147, #2316 🙃

Plus conda-forge

---

I do a lot of the maintenance around protobuf & boost in conda-forge, including windows, so it's not impossible that I could potentially help out, that's why I asked for specific problems you encountered.

From my POV, if newer libhdfspp needs boost, then that's just the way it is. Staying on an old version of the library is not a sustainable long-term strategy IMO. But if the core maintainers are against something (like a boost dependency) on principle, why would anyone invest time to go down a path that has almost no chance of getting merged?

[dongjoon-hyun]

Of course, I hope your PRs go well eventually.

Since you are here, may I ask if you are using C++ HDFS Support in the production, @h-vetinari ? Could you share your experience about how you use it in which environments (SW versions)?

[h-vetinari]

Since you are here, may I ask if you are using C++ HDFS Support in the production, @h-vetinari ?

I do use many of these technologies in my day job (spark, arrow, orc, hdfs, hadoop, etc.), but my main concern here was as a distributor with an interest in broad compatibility, and someone interested in the success of this ecosystem as a whole. If orc cannot be read from HDFS using C++ anymore, this would not be the end of the world, but it would be an annoying inconsistency for anyone running into this case.

Of course, I do understand the realities of volunteer-driven OSS projects. If you think the effort is not worth it, then that's a good reason to drop things. My question was about understanding the problems to see if they can be fixed (which I could potentially help with), but I cannot offer to help with maintenance long-term, so if you want to get rid of this combination, go for it.

[wgtmac]

I think the main reasons to drop HDFS support are:

1. Lack of active maintainers.
2. No test coverage.
3. It takes pain to upgrade HDFS library each time.
4. Need to bring in extra dependency, especially something like Boost.

We don't have builtin implementations for popular cloud object storage either. We do have an easy-to-implement InputStream interface so users can easily create an adaptation. If users are using Arrow, they can easily use ORC with HDFS support in it (which does not depend on HDFS support from the ORC side).

[dongjoon-hyun]

Apache ORC 2.3 entered Soft Feature Freeze status by creating branch-2.3 with 3-YEAR support plan from the initial release.

• https://github.com/apache/orc/tree/branch-2.3

Apache ORC 3.0.0 preparation starts on main branch via ORC-2100.

• ORC-2100: Setting version to 3.0.0-SNAPSHOT #2541