Update protobuf3 and use CMake by dstoup · Pull Request #561 · Kitware/fletch

dstoup · 2019-10-28T18:30:45Z

Currently this branch sets the default Protobuf version to 3 for testing purposes only. We have more work to do before version 3 can be the default.l If this branch passes dashboards, I will pull the version back to 2 and repush.

kwcvrobot · 2019-10-28T19:28:16Z

KWJenkins-FletchCentosPR-561 experienced errors.

kwcvrobot · 2019-10-28T20:24:06Z

KWJenkins-FletchLinuxGPUPR-561 experienced errors.

kwcvrobot · 2019-10-28T20:32:29Z

KWJenkins-FletchMacPR-561 experienced errors.

kwcvrobot · 2019-10-28T22:52:46Z

KwiverCheckWindowsPR experienced errors.

kwcvrobot · 2019-10-28T23:16:11Z

KWIVER against Fletch Windows build succeeded.

kwcvrobot · 2019-10-29T21:06:43Z

KWIVER against Fletch Centos build succeeded.

kwcvrobot · 2019-10-29T21:06:49Z

KWIVER against Fletch Linux build succeeded.

kwcvrobot · 2019-10-29T21:21:05Z

KWIVER against Fletch LinuxGPU build succeeded.

kwcvrobot · 2019-10-29T21:32:05Z

KwiverCheckMacPR experienced errors.

kwcvrobot · 2019-10-29T23:39:34Z

KwiverCheckWindowsPR experienced errors.

kwcvrobot · 2019-10-29T23:39:45Z

KWIVER against Fletch Windows build succeeded.

kwcvrobot · 2019-10-30T13:20:19Z

KWJenkins-FletchWindowsPR-561 experienced errors.

kwcvrobot · 2019-10-30T13:34:24Z

KWJenkins-FletchLinuxPR-561 experienced errors.

kwcvrobot · 2019-10-30T15:19:04Z

KWIVER against Fletch Centos build succeeded.

dstoup · 2019-10-30T15:38:25Z

Jenkins test this please

kwcvrobot · 2019-10-30T17:25:11Z

KWIVER against Fletch Centos build succeeded.

kwcvrobot · 2019-10-30T17:25:17Z

KWIVER against Fletch Linux build succeeded.

kwcvrobot · 2019-10-30T17:50:12Z

KWIVER against Fletch Mac build succeeded.

kwcvrobot · 2019-10-30T20:25:22Z

KWIVER against Fletch Windows build succeeded.

kwcvrobot · 2019-10-30T22:21:38Z

KWIVER against Fletch LinuxGPU build succeeded.

dstoup · 2019-10-31T11:23:37Z

@aaron-bray The dashboards passed but I think you'd mentioned something about kwiver's definitions needing to change as a result of pb3. Should I keep 2 as the default for now? Thinking that's the right thing anyway until we get buyin, but wanted to get a sense of the rest of the work left to do.

aaron-bray · 2019-11-15T14:14:15Z

+ ${COMMON_EP_ARGS}
+ ${COMMON_CMAKE_EP_ARGS}
+# UPDATE_COMMAND
+# COMMAND ${CMAKE_COMMAND} -DPULSE_IL2CPP_PATCH=${PULSE_IL2CPP_PATCH} -Dprotobuf_source=${protobuf_SRC} -Dprotobuf_patch=${protobuf_Patch} -P ${protobuf_Patch}/Patch.cmake


You can take this patch out. It was put in to work with the Unity IL2CPP compiler that converts C# to C++.
The IL2CPP compiler can be a bit too aggressive, so this patch was created
By the time you are supporting kwiver/fletch into C# and Unity, this will probably have changed
Just let me know then

Thanks ... what are you thoughts about making protobuf3 the default? It certainly works building/testing kwiver on all platforms.

aaron-bray · 2019-11-15T14:17:03Z

+ ${COMMON_CMAKE_ARGS}
+ -Dprotobuf_BUILD_TESTS:BOOL=OFF
+ -Dprotobuf_BUILD_EXAMPLES:BOOL=OFF
+ -Dprotobuf_BUILD_SHARED_LIBS:BOOL=${BUILD_SHARED_LIBS}


Please read this tid bit about protobuf linkage
https://github.com/protocolbuffers/protobuf/tree/master/cmake#dlls-vs-static-linking
I suggest we always build protobuf as static libs, and they should only be linked into the protobuf arrow and nothing else. Should work great in the kwiver architecture.

aaron-bray · 2019-11-15T14:41:09Z

protobuf3 can read in and use protobuf2 files
I have also built protobuf3 in fletch and used that in a Caffe build, which has protobuf2 proto files, and everything still worked fine.

So you really don't need to move the kwiver proto files fro 2 to 3 immediately
It does not look like protobuf2 supports JSON
And there are other differences between the versions
https://groups.google.com/forum/#!topic/protobuf/h-nwPLb42ac

You should be able to remove protobuf2 altogether from fletch and just use 3
I think we should move to protobuf3 proto files unless there is a compelling reason to stay at 2
But we can do that on our own time and still only build protobuf3

So I feel pretty confident we can nix proto2 from fletch all together and not sink any ships

dstoup · 2019-11-15T14:52:45Z

Perfect, thanks! I will keep protobuf3 as the default, leave protobuf2 for just a little while and look to remove it completely soon. I'm looking into the static thing as well. I suspect I will have to force PIC in those cases since it didn't build correctly out of the box that way (on Linux). I think VS was fine building static

aaron-bray · 2019-11-15T14:53:32Z

Note,
I think using the protobuf3 library (even with protobuf2 proto files) we should still get JSON support...
But I would want to test that just to be sure...

kwcvrobot · 2019-11-15T15:35:45Z

KWIVER against Fletch Centos build succeeded.

kwcvrobot · 2019-11-15T15:35:51Z

KWIVER against Fletch Linux build succeeded.

kwcvrobot · 2019-11-15T15:52:27Z

KWIVER against Fletch LinuxGPU build succeeded.

kwcvrobot · 2019-11-15T16:08:37Z

KWIVER against Fletch Mac build succeeded.

kwcvrobot · 2019-11-15T17:38:12Z

KWJenkins-FletchWindowsPR-561 experienced errors.

kwcvrobot · 2019-11-18T22:10:48Z

KWIVER against Fletch Centos build succeeded.

kwcvrobot · 2019-11-18T22:10:56Z

KWIVER against Fletch Linux build succeeded.

kwcvrobot · 2019-11-18T22:30:07Z

KWIVER against Fletch LinuxGPU build succeeded.

kwcvrobot · 2019-11-18T22:42:06Z

KWIVER against Fletch Mac build succeeded.

kwcvrobot · 2019-11-19T01:26:21Z

KwiverCheckWindowsPR experienced errors.

kwcvrobot · 2019-11-19T01:26:32Z

KWIVER against Fletch Windows build succeeded.

Update protobuf3 and use CMake
d7d95e7

dstoup requested review from aaron-bray and linus-sherrill October 28, 2019 18:30

Fix the build_shared flag to use protobuf custom flag
d8801bd

Open up Protofuf 3 build for WIN32
80fcb9e

dstoup force-pushed the cmake-upgrade-protobuf3 branch from e31efc8 to 80fcb9eCompare October 30, 2019 13:30

aaron-bray reviewed Nov 15, 2019
View reviewed changes

Remove unnecessary patch.
7c0f237

Build protobuf statically.
cafb37e

dstoup force-pushed the cmake-upgrade-protobuf3 branch from a69052e to cafb37eCompare November 18, 2019 19:59

Uh oh!

Conversation

dstoup commented Oct 28, 2019

Uh oh!

kwcvrobot commented Oct 28, 2019

Uh oh!

kwcvrobot commented Oct 28, 2019

Uh oh!

kwcvrobot commented Oct 28, 2019

Uh oh!

kwcvrobot commented Oct 28, 2019

Uh oh!

kwcvrobot commented Oct 28, 2019

Uh oh!

kwcvrobot commented Oct 29, 2019

Uh oh!

kwcvrobot commented Oct 29, 2019

Uh oh!

kwcvrobot commented Oct 29, 2019

Uh oh!

kwcvrobot commented Oct 29, 2019

Uh oh!

kwcvrobot commented Oct 29, 2019

Uh oh!

kwcvrobot commented Oct 29, 2019

Uh oh!

kwcvrobot commented Oct 30, 2019

Uh oh!

kwcvrobot commented Oct 30, 2019

Uh oh!

kwcvrobot commented Oct 30, 2019

Uh oh!

dstoup commented Oct 30, 2019

Uh oh!

kwcvrobot commented Oct 30, 2019

Uh oh!

kwcvrobot commented Oct 30, 2019

Uh oh!

kwcvrobot commented Oct 30, 2019

Uh oh!

kwcvrobot commented Oct 30, 2019

Uh oh!

kwcvrobot commented Oct 30, 2019

Uh oh!

dstoup commented Oct 31, 2019

Uh oh!

aaron-brayNov 15, 2019

Choose a reason for hiding this comment

Uh oh!

dstoupNov 15, 2019

Choose a reason for hiding this comment

Uh oh!

aaron-brayNov 15, 2019

Choose a reason for hiding this comment

Uh oh!

aaron-bray commented Nov 15, 2019

Uh oh!

dstoup commented Nov 15, 2019

Uh oh!

aaron-bray commented Nov 15, 2019

Uh oh!

kwcvrobot commented Nov 15, 2019

Uh oh!

kwcvrobot commented Nov 15, 2019

Uh oh!

kwcvrobot commented Nov 15, 2019

Uh oh!

kwcvrobot commented Nov 15, 2019

Uh oh!

kwcvrobot commented Nov 15, 2019

Uh oh!

kwcvrobot commented Nov 18, 2019

Uh oh!

kwcvrobot commented Nov 18, 2019

Uh oh!

kwcvrobot commented Nov 18, 2019

Uh oh!

kwcvrobot commented Nov 18, 2019

Uh oh!

kwcvrobot commented Nov 19, 2019