Add FFmpeg support to open video files by yongtang · Pull Request #30 · tensorflow/io

yongtang · 2018-12-16T17:45:13Z

This fix adds a preliminary video file support (VideoDataset) by linking and using FFmpeg's libraries.

The video.VideoDataset takes input of filename list, and generates outputs of RGB24 images (uint8) in (height, width, 3).

The reason to link against FFmpeg's libraries is because FFmpeg has a LGPL 2.1+ license.

On Ubuntu 18.04, FFmpeg libraries consists of:

libavformat.so.57
libavcodec.so.57
libavutil.so.55
libswscale.so.4

Because FFmpeg only guarantee's major version conpatibility, linking to a lower version (e.g., libavformat.so.54 on Ubuntu 14.04) may not work.

To support multiple versions of Ubuntu, multiple copies of header file combined with linking to different versions of .so might be needed.

It needs to be fully tested though and only limited major version should be supported.

~~At the moment, only ffmpeg libraries in Ubuntu 18.04 are used.~~
UPDATE:
At the moment, Ubuntu 14.04, 16.04, and 18.04 are all supported. We build 3 .so files and distributed at the same time:

_video_ops_ffmpeg_3.4.so (Ubuntu 18.04)
_video_ops_ffmpeg_2.8.so (Ubuntu 16.04)
_video_ops_libav_9.20.so (Ubuntu 14.04)

Additional platform could be added if there is enough need from the community.

To install required libraries on Ubuntu 18.04:

apt-get -y -qq install libavformat57 libavcodec57 libavutil55 libswscale4

To build the pip package:

bazel build -s --verbose_failures build_pip_pkg

NOTE: This PR fixed#11

Signed-off-by: Yong Tang yong.tang.github@outlook.com

googlebot · 2018-12-17T01:04:25Z

So there's good news and bad news.

👍 The good news is that everyone that needs to sign a CLA (the pull request submitter and all commit authors) have done so. Everything is all good there.

😕 The bad news is that it appears that one or more commits were authored or co-authored by someone other than the pull request submitter. We need to confirm that all authors are ok with their commits being contributed to this project. Please have them confirm that here in the pull request.

Note to project maintainer: This is a terminal state, meaning the cla/google commit status will not change from this state. It's up to you to confirm consent of all the commit author(s), set the cla label to yes (if enabled on your project), and then merge this pull request when appropriate.

googlebot · 2018-12-17T01:11:53Z

CLAs look good, thanks!

terrytangyuan · 2018-12-17T03:37:30Z

+# License for the specific language governing permissions and limitations under
+# the License.
+# ==============================================================================
+"""Tests for SequenceFileDataset."""


Update this to VideoDataset instead of SequenceFileDataset? Same for other places

Thanks @terrytangyuan, the PR has been updated.

yongtang · 2018-12-18T21:32:57Z

I think this PR is almost ready. I am working on adding Ubuntu 16.04 and 14.04 support (on top of 18.04 support that is already in place). Will update the PR shortly to completion.

yongtang · 2018-12-18T23:46:41Z

I think this PR is ready. We support Ubuntu 14.04, 16.04, and 18.04 at the moment, with one distributed package in pip. Tests also pass on those platforms.

martinwicke · 2018-12-19T00:32:11Z

This is a thing of beauty. I especially appreciate the multi-version tooling.

Since these are LGPL, we have to make sure to only release dynamically linked versions. Can we make sure (with a test/scripting) that this is always going ot be the case?

yongtang · 2018-12-19T19:37:25Z

@martinwicke The library only uses header files from ffmpeg to compile and build. In Bazel there is a check-license option though I don't know how reliable it is. I think some additional tooling might be possible (maybe a module in bazel) to enforce only header files are included for LGPL types of libraries .

I will take a look at the license tooling issue.

martinwicke · 2018-12-19T19:39:08Z

I think as long as we make sure all releases are built with dynamic linking, we should be ok.

This fix adds a preliminary video file support (`VideoDataset`) by linking and using FFmpeg's libraries. The `video.VideoDataset` takes input of filename list, and generates outputs of RGB24 images (`uint8`) in `(height, width, 3)`. The reason to link against FFmpeg's libraries is because FFmpeg has a LGPL 2.1+ license. On Ubuntu 18.04, FFmpeg libraries consists of: ``` libavformat.so.57 libavcodec.so.57 libavutil.so.55 libswscale.so.4 ``` Because FFmpeg only guarantee's major version conpatibility, linking to a lower version (e.g., libavformat.so.54 on Ubuntu 14.04) may not work. To support multiple versions of Ubuntu, multiple copies of header file combined with linking to different versions of .so might be needed. It needs to be fully tested though and only limited major version should be supported. At the moment, only ffmpeg libraries in Ubuntu 18.04 are used. To install required libraries on Ubuntu 18.04: ``` apt-get -y -qq install libavformat57 libavcodec57 libavutil55 libswscale4 ``` To build the pip package: ``` bazel build -s --verbose_failures build_pip_pkg ``` Signed-off-by: Yong Tang <yong.tang.github@outlook.com>

Signed-off-by: Yong Tang <yong.tang.github@outlook.com>

So that build could always succeed. In runtime system .so files will be used. Signed-off-by: Yong Tang <yong.tang.github@outlook.com>

…ave Ubuntu 18.04 support directly. Signed-off-by: Yong Tang <yong.tang.github@outlook.com>

Encountered quite some issues with ``` Use --sandbox_debug to see verbose messages from the sandbox /usr/bin/ld.gold: error: cannot find libavformat.so.57 /usr/bin/ld.gold: error: cannot find libavcodec.so.57 /usr/bin/ld.gold: error: cannot find libavutil.so.55 /usr/bin/ld.gold: error: cannot find libswscale.so.4 collect2: error: ld returned 1 exit status Target //tensorflow_io/video:video_py_test failed to build ``` Believe it is related to (or similiar): https://stackoverflow.com/questions/52386530/linker-fails-in-sandbox-when-running-through-bazel-but-works-when-sandboxed-comm?rq=1 Signed-off-by: Yong Tang <yong.tang.github@outlook.com>

(By default Travis CI uses Ruby) Signed-off-by: Yong Tang <yong.tang.github@outlook.com>

…ainer Signed-off-by: Yong Tang <yong.tang.github@outlook.com>

Signed-off-by: Yong Tang <yong.tang.github@outlook.com>

…nore error Signed-off-by: Yong Tang <yong.tang.github@outlook.com>

Signed-off-by: Yong Tang <yong.tang.github@outlook.com>

…brary and use `cc_import_library` rule to prevent including GPL code. Signed-off-by: Yong Tang <yong.tang.github@outlook.com>

Signed-off-by: Yong Tang <yong.tang.github@outlook.com>

yongtang · 2019-01-08T23:03:52Z

@terrytangyuan @martinwicke The PR has been updated.

I added a customized implementation of bazel rule def (cc_import_library) which controls the dependency so that library will only be linked to, and no source files (like GPL) are included or compiled.

It is pretty much similar to cc_import in Bazel's standard rule, except that cc_import does not support .so with soname. I created an issue in bazel repo: bazelbuild/bazel#7059

I think this FFmpeg PR is ready, please take a look and let me know if there are any issues.

terrytangyuan

LGTM!

yongtang · 2019-01-13T17:49:32Z

Thanks. Will merge this PR in, and have a follow up PR with an example.

* feat: reading from bigtable (#2) Implements reading from bigtable in a synchronous manner. * feat: RowRange and RowSet API. * feat: parallel read (#4) In this pr we make the read methods accept a row_set reading only rows specified by the user. We also add a parallel read, that leverages the sample_row_keys method to split work among workers. * feat: version filters (#6) This PR adds support for Bigtable version filters. * feat: support for other data types (#5) * fix: linter fixes (#8) * feat docs (#9) * fix: building on windows (#12) * fix: refactor bigtable package to api folder (#14) moved bigtable to tfensorflow_io.python.api * fix: tests hanging (#30) changed path to bigtable emulator and cbt in tests moved arguments' initializations to the body of the function in bigtable_ops.py fixed interleaveFromRange of column filters when using only one column * fix: temporarily disable macos tests (#32) * disable tests on macos Co-authored-by: Kajetan Boroszko <kajetan@unoperate.com> Co-authored-by: Kajetan Boroszko <kajetan.boroszko@gmail.com>

* feat: reading from bigtable (tensorflow#2) Implements reading from bigtable in a synchronous manner. * feat: RowRange and RowSet API. * feat: parallel read (tensorflow#4) In this pr we make the read methods accept a row_set reading only rows specified by the user. We also add a parallel read, that leverages the sample_row_keys method to split work among workers. * feat: version filters (tensorflow#6) This PR adds support for Bigtable version filters. * feat: support for other data types (tensorflow#5) * fix: linter fixes (tensorflow#8) * feat docs (tensorflow#9) * fix: building on windows (tensorflow#12) * fix: refactor bigtable package to api folder (tensorflow#14) moved bigtable to tfensorflow_io.python.api * fix: tests hanging (tensorflow#30) changed path to bigtable emulator and cbt in tests moved arguments' initializations to the body of the function in bigtable_ops.py fixed interleaveFromRange of column filters when using only one column * fix: temporarily disable macos tests (tensorflow#32) * disable tests on macos Co-authored-by: Kajetan Boroszko <kajetan@unoperate.com> Co-authored-by: Kajetan Boroszko <kajetan.boroszko@gmail.com>

yongtang mentioned this pull request Dec 16, 2018
Support video format for tensorflow-io #11
Closed

yongtang force-pushed the ffmpeg branch 2 times, most recently from 7c8fc9c to 66df035Compare December 17, 2018 01:04

yongtang force-pushed the ffmpeg branch from 66df035 to aabf043Compare December 17, 2018 01:11

terrytangyuan reviewed Dec 17, 2018
View reviewed changes

yongtang force-pushed the ffmpeg branch 11 times, most recently from 272d247 to abadf38Compare December 18, 2018 14:14

yongtang changed the title ~~[WIP] Add FFmpeg support to open video files~~Add FFmpeg support to open video filesDec 18, 2018

yongtang force-pushed the ffmpeg branch 2 times, most recently from a6c0ae3 to a928008Compare December 18, 2018 23:10

yongtang mentioned this pull request Dec 18, 2018
tf.contrib.ffmpeg: No mention of alternatives after deprecation tensorflow/tensorflow#24424
Closed

yongtang force-pushed the ffmpeg branch from 8567891 to e7ee4deCompare January 5, 2019 04:52

yongtang added 2 commits January 8, 2019 18:58

Add python wrapper for video.VideoDataset
9a500e5
Signed-off-by: Yong Tang <yong.tang.github@outlook.com>

yongtang added 16 commits January 8, 2019 18:58

Add test case for video.VideoDataset
fdf7499
Signed-off-by: Yong Tang <yong.tang.github@outlook.com>

Add BUILD file for tensorflow_io.video
cd22417
Signed-off-by: Yong Tang <yong.tang.github@outlook.com>

Add ffmpeg n3.4.4 dependency
6dcdaa5
Signed-off-by: Yong Tang <yong.tang.github@outlook.com>

Update BUILD file to include video.VideoDataset
15b2c43
Signed-off-by: Yong Tang <yong.tang.github@outlook.com>

Disable -//tensorflow_io/video:video_py_test test on Ubuntu 14.04
1df47e2
Signed-off-by: Yong Tang <yong.tang.github@outlook.com>

Update ffmpeg to use stub .so files
738b8ea
So that build could always succeed. In runtime system .so files will be used. Signed-off-by: Yong Tang <yong.tang.github@outlook.com>

Add Travis CI for Ubuntu 18.04, use docker run as Travis does not h…
d218d35
…ave Ubuntu 18.04 support directly. Signed-off-by: Yong Tang <yong.tang.github@outlook.com>

Use python as base vm image for Travis CI Ubuntu 18.04
29bb7dc
(By default Travis CI uses Ruby) Signed-off-by: Yong Tang <yong.tang.github@outlook.com>

Only Enable Video on Travis CI for Ubuntu 18.04, as it is inside cont…
f26e4ee
…ainer Signed-off-by: Yong Tang <yong.tang.github@outlook.com>

Add Ubuntu 16.04 support, by build separate so with ffmpeg 2.8
a113c69
Signed-off-by: Yong Tang <yong.tang.github@outlook.com>

Fix additional issue in ffmpeg 3.4 where cached packet should just ig…
bdb85d3
…nore error Signed-off-by: Yong Tang <yong.tang.github@outlook.com>

Add Ubuntu 14.04 support, with libav 9.20 built in.
f3884ae
Signed-off-by: Yong Tang <yong.tang.github@outlook.com>

Add libav dependency and enable video test on Ubuntu 14.04
ccc8a20
Signed-off-by: Yong Tang <yong.tang.github@outlook.com>

Enable Ubuntu 16.04 for video.VideoDataset
a5cbf32
Signed-off-by: Yong Tang <yong.tang.github@outlook.com>

Defined a bazel rule of cc_import_library to only accepts system li…
10a402d
…brary and use `cc_import_library` rule to prevent including GPL code. Signed-off-by: Yong Tang <yong.tang.github@outlook.com>

yongtang force-pushed the ffmpeg branch from e7ee4de to 10a402dCompare January 8, 2019 19:09

Reduce verbose logging output by bazel
4da4dd2
Signed-off-by: Yong Tang <yong.tang.github@outlook.com>

yongtang mentioned this pull request Jan 12, 2019
Support audio format for tensorflow-io #49
Closed

terrytangyuan approved these changes Jan 12, 2019
View reviewed changes

yongtang mentioned this pull request Jan 12, 2019
Support live video for tensorflow-io #51
Open

yongtang merged commit 5fac290 into tensorflow:masterJan 13, 2019

yongtang deleted the ffmpeg branch January 13, 2019 17:49

wookayin mentioned this pull request Mar 11, 2019
Support decoding video from byte stream for tensorflow-io #144
Closed

missaugustina mentioned this pull request May 8, 2019
Create docs/tutorial/ consistent with other TF projects #227
Closed

smellslikeml mentioned this pull request Apr 2, 2021
FFmpeg scene change detection eto-ai/rikai#156
Closed

Navigation Menu

Uh oh!

Add FFmpeg support to open video files - #30

Add FFmpeg support to open video files#30
yongtang merged 19 commits into
tensorflow:masterfrom
yongtang:ffmpeg

yongtang commented Dec 16, 2018•
edited
Loading

Uh oh!

googlebot commented Dec 17, 2018

Uh oh!

googlebot commented Dec 17, 2018

Uh oh!

terrytangyuanDec 17, 2018

Uh oh!

yongtangDec 17, 2018

Uh oh!

yongtang commented Dec 18, 2018

Uh oh!

yongtang commented Dec 18, 2018

Uh oh!

martinwicke commented Dec 19, 2018

Uh oh!

yongtang commented Dec 19, 2018

Uh oh!

martinwicke commented Dec 19, 2018

Uh oh!

yongtang commented Jan 8, 2019

Uh oh!

terrytangyuan left a comment •
edited
Loading

Uh oh!

yongtang commented Jan 13, 2019

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Conversation

yongtang commented Dec 16, 2018• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

googlebot commented Dec 17, 2018

Uh oh!

googlebot commented Dec 17, 2018

Uh oh!

terrytangyuanDec 17, 2018

Choose a reason for hiding this comment

Uh oh!

yongtangDec 17, 2018

Choose a reason for hiding this comment

Uh oh!

yongtang commented Dec 18, 2018

Uh oh!

yongtang commented Dec 18, 2018

Uh oh!

martinwicke commented Dec 19, 2018

Uh oh!

yongtang commented Dec 19, 2018

Uh oh!

martinwicke commented Dec 19, 2018

Uh oh!

yongtang commented Jan 8, 2019

Uh oh!

terrytangyuan left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

yongtang commented Jan 13, 2019

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

yongtang commented Dec 16, 2018•
edited
Loading

terrytangyuan left a comment •
edited
Loading