H264Sharp

Cisco's OpenH264 Native facade/wrapper for .Net with highly optimised SIMD color format conversion support. It is very suitable for realtime streaming over network.
This library is designed to give access to full control on Encoding and Decoding process instead of a blackbox approach, yet offers simplified API for quick setup and go.

SIMD color format converters are up to 2.9x faster than OpenCV implementation.

• Cross Platform
• Plug&Play
• Tested on .NetFramework and Net(up to 8), Windows & Linux & Android (x86 and Arm).
• No memory leaks or GC pressure.
• Compatible with WPF and OpenCV.(i.e. OpenCVsharp)

Cisco Openh264 is chosen for its unbeatible performance compared to other available software encoders. A paper involving performance metrics can be found here:
https://iopscience.iop.org/article/10.1088/1757-899X/1172/1/012036/pdf

Library consist of native dll which acts as OpenH264 wrapper/facade and color format converter (YUV <-> RGB,BGR,RGBA,BGRA)
C# library is .Net standard wrapper library for this dll and performs PInvoke to handle transcoding.

NuGet

Install the NuGet package. All native dependencies should be automatically installed and resolved(from version v1.6.0+).

• Tested on Windows(x86,x64), Linux(x86,x64,arm32,arm64), Android MAUI app(x64 on emulator, arm on Pixel phone).

Binaries also provided on Relases.

H264Sharp

H264SharpBitmapExtentions

For usage in Unity, manually place the dlls from releases. H264SharpNative will be resolved automatically. You have to specify the absolute path for openh264 dll(Cisco). (i.e. StreamingAssets)

Defines.CiscoDllName64bit="{YourPath}/openh264-2.4.0-win64.dll";

[Help Wanted]: Only MacOS and IOS binaries are left to complete this project. I dont own a Mac or an Iphone, and I want to test before shipping instead of just compiling. If you own this devices, it would be greatly appreciated to contribute this binaries.

Example

Examples codes can be found on examples directory on the repository.
For detailed information and documentation please check out Wiki page

Following code shows minimalist example of encoder and decoder in action.

staticvoidMain(string[]args){varimg=System.Drawing.Image.FromFile("ocean 1920x1080.jpg");intw=img.Width;inth=img.Height;varbitmap=newBitmap(img);H264Encoderencoder=newH264Encoder();H264Decoderdecoder=newH264Decoder();decoder.Initialize();encoder.Initialize(w,h,200_000_000,30,ConfigType.CameraCaptureAdvanced);RgbImagergbIn=bitmap.ToRgbImage();RgbImagergbOut=newRgbImage(H264Sharp.ImageFormat.Rgb,w,h);for(intj=0;j<100;j++){// Encodeif(!encoder.Encode(rgbIn,outvarencodedFrames))continue;//skipped// Decodeforeach(varencodedinencodedFrames){if(decoder.Decode(encoded,noDelay:true,outDecodingStateds,refrgbOut)){// Process rgbOut}}}}

Bitmaps are not included on library to keep it cross platform.

For the bitmaps and other image container types, an extention library is provided.

RgbImagergb=newRgbImage(H264Sharp.ImageFormat.Rgb,w,h);Bitmapbmp=rgb.ToBitmap();

And to extract bitmap data:

Bitmapbitmap;// some bitmapRgbImagergb=bitmap.ToRgbImage();

Quick Tips

For detailed information and documentation please check out Wiki page

Data

Data classes can use existing memory or allocate one for you

// Will allocate native memory and manage disposalpublicYuvImage(int width,int height)
public RgbImage(ImageFormatformat,intwidth,intheight)// Will refer to existing memorypublicYuvImage(IntPtr data,int width,int height);publicRgbImage(ImageFormat format,int width,int height,IntPtr imageBytes)
public RgbImage(ImageFormatformat,intwidth,intheight,byte[]data)

Encoder

Encoder has following Encode API:

publicboolEncode(RgbImageim,outEncodedData[]ed)publicboolEncode(YuvImageyuv,outEncodedData[]ed)publicboolEncode(YUVImagePointeryuv,outEncodedData[]ed)publicboolEncode(YUVNV12ImagePointeryuv,outEncodedData[]ed)

Where

• YuvImage and RgbImage may or may not own the memory depending on how its constructed.
  
• YUVImagePointer and YUVNV12ImagePointer are ref structs and only points to an existing memory.

EncodedData[] are the data frames of the encoder and refers to a ephemeral native memory and will be overwrtitten on next encode. You can get the native frames one by one, or all merged, into contigious managed memory:

..out EncodedData[] ec
// You can extract the bytes one by one.byte[] encodedbytes = ec[0].GetBytes();ec[0].CopyTo(buffer,offset);// or merge into single arraybyte[]encodedbytes=ec.GetAllBytes();ec.CopyAllTo(buffer,offset);

On single layer(standard use case) for IDR frames you get more than one EncodedData, the first frame is a metadata and it will produce neither an image nor an error when decoded.
Decoder can work with both frame by frame or merged.

Decoder

Decoder has the API:

publicboolDecode(byte[]encoded,intoffset,intcount,boolnoDelay,outDecodingStatestate,outYUVImagePointeryuv)publicboolDecode(byte[]encoded,intoffset,intcount,boolnoDelay,outDecodingStatestate,refYuvImageyuv)publicboolDecode(byte[]encoded,intoffset,intcount,boolnoDelay,outDecodingStatestate,refRgbImageimg)

Cisco decoder only supports YUV I420 Planar output

API Where:

• YUVImagePointer directly refers to decoder's native buffer, Cisco adds 64 byte padding.
  
• YuvImage Copies the native YUV image into provided container.
  
• RgbImage Output will convert native YUV into RGB container(RGB,BGR,RGBA or BGRA) provided.
  

If methods return false means there is no image and you need to check DecodingState. Otherwise there is an image but, on lossy link you still need to check DecodingState for error and perform necessary action(i.e. perform IDR refresh request to encoder).

Tips

• Raw image bytes are large, avoid allocating new ones and try to reuse same RgbImage or YuvImage or pool them in something like concurrent bag.

For more information refer to Tutorial

Advanced Configuration & Features

Advanced Setup

If you want to initialise your encoder and able to control everything, you can use provided API which is identical to Ciso C++ Release.

encoder=newH264Encoder();varparam=encoder.GetDefaultParameters();param.iUsageType=EUsageType.CAMERA_VIDEO_REAL_TIME;param.iPicWidth=w;param.iPicHeight=h;param.iTargetBitrate=1000000;param.iTemporalLayerNum=1;param.iSpatialLayerNum=1;param.iRCMode=RC_MODES.RC_BITRATE_MODE;param.sSpatialLayers[0].iVideoWidth=0;param.sSpatialLayers[0].iVideoWidth=0;param.sSpatialLayers[0].fFrameRate=60;param.sSpatialLayers[0].iSpatialBitrate=1000000;param.sSpatialLayers[0].uiProfileIdc=EProfileIdc.PRO_HIGH;param.sSpatialLayers[0].uiLevelIdc=0;param.sSpatialLayers[0].iDLayerQp=0;param.iComplexityMode=ECOMPLEXITY_MODE.HIGH_COMPLEXITY;param.uiIntraPeriod=300;param.iNumRefFrame=0;param.eSpsPpsIdStrategy=EParameterSetStrategy.SPS_LISTING_AND_PPS_INCREASING;param.bPrefixNalAddingCtrl=false;param.bEnableSSEI=true;param.bSimulcastAVC=false;param.iPaddingFlag=0;param.iEntropyCodingModeFlag=1;param.bEnableFrameSkip=false;param.iMaxBitrate=0;param.iMinQp=0;param.iMaxQp=51;param.uiMaxNalSize=0;param.bEnableLongTermReference=true;param.iLTRRefNum=1;param.iLtrMarkPeriod=180;param.iMultipleThreadIdc=1;param.bUseLoadBalancing=true;param.bEnableDenoise=false;param.bEnableBackgroundDetection=true;param.bEnableAdaptiveQuant=true;param.bEnableSceneChangeDetect=true;param.bIsLosslessLink=false;param.bFixRCOverShoot=true;param.iIdrBitrateRatio=400;param.fMaxFrameRate=30;encoder.Initialize(param);

Similarly for decoder

decoder=newH264Decoder();TagSVCDecodingParamdecParam=newTagSVCDecodingParam();decParam.uiTargetDqLayer=0xff;decParam.eEcActiveIdc=ERROR_CON_IDC.ERROR_CON_FRAME_COPY_CROSS_IDR;decParam.sVideoProperty.eVideoBsType=VIDEO_BITSTREAM_TYPE.VIDEO_BITSTREAM_SVC;decoder.Initialize(decParam);

Converter

Significant development effort has been spent here to deliver best possible performance. It was impactful to improve format conversions as much as possible especially for mobile platforms.

Color format conversion (RGB <-> YUV) has optional configuration where you can provide number of threads on parallelisation.
Using 1 thread consumes least cpu cycles(minimum context switch) and most efficient but it may be slower.
Performance of parallel proccesing depends on image size, memory speed, core IPC, cache size and many other factors, so your milage may vary.

You can configure on RGB to YUV conversion SIMD support. By default highest supported instruction set will be selected on runtime For example, if AVX2 is enabled it wont run SSE version Neon is only active on arm and does nothing on x86 systems.

varconfig=Converter.GetCurrentConfig();config.EnableSSE=1;config.EnableNeon=1;config.EnableAvx2=1;config.NumThreads=4;// default is 1config.EnableCustomThreadPool=1;Converter.SetConfig(config);// Or like..Converter.SetOption(ConverterOption.NumThreads,8);

Converter Bechmarks

H264Sharp conversion operations are up to 2.9x faster than OpenCV implementations.

1080p 5000 Iterations of RGB -> YUV and YUV -> RGB, CustomThreadPool AMD Ryzen 7 3700X Desktop CPU

#Threads	OpenCV (ms)	H264Sharp (ms)
1	11919	4899
2	6205	2479
4	3807	1303
8	2543	822
16	2462	824

Intel i7 10600U Laptop CPU 1080p 5000 Iterations of RGB -> YUV and YUV -> RGB, CustomThreadPool

#Threads	OpenCV (ms)	H264Sharp (ms)
1	11719	6010
2	6600	3210
4	4304	2803
8	3560	1839

Options

You can get and set options to decoder and encoder on runtime. All options API is implemented 1-1 with cisco api.

encoder.SetOption(ENCODER_OPTION.ENCODER_OPTION_IDR_INTERVAL,600);encoder.GetOption(ENCODER_OPTION.ENCODER_OPTION_IDR_INTERVAL,outintidrPeriod);decoder.GetOption(DECODER_OPTION.DECODER_OPTION_FRAME_NUM,outtempInt);// If you want to reuse your option structs for efficiency:SEncoderStatisticsss;SDecoderStatisticsss1;encoder.GetOptionRef(ENCODER_OPTION.ENCODER_OPTION_GET_STATISTICS,refss);decoder.GetOptionRef(DECODER_OPTION.DECODER_OPTION_GET_STATISTICS,refss1);

There are many possible options and they are commented on the enum fields as well as required types. If you want more detail, search as general H264 options.
Because you wont find any documentation on cisco side.

Example App

A simple example WPF application is provided. This app emulates advanced use cases for the lossy transfers(loss&jitter) leveraging LTR references.

here you can explore:

• Advanced Setup and their effects.
• Using LTR references and loss recovery.