SCDL

Description

This is the source code for ''A Comprehensive Survey on Video Saliency Detection with Auditory Information: the Audio-visual Consistency Perceptual is the Key!'' paper link

Please contact at "songsook@163.com" if you have any question about the paper and the repository.

Figure 1: The structure of our review.

Figure 2: Demonstrations of the differences between the conventional audio-visual saliency detection model training/testing pipeline (a) and the newly modified training/testing pipeline (b).

Figure 3: Demonstration of the differences regarding the scene contents of six wide-usedly datasets of AVAD, Coutrot1, Coutrot1, DIEM, ETMD and SumMe.

Getting Started

Requirements

• Python 3.7
• Pytorch 1.6.0
• CUDA v10.1, cudnn v.7.5.0
• torchvision

Usage

1.Clone git clone https://github.com/MengkeSong/SCDL.git cd SCDL/

2.Download the datasets

Google Drive link1 (annotation, audio, fold list, and av gt label) and link2 (video frame).

Baidu Netdisk link (r5ca).

Download the following datasets and unzip them into your_data folder. All datasets and labeled Audio-visual Consistency Degree can be downloaded at the Baidu Netdisk or Google Drive.

Then put them under the following directory:

-video_frame/
  -AVAD/
  -Coutrot1/
  -Coutrot2/
  -DIEM/
  -ETMD/
  -SumMe/

-annotation/
  -AVAD/
  -Coutrot1/
  -Coutrot2/
  -DIEM/
  -ETMD/
  -SumMe/

-audio/
  -AVAD/
  -Coutrot1/
  -Coutrot2/
  -DIEM/
  -ETMD/
  -SumMe/

-fold_list/
  -AVAD/
  -Coutrot1/
  -Coutrot2/
  -DIEM/
  -ETMD/
  -SumMe/
  
  -av_gt_label/
  -AVAD/
  -Coutrot1/
  -Coutrot2/
  -DIEM/
  -ETMD/
  -SumMe/

3.Training & Testing

Our method is based on the source codes of STANet, STAViS and AViNet. The original code of this paper will be publicaly available as soon as it has been recomposed.

To quickly train and test first, you can modify the used source codes of STANet, STAViS and AViNet by yourself referring to the technical details mentioned in our manuscript. The used source codes of STANet, STAViS and AViNet are provided in the zip files within /source codes/.

Results

Figure 3: Quantitative comparisons between our method with other fully-/weakly-/un-supervised methods on all 6 datasets.

Citation

Please cite the following article when referring to this method.

@ARTICLE{Chen2022SCDL,
      title={A Comprehensive Survey on Video Saliency Detection with Auditory Information: the Audio-visual Consistency Perceptual is the Key!}, 
      author={Chenglizhao Chen and Mengke Song and Wenfeng Song and Li Guo and Muwei Jian},
      year={2022},
}

Acknowledgement

Thanks to STANet, STAViS and AViNet.