VLAD

Synopsis

Python implementation of VLAD for a CBIR system. Reference:

---

H. Jégou, F. Perronnin, M. Douze, J. Sánchez, P. Pérez and C. Schmid, "Aggregating Local Image Descriptors into Compact Codes," in IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 34, no. 9, pp. 1704-1716, Sept. 2012. doi: 10.1109/TPAMI.2011.235

Code Example

A query (-q queries/0706.3046-img-1-22.jpg) looking for the seven most similar images (-r 7) using ORB descriptors (-d ORB) and a visual vocabulary of 16 words (-dV visualDictionary/visualDictionary16ORB.pickle) and a ball-tree data structure as index (-i ballTreeIndexes/index_ORB_W16.pickle) is given by:

pythonquery.py-qqueries/0706.3046-img-1-22.jpg-r7-dORB-dVvisualDictionary/visualDictionary16ORB.pickle-iballTreeIndexes/index_ORB_W16.pickle

You must compute the following first: Descriptors, Visual Dictionaries, ball-tree indexes and VLAD descriptors, see section "Computing VLAD features for a new dataset" below for details.

Another examples of queries:

SIFT

pythonquery.py-qqueries/0706.3046-img-1-22.jpg-r7-dSIFT-dVvisualDictionary/visualDictionary16SIFT.pickle-iballTreeIndexes/index_SIFT_W16.picklepythonquery.py-qqueries/1403.3290-img-5-14.jpg-r10-dSIFT-dVvisualDictionary/visualDictionary64SIFT.pickle-iballTreeIndexes/index_SIFT_W64.picklepythonquery.py-qqueries/0801.2442-img-2-21.jpg-r3-dSIFT-dVvisualDictionary/visualDictionary256SIFT.pickle-iballTreeIndexes/index_SIFT_W256.pickle

SURF

pythonquery.py-qqueries/1409.1047-img-3-06.jpg-r7-dSURF-dVvisualDictionary/visualDictionary256SURF.pickle-iballTreeIndexes/index_SURF_W256.picklepythonquery.py-qqueries/0903.1780-img-1-32.jpg-r7-dSURF-dVvisualDictionary/visualDictionary256SURF.pickle-iballTreeIndexes/index_SURF_W256.picklepythonquery.py-qqueries/1409.1047-img-3-06.jpg-r7-dSURF-dVvisualDictionary/visualDictionary16SURF.pickle-iballTreeIndexes/index_SURF_W16.pickle

ORB

pythonquery.py-qqueries/0706.3046-img-1-22.jpg-r7-dORB-dVvisualDictionary/visualDictionary16ORB.pickle-iballTreeIndexes/index_ORB_W16.picklepythonquery.py-qqueries/1506.05863-img-3-21.jpg-r7-dORB-dVvisualDictionary/visualDictionary16ORB.pickle-iballTreeIndexes/index_ORB_W16.pickle

Computing VLAD features for a new dataset

Example VLAD with ORB descriptors with a visual dictionary with 2 visual words and an a ball tree as index. (Of course, 2 visual words is not useful, instead, try 16, 32, 64, or 256 visual words)

Remark: Create folders: /ballTreeIndexes, /descriptors, /visualDictionary, /VLADdescriptors

1. compute descriptors from a dataset. The supported descriptors are ORB, SIFT and SURF:

   pythondescribe.py--datasetdataset--descriptordescriptorName--outputoutput

   *Example

   pythondescribe.py--datasetdataset--descriptorORB--outputdescriptors/descriptorORB

2. Construct a visual dictionary from the descriptors in path -d, with -w visual words:

   pythonvisualDictionary.py-ddescriptorPath-wnumberOfVisualWords-ooutput

   *Example :

   pythonvisualDictionary.py-ddescriptors/descriptorORB.pickle-w2-ovisualDictionary/visualDictionary2ORB

3. Compute VLAD descriptors from the visual dictionary:

   pythonvladDescriptors.py-ddataset-dVvisualDictionaryPath--descriptordescriptorName-ooutput

   *Example :

   pythonvladDescriptors.py-ddataset-dVvisualDictionary/visualDictionary2ORB.pickle--descriptorORB-oVLADdescriptors/VLAD_ORB_W2

4. Make an index from VLAD descriptors using a ball-tree DS:

   pythonindexBallTree.py-dVLADdescriptorPath-lleafSize-ooutput

   *Example :

   pythonindexBallTree.py-dVLADdescriptors/VLAD_ORB_W2.pickle-l40-oballTreeIndexes/index_ORB_W2

5. Query:

   pythonquery.py--queryimage--descriptordescriptor--indexindexTree--retrieveretrieve

    *Example
   ```python
   python query.py -q queries/0706.3046-img-1-22.jpg -r 11 -d ORB -dV visualDictionary/visualDictionary2ORB.pickle -i ballTreeIndexes/index_ORB_W2.pickle
   ```
   

Motivation

This project is part of the pipeline of DOCRetrieval project

Installation

First install conda , then:

condacreate--nameopenCV-Pythonnumpyscipyscikit-learnmatplotlibpython=3sourceactivateopenCV-Pythoncondainstall-cmenpoopencv3=3.1.0

Contributor

jorge.jorjasso@gmail.com