Color SIFT
Published October 30th, 2008
See:
G. J. Burghouts and J. M. Geusebroek.
Performance evaluation of local colour invariants.
Comput. Vision Image Understanding, 113:48-62, 2009.
The SIFT descriptor basically characterizes the local edge distribution
around keypoints. Extension to color sounds straightforward: just consider
color gradients, rather than intensity gradients, and put this into the
Gaussian derivative framework. However, one can make many choices to define
color gradients, the "physics based" approach being most promising. In my
earlier work
,
I addressed the topic of color gradients.
The SIFT implementation provided here is an adaptation of
David Lowe's original
code to include color
keypoints and color description, according to the physics-based approach
to color vision, all neatly put into the Gaussian scale-space paradigm.
In this color SIFT software, I have chosen to provide the few useful choices (among many alternatives), which can be selected by providing the option "-colordesc", to select an appropriate color descriptor. In any of the options, the descriptor will contain 3 vectors of 128 values, the first vector being exactly the original intensity based SIFT descriptor. The other two vectors are color based.
The default color SIFT descriptor is a chromatic descriptor "invC",
being one of the color SIFT descriptors
evaluated in the Burghouts and Geusebroek paper above
.
In this case, the first vector of 128 bytes in the
descriptor carries all intensity related information, whereas the second and
third vector contain the orthogonal chromatic information. Hence, intensity,
shadow, and shading effects are present in the intensity vector, whereas the
pure chromatic information is carried by the additional color vectors.
This color descriptor is proven to work very effective under many
circumstances, see the Burghouts and Geusebroek
performance evaluation
.
See also our
performance in the
TRECVID 2008 video retrieval benchmark, and our
winning position (UvA runs = color SIFT + various learning techniques)
in the
PASCAL-VOC 2008 object categorization benchmark.
Among other issues
,
,
the "InvC" color SIFT descriptor was an important asset to
achieve these results. See the excellent evaluation in Koen van de Sande's PAMI paper
.
Alternatively, one can choose for a normalized color descriptor "invW", a "raw" opponent color descriptor, and a computationally more intensive "hsv" descriptor.
Download Windows 32-bit executables and sample program
Download Linux (Intel 32-bit) executables and sample program
Download Linux (Intel 64-bit) executables and sample program
Download Mac (Intel 32-bit) executables and sample program
Below an adapted version of Krystian Mikolajczyk's "compute_descriptors" program (version 2006, see the Oxford vgg website) with the same color descriptors included. Due to implementation details, there may be slight differences between the output of this program with respect to the above code.
compute_descriptors Linux (Intel 32-bit) executable
For comparison with the results provided in the paper
,
you can download a text file containing the keypoints
used in the experiments:
Download keypoint locations and affine parameters for
ALOI.
(in Mikolajczyk's affine format)
Acknowledgements: Many thanks to David Lowe for kindly providing permission to redistribute the code adapted for color keypoints, and for the useful discussions on the topic. Many thanks to Krystian Mikolajczyk for providing permission to redistribute the "compute_descriptors" code adapted to include colour SIFT.
