The SIFT descriptor basically characterizes the local edge distribution around keypoints. Extension to color sounds straightforward: just consider color gradients, rather than intensity gradients, and put this into the Gaussian derivative framework. However, one can make many choices to define color gradients, the "physics based" approach being most promising. In my earlier work , I addressed the topic of color gradients. The SIFT implementation provided here is an adaptation of David Lowe's original code to include color keypoints and color description, according to the physics-based approach to color vision, all neatly put into the Gaussian scale-space paradigm.
In this color SIFT software, I have chosen to provide the few useful choices (among many alternatives), which can be selected by providing the option "-colordesc", to select an appropriate color descriptor. In any of the options, the descriptor will contain 3 vectors of 128 values, the first vector being exactly the original intensity based SIFT descriptor. The other two vectors are color based.
The default color SIFT descriptor is a chromatic descriptor "invC", being one of the color SIFT descriptors evaluated in the Burghouts and Geusebroek paper above . In this case, the first vector of 128 bytes in the descriptor carries all intensity related information, whereas the second and third vector contain the orthogonal chromatic information. Hence, intensity, shadow, and shading effects are present in the intensity vector, whereas the pure chromatic information is carried by the additional color vectors. This color descriptor is proven to work very effective under many circumstances, see the Burghouts and Geusebroek performance evaluation . See also our performance in the TRECVID 2008 video retrieval benchmark, and our winning position (UvA runs = color SIFT + various learning techniques) in the PASCAL-VOC 2008 object categorization benchmark. Among other issues , , the "InvC" color SIFT descriptor was an important asset to achieve these results. See the excellent evaluation in Koen van de Sande's PAMI paper .
Alternatively, one can choose for a normalized color descriptor "invW", a "raw" opponent color descriptor, and a computationally more intensive "hsv" descriptor.
Download Windows 32-bit executables and sample program
Download Linux (Intel 32-bit) executables and sample program
Download Linux (Intel 64-bit) executables and sample program
Download Mac (Intel 32-bit) executables and sample program
Below an adapted version of Krystian Mikolajczyk's "compute_descriptors" program (version 2006, see the Oxford vgg website) with the same color descriptors included. Due to implementation details, there may be slight differences between the output of this program with respect to the above code.
compute_descriptors Linux (Intel 32-bit) executable
Acknowledgements: Many thanks to David Lowe for kindly providing permission to redistribute the code adapted for color keypoints, and for the useful discussions on the topic. Many thanks to Krystian Mikolajczyk for providing permission to redistribute the "compute_descriptors" code adapted to include colour SIFT.