Some simple 3D model obtained using the work in progress Toolbox using images taken in my street.

As part of the reading group (Multiple View Geometry in Computer Vision, Hartley and Zisserman) I am following at the UvA every Friday, I have started a small project to implement all the relevant algorithms I encounter along the discussion of every chapter.

The objective is twofold. On one hand I will practice all the concepts that are discussed in every chapter in a “hands on” fashion. On the other hand, I will prepare the set of functions that I will need when I have to estimate egomotion using the LadyBug camera as part of my PhD research.

All the algorithms and functions are implemented following Zisserman’s book and details on each of them are available in the help. This is the list of currently implemented functions:

CHAPTER 04

• Computing the homography
• dlt2D.m - Implementation of the normalized Direct Linear Transform algorithm (Zisserman page 109)
• goldStd2DAffine.m - Implementation of the Gold Standard Algorithm for an affine homography estimation (Zisserman page 130)
• goldStd2D.m - Implementation of the Gold Standard Algorithm for an homography estimation (Zisserman page 114)
• ransac2d.m - Implementation of the RANSAC Algorithm for an homography estimation (Zisserman page 123). After determining the inliers, the homography is computed using the Golden Standard Algorithm (Levenberg-Marquardt)
• Related functions
• normalize2DPoints.m - Normalization of 2D image points based on the centroid and sqrt(2) distance
• normalize2DPointsCentroid.m - Normalization of 2D image points based on the centroid
• matchSIFT.m - Given the set of SIFT descriptors of 2 images, it determines the ones that match according to a distance ratio threshold
• symetricTransferError.m - Computation of the squared (not summed) symetric transfer error (Zisserman page 112) used in the Levenberg-Marquardt algorithm (lqsnonlin function in Matlab)
• stitchImages.m - Given two image files (RGB support added) it stitches them together for form a panoramic image
• normPanoramic.m - As the stitching procedure distorts the image, I included this simple function to normalize the image to a rectangle.

CHAPTER 07

• Camera Calibration
• getDistortionError.m - Given a set of collinear points in image coordinates, the radial distortion parameters (k1-k4) and the center of distortion, computes distance of each point to a line fitted to the points. Its used for estimation of radial distortion by minimization of the squared sum of the distances.
• getRadialDistortion.m - Given a set of collinear points in image coordinates, it computes the parameters of the radial distortion function L(r) up to 4th order by minimizing the previous error measure.
• getRectifiedPoints.m - Given a set of points in image coordinates and the radial distortion parameters (Zisserman page 191), it computes the rectified point coordinates.
• Related functions
• getRectifiedImage.m - Given an image and the radial distortion parameters it computes the rectified image.
• obtainCameraRadialDistortion.m - Given an image, an (optional) initial estimation of the radial distortion parameters and the number of desired points to select in a line (the user can select those points) it computes the radial distortion parameters.
• getDistancePointLine.m - Computes the distance from a point (or set of points) to the given line.
• correct_radial_distortion.m - Given an image and the radial distortion parameters it computes the rectified image. This function is different than the above in that it returns an image of the same size as the original where ALL the pixels contain information (as opposed to the portion in black from the other images). This is useful if rectangular images are needed. This function was kindly contributed by Richard den Hollander.

CHAPTER 11

• Epipolar Geometry Estimation
• ransacF.m - This function performs ransac for the estimation of the fundamental matrix F. The error used is the Sampson distance.
• getCameraMatrixHorn.m - Based on the paper of Horn, and given the essential matrix, the 4 possible solutions for rotation-translation are computed.
• getCameraMatrix.m - The same 4 rotation-translation solutions are computed but based on Zissermans method.
• getCorrectCameraMatrix.m - Given 4 possibvle solutions for a camera matrix (rotation-translation), the correct one is computed based on the fact the distance to the point must be positive.
• eightpoint.m - The eight point algorithm as described by Zisserman.
• getEssentialMatrix.m - A very silly multiplicacion to obtain the essential matrix given the fundamental matrix and the camera calibration matrices.
• Wrap Around functions
• getMultiImageVisualOdometry.m - Given a directory, a camera calibration matrix and a method, the visual odometry is computed.
• getVisualOdometry.m - The same but for just 2 images.

All the functions are available to download in a tar.gz file Ziss-Toolbox-r.0.2.tar.gz

Note that the current calibration code only accounts for radial distortion and not tangential distortion. A more interesting approach will be using more than one set of collinear points to estimate the parameters to better capture the radial distortion.

More functions will be available as the reading group continues.

A couple of days ago Gijs Dubbelman and I organized (after plenty of postponing) the recording of a new dataset (problably one of the largest to our knowledge in SLAM). We loaded up the Jeep with 3 different cameras (LadyBug2, hyperbolic mirror camera and a stereo system). Additionally, we recorded dGPS data, the measurements of a inertial measurement unit (IMU) and odometry of the jeep.

We recorded two large datasets and a couple of smaller ones for calibration of the odometry and error evaluation in egomotion estimation.

The first dataset was taken in Rotterdam, following a trajectory of about 12 km. Unfortunately the weather was not on our side and we had to stop due to the rain. Also, one of the computers run out of disk space and the hyperbolic mirror camera stopped recording images before closing the complete loop.

The second dataset was recorded in The Hague following a trajectory of about 14 km. This time we had more luck and the weather was beautiful. Everything worked out really nice and now we have plenty of data to process.

Below I attach the trajectory plotted over a google map.

Some example of the pictures recorded with the hyperbolic camera.

I cant wait to see the results of the SLAM with the hyperbolic mirror!!

An essential aspect in Computer Vision is camera calibration. For tasks such as 3D reconstruction, egomotion estimation, etc it’s important to have an accurate camera calibration to obtain decent results.

After some interesting sessions in our Zisserman reading group I started playing around with the Camera Calibration Toolbox from Caltech. With an intuitive GUI it is relatively easy to accurately estimate you camera parameters given a set of pictures of some sort of calibration pattern of known geometry. It is however a big hassle if you need to calibrate more than one camera (6 in my case) and you can only use with images of a known pattern.

I have written a set of functions that allow you to estimate the radial distortion from regular images (no need for a known pattern). The only constraint is that the image needs to contain a straight line (the closer to the borders the better). This is true in many cases such as surveillance cameras where almost certainly a piece of human built structure can be seen. The estimation process is relatively simple, given an image, and desired number of points, the user is asked to select points in the image that are known to be collinear. An iterative process takes care of minimizing the distance of such points after transformation to the best line fit.

For improvement in the estimation, the user can select a set of lines, in this case the optimization is performed over all the lines.

The images above show the original image and the corrected image after selecting a single line (the top of the roof of the houses) of 10 points. The estimation procedure works differently depending on the scene in the image. In this case, a single line was sufficient, for other images, 2 lines of 3 points would suffice.

This estimation procedure does not account for tangential distortion (usually small) but it will be interesting to see if it provides any improvement.

As soon as the functions are finished they will be available as part of CHAPTER-07 implementation of my Zisserman Toolbox.