3D pose estimation

Pose estimation in a motion capture system

3D pose estimation is a process of predicting the transformation of an object from a user-defined reference pose, given an image or a 3D scan. It arises in computer vision or robotics where the pose or transformation of an object can be used for alignment of a computer-aided design models, identification, grasping, or manipulation of the object.

The image data from which the pose of an object is determined can be either a single image, a stereo image pair, or an image sequence where, typically, the camera is moving with a known velocity. The objects which are considered can be rather general, including a living being or body parts, e.g., a head or hands. The methods which are used for determining the pose of an object, however, are usually specific for a class of objects and cannot generally be expected to work well for other types of objects.

From an uncalibrated 2D camera

[edit]

It is possible to estimate the 3D rotation and translation of a 3D object from a single 2D photo, if an approximate 3D model of the object is known and the corresponding points in the 2D image are known. A common technique developed in 1995 for solving this is POSIT,[1] where the 3D pose is estimated directly from the 3D model points and the 2D image points, and corrects the errors iteratively until a good estimate is found from a single image.[2] Most implementations of POSIT only work on non-coplanar points (in other words, it won't work with flat objects or planes).[3]

Another approach is to register a 3D CAD model over the photograph of a known object by optimizing a suitable distance measure with respect to the pose parameters.[4][5] The distance measure is computed between the object in the photograph and the 3D CAD model projection at a given pose. Perspective projection or orthogonal projection is possible depending on the pose representation used. This approach is appropriate for applications where a 3D CAD model of a known object (or object category) is available.

From a calibrated 2D camera

[edit]

Given a 2D image of an object, and the camera that is calibrated with respect to a world coordinate system, it is also possible to find the pose which gives the 3D object in its object coordinate system.[6] This works as follows.

Extracting 3D from 2D

[edit]

Starting with a 2D image, image points are extracted which correspond to corners in an image. The projection rays from the image points are reconstructed from the 2D points so that the 3D points, which must be incident with the reconstructed rays, can be determined.

Pseudocode

[edit]

The algorithm for determining pose estimation is based on the iterative closest point algorithm. The main idea is to determine the correspondences between 2D image features and points on the 3D model curve.

(a) Reconstruct projection rays from the image points (b) Estimate the nearest point of each projection ray to a point on the 3D contour (c) Estimate the pose of the contour with the use of this correspondence set (d) goto (b) 

The above algorithm does not account for images containing an object that is partially occluded. The following algorithm assumes that all contours are rigidly coupled, meaning the pose of one contour defines the pose of another contour.

(a) Reconstruct projection rays from the image points (b) For each projection ray R:      (c) For each 3D contour:           (c1) Estimate the nearest point P1 of ray R to a point on the contour           (c2) if (n == 1) choose P1 as actual P for the point-line correspondence           (c3) else compare P1 with P:                    if dist(P1, R) is smaller than dist(P, R) then                        choose P1 as new P (d) Use (P, R) as correspondence set. (e) Estimate pose with this correspondence set (f) Transform contours, goto (b) 

Estimating pose through comparison

[edit]

Systems exist which use a database of an object at different rotations and translations to compare an input image against to estimate pose. These systems accuracy is limited to situations which are represented in their database of images, however the goal is to recognize a pose, rather than determine it.[7]

Software

[edit]
  • posest, a GPL C/C++ library for 6DoF pose estimation from 3D-2D correspondences.
  • diffgeom2pose, fast Matlab solver for 6DoF pose estimation from only two 3D-2D correspondences of points with directions (vectors), or points at curves (point-tangents). The points can be SIFT attributed with feature directions.
  • MINUS: C++ package for (relative) pose estimation of three views. Includes cases of three corresponding points with lines at these points (as in feature positions and orientations, or curve points with tangents), and also for three corresponding points and one line correspondence.

See also

[edit]

References

[edit]
  1. ^ Javier Barandiaran (28 December 2017). "POSIT tutorial". OpenCV.
  2. ^ Daniel F. Dementhon; Larry S. Davis (1995). "Model-based object pose in 25 lines of code". International Journal of Computer Vision. 15 (1–2): 123–141. doi:10.1007/BF01450852. S2CID 14501637. Retrieved 2010-05-29.
  3. ^ Javier Barandiaran. "POSIT tutorial with OpenCV and OpenGL". Archived from the original on 20 June 2010. Retrieved 29 May 2010.
  4. ^ Srimal Jayawardena and Marcus Hutter and Nathan Brewer (2011). "A Novel Illumination-Invariant Loss for Monocular 3D Pose Estimation". 2011 International Conference on Digital Image Computing: Techniques and Applications. pp. 37–44. CiteSeerX 10.1.1.766.3931. doi:10.1109/DICTA.2011.15. ISBN 978-1-4577-2006-2. S2CID 17296505.
  5. ^ Srimal Jayawardena and Di Yang and Marcus Hutter (2011). "3D Model Assisted Image Segmentation". 2011 International Conference on Digital Image Computing: Techniques and Applications. pp. 51–58. CiteSeerX 10.1.1.751.8774. doi:10.1109/DICTA.2011.17. ISBN 978-1-4577-2006-2. S2CID 1665253.
  6. ^ Bodo Rosenhahn. "Foundations about 2D-3D Pose Estimation". CV Online. Retrieved 2008-06-09.
  7. ^ Vassilis Athitsos; Stan Sclarof (April 1, 2003). Estimating 3D Hand Pose from a Cluttered Image (PDF) (Technical report). Boston University Computer Science Tech. Archived from the original (PDF) on 2019-07-31.

Bibliography

[edit]
[edit]