Topology-aware non-rigid point cloud registration

Topology-aware registration overview In this project, we develop a non-rigid registration pipeline for pairs of unorganized point clouds that may be topologically different. Standard warp field estimation algorithms, even under robust, discontinuity-preserving regularization, tend to produce erratic motion estimates on boundaries associated with 'close-to-open' topology changes. We overcome this limitation by exploiting backward motion: in the opposite motion direction, a 'close-to-open' event becomes 'open-to-close', which is by default handled correctly. At the core of our approach lies a general, topology-agnostic warp field estimation algorithm, similar to those employed in recently introduced dynamic reconstruction systems from RGB-D input. We improve motion estimation on boundaries associated with topology changes in an efficient post-processing phase. Based on both forward and (inverted) backward warp hypotheses, we explicitly detect regions of the deformed geometry that undergo topological changes by means of local deformation criteria and broadly classify them as 'contacts' or 'separations'. Subsequently, the two motion hypotheses are seamlessly blended on a local basis, according to the type and proximity of detected events. Our method achieves state-of-the-art motion estimation accuracy on the MPI Sintel dataset. Experiments on a custom dataset with topological event annotations demonstrate the effectiveness of our pipeline in estimating motion on event boundaries, as well as promising performance in explicit topological event detection.


Topology-Aware Non-Rigid Point Cloud Registration [pdf] [bib]
Konstantinos Zampogiannis, Cornelia Fermüller, Yiannis Aloimonos, PAMI 2019

Topology-aware registration example

cilantro: a fast and flexible general purpose point cloud processing library

cilantro logo We introduce cilantro, a lean and fast open-source C++ library for working with point cloud data, with emphasis given to the 3D case. It includes efficient implementations for a variety of common operations that cover low-level processing, geometric registration, clustering/segmentation, spatial reasoning, robust model fitting, and powerful visualization utilities. The library is extensively templated, enabling operations on data of arbitrary numerical type and dimensionality (where applicable) and featuring a modular/extensible design of the more complex procedures. At the same time, convenience aliases/wrappers for the most common cases are provided within a clean and consistent API that attempts to minimize the amount of boilerplate code.

Supported functionality includes:

  • Connected component extraction Primitive operations: general dimension kd-trees, surface normal and curvature estimation, general dimension grid-based resampling, PCA, RGB-D image pair to/from 3D point cloud conversions
  • Clustering/segmentation: general dimension k-means clustering with multiple distance metric support, most common spectral clustering variants, mean-shift clustering with custom kernel support, parallel connected component segmentation based on arbitrary point-wise similarity functions
  • Spatial reasoning: a general dimension convex polytope representation computed from either vertex or half-space intersection input that allows for seamless switching between the respective representations, a representation of (general dimension) generic space regions as unions of convex polytopes that implements set operations
    Spatial reasoning
  • Point fusion based reconstruction Geometric registration: multiple generic Iterative Closest Point implementations that support arbitrary correspondence search methods in arbitrary point feature spaces for:
    • Rigid or affine alignment under the point-to-point metric (general dimension), point-to-plane metric (2D or 3D), or any combination thereof
    • Non-rigid alignment of 2D or 3D point sets, by means of a robustly regularized, locally rigid or locally affine deformation field, under any combination of the point-to-point and point-to-plane metrics; implementations for both densely and sparsely (by means of an Embedded Deformation graph) supported warp fields are provided

    Non-rigid point cloud registration
  • Robust model estimation: a RANSAC estimator template and instantiations thereof for robust, general dimension hyperplane estimation and rigid point cloud registration given noisy correspondences
  • Visualization: classical MDS, an extensible and easy to use 3D visualizer

Source code:

The library is released under the MIT license and is available here.


cilantro: A Lean, Versatile, and Efficient Library for Point Cloud Data Processing [pdf] [bib]
Konstantinos Zampogiannis, Cornelia Fermüller, Yiannis Aloimonos, ACM Multimedia 2018 (OSSC)

Box-shaped container/appliance part decomposition and fitting from partial 3D reconstructions

Microwave model fitting We develop a pipeline for fitting approximate models to partial 3D scans of box-shaped container and appliance objects. Our approach relies on a general cuboid detection and fitting algorithm that, guided by a normal clustering preprocessing step, efficiently segments a scene into planar surface patches and appropriately refines and groups the latter into box candidates. By combining box detections from partial reconstructions of the object exterior and interior (obtained in a common coordinate frame) with prior part knowledge (e.g., existence and/or relative location of door and/or door handle), we automatically obtain a decomposition of the object into box-shaped parts. Based on the detected interior box, we obtain a rectified render of the object interior which we use to reconstruct the container's shelf structure (if any) by means of a hybrid 2D/3D approach. Our final output is an approximate object model built with box shape primitives.

Source code:

Our box detection algorithm is implemented as part of a small library available here.


Combining Visual Learning with a Generic Cognitive Model for Appliance Representation [pdf] [bib]
Kanishka Ganguly, Konstantinos Zampogiannis, Cornelia Fermüller, Yiannis Aloimonos, EUCOG 2016

Fridge model fitting

Object-wise spatial relations and manipulation semantics

Spatial relations grounding In this project, we introduce an abstract representation for manipulation actions that is based on the evolution of the spatial relations between involved objects. Object tracking in RGBD streams enables straightforward and intuitive ways to model spatial relations in 3D space. Reasoning in 3D overcomes many of the limitations of similar previous approaches, while providing significant flexibility in the desired level of abstraction. At each frame of a manipulation video, we evaluate a number of spatial predicates for all object pairs and treat the resulting set of sequences (Predicate Vector Sequences, PVS) as an action descriptor. As part of our representation, we introduce a symmetric, time-normalized pairwise distance measure that relies on finding an optimal object correspondence between two actions. We experimentally evaluate the method on the classification of various manipulation actions in video, performed at different speeds and timings and involving different objects. The results demonstrate that the proposed representation is remarkably descriptive of the high-level manipulation semantics.

Source code:

Original implementation with sample data available here.


RGBD dataset of manipulation action videos with sample object (point cloud) tracking results available here.


Learning the Spatial Semantics of Manipulation Actions through Preposition Grounding [pdf] [bib] [slides]
Konstantinos Zampogiannis, Yezhou Yang, Cornelia Fermüller, Yiannis Aloimonos, ICRA 2015

Spatial relations tracking