FastPoints2Mesh

Data-Driven Inference of Mesh-based Representations for Deformable Objects from Unstructured Point Clouds

Started
June 1, 2023
Status
In Progress
Share this project

Abstract

Combining robotic manipulation with computer vision holds great promise for automating daily tasks of ever-increasing complexity. However, versatile manipulation of complex or deformable objects is still beyond current robotic capabilities, primarily because fast and robust shape estimation for soft materials remains a challenging and largely open problem. We propose to tackle this challenge with a physics-informed mesh representation, building on recent advances in machine learning to achieve real-time point cloud to mesh reconstruction. Recent methods make it efficient and effective to learn from point cloud representation and show robustness to input perturbation and corruption. We aim to reconstruct in real-time the mesh of a deforming object by bringing suitable physical properties such as deformability, inertia, and density conservation into the loss function of the point cloud to mesh reconstruction neural network. We will procure a large-scale simulated dataset and enhance it with real-world data for training neural networks. The simulated dataset will be based on datasets of CAD object models (ShapeNet, PartNet, etc.) that are available online. The resulting pre-trained network is expected to greatly improve both the surface reconstruction accuracy and the frame rate of real-time mesh reconstruction. Such a quick and accurate meshing solution will be an important foundation for computer vision and robotics. The tracking of complex objects in unstructured environments is crucial for dexterous robotic manipulation and real-world interactions, which will strongly impact many application domains, including medicine, logistics, food processing, and agriculture.

People

Collaborators

SDSC Team:
Firat Ozdemir
Alessandro Maissen
Luis Salamanca
Mathieu Salzmann

PI | Partners:

ETH Zurich, Soft Robotics Lab:

  • Prof. Robert Katzschmann
  • Hehui Zheng
  • Chenyu Yang

More info

ETH Zurich, Computational Robotics Lab:

  • Prof. Stelian Coros
  • Miguel Angel Zamora Mora

More info

description

Motivation

Recent advancements in computer vision could allow for robotic manipulation of complex objects (see Fig. 1 for some robotic manipulators). Along this line, inferring functional and physical properties of previously unseen objects from visual data play a key role to assess effective strategies for manipulation.

Proposed Approach / Solution

The project explores fast point cloud to mesh reconstruction of both soft robot arms (manipulators) and objects of interest (objects in the scene). A parallel effort is for generating a large-scale dataset with complex objects being manipulated. This dataset includes two types of data. One is the real-world collected data with both daily objects such as plush toys shown in Figure 2, and 3D printed deformable objects for easy reproduction [1]. The other is the simulated data based on large-scale 3D model datasets (ShapeNet, PartNet, etc.) that are publicly available online. This would allow for optimizing data-driven models that can infer functional and/or physical properties based on vision alone. We are also exploring approaches for simulating capture systems in an effort to bridge the gap between simulated and real-world data.

Impact

Fast mesh reconstruction is an important foundation for tracking and interacting with complex objects in unstructured scenes. Its impact would surpass the robotics field alone, with applications in medicine, logistics, food processing and agriculture. The generated dataset can promote future novel work from the community and allow for benchmarking findings.

Figure 1: A glimpse into robotic manipulators developed by the collaborators at SRL and CRL of ETH Zurich.
Figure 2: Real-world collected plush toy data. The deformations are captured with a professional volumetric capture system that allows for complete 360-degree reconstruction of the object. The deformable objects are poked by a Husky dual-arm robot.

Gallery

Annexe

Additional resources

Bibliography

  1. Obrist, Jan, et al. "PokeFlex: A Real-World Dataset of Volumetric Deformable Objects for Robotics." arXiv preprint arXiv:2410.07688 (2025).

Publications

Related Pages

More projects

LUCID National Data Stream

In Progress
Low Value of Care in Medical Hospitalized Patients - a National Data Stream on Quality of Care in Swiss University Hospitals
Health & Biomedical

Syngenta: Steam consumption optimization

Completed
Reliable strategies to save energy in Syngenta’s Kaisten plant
Energy & Sustainability
Private sector

Pilot project ENERBAT

Completed
Data-Driven Pathways to Net Zero for the Canton of Vaud’s Building Portfolio
Energy & Sustainability
Climate & Environment
Public sector

EKZ: Synthetic Load Profile Generation

Completed
Reliable electricity load monitoring for non-metered nodes
Energy & Sustainability
Public sector

News

Latest news

Coding the Future: Energy Data Hackdays Expand to French-speaking Switzerland
May 7, 2026

Coding the Future: Energy Data Hackdays Expand to French-speaking Switzerland

Coding the Future: Energy Data Hackdays Expand to French-speaking Switzerland

Held at the SDSC headquarters at Biopôle, the Energy Data Hackdays gather 100 experts to tackle 5 energy and grid challenges.
Science des données : le SDSC et le Canton de Vaud soutiennent quatre projets appliqués
April 30, 2026

Science des données : le SDSC et le Canton de Vaud soutiennent quatre projets appliqués

Science des données : le SDSC et le Canton de Vaud soutiennent quatre projets appliqués

Le SDSC et le Canton de Vaud ont retenu quatre projets parmi les 57 soumissions reçues lors de leur deuxième appel à projets.
Le Swiss Data Science Center inaugure son siège au Biopôle de Lausanne
March 12, 2026

Le Swiss Data Science Center inaugure son siège au Biopôle de Lausanne

Le Swiss Data Science Center inaugure son siège au Biopôle de Lausanne

Le SDSC inaugure aujourd'hui son siège au campus Biopôle de Lausanne, dans le cadre d'un partenariat stratégique avec l'État de Vaud.

Contact us

Let’s talk Data Science

Do you need our services or expertise?
Contact us for your next Data Science project!