3D深層学習

本資料は2019年9月12日に社内共有資料として展開していたものをWEBページ向けにリニューアルした内容になります。

■Purpose

Purpose of this material

• Overview of 3D deep learning
• Comparison b/w each method of 3D deep learning
• Main papers (In this material, I have summarized the material based on following materials and cited papers therein.)
  • E. Ahmed et al, “A survey on Deep Learning Advances on Different 3D Data Representations”, 2018
  • M. M. Bronstein et al., “Geometric deep learning: going beyond Euclidean data”, 2016

■Application

Application of 3D Deep Learning

■Agenda

Methods of 3D Deep Learning

• Euclidean vs Non-Euclidean
• Euclidean Method
  • Projections / Multi-View
  • Voxel
• Non-Euclidean Method
  • Point Cloud / Mesh / Graph
• Accuracy
• Dataset / Material
• Appendix
  • Mesh Generation
  • Laplacian on Graph
  • Correspondence

■3D Data

■Representation

Representation of 3D data

■Euclidean vs Non-Euclidean

Euclidean

Euclidean (detail of feature)

Non-Euclidean

Non-Euclidean (detail of feature)

■Euclidean

Representation of 3D data

Each Euclidean Method (Projections / RGB-D / Volumetric / Multi-View)

Deep Pano [1]

• Projection to Panoramic image
• Row-wise max-pooling for rotational invariant

Two-stream CNNs on RGB-D [1]

• Concatenate CNN of RGB and CNN of depth map

VoxNet [1]

• Voxelization of 3D point cloud to voxel
• Not robust for data loss

MVCNN [1]

• Merge CNN of each images

■Non-Euclidean (Point Clouds)

Representation of 3D data

Each Non-Euclidean Method (Point Cloud)

PointNet [1]

PointNet

T-Net [1]

• Similar to Spatial Transformer Networks in 2D
• Spatial Transformer Networks
  • Alignment of image (transformation, rotation, distortion etc.) by spatial transformation
  • Learn affine transformation from input data (not necessarily special data)
  • Can insert this networks at each point b/w networks

T-Net

• 3D ver. of Spatial Transformer Networks in 2D
• Not need sampling grid (There are no gird structure in 3D)
  • Directly apply transformation to each point cloud
• Output parameter
  • 3 × 3 in first T-Net
  • 64 × 64 in second T-Net

PointNet++ [1]

• Comparison b/w PointNet
• Detailed information is kept
• Can treat different density of point cloud

Set abstraction

• Grouping in one scale + feature extraction
  • Sampling Layer : Extraction of sampling points by farthest point sampling (FPS)
  • Grouping Layer : Grouping points around sampling points
  • PointNet Layer : Applying PointNe

• Point Feature Propagation for segmentation
  • Interpolation : interpolation from k neighbor points
  • Concatenation

• Single scale grouping
• Multi scale/resolution grouping
  • Combination of features from different scales
  • Combination of features from different scales
  • Modifying architecture in set abstraction level

• Detail of architecture
• Note: #vertex is fixed

Architecture for classification and part segmentation of ModelNet using single scale grouping

• Detail of architecture
• Note: #vertex is fixed

Architecture classification of ModelNet using multi-resolution grouping (MRG)

Dynamic Graph CNN (DGCNN) [1]

• PointNet + w/ Edge Conv.
• Edge Conv.
  • Create local edge structure dynamically (not fixed in each layer)

PointCNN [1]

Downsampling information from neighborhoods into fewer representative points

■Non-Euclidean (Mesh)

Representation of 3D data

Each Non-Euclidean Method (Mesh)

MeshCNN [1]

• Edge collapse by pooling
• Can apply only the manifold mesh

MeshNet

• Input feature
  • Center, corner, normal, neighbor inde

■Non-Euclidean (Graph)

Each Non-Euclidean Method (Graph)

Spectral / Spatial Method

Each Non-Euclidean Method (Graph)

Spatial method is more useful than spectral method.

※Some equations from following pages are referred to the documents in PyTorch-geometric.
(https://pytorch-geometric.readthedocs.io/en/latest/modules/nn.html)
I will explain PyTorch-geometric in later page.

Each Non-Euclidean Method (Graph)

Spectral, Spectral free

Spectral CNN [2]

• cannot use different shape
  • Spectral filter coefficients is base dependent
• High computational cost
• No locality

Chebyshev Spectral CNN (ChebNet) [1]

• Not calculate Laplacian eigenvectors directly
• Locality (K hops)
• Approximate filter as polynomial

Graph Convolutional Network (GCN) [2]

• Special ver. of ChebNet (𝐾 = 2)

Each Non-Euclidean Method (Graph)

• Charting

Geodesic CNN (GCNN)

• Create local coordinate
• Do not verify the meaningful chart (need to create small radius chart)

Anisotropic CNN (ACNN)

• Fourier basis is based on anisotropic heat diffusion eq.

MoNet

• Learn filter as parametric kernel
• Generalization of geodesic CNN and anisotropic CNN

SplineCNN [1]

• Filter based on B-spline fun

FeaStNet [1]

• Dynamically determine relation b/w filter weight and local graph of a node

PyTorch-geometric

• https://github.com/rusty1s/pytorch_geometric
• Library based on PyTorchLibrary based on PyTorch
• For point cloud, mesh (not only graph)
• Include Point cloud, graph-type approach code
  • PointNet++, DGCNN, PointCNN
  • ChebNet, GCN, MoNet, SplineCNN, FeaStNet
• Easy to get the famous sample data and transform same data format
  • ModelNet, ShapeNet, etc.
• Many example and benchmark

■Accuracy

Accuracy (Classification)

• around 90% in any method (except VoxNet)

Accuracy (Segmentation)

■Dataset

3D Dataset

ScanNet：registration required
S3DIS : registration required (for original)
FAUST(Note) : registration required

■Material

Material of 3D deep learning (3D / point cloud)

Material of 3D deep learning (graph)

■Summary

There are many methods of 3D deep learning.

Two main method

• Euclidean vs Non-Euclidean
• Euclidean Method
  • Projections / Multi-View / Voxel
• Non-Euclidean Method
  • Point Cloud / Mesh / Graph

Each method have merit and demerit.

• We need to choose the better method for each data type and application.

The research about 3D deep learning is growing.

■Appendix

Appendix

• Mesh Generation
• Laplacian on Graph
• Correspondence

■Appendix : Mesh Generation

Mesh Generation

• In this material, I have summarized these materials.

Difficulty of Mesh Generation

Kinds of Mesh Generation

Classification of the method

In general, it is easier to use the implicit method, since there are noise of point cloud.

Kinds of Mesh Generation (Detail)

Direct Triangulation (example of built-in function in MeshLab)

Voronoi-Based Surface Reconstruction

Voronoi diagram

• Region divided by the bisector of each vertices (in 2D)

Delaunay triangulation

• riangulation by connection of vertices

Ball-Pivoting Algorithm

Kinds of Mesh Generation (Detail)

Surface Smoothness (example of built-in function in MeshLab)

Signed distance function + Marching Cubes

Screened Poisson surface reconstruction

get Indicator Function by solving the Poisson eq.

■Appendix : Laplacian on Graph

Laplacian on Graph [1]

Convolution

■Appendix :Correspondence

Correspondence [1]

■ダウンロード

3D深層学習.pdf