Skip to content

Commit

Permalink
adding evaluation metrics
Browse files Browse the repository at this point in the history
  • Loading branch information
Panos Achlioptas committed Mar 18, 2018
1 parent 846c628 commit 70177f0
Show file tree
Hide file tree
Showing 6 changed files with 1,183 additions and 24 deletions.
7 changes: 6 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# Learning Representations And Generative Models For 3D Point Clouds
# Learning Representations and Generative Models For 3D Point Clouds
Created by <a href="http://web.stanford.edu/~optas/" target="_blank">Panos Achlioptas</a>, <a href="http://web.stanford.edu/~diamanti/" target="_blank">Olga Diamanti</a>, <a href="http://mitliagkas.github.io" target="_blank">Ioannis Mitliagkas</a>, <a href="http://geometry.stanford.edu/member/guibas/" target="_blank">Leonidas J. Guibas</a>.

![representative](https://github.com/optas/latent_3d_points/blob/master/doc/images/teaser.jpg)
Expand Down Expand Up @@ -64,6 +64,11 @@ To train a point-cloud AE look at:

latent_3d_points/notebooks/train_single_class_ae.ipynb

To use the evaluation metrics (MMD, Coverage, JSD) between two point-cloud sets look at:

latent_3d_points/notebooks/compute_evaluation_metrics.ipynb



## License
This project is licensed under the terms of the MIT license (see LICENSE.md for details).
2 changes: 1 addition & 1 deletion download_data.sh
Original file line number Diff line number Diff line change
Expand Up @@ -5,5 +5,5 @@ wget https://www.dropbox.com/s/vmsdrae6x5xws1v/shape_net_core_uniform_samples_20
mv shape_net_core_uniform_samples_2048.zip\?dl\=0 shape_net_core_uniform_samples_2048.zip
unzip shape_net_core_uniform_samples_2048.zip
rm shape_net_core_uniform_samples_2048.zip
mkdir data
mkdir -p data
mv shape_net_core_uniform_samples_2048 data
242 changes: 242 additions & 0 deletions notebooks/compute_evaluation_metrics.ipynb
Original file line number Diff line number Diff line change
@@ -0,0 +1,242 @@
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Assuming 2 sets of point-clouds, we will compute the MMD, Coverage and JSD as done in the paper.\n",
"\n",
"(To compute these metrics you __don't need__ to have tflearn installed, only the structural: EMD, Chamfer losses and sklearn for the JSD.)"
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"import numpy as np\n",
"import os.path as osp\n",
"\n",
"from latent_3d_points.src.evaluation_metrics import minimum_mathing_distance, \\\n",
"jsd_between_point_cloud_sets, coverage\n",
"\n",
"from latent_3d_points.src.in_out import snc_category_to_synth_id,\\\n",
" load_all_point_clouds_under_folder"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Load some point-clouds and make two sets (sample_pcs, ref_pcs) from them. The ref_pcs is considered as the __ground-truth__ data while the sample_pcs corresponds to a set that is matched against it, e.g. comes from a generative model."
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"%load_ext autoreload\n",
"%autoreload 2\n",
"%matplotlib inline"
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {
"collapsed": false
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Give me the class name (e.g. \"chair\"): chair\n",
"6778 pclouds were loaded. They belong in 1 shape-classes.\n"
]
}
],
"source": [
"top_in_dir = '../data/shape_net_core_uniform_samples_2048/' # Top-dir of where point-clouds are stored.\n",
"class_name = raw_input('Give me the class name (e.g. \"chair\"): ').lower()\n",
"syn_id = snc_category_to_synth_id()[class_name]\n",
"class_dir = osp.join(top_in_dir , syn_id)\n",
"all_pc_data = load_all_point_clouds_under_folder(class_dir, n_threads=8, file_ending='.ply', verbose=True)"
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"n_ref = 100 # size of ref_pcs.\n",
"n_sam = 150 # size of sample_pcs.\n",
"all_ids = np.arange(all_pc_data.num_examples)\n",
"ref_ids = np.random.choice(all_ids, n_ref, replace=False)\n",
"sam_ids = np.random.choice(all_ids, n_sam, replace=False)\n",
"ref_pcs = all_pc_data.point_clouds[ref_ids]\n",
"sample_pcs = all_pc_data.point_clouds[sam_ids]"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Compute the three metrics."
]
},
{
"cell_type": "code",
"execution_count": 11,
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"ae_loss = 'chamfer' # Which distance to use for the matchings.\n",
"\n",
"if ae_loss == 'emd':\n",
" use_EMD = True\n",
"else:\n",
" use_EMD = False # Will use Chamfer instead.\n",
" \n",
"batch_size = 100 # Find appropriate number that fits in GPU.\n",
"normalize = True # Matched distances are divided by the number of \n",
" # points of thepoint-clouds.\n",
"\n",
"mmd, matched_dists = minimum_mathing_distance(sample_pcs, ref_pcs, batch_size, normalize=normalize, use_EMD=use_EMD)\n",
"\n",
"cov, matched_ids = coverage(sample_pcs, ref_pcs, batch_size, normalize=normalize, use_EMD=use_EMD)\n",
"\n",
"jsd = jsd_between_point_cloud_sets(sample_pcs, ref_pcs, resolution=28)"
]
},
{
"cell_type": "code",
"execution_count": 12,
"metadata": {
"collapsed": false
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"0.0714721 0.73 0.0396569736382\n"
]
}
],
"source": [
"print mmd, cov, jsd"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"For a detailed breakdown of the evaluation functions, inspect their docs."
]
},
{
"cell_type": "code",
"execution_count": 24,
"metadata": {
"collapsed": false
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Computes the Coverage between two sets of point-clouds.\n",
"\n",
" Args:\n",
" sample_pcs (numpy array SxKx3): the S point-clouds, each of K points that will be matched\n",
" and compared to a set of \"reference\" point-clouds.\n",
" ref_pcs (numpy array RxKx3): the R point-clouds, each of K points that constitute the\n",
" set of \"reference\" point-clouds.\n",
" batch_size (int): specifies how large will the batches be that the compute will use to\n",
" make the comparisons of the sample-vs-ref point-clouds.\n",
" normalize (boolean): When the matching is based on Chamfer (default behavior), if True,\n",
" the Chamfer is computed as the average of the matched point-wise squared euclidean\n",
" distances. Alternatively, is their sum.\n",
" use_sqrt (boolean): When the matching is based on Chamfer (default behavior), if True,\n",
" the Chamfer is computed based on the (not-squared) euclidean distances of the matched\n",
" point-wise euclidean distances.\n",
" sess (tf.Session): If None, it will make a new Session for this.\n",
" use_EMD (boolean): If true, the matchings are based on the EMD.\n",
" ret_dist (boolean): If true, it will also return the distances between each sample_pcs and\n",
" it's matched ground-truth.\n",
" Returns: the coverage score (int),\n",
" the indices of the ref_pcs that are matched with each sample_pc\n",
" and optionally the matched distances of the samples_pcs.\n",
" \n",
"Computes the MMD between two sets of point-clouds.\n",
"\n",
" Args:\n",
" sample_pcs (numpy array SxKx3): the S point-clouds, each of K points that will be matched and\n",
" compared to a set of \"reference\" point-clouds.\n",
" ref_pcs (numpy array RxKx3): the R point-clouds, each of K points that constitute the set of\n",
" \"reference\" point-clouds.\n",
" batch_size (int): specifies how large will the batches be that the compute will use to make\n",
" the comparisons of the sample-vs-ref point-clouds.\n",
" normalize (boolean): When the matching is based on Chamfer (default behavior), if True, the\n",
" Chamfer is computed as the average of the matched point-wise squared euclidean distances.\n",
" Alternatively, is their sum.\n",
" use_sqrt: (boolean): When the matching is based on Chamfer (default behavior), if True, the\n",
" Chamfer is computed based on the (not-squared) euclidean distances of the matched point-wise\n",
" euclidean distances.\n",
" sess (tf.Session, default None): if None, it will make a new Session for this.\n",
" use_EMD (boolean: If true, the matchings are based on the EMD.\n",
"\n",
" Returns:\n",
" A tuple containing the MMD and all the matched distances of which the MMD is their mean.\n",
" \n",
" JSD between two sets of point-clouds, as introduced in the paper ```Learning Representations And Generative Models For 3D Point Clouds```. \n",
" Args:\n",
" sample_pcs: (np.ndarray S1xR2x3) S1 point-clouds, each of R1 points.\n",
" ref_pcs: (np.ndarray S2xR2x3) S2 point-clouds, each of R2 points.\n",
" resolution: (int) grid-resolution. Affects granularity of measurements.\n",
" \n"
]
}
],
"source": [
"print coverage.__doc__\n",
"print minimum_mathing_distance.__doc__\n",
"print jsd_between_point_cloud_sets.__doc__"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "TensorFlow1",
"language": "python",
"name": "tf1"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 2
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython2",
"version": "2.7.6"
}
},
"nbformat": 4,
"nbformat_minor": 2
}
Loading

0 comments on commit 70177f0

Please sign in to comment.