This directory contains scripts needed to run PixSfM experiments for Sk3D or custom data.

You can read a report about results here.

You can run build the image, which is located in docker/Dockerfile, directly via

docker build -t sk3d/pixsfm_experiments:latest -f Dockerfile .

To run PixSfM experiments using some sample data from Sk3D or custom dataset, you can

• run script scripts/copy_dataset.sh (we can edit locations in this file, such as $home_path)
• copy custom dataset (just images) to pixel-perfect-sfm/datasets/{scene_name}/mapping
• copy ready reconstruction (COLMAP outputs) in pixel-perfect-sfm/outputs/{scene_name}/exp_N for required experiment

Run the PixSfM docker image

docker run \
    --gpus all --rm -it  \
    -v /home/user_name/workspace:/workspace \
    --name sk_pixsfm sk3d/pixsfm_experiments:latest /bin/bash

Then, run experiments in pixsfm-experiments/experiments/ to obtain sparse SfM results.

• If you want to run reconstruction using some images, you should use the following format of dataset

pixel-perfect-sfm   
└── datasets
    └── {scene_name}    
       └── mapping
           ├── 0000.jpg       
           ├── 0001.jpg       
           └── ...  
       └── query
           ├── 0000.jpg       
           ├── 0001.jpg       
           └── ...       

• If you have ready reconstruction (COLMAP output), you should use the following format of data

pixel-perfect-sfm   
└── outputs
    └── {scene_name}                 
       └── exp        
           ├── images.txt
           ├── points3D.txt     
           └── cameras.txt 

Here is a rough explanation about what goes in experimnets folder:

• experiments/check_projected_texture.ipynb: helps to project texture on the mesh with the help of nvdiffrast
• experiments/icp_coordinates.ipynb: finds transition matrix from one coordinate system to another using the centers of cameras from images.txt files, saves updated camera poses to a new file.
• experiments/Query_Localizer.ipynb: runs query localization and refinement for query images from pixel-perfect-sfm/datasets/{scene_name}/query by building SfM using images from pixel-perfect-sfm/datasets/{scene_name}/mapping
• experiments/Query_Localizer_known_camera_poses_sfm.ipynb*: runs query localization and refinement for query images by building SfM from known camera poses.
• experiments/Query_Localizer_optimum.ipynb: runs query localization and refinement for query images by building SfM and updating it with previously found camera poses of query images.
• experiments/raytracing_example.ipynb: fifth experiment, where we tried to use the information about tentative matching graphs. Using refined 2D key points after performing FMKA to build a graph of matches. Building equivalent graph, but with 3D points obtained from ray casting on surface of our scan (finding 2D-3D correspondances). Removing 3D points outliers within same track with the help of algorithm for finding smallest bounding sphere. Performing FMBA (not completed).
• experiments/raytracing_example_exhaustive.ipynb: fifth experiment, where we tried to use the information about tentative matching graphs. Here estimation_and_geometric_verification is added for experiment in order to get rid of false matches. Information about matches was taken from two_view_geometries table. Pairs from exhaustive.
• experiments/raytracing_example_retrieval.ipynb: fifth experiment, where we tried to use the information about tentative matching graphs. Pairs from retrieval. (not completed)
• experiments/run_1st_experiment.ipynb: default pipeline with different configurations to obtain SfM sparse model from given images.
• experiments/run_2nd_experiment.ipynb: reconstruct sparse model from known camera poses using hloc and PixSfM.
• experiments/run_3rd_experiment.ipynb: substitute 3d points of given SfM reconstruction by another GT reconstruction, run feature-metric BA (Bundle Adjustment)

• Query Localizer with known refined camera poses (reference SfM was created from the first 3 known refined camera poses, we used GT parameters for camera intrinsics to build SfM and perform query localization)

Query Localizer w/refined images: Dragon

Query Localizer w/refined images: Ceramic girl in hat

Query Localizer w/refined images: White piggybank

Query Localizer w/refined images: Amber vase

• Query Localizer with known refined camera poses (optimum) (reference SfM was created from known camera poses, and iteratively added with a new camera pose after query localization step to find the next query’s camera pose. This experiment takes too much time, approximately 8 hours for 97 images!)

Query Localizer w/refined images (opt): Dragon

Query Localizer w/refined images (opt): Ceramic girl in hat

Query Localizer w/refined images (opt): White piggybank

• Comparison between previous two experiments

Query Localizer comparison: Dragon

Query Localizer comparison: Ceramic girl in hat

Query Localizer comparison: White piggybank

Fifth experiment is not completed:

• use retrieval matching and pairs from covisibility instead of exhaustive matching
• check distribution of distances between all matches in all tracks
• perform geometric verification with specific threshold without using COLMAP geometric verification
• after all experiments above make a conclusion of how far 3d points from each other on the scan