openMVG source code learning (I) main_SfMInit_ImageListing

This study note will use openmvg as the third-party library, the method of restoring motion structure recommended in the official documents, the meaning of the code and the experience I found in my study. Welcome to discuss!

Code and platform included

ubuntu18.04
The installation of openMVG will not be repeated here. Just refer to the official documents

main_SfMInit_ImageListing.cpp
main_ComputeFeatures.cpp
main_ComputeMatches.cpp
main_IncrementalSfM.cpp
main_GlobalSfM.cpp

main_SfMInit_ImageListing.cpp

Then let's officially start!
First, create a new project and paste the source code.
We need to configure the engineering environment ourselves:

cmakelist.txt

Let's start with cmakelist How to write TXT?

cmake_minimum_required(VERSION 3.0.0)
project(mvgtest1)
set(CMAKE_CXX_STANDARD 17)

add_executable(mvgtest1 img_list.cpp ComputeFeatures.cpp ComputeMatches.cpp Globalsfm.cpp)

#OpenMVG,OpenCV,Ceres

find_package(OpenCV REQUIRED)
find_package(OpenMVG REQUIRED)

find_package(Ceres REQUIRED PATHS "${CERES_PATH}/Thirdparty/ceres-solver")
include_directories(${CERES_INCLUDE_DIRS})
set(LIBS ${Ceres_LIBS_DIR}/libceres.a umfpack cxsparse glog gflags gomp
        ccolamd btf klu cholmod lapack blas camd amd pthread)
include_directories(${OPENMVG_INCLUDE_DIRS})

target_link_libraries(mvgtest1
        PRIVATE
        OpenMVG::openMVG_sfm
        OpenMVG::openMVG_matching
        OpenMVG::openMVG_camera
        OpenMVG::openMVG_exif
        OpenMVG::openMVG_features
        OpenMVG::openMVG_geodesy
        OpenMVG::openMVG_geometry
        OpenMVG::openMVG_graph
        OpenMVG::openMVG_image
        OpenMVG::openMVG_linearProgramming
        OpenMVG::openMVG_matching
        OpenMVG::openMVG_matching_image_collection
        OpenMVG::openMVG_multiview
        OpenMVG::openMVG_numeric
        OpenMVG::openMVG_robust_estimation
        OpenMVG::openMVG_sfm
        OpenMVG::openMVG_system
        ${OpenCV_LIBS}
        )
FIND_PACKAGE( OpenMP REQUIRED)
if(OPENMP_FOUND)
    message("OPENMP FOUND")
    set(CMAKE_C_FLAGS "${CMAKE_C_FLAGS} ${OpenMP_C_FLAGS}")
    set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} ${OpenMP_CXX_FLAGS}")
    set(CMAKE_EXE_LINKER_FLAGS "${CMAKE_EXE_LINKER_FLAGS} ${OpenMP_EXE_LINKER_FLAGS}")
endif()

FIND_PACKAGE( OpenMP REQUIRED)
if(OPENMP_FOUND)
    message("OPENMP FOUND")
    set(CMAKE_C_FLAGS "${CMAKE_C_FLAGS} ${OpenMP_C_FLAGS}")
    set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} ${OpenMP_CXX_FLAGS}")
endif()

But explain more

Preparation before code running

Delete cmd related things. Anyway, there will be an error when running in the clion. Deleting cmd related things will not affect the integrity of the code. After deletion, the parameter passed by the main function is int main()

Then, sort out the structure of the code to make it easier for you to understand the logical arrangement.

Data preparation

Well, let's go into the explanation of the code,

Firstly, the main function of the cpp is to read the picture, obtain the path, and the internal parameters of the camera to generate sfm_data.json file is the preparation file for later feature matching and reconstruction, which should be used in subsequent processes.

The beginning of the mian function defines the folder path of the input and output of the program

std::string sImageDir = "../imgdata";

Picture folder path

std::string sOutputDir = "../output";		

Output file folder

std::string sKmatrix = "fx;0;ppx;0;fy;ppy;0;0;1";	

Camera internal parameter matrix, read in the string, and obtain the f, ppx, ppy data by the checkIntrinsicStringValidity function

std::string sfileDatabase = "";	

Read the data database file from the file and read it into the vector < < datasheet >
Datasheet stores the database structure of camera model and sensor size
There are two members std::string model_; And double sensorSize_;

std::pair<bool, Vec3> prior_w_info(false, Vec3(1.0,1.0,1.0));

Priority of predefined rotation

int i_User_camera_model = PINHOLE_CAMERA_RADIAL3;

Defines the type of camera model

enum EINTRINSIC
{
  PINHOLE_CAMERA_START = 0,
  PINHOLE_CAMERA,         //No distortion
  PINHOLE_CAMERA_RADIAL1, // Radial distortion K1
  PINHOLE_CAMERA_RADIAL3, // Radial distortion K1,K2,K3
  PINHOLE_CAMERA_BROWN, //Radial distortion K1, K2, K3, tangential distortion T1, T2
  PINHOLE_CAMERA_FISHEYE, //A simple fisheye distortion model with four distortion coefficients
  PINHOLE_CAMERA_END,
  CAMERA_SPHERICAL = PINHOLE_CAMERA_END + 1
};

This is the camera model given by the source code

bool b_Group_camera_model = true;

If necessary, cameras with the same characteristics can be combined together (resulting in a faster and more stable BA).

 int i_GPS_XYZ_method = 0;

In the function checkGPS, 1 is UTM, that is (function lla_to_utm):
Convert WGS84 lon, lat and alt data into UTM data (universal horizontal axis Mercator).
0 is wgs84 (function lla_to_ecef)
Convert WGS84 lon, lat and alt data into ECEF data (geocentric fixation).
And the function ecef_to_lla
Convert the ECEF (XYZ) of WGS84 ellipsoid to lon, lat and alt values.

data fetch

So with these data:

Image file
 Output folder
 Camera internal parameter matrix
 Sensor model and size
 Coordinate Datum
 Distortion model

We can do sfm_data.json has been established. The inspection of file data and internal parameter matrix will not be repeated here.

After reading various data:

 double width = -1, height = -1, focal = -1, ppx = -1,  ppy = -1;

The internal parameters defined to represent each picture are checked and read in through the checkIntrinsicStringValidity function

const auto e_User_camera_model = EINTRINSIC(i_User_camera_model);

Defined distortion model

  std::vector<std::string> vec_image = stlplus::folder_files( sImageDir );
  std::sort(vec_image.begin(), vec_image.end());	  

It is defined as the vector of each picture path and sorted by name

std::vector<Datasheet> vec_database;

The sensor type and size are stored and read in using parseDatabase

Read in and generate sfm_data.json

First, let's take a look at SfM_Data class, which is used to define the general SfM data container, storage structure and camera attributes
Include members:

Views

 It's a Hash_Map<IndexT, std::shared_ptr<View>>，storage View type
View A view defines an image through a string and a unique index of the view, camera, and pose
  // image path on disk
  std::string s_Img_path;
  // Id of the view
  IndexT id_view;
  // Index of intrinsics and the pose
  IndexT id_intrinsic, id_pose;
  // image size
  IndexT ui_width, ui_height;

Poses

using Poses = Hash_Map<IndexT, geometry::Pose3>;      //Define pose collection (indexed by View::id_Pose)
Direction matrix and rotation center Mat3 rotation_;   Vec3 center_;			Eigen type

Intrinsics

using Intrinsics = Hash_Map<IndexT, std::shared_ptr<cameras::IntrinsicBase>>;
Is the internal parameter attribute of the camera

Landmarks

Defined by TrackId Indexed landmark collection, Landmark Contains two members,
3d Points and their corresponding coordinates on the image,
Because the coordinates in a world can be observed by multiple cameras.
Landmarks Points are divided into points obtained by triangulation (for BA)And ground control points (for GCP)

std::string s_root_path;
The root directory path of the picture

Next, let's take a look at how this program is arranged for reading
Create an SfM_Data, read the root directory
VEC_ Iterator of image to loop C_Progress_display displays progress on the console

Take a cycle as an example:

Initialize internal parameters

width = height = ppx = ppy = focal = -1.0;

Merge folder with file name (basename.extension)

const std::string sImageFilename = stlplus::create_filespec( sImageDir, *iter_image );

Gets the file name - that is, the file name without the folder part but with the extension

const std::string sImFilenamePart = stlplus::filename_part(sImageFilename);

Detect whether it is an image

if (openMVG::image::GetFormat(sImageFilename.c_str()) == openMVG::image::Unknown)
{
  error_report_stream
      << sImFilenamePart << ": Unkown image file format." << "\n";
  continue; // image cannot be opened
}
//string::npos is a constant used to indicate the non-existent location
//Determine whether it is a mask image
if (sImFilenamePart.find("mask.png") != std::string::npos
   || sImFilenamePart.find("_mask.png") != std::string::npos)
{
  error_report_stream
      << sImFilenamePart << " is a mask image" << "\n";
  continue;
}
//Skip this loop in both cases


ImageHeader imgHeader;
if (!openMVG::image::ReadImageHeader(sImageFilename.c_str(), &imgHeader))
  continue; // image cannot be read

width = imgHeader.width;
height = imgHeader.height;
ppx = width / 2.0;
ppy = height / 2.0;

Consider providing focus manually

if (sKmatrix.size() > 0) // Known user calibration K matrix
{
  if (!checkIntrinsicStringValidity(sKmatrix, focal, ppx, ppy))
    focal = -1.0;
}
else //User supplied focal length value
  if (focal_pixels != -1 )
    focal = focal_pixels;

// If not provided manually or incorrectly
if (focal == -1)
{
  std::unique_ptr<Exif_IO> exifReader(new Exif_IO_EasyExif);
  //Open the file for inspection and analysis, and return bool
  exifReader->open( sImageFilename );
  //Verify whether the file has metadata and the model of the obtained camera is not empty
  const bool bHaveValidExifMetadata =
    exifReader->doesHaveExifInfo()
    && !exifReader->getModel().empty();
//If there is an error, an error will be reported, and if there is a right, it will be referenced
  if (bHaveValidExifMetadata) // If image contains meta data
  {
    const std::string sCamModel = exifReader->getModel();
    // Handle the case where the focal length is equal to 0
    if (exifReader->getFocal() == 0.0f)
    {
      error_report_stream
        << stlplus::basename_part(sImageFilename) << ": Focal length is missing." << "\n";
      focal = -1.0;
    }
    else
    // Create an image entry in a list file
    {
      Datasheet datasheet;
      if ( getInfo( sCamModel, vec_database, datasheet ))
      {
        // The camera model is found in the database, so we can calculate its approximate focal length
        const double ccdw = datasheet.sensorSize_;
        focal = std::max ( width, height ) * exifReader->getFocal() / ccdw;
      }
      else
      {
        error_report_stream
          << stlplus::basename_part(sImageFilename)
          << "\" model \"" << sCamModel << "\" doesn't exist in the dataelse		base" << "\n"
          << "Please consider add your camera model and sensor width in the database." << "\n";
      }
    }
  }
}

Obtain the camera model and construct the internal parameters related to the view

std::shared_ptr<IntrinsicBase> intrinsic;
if (focal > 0 && ppx > 0 && ppy > 0 && width > 0 && height > 0)
{
  // Create the required camera type, as described earlier
  switch (e_User_camera_model)
  {
    case PINHOLE_CAMERA:
      intrinsic = std::make_shared<Pinhole_Intrinsic>
        (width, height, focal, ppx, ppy);
    break;
    case PINHOLE_CAMERA_RADIAL1:
      intrinsic = std::make_shared<Pinhole_Intrinsic_Radial_K1>
        (width, height, focal, ppx, ppy, 0.0); // setup no distortion as initial guess
    break;
    case PINHOLE_CAMERA_RADIAL3:
      intrinsic = std::make_shared<Pinhole_Intrinsic_Radial_K3>
        (width, height, focal, ppx, ppy, 0.0, 0.0, 0.0);  // setup no distortion as initial guess
    break;
    case PINHOLE_CAMERA_BROWN:
      intrinsic = std::make_shared<Pinhole_Intrinsic_Brown_T2>
        (width, height, focal, ppx, ppy, 0.0, 0.0, 0.0, 0.0, 0.0); // setup no distortion as initial guess
    break;
    case PINHOLE_CAMERA_FISHEYE:
      intrinsic = std::make_shared<Pinhole_Intrinsic_Fisheye>
        (width, height, focal, ppx, ppy, 0.0, 0.0, 0.0, 0.0); // setup no distortion as initial guess
    break;
    case CAMERA_SPHERICAL:
       intrinsic = std::make_shared<Intrinsic_Spherical>
         (width, height);
    break;
    default:
      std::cerr << "Error: unknown camera model: " << (int) e_User_camera_model << std::endl;
      return EXIT_FAILURE;
  }
}

Build the view corresponding to the image. If there is gps weight (it needs to be defined as priority rotation)

const std::pair<bool, Vec3> gps_info = checkGPS(sImageFilename, i_GPS_XYZ_method);
if (gps_info.first)
{
//The subclass of Views. You can choose whether to preferentially rotate or adjust the position
  ViewPriors v(*iter_image, views.size(), views.size(), views.size(), width, height);

  // Add internal files related to the image (if any)
  if (intrinsic == nullptr)
  {
//Because the view has invalid internal data
//(export view with invalid intrinsic field value)
    v.id_intrinsic = UndefinedIndexT;
  }
  else
  {
    // Add the defined intrinsic to the sfm_container
    intrinsics[v.id_intrinsic] = intrinsic;
  }

  v.b_use_pose_center_ = true;
  v.pose_center_ = gps_info.second;

  //Previous weights
  if (prior_w_info.first == true)
  {
    v.center_weight_ = prior_w_info.second;
  }
  //Add view to sfm container
  views[v.id_view] = std::make_shared<ViewPriors>(v);
}

When there is no gps information

else		
{
  View v(*iter_image, views.size(), views.size(), views.size(), width, height);

  // Add intrinsic related to the image (if any)
  if (intrinsic == nullptr)
  {
    //Since the view have invalid intrinsic data
    // (export the view, with an invalid intrinsic field value)
    v.id_intrinsic = UndefinedIndexT;
  }
  else
  {
    // Add the defined intrinsic to the sfm_container
    intrinsics[v.id_intrinsic] = intrinsic;
  }

  // Add the view to the sfm_container
  views[v.id_view] = std::make_shared<View>(v);
}

Error reporting and preservation

Report mistakes without telling
Save SFM_ Name the data of data and save it to the output file

Save(  sfm_data, stlplus::create_filespec( sOutputDir, "sfm_data.json" ).c_str(),  ESfM_Data(VIEWS|INTRINSICS))

Examples

This example is very simple. Read a picture. Because this process does not make gps constraints, there is no need to add an advanced pose