#+TITLE: mrcal 2.5.1 release notes
#+OPTIONS: toc:nil

* New in mrcal 2.5.1
This is yet another mostly-maintenance release. /Most/ of the implementation of
some new big features is written and committed, but it's still incomplete. The
new stuff is there, but is lightly tested and documented. This will be completed
eventually in mrcal 3.0:

- [[file:uncertainty-cross-reprojection.org][Cross-reprojection uncertainty]], to be able to perform full calibrations with a
  splined model and /without/ a chessboard. [[file:mrcal-show-projection-uncertainty.html][=mrcal-show-projection-uncertainty
  --method cross-reprojection-rrp-Jfp=]] is available today, and works in the
  usual moving-chessboard-stationary camera case. Fully boardless coming later.
- More general view of uncertainty and diffs. I want to support extrinsics-only
  and/or intrinsics computations-only in lots of scenarios. Uncertainty in point
  solves is already available in some conditions, for instance if the points are
  fixed. New [[file:mrcal-show-stereo-pair-diff.html][=mrcal-show-stereo-pair-diff= tool]] reports an extrinsics+intrinsics
  diff between two calibrations of a stereo pair; experimental
  [[https://www.github.com/dkogan/mrcal/blob/master/analyses/extrinsics-stability.py][=analyses/extrinsics-stability.py=]] tool reports an extrinsics-only diff. These
  are in contrast to the intrinsics-only uncertainty and diffs in the existing
  [[file:mrcal-show-projection-diff.html][=mrcal-show-projection-diff=]] and [[file:mrcal-show-projection-uncertainty.html][=mrcal-show-projection-uncertainty=]] tools.
  Some documentation in the [[file:uncertainty.org][uncertainty]] and [[file:differencing.org][differencing]] pages.
- Implicit point solves, using the triangulation routines in the optimization
  cost function. Should produce much more efficient structure-from-motion
  solves. This is all the "triangulated-features" stuff. The cost function is
  primarily built around =_mrcal_triangulated_error()=. This is demoed in
  [[https://www.github.com/dkogan/mrcal/blob/master/test/test-sfm-triangulated-points.py][=test/test-sfm-triangulated-points.py=]]. And I've been using
  =_mrcal_triangulated_error()= in structure-from-motion implementations within
  other optimization routines.

** Experimental tools
A number of new tools have been written that are very useful, but are still a
bit rough, and need to be finished

- A new [[https://www.github.com/dkogan/mrcal/blob/master/analyses/mrcal-pick-features][=analyses/mrcal-pick-features=]] tool is available to interactively pick
  matching features from a stereo pair's images, and to compute the relative
  extrinsics off them. This is very useful for a narrow application, and the
  optimal solver details are unclear, so this is not-yet released today. The
  assisted gui feature-picker and solver will be used for /something/
  eventually, but today it's there, waiting to be finished.

- New validation tools [[https://www.github.com/dkogan/mrcal/blob/master/analyses/validate-input-noise.py][=analyses/validate-input-noise.py=]] and
  [[https://www.github.com/dkogan/mrcal/blob/master/analyses/validate-uncertainty.py][=analyses/validate-uncertainty.py=]] and [[https://www.github.com/dkogan/mrcal/blob/master/analyses/validate-noncentral.py][=analyses/validate-noncentral.py=]] are
  useful to check the various assumptions made by the mrcal processing. Used for
  sanity-checking a solve and for algorithm development. Documented somewhat
  [[file:tour-cross-validation.org::#cross-validation-comparing-with-uncertainty][here]] and [[file:formulation.org::#estimating-input-noise][here]].

- The mrcal library has now been used in multiple projects to align non-camera
  sensors with great success:

  - A geometry-only routine to align a set of cameras and a set of LIDARs is
    available in the [[https://github.com/dkogan/camera-lidar-calibration/][=camera-lidar-calibration= repo]]. This aligns the geometry
    /only/ and works for a LIDAR-only camera-less system as well. This tool
    assumes the camera intrinsics are /perfect/ and it does uncertainty
    propagation of the output. It is decently well-tested and works well.
    Documentation and interfaces are still unideal, but improvements are
    welcome!

  - A routine to align a set of cameras and an IMU is available in
    [[https://www.github.com/dkogan/mrcal/blob/master/analyses/calibrate-camera-imu.py][=analyses/calibrate-camera-imu.py=]]. This is also geometry-only and also
    trusts the intrinsics 100%. It's fairly simplistic, but highly usable and
    on-par with existing methods. If needed a full kalibr-style
    sensor-bias-fitting thing will probably be implemented eventually.
    Improvements welcome!

** General fixes

- Thanks to [[https://github.com/thatcomputerguy0101][Alan Everett]], mrcal and the adjacent tooling now all build with
  macos. If anybody wants to see support for more OSs and distribution
  mechanisms (homebrew? fedora? pip?), and is willing to help with the
  implementation and the maintenance, please get in touch with me.

- Poseutils: more careful handling of rotations near singularities. Rotations by
  ~ 180deg in particular weren't implemented properly, and had bugs that are now
  fixed.

- Renamed "residuals" -> "measurements" to clarify the nomenclature. These are
  similar except for scaling and the "measurements" may also contain
  regularization.
  - =residuals_point()= -> =measurements_point()=
  - =residuals_board()= -> =measurements_board()=
  - =show_residuals_....()= functions now ask for a measurement =x= argument;
    the old =residuals= argument is still accepted for backwards compatibility

- Added =mrcal_traverse_sensor_links()= in C and =mrcal.traverse_sensor_links()=
  in Python. These compute the best path through a graph of sensors, trying to
  maximize the common observations between each successive pair of sensors in
  the path. This is a part of the seeding algorithm, and is now exposed for
  other usages. The general min-heap implementation at the core of this
  computation is available in =mrcal/heap.h=

- [[file:mrcal-calibrate-cameras.html][=mrcal-calibrate-cameras=]] and [[file:mrcal-python-api-reference.html#-compute_chessboard_corners][=mrcal.compute_chessboard_corners()=]] have a
  higher minimum points-per-chessboard-observation threshold (new
  =Npoint_observations_per_board_min= kwarg in the function). We now throw out
  any images where a chessboard detection has fewer than
  =2*min(gridn_width,gridn_height)= points. This is irrelevant for =mrgingham=,
  which produces all-or-nothing detections, but other detectors could produce
  barely-adequate detections that could cause problems previously

- [[file:mrcal-python-api-reference.html#-compute_chessboard_corners][=mrcal.compute_chessboard_corners()=]] has new =image_path_prefix= and
  =image_directory= kwargs to provide more options for interpreting paths in the
  given =corners_cache_vnl=. This is analogous to how other parts of mrcal work
  (for instance the [[file:mrcal-show-residuals-board-observation.html][=mrcal-show-residuals-board-observation=]] tool).

- The [[file:mrcal-calibrate-cameras.html][=mrcal-calibrate-cameras=]] use that function, and now has
  =--image-path-prefix= and =--image-directory=. Useful to make =--imagersize=
  unnecessary and to have image overlays working with =--explore=. This is
  especially helpful if calibrating a system with different-resolution cameras.
  In that case =--imagersize= doesn't work, and finding the images on disk is
  required

- The [[file:mrcal-convert-lensmodel.html][=mrcal-convert-lensmodel=]] tool has =--all= to write out /all/ the models
  from a joint re-solve

- Added new [[file:mrcal-python-api-reference.html#-model_resolution__deg_pixel][=mrcal.model_resolution__deg_pixel()=]] function and the
  [[file:mrcal-show-model-resolution.html][=mrcal-show-model-resolution=]] tool

- [[file:mrcal-python-api-reference.html#-show_distortion_off_pinhole_radial][=mrcal.show_distortion_off_pinhole_radial()=]] works with all models, not just
  opencv ones

- The image load/save functions no longer use [[https://freeimage.sourceforge.io/][libfreeimage]], since that library
  is unmaintained. I'm using [[https://github.com/nothings/stb][libstb]] for image-reading and [[https://www.libpng.org/pub/png/libpng.html][libpng]] and
  [[https://libjpeg-turbo.org/][libjpeg-turbo]] for image-writing. libstb can write images too, but there were
  several limitations that made it necessary to use the dedicated libraries.
  This means that currently I can write /only/ =.png= and =.jpg= images. This
  can be extended if needed.

- The [[file:mrcal-cull-corners.html][=mrcal-cull-corners=]] tool has =--cull-rad-off-center-board= to throw out
  chessboard edges. Useful if we dont trust the chessboard shape

- Added [[file:mrcal-python-api-reference.html#cameramodel-optimization_inputs_reset][=mrcal.cameramodel.optimization_inputs_reset()=]] and
  [[file:mrcal-python-api-reference.html#cameramodel-valid_intrinsics_region_reset][=mrcal.cameramodel.valid_intrinsics_region_reset()=]] to unset these fields in a
  [[file:mrcal-python-api-reference.html#cameramodel][=mrcal.cameramodel=]].

- Added the [[https://www.github.com/dkogan/mrcal/blob/master/analyses/mrcal-convert-lensmodel-from-kalibr-fov][=analyses/mrcal-convert-lensmodel-from-kalibr-fov=]] tool to fit a
  Kalibr "fov" model into something that mrcal supports. This isn't
  general-enough or tested-enough to be fully distributed yet, so run it from
  the mrcal source tree if you need it

- Added =mrcal_cameramodel_converter()= "converter" function that can be used
  with =O&= conversions in [[https://docs.python.org/3/c-api/arg.html#c.PyArg_ParseTupleAndKeywords][=PyArg_ParseTupleAndKeywords()=]] calls. Can interpret
  either path strings or [[file:mrcal-python-api-reference.html#cameramodel][=mrcal.cameramodel=]] objects as
  =mrcal_cameramodel_VOID_t= C structures. Useful for writing Python extension
  modules for C code that uses =mrcal_cameramodel_VOID_t=.

- Some fields in the [[file:mrcal-python-api-reference.html#cameramodel][=mrcal.cameramodel=]] type and the saved =.cameramodel= files
  and the =optimization_inputs= C and Python interfaces were renamed:

  \begin{aligned}
  \mathrm{extrinsics\_rt\_fromref} & \to \mathrm{rt\_cam\_ref}   \\
  \mathrm{extrinsics\_Rt\_fromref} & \to \mathrm{Rt\_cam\_ref}   \\
  \mathrm{extrinsics\_rt\_toref}   & \to \mathrm{rt\_ref\_cam}   \\
  \mathrm{extrinsics\_Rt\_toref}   & \to \mathrm{Rt\_ref\_cam}   \\
  \mathrm{frames\_rt\_fromref}     & \to \mathrm{rt\_frame\_ref} \\
  \mathrm{frames\_Rt\_fromref}     & \to \mathrm{Rt\_frame\_ref} \\
  \mathrm{frames\_rt\_toref}       & \to \mathrm{rt\_ref\_frame} \\
  \mathrm{frames\_Rt\_toref}       & \to \mathrm{Rt\_ref\_frame} \\
  \end{aligned}

  This makes things more consistent. Compatibility logic is in place, so old
  code and data should keep working. New =.cameramodel= files write both the new
  and old fields, so old tools can read the new files.

- Transform composition functions have new arguments: =inverted0= and
  =inverted1=, to make it easier to use inverted transforms. The C macros
  =mrcal_compose_Rt()=, =mrcal_compose_rt()=, =mrcal_compose_r()= are unchanged,
  with new macros available to apply the inverses: =...._inverted0()=,
  =...._inverted1()=, =...._inverted01()=. The corresponding
  =mrcal_compose_..._full()= C functions have new arguments; *this breaks the
  API and ABI*. The Python functions [[file:mrcal-python-api-reference.html#-compose_r][=mrcal.compose_r()=]] and
  [[file:mrcal-python-api-reference.html#-compose_rt][=mrcal.compose_rt()=]] and [[file:mrcal-python-api-reference.html#-compose_R][=mrcal.compose_R()=]] have new kwargs with default
  values, so this API remains compatible.

** Stereo
- =mrcal_rectified_system2()= is a new flavor of =mrcal_rectified_system()= to
  provide a new =az_edge_margin_deg= argument, to set the closest the rectified
  view is allowed to get to $\mathrm{az}=\pm 90^\circ$. The legacy function
  still exists, defaulting to =az_edge_margin_deg=10=. [[file:mrcal-stereo.html][=mrcal-stereo=]] controls
  this via =--az-edge-margin-deg=. If =az0_deg= is being auto-detected, it will
  be shifted to avoid looking along the baseline

- Similarly the Python [[file:mrcal-python-api-reference.html#-rectified_system][=mrcal.rectified_system()=]] function now takes this
  =az_edge_margin_deg= argument also

- The C function =mrcal_rectified_system2()= now supports pinhole rectification,
  something only the Python [[file:mrcal-python-api-reference.html#-rectified_system][=mrcal.rectified_system()=]] call did previously. This
  is strictly worse than the [[file:stereo.org::#stereo-rectification-models][=LENSMODEL_LATLON= rectification]] used by default,
  but is good to have for testing, since this is the traditional scheme used by
  every other tool.

- [[file:mrcal-stereo.html][=mrcal-stereo=]]: =--equalization= implies =--force-grayscale=

- [[file:mrcal-python-api-reference.html#-stereo_range][=mrcal.stereo_range()=]] has a better default for disparity_max to handle
  invalid disparities reliably in the common case where the disparity is a
  16-bit signed integer

** C API 
- Added C implementations of Procrustes fits:
  =mrcal_align_procrustes_vectors_R01()= and
  =mrcal_align_procrustes_points_Rt01()=. These are now the internal
  implementations of the Python [[file:mrcal-python-api-reference.html#-align_procrustes_vectors_R01][=mrcal.align_procrustes_vectors_R01()=]] and
  [[file:mrcal-python-api-reference.html#-align_procrustes_points_Rt01][=mrcal.align_procrustes_points_Rt01()=]] functions

- Added =mrcal_R_aligned_to_vector()= C function. This is now the internal
  implementation of [[file:mrcal-python-api-reference.html#-R_aligned_to_vector][=mrcal.R_aligned_to_vector()=]]

- Added simple math operation functions to the C API:
  - =double         mrcal_point3_inner(const mrcal_point3_t a, const mrcal_point3_t b)=
  - =double         mrcal_point3_norm2(const mrcal_point3_t a)=
  - =double         mrcal_point3_mag  (const mrcal_point3_t a)=
  - =mrcal_point3_t mrcal_point3_add  (const mrcal_point3_t a, const mrcal_point3_t b)=
  - =mrcal_point3_t mrcal_point3_sub  (const mrcal_point3_t a, const mrcal_point3_t b)=
  - =mrcal_point3_t mrcal_point3_scale(const mrcal_point3_t a, const double s)=
  - =mrcal_point3_t mrcal_point3_cross(const mrcal_point3_t a, const mrcal_point3_t b)=

  And similar for =mrcal_point2_t=, except there's no =mrcal_point2_cross()=

- Added simple point and pose printing utilities:
  - =mrcal_point2_print(p)=
  - =mrcal_point3_print(p)=
  - =mrcal_Rt_print(Rt)=
  - =mrcal_rt_print(rt)=

- =mrcal_image_uint8_load()= applies stretch equalization if given a 16-bit
  image. This is a reasonable default. If more specific processing is needed,
  call =mrcal_image_uint16_load()=. Applies to [[file:mrcal-python-api-reference.html#-load_image][=mrcal.load_image()=]] also.

- Added =mrcal_image_void_t= image type for generic functions that aren't meant
  to interface with any particular image type. The [[https://www.github.com/dkogan/mrcal/blob/master/image.h][=mrcal_image_anytype_load()=]]
  function now uses that type.

- All headers have C++ =extern "C"= wrappers for direct =#include= in C++
  projects

- Renamed =mrcal_cameramodel_t= -> =mrcal_cameramodel_VOID_t= to clarify that
  the specific lensmodel type is unknown here. The legacy alias
  =mrcal_cameramodel_t= is still available for backwards-compatibility

- Added =mrcal_intrinsics_XXX_t= structures to represent camera intrinsics.
  These are exactly like =mrcal_cameramodel_XXX_t= but without the extrinsics

- The C API can read models into a preallocated buffer intead of forcing
  allocation, as before. New functions:
  #+begin_src c
  bool mrcal_read_cameramodel_string_into(// out
                                          mrcal_cameramodel_VOID_t* model,
                                          // in,out
                                          int* Nintrinsics_max,
                                          // in
                                          const char* string,
                                          const int len);

  bool mrcal_read_cameramodel_file_into  (// out
                                          mrcal_cameramodel_VOID_t* model,
                                          // in,out
                                          int* Nintrinsics_max,
                                          // in
                                          const char* filename);
  #+end_src

* Migration notes 2.4 -> 2.5

The C API got a few updates that require a few minor changes to user code:

- =mrcal_compose_rt_full()= and =mrcal_compose_Rt_full()= C functions have two
  new arguments: =inverted0=, =inverted1=. Most callers use the
  =mrcal_..._compose()= macros, so for them only the ABI has changed, and a
  rebuild is sufficient.

- =mrcal_compose_rt_full()= C function can return =dt01_dr1= and =dt01_dt0=.
  Most callers use the =mrcal_..._compose()= macros, so for them only the ABI
  has changed, and a rebuild is sufficient.

- The C types =mrcal_cameramodel_XXX_t= no longer have a generic member
  =mrcal_cameramodel_t m=. If you need a generic alias of
  =mrcal_cameramodel_XXX_t* model= you now need to
  =(mrcal_cameramodel_VOID_t*)model= instead of =&model->m=

- All the headers renamed to remove =mrcal-= from their name. Everything is
  intended to be included as

  #+begin_src c
  #include <mrcal/thing.h>
  #+end_src

  So the =mrcal-= in the name was superfluous. If you were including either of:

  - =mrcal-image.h=
  - =mrcal-types.h=

  You should now include:

  - =mrcal/image.h=
  - =mrcal/types.h=

- The [[https://www.github.com/dkogan/mrcal/blob/master/image.h][=mrcal_image_anytype_load()=]] function uses the =mrcal_image_void_t= image
  type. The semantics are identical, but older code might need a cast or a type
  change to build.

And some nomenclature changed: some fields in the [[file:mrcal-python-api-reference.html#cameramodel][=mrcal.cameramodel=]] type and
the saved =.cameramodel= files and the =optimization_inputs= C and Python
interfaces were renamed. The old naming mostly still works, but transitioning to
the new naming would be a good thing to do:

  \begin{aligned}
  \mathrm{extrinsics\_rt\_fromref} & \to \mathrm{rt\_cam\_ref}   \\
  \mathrm{extrinsics\_Rt\_fromref} & \to \mathrm{Rt\_cam\_ref}   \\
  \mathrm{extrinsics\_rt\_toref}   & \to \mathrm{rt\_ref\_cam}   \\
  \mathrm{extrinsics\_Rt\_toref}   & \to \mathrm{Rt\_ref\_cam}   \\
  \mathrm{frames\_rt\_fromref}     & \to \mathrm{rt\_frame\_ref} \\
  \mathrm{frames\_Rt\_fromref}     & \to \mathrm{Rt\_frame\_ref} \\
  \mathrm{frames\_rt\_toref}       & \to \mathrm{rt\_ref\_frame} \\
  \mathrm{frames\_Rt\_toref}       & \to \mathrm{Rt\_ref\_frame} \\
  \end{aligned}
