#+TITLE: mrcal 3.0 release notes
#+OPTIONS: toc:nil

* New in mrcal 3.0

cross-reprojection

mrcal.drt_ref_refperturbed__dbpacked()
mrcal.compose_rt_tinyrt0_gradientrt0()
mrcal.compose_rt_tinyrt1_gradientrt1
compose_r_tinyr1_gradientr1

75893eea: Added initialy EARLY version of mrcal-show-stereo-pair-diff (this
needs to be tested, documented)

faster uncertainty computation:
  cecab50a CHOLMOD_factorization.solve_xt_JtJ_bt() can solve other problems
  9393ef79 Much faster uncertainty computation

2024-11-27 bestq uncertainty method
  - works
  - needs documentation
  - MUCH slower than the mean-pcam, since it solves the linear problem (Nframes
    or Ncameras_extrinsics) more times than mean-pcam

* Migration notes 2.4 -> 3.0

* todo
- add notes about the triangulated-points merge

** triangulated features merge
*** =git grep -n Noutliers= Noutliers has change meaning: it's now =Nmeasurements_outliers=

   #+begin_src diff
   diff --git a/doc/c-api.org b/doc/c-api.org
   index 7d3c8939..2ae7d9d5 100644
   --- a/doc/c-api.org
   +++ b/doc/c-api.org
        /* The RMS error of the optimized fit at the optimum. Generally the residual */
        /* vector x contains error values for each element of q, so N observed pixels */
        /* produce 2N measurements: len(x) = 2*N. And the RMS error is */
        /*   sqrt( norm2(x) / N ) */
        double rms_reproj_error__pixels;

   -    /* How many pixel observations were thrown out as outliers. Each pixel */
   -    /* observation produces two measurements. Note that this INCLUDES any */
   -    /* outliers that were passed-in at the start */
   +    /* How many measurements were thrown out as outliers. Each pixel */
   +    /* observation of a chessboard point produces two measurements (x,y). */
   +    /* Note: this INCLUDES any outliers that were passed-in at the start */
        int Noutliers;
    } mrcal_stats_t;


   --- a/mrcal-calibrate-cameras
   +++ b/mrcal-calibrate-cameras
   @@ -745,13 +745,13 @@ Npoints_chessboard = args.object_width_n*args.object_height_n*Nobservations
    residuals = \
        stats['x'][:Npoints_chessboard*2]. \
        reshape(Nobservations, args.object_height_n, args.object_width_n, 2)
    worst_point_err = np.sqrt(np.max(nps.norm2( nps.clump(residuals, n=3) )))
    report += f"Worst residual (by measurement): {worst_point_err:.01f} pixels\n"
    if not args.skip_outlier_rejection:
   -    report += "Noutliers: {} out of {} total points: {:.01f}% of the data\n". \
   +    report += "Noutliers: {} out of {} total measurements: {:.01f}% of the data\n". \
            format(stats['Noutliers'],
                   args.object_height_n*args.object_width_n*len(observations),
                   100.0 * stats['Noutliers'] / (args.object_height_n*args.object_width_n*len(observations)))
    if calobject_warp is not None:
        report += f"calobject_warp = {calobject_warp}\n"
 
   #+end_src

*** divergent-rays-are outlier logic is weird
I declare and outlier on the first pass. That pass is for getting the threshold

** patches deferred for next release

#+begin_src diff
diff --git a/mrcal-show-projection-diff b/mrcal-show-projection-diff
index 572d701..6cb48dc 100755
--- a/mrcal-show-projection-diff
+++ b/mrcal-show-projection-diff
@@ -503,3 +503,7 @@ if not args.intrinsics_only and args.radius != 0 and \
 
 if args.hardcopy is None:
     plot.wait()
+
+
+# should --unset key be the default? And for the uncertainty plot?
+
diff --git a/mrcal-show-residuals-board-observation b/mrcal-show-residuals-board-observation
index 76ce4db..b8c17eb 100755
--- a/mrcal-show-residuals-board-observation
+++ b/mrcal-show-residuals-board-observation
@@ -365,3 +365,8 @@ The optimization inputs are available in the optimization_inputs dict
 for i in range(Nplots):
     os.waitpid(pids[i], 0)
 sys.exit()
+
+
+
+
+### add auto-vector-scale
#+end_src

** _propagate_calibration_uncertainty() needs to be exported in the API
** I should check the camera extrinsics uncertainty
If the camera geometry is very uncertain, the calibration isn't successful; even
if the variance in the other state variables compensates for these perfectly.
The _propagate_calibration_uncertainty() function can easily do this. I should
rename it. And I should expose it as part of the API. This code works to detect
uncertain extrinsics for a camera pair:

#+begin_src python

model_filename = sys.argv[1]
m = mrcal.cameramodel(model_filename)
optimization_inputs = m.optimization_inputs()

istate_extrinsics0 = mrcal.state_index_extrinsics(0, **optimization_inputs)
Nstate_extrinsics  = mrcal.num_states_extrinsics(    **optimization_inputs)

Nstate = mrcal.num_states( **optimization_inputs)

if Nstate_extrinsics != 6:
    raise Exception(f"Unexpected {Nstate_extrinsics=}")

dF_db = np.zeros((Nstate_extrinsics, Nstate), dtype=float)
dF_db[:,istate_extrinsics0:istate_extrinsics0+Nstate_extrinsics] = \
    np.eye(Nstate_extrinsics)

Var_rt_cam_ref = \
    mrcal.model_analysis._propagate_calibration_uncertainty('covariance',
                                                            dF_db = dF_db,
                                                            observed_pixel_uncertainty = 1.,
                                                            optimization_inputs = optimization_inputs)

print(f"stdev(rt_cam_ref) = {np.sqrt(np.diag(Var_rt_cam_ref))}")

#+end_src

** uncertainty regression
The triangulated-features merge caused the uncertainty reporting to be a bit
different for some reason. I need to chase it down to see what happened. I'm
looking at

~/projects/mrcal.old/out0.cameramodel

This command is returning slightly different results before/after the merge:

~/projects/mrcal.old/mrcal-show-projection-uncertainty out0.cameramodel --cbmax 30

** uncertainty strongly affected by regularization weight
Computing the uncertainty of the results of stationary-calibration.py can
produce wildly different output if I tweak the regularization weight

** regularization scaling
I should aim for specific number of pixels instead of for some ratio. This will
probably break loading optimization_inputs from model files: they'd need
reoptimization

** point range normalization
I removed it here: 0e727189. Do I want it back in some form? I do still require
point_min_range and point_max_range. Do I really need these?

** XyJax loaded in too many doc pages
I need it everywhere I use \xymatrix (currently uncertainty.org only). So that's
the only place I should use it. Loading it needlessly is slow

** mrcal-convert-lensmodel
This needs to support points:
- search for indices_point_camintrinsics_camextrinsics
- solving without --sampled fails with points: no logic to do point culling

** mrcal-cull-corners should be able to cull board edges
Need new option like =--cull-board-rowscols L,T,R,B=

Can hack it on the commandline:

#+begin_src sh
R=1; < $C vnl-filter --sub 'ii() { if(filename != prev(filename)) { i=0; return i; } return ++i; }' -p .,'i=ii()' | vnl-filter -p .,\!i,'i=int(i/14)',j='i % 14' | vnl-filter -p filename,x,y,level="(i<$R || i>=14-$R || j<$R || j>=14-$R) ? \"-\" : level" > /tmp/corners-board-edge-cut$R.vnl
#+end_src

** mrcal-stereo should have an anti-aliasing filter
When I downsample. Just before =mrcal.transform_image()= it should

#+begin_src python
for i in range(len(images)):
    images[i] = cv2.GaussianBlur(images[i],
                                 ksize=(0,0), # auto-select
                                 # sigmaX = 2 ^ -pixels_per_deg,
                                 sigmaX = 2 )
#+end_src

** I should support more lens models
Being compatible with at least ROS would be nice. Their models are:

- =plumb_bob=: This is =LENSMODEL_OPENCV5=
- =rational_polynomial=: This is =LENSMODEL_OPENCV8=
- =equidistant=: mrcal does not support this today. It should. This is
  [[https://docs.opencv.org/3.4/db/d58/group__calib3d__fisheye.html][cv::fisheye]]

** _mrcal_drt_ref_refperturbed__dbpacked() currently is hardcoded to use the Jfpcw formulation
Give it an argument to select the formulation. And rename the function. Or
something
** other stuff
- "pydoc3 mrcal" should show everything. It doesn't. "compose_rt" isn't there,
  for instance

- mrcal-stereo: during the rectification (or maybe disparity search) stage C-c
  doesn't work.

** Expose _options_heatmap_with_contours()
** mrcal.stereo_range() does uint16 based on qrect0 is None. It should look at the type
Currently a full-image float range image doesn't work right: it casts to uint16,
and we lose accuracy
** Port to mrcal 2.5
commit 317b4b904f1f1fa3c983e48d86104dca50893a6e
Author: Dima Kogan <dima@secretsauce.net>
Date:   Thu Aug 10 14:30:58 2023 -0700

    r_from_R_core() better implementation around th=180deg

 And mrcal-stereo --single-buffered

** document valid-intrinsics-region

** new observed_pixel_uncertainty
  - apply sqrt() factor. Documented in 68789474. git grep -n
    observed_pixel_uncertainty ..
  - Does too little data give smaller residuals? It should. Is this reflected in
    the sqrt() factor?

** test-projection-uncertainty improvements
"--fixed frames" and --moving-camera and --points should be consolidated into a
clearer set of things. I should be able to have the surveyed calibration
uncertainty test in test-projection-uncertainty, but that doesn't currently
work. The best-effort is the below patch:

#+begin_src diff
diff --git a/test/test-projection-uncertainty.py b/test/test-projection-uncertainty.py
index 8cc40962..2ac9d2c7 100755
--- a/test/test-projection-uncertainty.py
+++ b/test/test-projection-uncertainty.py
@@ -379,17 +379,72 @@ x_baseline_optimized = \
 
 lensmodel       = optimization_inputs_baseline['lensmodel']
 imagersizes     = optimization_inputs_baseline['imagersizes']
 intrinsics_true = nps.cat( *[m.intrinsics()[1] \
                              for m in models_true] )
 
-models_baseline = \
-    [ mrcal.cameramodel( optimization_inputs = optimization_inputs_baseline,
-                         icam_intrinsics     = i) \
-      for i in range(args.Ncameras) ]
 
+print(f"rt_cam_ref.shape: {optimization_inputs_baseline['rt_cam_ref'].shape}")
+if optimization_inputs_baseline.get('indices_frame_camintrinsics_camextrinsics') is not None:
+    print(f"indices_frame_camintrinsics_camextrinsics.shape: {optimization_inputs_baseline['indices_frame_camintrinsics_camextrinsics'].shape}")
+else:
+    print(f"indices_point_camintrinsics_camextrinsics.shape: {optimization_inputs_baseline['indices_point_camintrinsics_camextrinsics'].shape}")
+
+
+if False:
+    ####### this can't be written in all cases:
+    # test/test-projection-uncertainty.py \
+    #   --fixed frames \
+    #   --model opencv4 \
+    #   --Ncameras 1 \
+    #   --range-to-boards 4 \
+    #   --points
+    # mrcal.c(3944): Cannot compute icam_extrinsics. I don't have a vanilla calibration problem (stationary cameras, cam0 is reference)
+    # Traceback (most recent call last):
+    #   File "/home/dima/projects/mrcal/test/test-projection-uncertainty.py", line 389, in <module>
+    #     [ mrcal.cameramodel( optimization_inputs = optimization_inputs_baseline,
+    #       ~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+    #                          icam_intrinsics     = i) \
+    #                          ^^^^^^^^^^^^^^^^^^^^^^^^
+    #   File "/home/dima/projects/mrcal/test/../mrcal/cameramodel.py", line 1249, in __init__
+    #     icam_extrinsics = mrcal.corresponding_icam_extrinsics(icam_intrinsics,
+    #                                                           **optimization_inputs)
+    # RuntimeError: mrcal-pywrap.c:3277 callback_corresponding_icam_extrinsics(): Error calling mrcal_corresponding_icam_extrinsics()
+    models_baseline = \
+        [ mrcal.cameramodel( optimization_inputs = optimization_inputs_baseline,
+                             icam_intrinsics     = i) \
+          for i in range(args.Ncameras) ]
+else:
+    # This path causes a crash:
+    # test/test-projection-uncertainty.py \
+    #   --fixed cam0 \
+    #   --model opencv4 \
+    #   --do-sample \
+    #   --observed-pixel-uncertainty 0.03 \
+    #   --reproject-perturbed cross-reprojection-rrp-Jfp \
+    #   --points
+    # Traceback (most recent call last):
+    #   File "/home/dima/projects/mrcal/test/test-projection-uncertainty.py", line 394, in <module>
+    #     [ mrcal.cameramodel( optimization_inputs = optimization_inputs_baseline,
+    #       ~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+    #                          icam_intrinsics = 0,
+    #                          ^^^^^^^^^^^^^^^^^^^^
+    #                          icam_extrinsics = i) \
+    #                          ^^^^^^^^^^^^^^^^^^^^
+    #   File "/home/dima/projects/mrcal/test/../mrcal/cameramodel.py", line 1254, in __init__
+    #     self.rt_cam_ref(optimization_inputs['rt_cam_ref'][icam_extrinsics])
+    models_baseline = \
+        [ mrcal.cameramodel( optimization_inputs = optimization_inputs_baseline,
+                             icam_intrinsics = 0,
+                             icam_extrinsics = i) \
+          for i in range(args.Ncameras) ]
+##### the extra test in Makefile.tests does not pass. I try to do
+##### fixed-points-only uncertainty. It should work. test-surveyed-calibration
+##### does the same thing (I think) and that does work. The only difference is
+##### that that test has only a single camera pose
 
 # I evaluate the projection uncertainty of this vector. In each camera. I'd like
 # it to be center-ish, but not AT the center. So I look at 1/3 (w,h). I want
 # this to represent a point in a globally-consistent coordinate system. Here I
 # have fixed frames, so using the reference coordinate system gives me that
 # consistency. Note that I look at q0 for each camera separately, so I'm going
@@ -2332,12 +2397,69 @@ else:                                                          reproject_perturb
 # "method" argument for mrcal.projection_uncertainty()
 if   re.match('^(mean-pcam|bestq)$', args.reproject_perturbed): method = args.reproject_perturbed
 elif re.match('cross-reprojection',  args.reproject_perturbed): method = 'cross-reprojection-rrp-Jfp'
 # default
 else:                                                           method = 'mean-pcam'
 
+
+
+
+
+
+
+
+
+distance = args.distances[0]
+# shape (Ncameras, 3)
+p_cam_baseline = mrcal.unproject(q0_baseline, lensmodel, intrinsics_baseline,
+                                 normalize = True) * distance
+
+# shape (Ncameras, 3)
+p_ref_baseline = \
+    mrcal.transform_point_rt( mrcal.invert_rt(extrinsics_baseline_mounted),
+                              p_cam_baseline )
+p_ref_query = p_ref_baseline
+p_cam_query = \
+    mrcal.transform_point_rt(rt_cam_ref_true, p_ref_query)
+q = mrcal.project(p_cam_query, lensmodel, intrinsics_true)
+print(q[0])
+
+import IPython
+IPython.embed()
+sys.exit()
+
+# THE PROBLEM:
+#
+# extrinsics_baseline_mounted should be the reoptimized rt_cam_ref_true, but
+# they don't even have the same shape:
+r'''
+In [5]: extrinsics_baseline_mounted.shape
+Out[5]: (50, 6)
+
+In [10]: extrinsics_baseline_mounted[0]
+Out[10]: array([ 0.05111173,  0.22532253,  0.07174262, -0.00834507, -0.03635452,  0.20792226])
+
+In [11]: rt_cam_ref_true[0]
+Out[11]: array([0., 0., 0., 0., 0., 0.])
+
+In [12]: rt_cam_ref_true.shape
+Out[13]: (1, 6)
+
+'''
+
+
+
+
+
+
+
+
+
+
+
+
 q0_true = dict()
 for distance in args.distances:
 
     # shape (Ncameras, 2)
     q0_true_here = \
         reproject_perturbed(q0_baseline,
@@ -2401,13 +2525,13 @@ for icam in (0,3):
                                       atinfinity = True,
                                       method     = method,
                                       observed_pixel_uncertainty = args.observed_pixel_uncertainty )
 
     # I move the extrinsics of a model, write it to disk, and make sure the same
     # uncertainties come back
-    if True:
+    if False:
         model_moved = mrcal.cameramodel(models_baseline[icam])
         model_moved.rt_cam_ref([1., 2., 3., 4., 5., 6.])
         model_moved.write(f'{workdir}/out.cameramodel')
         model_read = mrcal.cameramodel(f'{workdir}/out.cameramodel')
 
         icam_intrinsics_read = model_read.icam_intrinsics()
#+end_src

Running:

#+begin_src sh
test/test-projection-uncertainty.py \
  --fixed frames \
  --model opencv4 \
  --Ncameras 1 \
  --range-to-boards 4 \
  --points
#+end_src

Doesn't work. q_baseline diverges a LOT from q_true. Something isn't right. This
is already implemented and working in test-surveyed-calibration --do-sample. So
the functionality is validated, it's the test that needs to be improved.

* C stereo unproject

* triangulate() should report stuff in the ref coords, not camera0 coords
It doesn't make a whole lot of sense the way I'm doing it right now

* mrcal c api stuff should mostly be double* instead of mrcal_point3_t

* m-c-c should be able to do extrinsics-only solves better
- I need to pass in intrinsics. Currently I do that with --seed, which is bad
  because:
  - It's ,-separated which breaks tab-completion
  - I don't want to seed the extrinsics. I want to load the intrinsics and use
    the normal seeding algorithm for extrinsics
- In this case I should be able to have non-matching --lensmodel


* release checklist
These are notes to myself containing the steps needed to roll a new release

- docs: make sure all new python functions are described in python.org
- test building with clang
- new [[file:versions.org][versions]]
- new [[file:news-2.2.org][news]]
- [[file:~/projects/mrcal/Makefile::PROJECT_NAME := mrcal][Makefile ABI version]]
- package build and upload
- versioned docs:
  - on the server move =docs-latest-release= symlink. This controls what =make publish-docs= writes to
  - on the server populate the =docs-latest-release/external= symlink
  - on the server move =docs-default= symlink. This controls what
    https://mrcal.secretsauce.net sees
- git tag
