#+TITLE: Making and using camera models with mrcal: the practical details
#+AUTHOR: Dima Kogan

#+OPTIONS: toc:nil H:2

#+LATEX_CLASS_OPTIONS: [presentation]

# Org adds this: \usepackage[T1]{fontenc} On my box this has the effect of
# asking for fonts that I don't have, which results in very ugly rendering with
# mupdf, where it uses bitmapped fonts, and scales them in ugly ways. Adding the
# below makes it pick the nice fonts
#+LaTeX_HEADER: \usepackage{lmodern}

#+LaTeX_HEADER: \setbeamertemplate{navigation symbols}{}

# I want clickable links to be blue and underlined, as is custom
#+LaTeX_HEADER: \usepackage{letltxmacro}
#+LaTeX_HEADER: \LetLtxMacro{\hreforiginal}{\href}
#+LaTeX_HEADER: \renewcommand{\href}[2]{\hreforiginal{#1}{\color{blue}{\underline{#2}}}}
#+LaTeX_HEADER: \renewcommand{\url}[1]{\href{#1}{\tt{#1}}}

# I want a visible gap between paragraphs
#+LaTeX_HEADER: \setlength{\parskip}{\bigskipamount}

* Overview
** What is mrcal for?
mrcal is a generic toolkit, applicable to anything that makes or uses camera
models.

Common applications:

- Calibration
- Triangulation
- Photogrammetry
- SFM

** Why does mrcal exist?

I couldn't find a set of tools precise-enough to make my visual ranging work
possible, so I wrote my own

By necessity, mrcal is a complete re-design and re-implementation of camera
modeling tooling from the ground-up:

- Richer camera model available to precisely model lens behavior
- Uncertainty propagation and cross-validation techniques available to
  validate a calibration
- No implicit pinhole assumption anywhere (no homogeneous coordinates,
  essential, fundamental matrices, etc)
- Fisheye-friendly stereo rectification

** Computing details

- mrcal is a library (C and Python)
- Many commandline tools available, so no coding required for common tasks
- /Thoroughly/ documented
- Open-source, available in stock Debian

Contributions welcome!

** Where is it?

Detailed documentation is available at https://mrcal.secretsauce.net/

- This talk is a repackaging of the documentation on that page

- See the docs for more detail, and for links to all the data and commands that
  produced the presented results

* Tour of mrcal
** Demo calibration
Let's start by looking at the "tour of mrcal":
https://mrcal.secretsauce.net/tour.html

We follow a real-world data flow, starting with chessboard observations. Images
captured using

- Sony Alpha 7 III full-frame SLR. 6000x3376 imager
- /Very/ wide lens: Samyang 12mm F2.8 fisheye. 180deg field of view
  corner-corner
- Just one camera (in this demo; mrcal supports multiple cameras)
- Outdoor images captured in downtown Los Angeles

** Sample image of the scene
#+ATTR_LATEX: :width 0.9\textwidth :height 0.7\textheight :options keepaspectratio
[[file:../out/external/figures/stereo/0.downsampled.jpg]]

** Gathering and detecting chessboard corners
We capture images, and use [[https://github.com/dkogan/mrgingham/][mrgingham]] to detect chessboard corners. For an
arbitrary image:

#+ATTR_LATEX: :width 0.9\textwidth :height 0.7\textheight :options keepaspectratio
[[file:../out/external/figures/calibration/mrgingham-results.png]]

* Tour of mrcal: =LENSMODEL_OPENCV8=
** Let's run a calibration!
This is a wide lens, so we need a lens model that can handle it. Let's use the
8-parameter OpenCV model: =LENSMODEL_OPENCV8=

#+begin_example
$ mrcal-calibrate-cameras --lensmodel LENSMODEL_OPENCV8 ...
...

RMS reprojection error: 0.4 pixels
Worst residual (by measurement): 1.8 pixels
Noutliers: 564 out of 16464 total points: 3.4% of the data
calobject_warp = [-0.00012726 -0.00014325]
#+end_example

** Why is the reprojection error > 0?
We have two sources of error:

- *Sampling error*: chessboard observations are noisy. We can study this, but we
  cannot reduce it

- *Model error*: our model of the lens behavior, chessboard shape and everything
  else isn't perfect. We can and must suppress this as much as possible

We want the model error to be negligible. If it is and if the sampling error is
normal and i.i.d., then we get a bias-free maximum-likelihood calibration result

_Patterns_ in the residuals indicate the presence of model errors

** Does the solved geometry look right?                            :noexport:

#+ATTR_LATEX: :width 0.9\textwidth :height 0.7\textheight :options keepaspectratio
[[file:../out/external/figures/calibration/calibration-chessboards-geometry-crop.pdf]]

Yes. That's how I danced.

** =LENSMODEL_OPENCV8= residuals histogram
What does the error distribution look like?

#+ATTR_LATEX: :width 0.9\textwidth :height 0.7\textheight :options keepaspectratio
[[file:../out/external/figures/calibration/residuals-histogram-opencv8-crop.pdf]]

** =LENSMODEL_OPENCV8= worst-observation residuals                 :noexport:
The worst-fitting observations are a great way to see common issues such as:

- out-of focus images
- images with motion blur
- rolling shutter effects
- synchronization errors
- chessboard detector failures
- insufficiently-rich models (of the lens or of the chessboard shape or anything
  else)

We look for _patterns_ in the residuals: patterns indicate the presence of model
errors

** =LENSMODEL_OPENCV8=: the worst image
#+ATTR_LATEX: :width 0.9\textwidth :height 0.7\textheight :options keepaspectratio
[[file:../out/external/figures/calibration/worst-opencv8.png]]

** =LENSMODEL_OPENCV8=: the worst image in a corner                :noexport:
#+ATTR_LATEX: :width 0.9\textwidth :height 0.7\textheight :options keepaspectratio
[[file:../out/external/figures/calibration/worst-incorner-opencv8.png]]

** =LENSMODEL_OPENCV8=: residual directions
#+ATTR_LATEX: :width 0.9\textwidth :height 0.7\textheight :options keepaspectratio
[[file:../out/external/figures/calibration/directions-opencv8-crop.pdf]]

** =LENSMODEL_OPENCV8=: conclusions
We see clear patterns in the residuals, so

- =LENSMODEL_OPENCV8= does not fit our data

Let's fix it.

* Tour of mrcal: =LENSMODEL_SPLINED_STEREOGRAPHIC=
** =LENSMODEL_SPLINED_STEREOGRAPHIC= definition                    :noexport:
- We need a more flexible lens model to represent our lens.
- mrcal currently supports a /splined/ model that is configurable to be as rich
  as we like

We compute a normalized /stereographic/ projection:

\[ \vec u = \mathrm{project}_\mathrm{stereographic}\left(\vec p\right) \]

This maps a 3D direction vector to a 2D point $\vec u$. This works behind the
camera, so wide-angle lenses are supported well.

** =LENSMODEL_SPLINED_STEREOGRAPHIC= definition                    :noexport:
Then use $\vec u$ to look-up an adjustment factor $\Delta \vec u$ using two
splined surfaces: one for each of the two elements of

\[ \Delta \vec u \equiv
\left[ \begin{aligned}
\Delta u_x \left( \vec u \right) \\
\Delta u_y \left( \vec u \right)
\end{aligned} \right] \]

We can then define the rest of the projection function:

\[\vec q =
 \left[ \begin{aligned}
 f_x \left( u_x + \Delta u_x \right) + c_x \\
 f_y \left( u_y + \Delta u_y \right) + c_y
\end{aligned} \right] \]

** Doing it again with =LENSMODEL_SPLINED_STEREOGRAPHIC=
Let's re-process the same calibration data using the splined model. We run the
same command as before, but using the =LENSMODEL_SPLINED_STEREOGRAPHIC_= ...
=order=3_Nx=30_Ny=18_fov_x_deg=150= model. This is one long string.

This model has 1084 parameters.

#+begin_example
$ mrcal-calibrate-cameras
    --lensmodel LENSMODEL_SPLINED_STEREOGRAPHIC_ ...
    ... order=3_Nx=30_Ny=18_fov_x_deg=150 ...
...
RMS reprojection error: 0.2 pixels
Worst residual (by measurement): 1.3 pixels
Noutliers: 28 out of 16464 total points: 0.2% of the data
calobject_warp = [-1.26851438e-04 -8.03269701e-05]
#+end_example

** =LENSMODEL_OPENCV8= residuals histogram

#+ATTR_LATEX: :width 0.9\textwidth :height 0.7\textheight :options keepaspectratio
[[file:../out/external/figures/calibration/residuals-histogram-opencv8-crop.pdf]]

** =LENSMODEL_SPLINED_...= residuals histogram

#+ATTR_LATEX: :width 0.9\textwidth :height 0.7\textheight :options keepaspectratio
[[file:../out/external/figures/calibration/residuals-histogram-splined-crop.pdf]]

** =LENSMODEL_OPENCV8= worst-observation residuals                 :noexport:
The worst-fitting observations are a great way to see common issues such as:

- out-of focus images
- images with motion blur
- rolling shutter effects
- synchronization errors
- chessboard detector failures
- insufficiently-rich models (of the lens or of the chessboard shape or anything
  else)

We look for _patterns_ in the residuals: patterns indicate the presence of model
errors

** =LENSMODEL_OPENCV8=: the worst image
#+ATTR_LATEX: :width 0.9\textwidth :height 0.7\textheight :options keepaspectratio
[[file:../out/external/figures/calibration/worst-opencv8.png]]

** =LENSMODEL_SPLINED_...=: the worst image
#+ATTR_LATEX: :width 0.9\textwidth :height 0.7\textheight :options keepaspectratio
[[file:../out/external/figures/calibration/worst-splined.png]]

** =LENSMODEL_OPENCV8=: the worst image in a corner                :noexport:
#+ATTR_LATEX: :width 0.9\textwidth :height 0.7\textheight :options keepaspectratio
[[file:../out/external/figures/calibration/worst-incorner-opencv8.png]]

** =LENSMODEL_SPLINED_...=: the worst image in a corner            :noexport:
#+ATTR_LATEX: :width 0.9\textwidth :height 0.7\textheight :options keepaspectratio
[[file:../out/external/figures/calibration/worst-incorner-splined.png]]

** =LENSMODEL_OPENCV8=: residual directions
#+ATTR_LATEX: :width 0.9\textwidth :height 0.7\textheight :options keepaspectratio
[[file:../out/external/figures/calibration/directions-opencv8-crop.pdf]]
** =LENSMODEL_SPLINED_...=: residual directions
#+ATTR_LATEX: :width 0.9\textwidth :height 0.7\textheight :options keepaspectratio
[[file:../out/external/figures/calibration/directions-splined-crop.pdf]]

** Conclusion
We have good evidence that =LENSMODEL_SPLINED_STEREOGRAPHIC= fits this lens much
better than =LENSMODEL_OPENCV8=

* Differencing
** Differencing
We computed the calibration two different ways. How different are the two
models?

** Differencing                                                    :noexport:
Let's compute the difference using an obvious algorithm:

Given a pixel $\vec q_0$,

- Unproject $\vec q_0$ to a fixed point $\vec p$ using lens 0
- Project $\vec p$ back to pixel coords $\vec q_1$ using lens 1
- Report the reprojection difference $\vec q_1 - \vec q_0$

#+ATTR_LATEX: :width 0.9\textwidth :height 0.7\textheight :options keepaspectratio
[[file:../out/figures/diff-notransform.pdf]]

** Differencing                                                    :noexport:
#+ATTR_LATEX: :width 0.9\textwidth :height 0.7\textheight :options keepaspectratio
[[file:../out/external/figures/diff/diff-radius0-heatmap-splined-opencv8-crop.pdf]]

** Differencing                                                    :noexport:
#+ATTR_LATEX: :width 0.9\textwidth :height 0.7\textheight :options keepaspectratio
[[file:../out/external/figures/diff/diff-radius0-vectorfield-splined-opencv8-crop.pdf]]

** Differencing                                                    :noexport:
So with a motion of the camera, we can make the errors disappear.

The issue is that each calibration produces noisy estimates of all the
intrinsics and all the coordinate transformations:

[[file:../out/figures/uncertainty.pdf]]

And the point $\vec p$ we were projecting wasn't truly fixed.

** Differencing                                                    :noexport:
We want to add a step:

- Unproject $\vec q_0$ to a fixed point $\vec p_0$ using lens 0
- *Transform $\vec p_0$ from the coordinate system of one camera to the coordinate
  system of the other camera*
- Project $\vec p_1$ back to pixel coords $\vec q_1$ using lens 1
- Report the reprojection difference $\vec q_1 - \vec q_0$

[[file:../out/figures/diff-yestransform.pdf]]

** Differencing
#+ATTR_LATEX: :width 0.9\textwidth :height 0.7\textheight :options keepaspectratio
[[file:../out/external/figures/diff/diff-splined-opencv8-crop.pdf]]

** Differencing                                                    :noexport:
/Much/ better. As expected, the two models agree relatively well in the center,
and the error grows as we move towards the edges.

This differencing method has numerous applications:

- evaluating the manufacturing variation of different lenses
- quantifying intrinsics drift due to mechanical or thermal stresses
- testing different solution methods
- underlying a cross-validation scheme

* Uncertainty
** Uncertainty
- All calibrations are based on observations of the calibration object
  (chessboard corners)
- These observations /always/ contain some noise (sampling error)
- A calibration result is trustworthy /only/ if it is insensitive to this noise

We quantify this sensitivity by computing a _projection uncertainty_

** Uncertainty                                                     :noexport:
When we project a point $\vec p$ to a pixel $\vec q$, it would be /really/ nice
to get an uncertainty estimate $\mathrm{Var} \left(\vec q\right)$. The we could

- Propagate this uncertainty downstream to whatever uses the projection
  operation, for example to get the uncertainty of ranges from a triangulation
- Evaluate how trustworthy a given calibration is, and to run studies about how
  to do better
- Quantify overfitting effects
- Quantify the baseline noise level for informed interpretation of model
  differences

Since splined models can have 1000s of parameters (the one we just demoed has
1084), they are prone to overfitting, and it's critically important to gauge
those effects.

** Uncertainty                                                     :noexport:
A grand summary of how we do this:

1. We are assuming a particular distribution of observation input noise
   $\mathrm{Var}\left( \vec q_\mathrm{ref} \right)$
2. We propagate it through the optimization to get the variance of the
   optimization state $\mathrm{Var}(\vec b)$
3. For any /fixed/ point, its projection $\vec q = \mathrm{project}\left(
   \mathrm{transform}\left( \vec p_\mathrm{fixed} \right)\right)$ depends on
   parameters of $\vec b$, whose variance we know. So

\[ \mathrm{Var}\left( \vec q \right) =
\frac{\partial \vec q}{\partial \vec b}
\mathrm{Var}\left( \vec b \right)
\frac{\partial \vec q}{\partial \vec b}^T
\]

** Uncertainty simulation                                          :noexport:
The mrcal test suite contains a simulation to validate the approach.

- 4 cameras
- Placed side by side + noise in pose
- =LENSMODEL_OPENCV4= lens model
- looking at 50 chessboard poses, with randomized pose

** Uncertainty simulation                                          :noexport:
The geometry looks like this:

#+ATTR_LATEX: :width 0.9\textwidth :height 0.7\textheight :options keepaspectratio
[[file:../out/external/figures/uncertainty/simulated-uncertainty-opencv4--simulated-geometry-crop.pdf]]

** Uncertainty simulation                                          :noexport:
Each camera sees this:

#+ATTR_LATEX: :width 0.9\textwidth :height 0.7\textheight :options keepaspectratio
[[file:../out/external/figures/uncertainty/simulated-uncertainty-opencv4--simulated-observations-crop.pdf]]

The red *$\ast$* is a point we will examine.

** Uncertainty simulation                                          :noexport:
#+ATTR_LATEX: :width 0.9\textwidth :height 0.7\textheight :options keepaspectratio
[[file:../out/external/figures/uncertainty/simulated-uncertainty-opencv4--distribution-onepoint-crop.pdf]]

** Uncertainty simulation                                          :noexport:
Let's look at the uncertainty everywhere in the imager

#+ATTR_LATEX: :width 0.9\textwidth :height 0.7\textheight :options keepaspectratio
[[file:../out/external/figures/uncertainty/simulated-uncertainty-opencv4--uncertainty-wholeimage-noobservations-crop.pdf]]

This confirms the expectation: the sweet spot of low uncertainty follows the
region where the chessboards were

** Uncertainty simulation                                          :noexport:
- The worst uncertainty-at-*$\ast$* camera claims an uncertainty of 0.8 pixels.
  That's pretty low. But we had no chessboard observations there; is this
  uncertainty realistic? _No_

- =LENSMODEL_OPENCV4= is stiff, so the projection doesn't move much due to
  noise. And we interpreted that as low uncertainty. But that comes from our
  choice of model, and /not/ from the data. So

*Lean models always produce overly-optimistic uncertainty estimates*

Solution: use splined models! They are very flexible, and don't have this issue.

** Uncertainty simulation                                          :noexport:
Running the same simulation with a splined model, we see the /real/ projection
uncertainty:

#+ATTR_LATEX: :width 0.9\textwidth :height 0.7\textheight :options keepaspectratio
[[file:../out/external/figures/uncertainty/simulated-uncertainty-splined--uncertainty-wholeimage-noobservations-crop.pdf]]

So /only/ the first camera actually had usable projections.

** Uncertainty simulation                                          :noexport:
Let's overlay the observations:

#+ATTR_LATEX: :width 0.9\textwidth :height 0.7\textheight :options keepaspectratio
[[file:../out/external/figures/uncertainty/simulated-uncertainty-splined--uncertainty-wholeimage-observations-crop.pdf]]

** Uncertainty from the DTLA data
Computing the uncertainty map from the earlier =LENSMODEL_OPENCV8= calibration:

#+ATTR_LATEX: :width 0.9\textwidth :height 0.7\textheight :options keepaspectratio
[[file:../out/external/figures/uncertainty/uncertainty-opencv8-crop.pdf]]
** Uncertainty from the DTLA data
And from the =LENSMODEL_SPLINED_STEREOGRAPHIC_...= calibration:

#+ATTR_LATEX: :width 0.9\textwidth :height 0.7\textheight :options keepaspectratio
[[file:../out/external/figures/uncertainty/uncertainty-splined-crop.pdf]]

** Uncertainty conclusion                                          :noexport:
The splined model promises double the uncertainty that =LENSMODEL_OPENCV8= does.

Conclusions:

- We have a usable uncertainty-quantification method
- It is over-optimistic when applied to lean models

So splined models have a clear benefit even for long lenses, where the lean
models are expected to fit

* Ranging note
** Ranging note
Let's revisit an important detail I glossed-over when talking about differencing
and uncertainties. Both computations begin with $\vec p =
\mathrm{unproject}\left( \vec q \right)$

But an unprojection is ambiguous in range, so *diffs and uncertainties are
defined as a function of range*

#+ATTR_LATEX: :width 0.9\textwidth :height 0.7\textheight :options keepaspectratio
[[file:../out/figures/projection-scale-invariance.pdf]]

All the uncertainties reported so far were at $\infty$

** The uncertainty figure
The uncertainty of our =LENSMODEL_OPENCV8= calibration at the center as a
function of range:

#+ATTR_LATEX: :width 0.9\textwidth :height 0.7\textheight :options keepaspectratio
[[file:../out/external/figures/uncertainty/uncertainty-vs-distance-at-center-crop.pdf]]

* Let's apply these techniques
** Let's apply these techniques
We described several analysis techniques:

- Visualizing the solve residuals
- Computing projection differences between two models
- Evaluating projection uncertainty

Let's use these to answer practical questions

** Optimal choreography overview
For many of the following analyses we study the effects of sampling error. We

- Set up a simulated world with some baseline geometry
- Scan some parameter
- Calibrate
- Look at the uncertainty-vs-range plots as a function of that parameter

* What kind of calibration object do we want?
** How dense should our chessboard be?
#+ATTR_LATEX: :width 0.9\textwidth :height 0.7\textheight :options keepaspectratio
[[file:../out/external/figures/dance-study/dance-study-scan-object_width_n-crop.pdf]]

** What should the chessboard corner spacing be?
#+ATTR_LATEX: :width 0.9\textwidth :height 0.7\textheight :options keepaspectratio
[[file:../out/external/figures/dance-study/dance-study-scan-object_spacing-crop.pdf]]

** Do we want tiny boards nearby or giant boards faraway?
#+ATTR_LATEX: :width 0.9\textwidth :height 0.7\textheight :options keepaspectratio
[[file:../out/external/figures/dance-study/dance-study-scan-object_spacing-compensated-range-crop.pdf]]

** How far should the chessboards be placed?
#+ATTR_LATEX: :width 0.9\textwidth :height 0.7\textheight :options keepaspectratio
[[file:../out/external/figures/dance-study/dance-study-scan-range-crop.pdf]]

** How many chessboard observations should we get?
#+ATTR_LATEX: :width 0.9\textwidth :height 0.7\textheight :options keepaspectratio
[[file:../out/external/figures/dance-study/dance-study-scan-Nframes-crop.pdf]]

** What kind of calibration object do we want? Guidelines
- More data is good
  - More chessboard corners
  - More chessboard observations
- The chessboard should fill the imager
  - Close-ups
  - Big chessboards

Questions:

- So what kind of calibration object do we want? Are /chessboards/ the right
  choice?

- Should we place the chessboard immediately in front of the lens? Should we use
  a /giant/ chessboard?

** Chessboards? Circles? AprilTags? Charuco?
mrcal doesn't care!

- Grids of circles (and possibly AprilTags) don't directly observe the point, so
  they could be biased. mrcal has a visual validation tool:
  =mrcal-reproject-to-chessboard= that produces a [[file:~/projects/mrcal/doc/out/external/figures/reprojected-to-chessboard/reprojected-to-chessboard.mp4][validation sequence]]
- Anything with AprilTags needs a high-resolution-enough image to resolve the
  AprilTag. This resolution could instead be used to cram extra chessboard
  squares into the image

I use chessboards with the mrgingham detector

** The downsides of extreme closeups
*** Corners out of focus
- If the blur is unbiased and gaussian: this will increase the noise, but we can
  compensate by gathering more data
- It looks like the blur mostly /is/ unbiased and gaussian, but don't push it

*** Noncentral effects become significant
Core assumption of almost all camera modeling and processing:

- All rays of light intersect at a single point

*This is not a valid assumption near the lens*

** Noncentrality
The size of the glass in the lens becomes non-negligible as we observe nearby
objects

#+ATTR_LATEX: :width 0.9\textwidth :height 0.7\textheight :options keepaspectratio
[[file:../out/figures/noncentral.pdf]]

** Noncentrality
- Most triangulation and stereo routines assume a central projection. This is
  true for non-closeups

- If necessary, noncentral behavior /can/ be modeled:
  - mrcal has partial support, which was critically important for some projects
  - CAHVORE is noncentral with most people throwing away the noncentrality when
    they use it

- We should try to calibrate and use the cameras beyond where noncentral effects
  are significant. mrcal cross-validation will tell you if you're too close.

** Noncentrality                                                   :noexport:
There's limited awareness that this is what's being done. I've seen people carry
around code like this to centralize CAHVORE unprojections:

#+begin_src c
  cmod_2d_to_3d(p_in, &(model.core), ow, nw, NULL, NULL);
  udmxs_(nw, nw, 1000000, 3);  // nw = 1000000*nw
  udadd_(nw, ow, nw, 3);       // nw = ow + nw
#+end_src

Don't do it: just set $\vec E = 0$

** The downsides of huge chessboards
- Difficult to manufacture
- Expensive
- Unstable

mrcal has a simple /static/ deformation model: a parabolic deformation in x and
in y. *Usually this isn't enough to accurately represent foam boards*

** The downsides of huge chessboards
Because *intrinsics are sensitive to chessboard shape errors*. Simulated
intrinsics calibration error due to a board shape error of 1mm in the center in
one direction, and 0.5mm in the center in the other direction. No other noise
present.

#+ATTR_LATEX: :width 0.9\textwidth :height 0.7\textheight :options keepaspectratio
[[file:../out/external/figures/residuals/unstable-chessboard-shape-diff-crop.pdf]]

** The downsides of huge chessboards
I usually use an Aluminum-honeycomb-backed 1m x 1m square board. This works
well.

** What kind of calibration object do we want? Conclusions
- Chessboard as large as possible
- Placed as close to the camera as possible
- With as dense a chessboard grid as possible

Using the mrcal tools to verify that we didn't go too far

* How should we capture images?                                    :noexport:
** How should we capture images?
- Same physical settings as when using the system: aperture size, zoom, focus
- Features should be in-focus: trade-off with uncertainty requirements from
  before. Some out-of-focus blur is ok
- Self-consistent timing
  - No motion blur. Use a tripod or capture images with lots of light: doing it
    outside is great
  - No rolling shutter effects. If you have such a camera, use a tripod
  - If calibrating multiple cameras, hardware sync is a requirement

mrcal will clearly identify these issues, if they're present in the data

* How should we dance?
** How much should we tilt the chessboards?                        :noexport:
We already saw that we want

- Closeups
- Lots of images

Should we tilt the chessboards?

#+ATTR_LATEX: :width 0.9\textwidth :height 0.6\textheight :options keepaspectratio
[[file:../out/external/figures/dance-study/dance-study-scan-tilt_radius-crop.pdf]]

** How should we dance? Conclusions

#+ATTR_LATEX: :width 0.9\textwidth :height 0.4\textheight :options keepaspectratio
[[file:../out/figures/observation-usefulness.pdf]]

Use mrcal tools to validate

* Which model should we use for the lenses?
** Which model should we use for the lenses?
Today mrcal supports

- OpenCV models with 4,5,8,12 parameters
- CAHVOR, CAHVORE
- =LENSMODEL_SPLINED_STEREOGRAPHIC=: the rich, splined model

Unless you really need compatibility with a legacy system or you have low
accuracy requirements, *=LENSMODEL_SPLINED_STEREOGRAPHIC= is strongly
recommended*.

* Calibrating                                                      :noexport:
** Computing the calibration
We are ready to compute the calibration!

- Run the =mrcal-calibrate-cameras= tool
- For cross-validation you want to split your data into 2 (or more) independent
  sets, and process those independently
- If using a lesser lens model, calibrate both with the model you selected, and
  with =LENSMODEL_SPLINED_STEREOGRAPHIC=. Use that to see how much error you get
  from your model choice

* Results interpretation
** Interpreting the calibration results

Once we have a calibration, we should see how well we did:

- We examine the projection uncertainty to make sure we have enough good data in
  the right places
- We examine the cross-validation diffs to confirm that the model fits
- If these diffs are too high, we examine the residuals to find the cause of our
  model errors

* Uncertainty
** Projection uncertainty
- Projection uncertainty gauges the effect of sampling error
- This is directly affected by the quality of the data we gathered. Problems
  with the chessboard dance will show up here
- Lean lens models (anothing other than =LENSMODEL_SPLINED_STEREOGRAPHIC=) will
  produce an overly-optimistic uncertainty report
- *A low projection uncertainty is a necessary, but not sufficient condition for
  a good calibration*: uncertainty reporting samples the input pixel noise, but
  not the model noise

If the uncertainty is unacceptable, stop there, and fix that first.

** DTLA projection uncertainty: OPENCV8

#+ATTR_LATEX: :width 0.9\textwidth :height 0.7\textheight :options keepaspectratio
[[file:../out/external/figures/uncertainty/uncertainty-opencv8-crop.pdf]]

** DTLA projection uncertainty: splined model

#+ATTR_LATEX: :width 0.9\textwidth :height 0.7\textheight :options keepaspectratio
[[file:../out/external/figures/uncertainty/uncertainty-splined-crop.pdf]]

* Cross-validation
** Cross-validation diffs
Now we look for model errors

- We split our input dataset, and process the subsets independently: this
  samples the model error
- We use the differencing method to compare the projection behaviors
- Unlike the uncertainty reporting, interpreting these requires some thought

** Cross-validation diffs: detecting model errors
I want to see

\[E_{\mathrm{uncertainty}_0} + E_{\mathrm{uncertainty}_1} \approx E_{\mathrm{crossvalidation}}\]

Let's look at the downtown LA data. We want to see a cross-validation diff of ~
0.2 pixels.

** DTLA cross-validation diffs: OPENCV8
#+ATTR_LATEX: :width 0.9\textwidth :height 0.7\textheight :options keepaspectratio
[[file:../out/external/figures/cross-validation/diff-cross-validation-opencv8-crop.pdf]]

** DTLA cross-validation diffs: splined model
#+ATTR_LATEX: :width 0.9\textwidth :height 0.7\textheight :options keepaspectratio
[[file:../out/external/figures/cross-validation/diff-cross-validation-splined-crop.pdf]]

** DTLA cross-validation diffs
- Clearly the =LENSMODEL_OPENCV8= result has issues
- But the =LENSMODEL_SPLINED_STEREOGRAPHIC= result has too-high errors too

Because I captured images from too close to the lens, and we're seeing
non-negligible noncentral behavior. Asking mrcal to model that behavior
produces:

** DTLA cross-validation diffs: splined model, noncentral
#+ATTR_LATEX: :width 0.9\textwidth :height 0.7\textheight :options keepaspectratio
[[file:../out/external/figures/cross-validation/diff-cross-validation-splined-noncentral-crop.pdf]]

** DTLA cross-validation diffs
- If this calibration was important, I would get a different dataset from
  further out

** DTLA cross-validation diffs
- Here the cross-validation diffs alerted us to the presense of a problem. They
  are /very/ good at that

- Finding the cause of the problem requires some intuition and experimentation

* Residuals
** Residuals
- One technique is available to help diagnose problems: examining the solve
  residuals

** Residuals

We usually have a /lot/ of images and a /lot/ of residuals. I look at the few
worst-fitting images. Usually I only look at the residuals if

- I'm calibrating an unfamiliar system
- I don't trust something about the way the data was collected
- Something unknown is causing issues (we're seeing too-high cross-validation
  diffs), and we need to debug

Model errors are indicated with noise that is correlated or heteroscedastic, so
*we look for patterns in the residuals*.

Let's examine the residuals we get from common problems

** Residuals: poorly-fitting lens model
We saw this in the downtown Los Angeles data

- We looked at both the =LENSMODEL_OPENCV8= and
  =LENSMODEL_SPLINED_STEREOGRAPHIC= residuals

- The latter was much better, but still showed patterns

Earlier residual plots follow below

** Residuals: =LENSMODEL_OPENCV8=: the worst image
#+ATTR_LATEX: :width 0.9\textwidth :height 0.7\textheight :options keepaspectratio
[[file:../out/external/figures/calibration/worst-opencv8.png]]

** Residuals: =LENSMODEL_OPENCV8=: residual directions
#+ATTR_LATEX: :width 0.9\textwidth :height 0.7\textheight :options keepaspectratio
[[file:../out/external/figures/calibration/directions-opencv8-crop.pdf]]

** Residuals: rolling shutter

Some cameras save money on memory by sending pixel data as it is captured. The
result: *rolling shutter cameras capture different parts of the image at
different times*.

This produces funky residuals

** Residuals: rolling shutter
#+ATTR_LATEX: :width 0.9\textwidth :height 0.7\textheight :options keepaspectratio
[[file:../out/external/figures/residuals/rolling-shutter-0.pdf]]

** Residuals: rolling shutter
#+ATTR_LATEX: :width 0.9\textwidth :height 0.7\textheight :options keepaspectratio
[[file:../out/external/figures/residuals/rolling-shutter-2.pdf]]

** Residuals: rolling shutter
#+ATTR_LATEX: :width 0.9\textwidth :height 0.7\textheight :options keepaspectratio
[[file:../out/external/figures/residuals/rolling-shutter-3.pdf]]

** Residuals: syncronization errors
- In a multi-camera calibration we assume that sets of images were captured at
  the same instant in time
- This requires a shared physical wire that each camera uses to initiate image
  capture

If this doesn't work right we get the tell-tale residuals, and we can examine
the solution to find the smoking-gun images that prove the breakage

** Residuals: syncronization errors
#+ATTR_LATEX: :width 0.9\textwidth :height 0.7\textheight :options keepaspectratio
[[file:../out/external/figures/residuals/sync-errors-0.pdf]]

** Residuals: syncronization errors
#+ATTR_LATEX: :width 0.9\textwidth :height 0.7\textheight :options keepaspectratio
[[file:../out/external/figures/residuals/sync-errors-1.pdf]]

** Residuals: chessboard shape errors
- Errors in chessboard shape are difficult to disentangle from errors in
  intrinsics
- We can have static and dynamic shape errors:
  1. The chessboard is non-flat, but in a way not modeled by the solver
  2. The chessboard shape changes over the course of the chessboard dance

** Residuals: errors due to unmodeled chessboard shape             :noexport:

#+ATTR_LATEX: :width 0.9\textwidth :height 0.7\textheight :options keepaspectratio
[[file:../out/external/figures/residuals/unmodeled-chessboard-shape.pdf]]

** Residuals: errors due to unstable chessboard shape

#+ATTR_LATEX: :width 0.9\textwidth :height 0.7\textheight :options keepaspectratio
[[file:../out/external/figures/residuals/unstable-chessboard-shape.pdf]]

** Residuals: chessboard shape errors. Conclusions
- These are hard to conclusively pick out from residual plots
- It's helpful to look at more than just 1 or 2 worst-case images
- The most tilted chessboard observations usually show very consistent residual
  vectors along the far edge of the chessboard

** Perfectly-corrupted solves
mrcal can report the errors from a solve containing /only one kind/ of
hypothetical error. This measures the effect of problems we think may exist

A board shape error of 1mm in the center in one direction, and 0.5mm in the
center in the other direction does this:

#+ATTR_LATEX: :width 0.9\textwidth :height 0.7\textheight :options keepaspectratio
[[file:../out/external/figures/residuals/unstable-chessboard-shape-diff-crop.pdf]]

* Recipes
** Camera stability
Let's switch gears, and look at some applications

Let's use the differencing method to gauge stability of intrinsics:

- If we stress a camera system (mechanically, thermally, etc), does its behavior
  change?

** Lens stability
As a baseline, once again here's the cross-validation diff from the downtown Los
Angeles dataset. This is the /difference between two subsequent solves without
touching anything/

#+ATTR_LATEX: :width 0.9\textwidth :height 0.7\textheight :options keepaspectratio
[[file:../out/external/figures/cross-validation/diff-cross-validation-splined-noncentral-crop.pdf]]

** Lens stability
- Then I moved the camera and tripod over by 2m or so, and gathered more
  chessboard images. Comparison from before:

#+ATTR_LATEX: :width 0.9\textwidth :height 0.7\textheight :options keepaspectratio
[[file:../out/external/figures/lens-stability/diff-dance34-splined-noncentral-crop.pdf]]

* Interoperating with other tools                                  :noexport:
** Interoperating with other tools
What if your system is built around CAHVOR, but you want to use
=LENSMODEL_SPLINED_STEREOGRAPHIC= for more accuracy?

- mrcal libraries are easy to use and integrate: it's not a lot of work to
  transition

But if you really don't want to....

** What is the cost of a too-lean model?
mrcal gives us tools to quantify this.

The usual process is:

- Calibrate using the production model
- Look at the projection uncertainty and cross-validation plots to estimate the
  expected calibration errors

** What is the cost of a too-lean model?
With the =LENSMODEL_SPLINED_STEREOGRAPHIC= we can go one step further:

- /Also/ calibrate using =LENSMODEL_SPLINED_STEREOGRAPHIC=
- Use the metrics to show that this fits very well
- Use =LENSMODEL_SPLINED_STEREOGRAPHIC= as the ground truth to get more
  informative diffs

** Having our cake and eating it too
What if we want the accuracy of =LENSMODEL_SPLINED_STEREOGRAPHIC= but we also
don't want to rearchitect our code? With a bit of extra computation we can.

- We have a $\left(\mathrm{model},\mathrm{image}\right)$ tuple that faithfully
  describes the world, if we use =LENSMODEL_SPLINED_STEREOGRAPHIC=
- We can generate another tuple
  $\left(\mathrm{model}_\mathrm{production},\mathrm{image}_\mathrm{reprojected}\right)$
  that reprojects the input image to one that follows the production model. We
  use this tuple everywhere, and it /also/ faithfully describes the world
- Pixel noise variances will need to be adjusted

* Model evaluation
** Estimating ranging errors caused by calibration errors
- Projection errors aren't what we ultimately care about

- mrcal allows us to propagate these to what we care about

Propagating errors to triangulation:

#+ATTR_LATEX: :width 0.9\textwidth :height 0.7\textheight :options keepaspectratio
[[file:../out/external/figures/triangulation/sample--ellipses-crop.pdf]]

** Scene-aware error propagation
If the rough geometry of an observed scene is known beforehand, we can make a
rough expected-error map:

1. Compute $\frac{\partial\mathrm{range}}{\partial\mathrm{azimuth}}$ from the
   triangulation expression
2. Estimate $\Delta \mathrm{azimuth}$ by combining expected sampling error and
   calibration error
3. $\Delta \mathrm{range} \approx \frac{\partial\mathrm{range}}{\partial\mathrm{azimuth}} \Delta \mathrm{azimuth}$

** Scene-aware error propagation
In the downtown Los Angeles scene we observed this calibration error using
=LENSMODEL_OPENCV8=:

#+ATTR_LATEX: :width 0.9\textwidth :height 0.7\textheight :options keepaspectratio
[[file:../out/external/figures/diff/diff-splined-opencv8-crop.pdf]]

** Scene-aware error propagation
- Rough median of calibration error: 0.5 pixels per camera
- At worst: 1.0 pixels of calibration error
- Noise in stereo matching is ~ 0.3 pixels

So we assume a disparity error of 1.0 + 0.3 = 1.3 pixels

** Scene-aware error propagation: left-rectified image
#+ATTR_LATEX: :width 0.9\textwidth :height 0.7\textheight :options keepaspectratio
[[file:../out/external/figures/stereo-range-sensitivity/0-rectified.downsampled.png]]

** Scene-aware error propagation: disparity
#+ATTR_LATEX: :width 0.9\textwidth :height 0.7\textheight :options keepaspectratio
[[file:../out/external/figures/stereo-range-sensitivity/disparity-crop.pdf]]

** Scene-aware error propagation: propagated range error
#+ATTR_LATEX: :width 0.9\textwidth :height 0.7\textheight :options keepaspectratio
[[file:../out/external/figures/stereo-range-sensitivity/sensitivity-crop.pdf]]

** Scene-aware error propagation
These errors are correlated and will /not/ average out. They should be
minimized.

* Conclusion
** Conclusion
- mrcal solves many pervasive issues in traditional camera-modeling toolkits
- Allows many practical questions to be addressed directly
- Many improvements and extensions and applications planned and in development

* Stereo                                                           :noexport:
** Overview
mrcal can do some basic stereo processing. At its core, it's the usual epipolar
geometry process:

1. Ingest two camera models
2. Ingest images captured by these two cameras
3. Transform the images to construct "rectified" images
4. Perform "stereo matching"

Each pair of corresponding rows in the rectified images represents a plane in
space:

#+ATTR_LATEX: :width 0.9\textwidth :height 0.4\textheight :options keepaspectratio
[[file:../out/figures/rectification.pdf]]

** Input images
I used the lens I calibrated at the start to capture a pair of images in
downtown Los Angeles. The left image:

#+ATTR_LATEX: :width 0.9\textwidth :height 0.7\textheight :options keepaspectratio
[[file:../out/external/figures/stereo/0.downsampled.jpg]]

We're on a catwalk between 2nd and 3rd, looking S over Figueroa St.

** Rectification
I then used mrcal's rectification function to produce the rectified image. The
left:

#+ATTR_LATEX: :width 0.9\textwidth :height 0.7\textheight :options keepaspectratio
file:../out/external/figures/stereo/0-rectified.downsampled.png

** Disparity
And the resulting disparity, as computed by the OpenCV matcher:

#+ATTR_LATEX: :width 0.9\textwidth :height 0.7\textheight :options keepaspectratio
file:../out/external/figures/stereo/0-disparity.downsampled.png

** JPLV

What if we wanted to use JPLV stereo with splined models?

We can use mrcal to remap to another projection and feed /that/ to jplv. For
instance, let's

- Remap to a pinhole model (with some arbitrary zoom factor)
- Use jplv to compute the rectified image

** Narrow virtual cameras
Another way to do stereo processing of wide images using tools that aren't built
for it is to

- split the wide-angle stereo pair into a set of narrow-view stereo pairs

This generates a skewed geometry, but mrcal can still use it just fine. Due to a
bug, jplv cannot.

** Narrow virtual cameras
#+ATTR_LATEX: :width 0.9\textwidth :height 0.7\textheight :options keepaspectratio
file:../out/external/figures/stereo/stereo-geometry-narrow.pdf

** Narrow virtual cameras
One of the resulting resampled /pinhole/ images:

#+ATTR_LATEX: :width 0.9\textwidth :height 0.7\textheight :options keepaspectratio
file:../out/external/figures/stereo/narrow-left.downsampled.jpg

** Narrow virtual cameras
Rectified using mrcal

#+ATTR_LATEX: :width 0.9\textwidth :height 0.7\textheight :options keepaspectratio
file:../out/external/figures/stereo/rectified0-narrow.downsampled.jpg

** Narrow virtual cameras
Disparity from OpenCV

#+ATTR_LATEX: :width 0.9\textwidth :height 0.7\textheight :options keepaspectratio
file:../out/external/figures/stereo/disparity-narrow.downsampled.png

