[ Camera calibration ] NaNs in camera matrix after intrinsics initialization

[vonaviv]

Hello,

I am having a problem with camera calibration for some sessions. The equivalent calibration procedure works for most sessions, but for some this error appears. I can not identify any significant difference in the calibration videos for this session in comparison to the ones without the error. I only see that for Cam_03 there are some NaNs in a matrix .... 'name': 'Cam_03', 'size': [1280, 1024], 'matrix': [[nan, 0.0, 639.5], [0.0, nan, 511.5], [0.0, 0.0, 1.0]] .... As you can see below there are plenty of board detection in this specific camera, so I am not sure what is causing this behavior.

Computing calibration for: calib_20230524
Using cameras: ['Cam_03', 'Cam_04', 'Cam_10', 'Cam_11', 'Cam_14']
Cam_03.mp4
100%|███████████████████████████| 17776/17776 [03:28<00:00, 85.38it/s]
1639 boards detected
Cam_04.mp4
100%|███████████████████████████| 17776/17776 [04:09<00:00, 71.32it/s]
1818 boards detected
Cam_10.mp4
100%|███████████████████████████| 17776/17776 [05:18<00:00, 55.88it/s]
6489 boards detected
Cam_11.mp4
100%|███████████████████████████| 17776/17776 [03:53<00:00, 76.09it/s]
1686 boards detected
Cam_14.mp4
100%|███████████████████████████| 17776/17776 [03:00<00:00, 98.42it/s]
700 boards detected
[{'name': 'Cam_03', 'size': [1280, 1024], 'matrix': [[nan, 0.0, 639.5], [0.0, nan, 511.5], [0.0, 0.0, 1.0]], 'distortions': [0.0, 0.0, 0.0, 0.0, 0.0], 'rotation': [0.0, 0.0, 0.0], 'translation': [0.0, 0.0, 0.0]}, {'name': 'Cam_04', 'size': [1280, 1024], 'matrix': [[1104.7618050259102, 0.0, 639.5], [0.0, 1104.7618050259102, 511.5], [0.0, 0.0, 1.0]], 'distortions': [0.0, 0.0, 0.0, 0.0, 0.0], 'rotation': [0.0, 0.0, 0.0], 'translation': [0.0, 0.0, 0.0]}, {'name': 'Cam_10', 'size': [1280, 1024], 'matrix': [[752.3003203201827, 0.0, 639.5], [0.0, 752.3003203201827, 511.5], [0.0, 0.0, 1.0]], 'distortions': [0.0, 0.0, 0.0, 0.0, 0.0], 'rotation': [0.0, 0.0, 0.0], 'translation': [0.0, 0.0, 0.0]}, {'name': 'Cam_11', 'size': [1280, 1024], 'matrix': [[772.6256344956118, 0.0, 639.5], [0.0, 772.6256344956118, 511.5], [0.0, 0.0, 1.0]], 'distortions': [0.0, 0.0, 0.0, 0.0, 0.0], 'rotation': [0.0, 0.0, 0.0], 'translation': [0.0, 0.0, 0.0]}, {'name': 'Cam_14', 'size': [1280, 1024], 'matrix': [[780.5477950413123, 0.0, 639.5], [0.0, 780.5477950413123, 511.5], [0.0, 0.0, 1.0]], 'distortions': [0.0, 0.0, 0.0, 0.0, 0.0], 'rotation': [0.0, 0.0, 0.0], 'translation': [0.0, 0.0, 0.0]}]
defaultdict(<class 'int'>,
            {('Cam_03', 'Cam_04'): 462,
             ('Cam_03', 'Cam_10'): 73,
             ('Cam_04', 'Cam_03'): 462,
             ('Cam_04', 'Cam_10'): 274,
             ('Cam_10', 'Cam_03'): 73,
             ('Cam_10', 'Cam_04'): 274,
             ('Cam_10', 'Cam_11'): 1044,
             ('Cam_10', 'Cam_14'): 530,
             ('Cam_11', 'Cam_10'): 1044,
             ('Cam_11', 'Cam_14'): 180,
             ('Cam_14', 'Cam_10'): 530,
             ('Cam_14', 'Cam_11'): 180})
Traceback (most recent call last):
  File "camera_calibration.py", line 67, in <module>
    cgroup.calibrate_videos(videos, board)
  File "/home/vivanov/anaconda3/envs/anipose/lib/python3.7/site-packages/aniposelib/cameras.py", line 1653, in calibrate_videos
    verbose=verbose, **kwargs)
  File "/home/vivanov/anaconda3/envs/anipose/lib/python3.7/site-packages/aniposelib/cameras.py", line 1610, in calibrate_rows
    rvecs, tvecs = get_initial_extrinsics(rtvecs, self.get_names())
  File "/home/vivanov/anaconda3/envs/anipose/lib/python3.7/site-packages/aniposelib/utils.py", line 180, in get_initial_extrinsics
    extrinsics = compute_camera_matrices(rtvecs, pairs)
  File "/home/vivanov/anaconda3/envs/anipose/lib/python3.7/site-packages/aniposelib/utils.py", line 173, in compute_camera_matrices
    ext = get_transform(rtvecs, b, a)
  File "/home/vivanov/anaconda3/envs/anipose/lib/python3.7/site-packages/aniposelib/utils.py", line 73, in get_transform
    M = np.matmul(M_left, inv(M_right))
  File "/home/vivanov/anaconda3/envs/anipose/lib/python3.7/site-packages/scipy/linalg/basic.py", line 939, in inv
    a1 = _asarray_validated(a, check_finite=check_finite)
  File "/home/vivanov/anaconda3/envs/anipose/lib/python3.7/site-packages/scipy/_lib/_util.py", line 293, in _asarray_validated
    a = toarray(a)
  File "/home/vivanov/anaconda3/envs/anipose/lib/python3.7/site-packages/numpy/lib/function_base.py", line 489, in asarray_chkfinite
    "array must not contain infs or NaNs")
ValueError: array must not contain infs or NaNs

[vonaviv]

In another session it is a different camera, which causes the problem.

{
'name': 'Cam_04', 'size': [1280, 1024], 
'matrix': [[nan, 0.0, 639.5], [0.0, nan, 511.5], [0.0, 0.0, 1.0]], 
'distortions': [0.0, 0.0, 0.0, 0.0, 0.0], 
'rotation': [0.0, 0.0, 0.0], 
'translation': [0.0, 0.0, 0.0]
}

[vonaviv]

Even though the error comes from rvecs, tvecs = get_initial_extrinsics(rtvecs, self.get_names()), it seems that the initialization of the intrinsics is to blame for this. I have found a workaround, which is to initialize CameraGroup from another session with a successful calibration and skip the intrinsics initialization (skipping just the extrinsics initialization doesn't solve the issue).
Here is a code snippet for this:

cgroup = CameraGroup.load('another_session.toml')
cgroup.calibrate_videos(videos, board, init_extrinsics=True, init_intrinsics=False)