how to reduce reprojection error my camera calibration with scene ?

[hunminkim98]

Hello, sir.
I have been facing an issue for the last 2 months where I am unable to reduce the camera calibration reprojection error below a certain number.
Here are the results of the most recent calibration attempt. ( with 4 cameras)
--> Residual (RMS) calibration errors for each camera are respectively [11.431, 8.938, 12.383, 11.874] px,
which corresponds to [30.368, 25.741, 32.276, 20.572] mm.
I'm calibrating with a total of 12 or 16 points in the scene. This is a homemade calibration cage, so it's not made to very precise dimensions, but I tried to get the numbers as accurate as possible (to 4 decimal places).
The intrinsic parameter was measured using a matlab. ( below 1px )
below is example image that using my calibration

i don't know how i doing to advanced calibration... What could I be missing?

• additionally, i use size of image or video 2704 x 2028 ! with Gopro black 7 !
  Thank you so much for reading.

[davidpagnon]

Hi,

• What do you mean by, you calibrated intrinsics with a Matlab? Do you mean with a checkerboard? Did you use Pose2Sim, or something else? Can you give me a screenshot of your calibration file so that I can see if there is an obvious error?
• concerning extrinsic, does the 3D plot look like the cage? And are you sure you clicked the objects in the right order? (also, did you see that you could zoom in the image in order to click very precisely on your point?)
• lastly, did you try triangulation? Did it look bad? Your calibration results are not perfect but not bad, and I demonstrated in another paper that a centimetric calibration error usually did not hinder results in the case of full-body human movement (I am not the first to have noticed that). What actually matters more is the reprojection error after triangulating the detected keypoints, and from my experience this error is not increased in such cases. The calibration error actually depends heavily on the volume used for calibration : you can obtain a artificially tiny error by calibrating a very small volume.

With this kind of calibration (ie with a single image) , you will not get sub-millimetric reprojection error. On the one hand, it does not make a sensible difference; and on the other hand, in the future we could calibrate on each frame and without it being computationally intensive. This means that we could triangulate data from moving cameras.

[hunminkim98]

Sorry I'm late.
First of all, thank you for your detailed reply.
I'll answer your questions one by one. (My questions are in bold.)

Q.
What do you mean by, you calibrated intrinsics with a Matlab? Do you mean with a checkerboard? Did you use Pose2Sim, or something else? Can you give me a screenshot of your calibration file so that I can see if there is an obvious error?

A.
I used camera calibration app in Mtlab to calculate GoPro7 intrinsic parameter (desipte pose2sim method more simple and useful than Matlab, it can see reprojection error of only intrinsic parameter and manually except have high error ).
below is camera calibration app in MTLAB

and next are outcome of intrinsic calibration K, distortion, image size .. ( I tried both calculating tangent distortion and not calculating tangent distortion, I don't know which is better. ), ( On average 500~1000 image are used )


Finally, the next step is to put the value into the .toml file.

Q.
concerning extrinsic, does the 3D plot look like the cage? And are you sure you clicked the objects in the right order? (also, did you see that you could zoom in the image in order to click very precisely on your point?)

A
I think I tried to click the same point, same order as accurately as possible and see the picture, it seems pretty good shape like cage.

Q
lastly, did you try triangulation? Did it look bad? Your calibration results are not perfect but not bad, and I demonstrated in another paper that a centimetric calibration error usually did not hinder results in the case of full-body human movement (I am not the first to have noticed that). What actually matters more is the reprojection error after triangulating the detected keypoints, and from my experience this error is not increased in such cases. The calibration error actually depends heavily on the volume used for calibration : you can obtain a artificially tiny error by calibrating a very small volume.

A
I've done a lot of triangulation in a lot of calibration environments, and your comment reminds me of my past experiences: sometimes the triangulation is messed up even when the calibration error is low, and sometimes the triangulation is reasonable even when the calibration error is high. I think I read your Part 1: Robustness paper and was obsessing about getting a calibration result of 1px or less. Then i'll have to let go of that obsession and focus on getting the error of the triangulation down.
So, what do you think is a reasonable range of triangulation error for keypoints? Also, the other paper you mentioned is Part 2: Accuracy ?
and you mentioned "The calibration error actually depends heavily on the volume used for calibration: you can obtain an artificially tiny error by calibrating a very small volume". i have a question here, i understood intrinsic parameter is rely on volume of using images, but, extrinsic calibration is using only one image... isn't it?
(i mean, put the video in the extrinsic camera calibration folder, then python code will caputre the first frame of the video.), (When I first inserted the video, I got an error like the image below where the python code was not able to load and read the image from the MP4 file, so I made some modifications, is this out of the developer's intention or did I download the most recently released version of pose2sim0.4.0 and I downloaded the wrong calibration.py file? )

Sorry, that's a lot of intermediate questions, what's the volume mean in extrinsic calibration?

Finally, i apologize if I seem like I'm asking too many questions, but I'm not a computer vision major and I'm studying on my own with no experts around to help me, so the opportunity to get answers directly from the developers is invaluable, so I have a lot of questions.
If calibrating with single image can't give me values below mm, is there any way to use multi image?
And by moving camera, does it mean that triangulation is possible even if the position of the camera changes? If I understand correctly, this is really an innovation !!!!

I am very honored to receive your reply, it is very interesting to study pose2sim. Hopefully I will have a successful experience with it.

Thank you for reading my long post.

[davidpagnon]

1. Intrinsics: This Matlab tools looks quite interesting, I should have a look! What is its name?
   Results look good, but I think you switched width and height in size in your Calib.toml file
2. Extrinsics: This looks good
3. Triangulation: - The most obvious indicator for the validity of your triangulation is visual: open your resulting trc file in OpenSim (File -> Preview experimental data) and see if this looks okay.
   -A more objective way is to look at the results of your triangulation (which are stored in User\logs.txt). An error of about 1 cm is okay, 5 cm starts to be a little much depending on your application.
   -But this also heavily depends on your pose detection algorithm: a good way to compensate for errors is to increase the number of cameras.

The calibration volume has to be understood literally: if your object points are very close to each other and/or represent a plane (2D instead of 3D), then it makes a small volume. This will give you a small calibration error, but this does not guarantee that your triangulation results will be good. In your case, the cage is 3D and probably large enough.

Other calibration methods use multiple images for calibration, most of them use bundle adjustment: they optimize intrinsics and extrinsics at the same time. On the other hand, here we use multiple images for intrinsics once for all, and then one single image for extrinsics. This is less computationally intensive, and actually I was wrong, if done correctly this is just as accurate.

Yes, it would be possible to live-calibrate for a moving camera! We would need to click (or detect) the object points on the first frame, and then they could be automatically tracked on the next frame. I could implement it, if only I had time...

Thank you for your feedback! I was in the exact same situation as you, not a computer major and mostly working alone, so I understand how hard it can be sometimes :)

[davidpagnon]

So again, try to switch width and height in your Calib.toml file, check triangulation results, and get back to me :)

Also you are right, there was a bug when you chose videos for extrinsic calibration! I should have solved it now, I will make it available through pip install pose2sim --upgrade in the next release. You can check if this works (and tell me if it does not) by replacing your calibration.py file by this one.

[hunminkim98]

Again, I appreciate your response.

1. Intrinsic : Detailed information about the intrinsic calibrator application in Matlab is here. You can see many functions used for calibration!
   i tried new calibration after switching width and height in size of my calib.toml file. hopefully work properly then get a smaller error than not switched. but actullay, i couldn't got significant difference..nevertheless, i had get a slightly lower error value. below picture is result of both settings ( not switched, switched ).

• not switched
  

• switched (size = [ 2028.0, 2704.0] to [ 2704.0, 2028.0])
  

I'm attaching a representative photo of the reprojection error results, as camera3 continues to show high error values.

Also, in your experience, what is a good camera-to-object distance? In my experience, close distance (under 2m) will increase reprojection error (maybe not, far distance (over 5~6m) will decrease reprojection error (but, in this case, hard for me to click the same points exactly so, sometimes get higher error value and poor result of openpose detection). Also, as the distance between camera and object increases, the points should be spread out in the photo, so it seems reasonable to use a larger cage. Is my approach correct?

2. Triangulation : In my opinion, it usually shows a pretty good visualization of the preliminary experimental data. However, when scaled or inverse kinematic, they are often poor (neck rotated, pelvis twisted, etc.).
   As you said, looking at the logs.txt file, I can understand why this is the case. The following pictures are the results of triangulating the T-pose static and the Conventional Deadlift.
   (I was tempted to use a lower reproj_error_threshold_triangulation value to get lower error values, but there are a lot of weird shapes and data breaks, so I used a rather high threshold for the most reasonable visualization and data preservation).

• using OpenPoseDemo.exe --model_pose BODY_25B --net_resolution -1x480

• T-pose static ( likelihood : 0.3 , reprojection error threshold : 30 px )
  
  

• T-pose static ( likelihood : 0.3 , reprojection error threshold : 100 px )
  
  

Deadlift ( likelihood : 0.3 , reprojection error threshold : 30 px )

Deadlift ( likelihood : 0.3 , reprojection error threshold : 100 px )

I don't actually need any information about the hands, so it's okay if they look weird. My goal is to reduce the error values for Hip, Knee, and Ankle.
I only used 12 points for low calibration error, but with 16 points it would make sense to use more points despite the slightly higher error value as you say ?
Actually, before I knew about pose2sim, I tried bundle adjustment as well and it didn't work, so I'm glad to hear that I can get the same result.

By the way, the moving camera calibration is very impressive, I've always imagined it, if it's implemented, I'll study hard to try it and give you feedback!

[hunminkim98]

Additionally, thank you so much for being so active.
I tried the newly updated calibration.py and it was a bit buggy ( it could be a bug caused by my mistake, but I tried several times and faced the same error ).

The imgp_confirmed and objp_confirmed in calibration.py were initialized at the end of all clicks and were not returning the values appropriately, resulting in the error shown in the following picture.

Looking at the code, I could see that the on_close function was also being called when all the clicks were done, which was the correct intention, but it was initializing the saved values via the left mouse button to an empty list, which caused the error message that there were less than 6 click points.

In hopes of being of some help to you, I'm sending you the following.

1. This was a persistent error on the first few attempts.
   

2. I've used the print function to see how imgp and objp are returned, and i also checked how on_close, the function that initializes them, is used.
   

3. elif count == len(objp)-1: part, it seemed to call plt.close('all') when all the clicks were done, resetting it to imgp_confirmed = [] , objp_confirmed = [] and calling an error message that there were less than 6 clicks or no clicks. (I saved the objp_confirmed and imgp_confirmed values earlier, but didn't reload them after they were initialized, so I think they were treated as []).
   
   

4. elif count == len(objp) - 1: Below, I added code to call objp_confirmed and imgp_confirmed again to save the value.
   
   

I still didn't get a perfect calibration value, but it seems to be working properly!
It's a small piece of feedback, but I hope it helps, and again, thank you for being so proactive in your feedback to me.

[davidpagnon]

Also, in your experience, what is a good camera-to-object distance? In my experience, close distance (under 2m) will increase reprojection error (maybe not, far distance (over 5~6m) will decrease reprojection error (but, in this case, hard for me to click the same points exactly so, sometimes get higher error value and poor result of openpose detection). Also, as the distance between camera and object increases, the points should be spread out in the photo, so it seems reasonable to use a larger cage. Is my approach correct?

Actually, you should not focus on calibration reprojection error too much. It can be made artificially large or small, and should not be overly trusted. If you want a really meaningful reprojection error, check the reprojection error after triangulation. This is what you are interested in after all.

Now, if you want this triangulation error to decrease, you need to have

1. as many object points as possible;
2. they need to cover as much of the capture volume as possible. Or if you cannot calibrate the whole space, try to cover the space where the person is going to be.

I don't think you answered, did you try to zoom in the image to better click the points? You can try to add "--scale_number 4 --scale_gap 0.25" when you run OpenPose to make it slightly more accurate.

But again, your calibration can be perfect, and yet your results might not be great for two reasons:

1. As all current pose estimation models, OpenPose is not very accurate;
2. One way to mitigate its variability (but not its accuracy) is to increase the number of cameras, which gives better results much more easily than improving the calibration. A phone can be a perfect addition to your system for example (however, you'll have to deal with synchronization).

2. Triangulation : In my opinion, it usually shows a pretty good visualization of the preliminary experimental data. However, when scaled or inverse kinematic, they are often poor (neck rotated, pelvis twisted, etc.).

Now, looking at your triangulation results: your camera 3 is almost always excluded, this is strange.

1. Are you sure of its calibration?
2. Is it positioned in a way that it correctly detects the person? Check its output to make sure that not too many points were excluded, and that right and left limbs were not switched
   In any case, I think you did the right thing by trying to increase the reprojection error threshold. Did you try lowering the likelihood threshold?

The pelvis will always look weird, because you just have the hips and the shoulder keypoints to determine hip angles, pelvis angles, and trunk angles. Three solutions there:

1. detecting more intermediate points. Unfortunately, there are currently no pose estimation models that allow this.
2. adding joint limits in your model (.osim) file
3. adding a weight of 1-2 for a pelvis angle of 0 to the pelvis coordinate in your inverse kinematics (second tab in the tool of the GUI, on the bottom)

Same idea for the hands, although you can actually add more points to detect by using the OpenPose body_135 model, which gives hand keypoints.

Be aware, however, that sparse points always look better than the skeleton, but it is just an illusion :)

---

Main things to do:

• Try adding object points, and if you can, make a larger cage
• Try adding one or several cameras.

[davidpagnon]

By the way, the moving camera calibration is very impressive, I've always imagined it, if it's implemented, I'll study hard to try it and give you feedback!

I would like to do it but I just started a post-doc and unfortunately, I have limited time :/

Wow, thanks for the bug report! That's what happens when you do not correctly synchronize your code between 2 different computers... I had actually solved it earlier, but then made modifications on the wrong version... Hopefully it will be solved now, and will work on videos too.

There is an error message that I don't manage to get rid of (in spite of trying that), but it does not affect results.
In order to check whether the input for calibration is video or image, I try to open the file as a video. If it does not work, I open it as an image and process it differently. When this happens, the ffmpeg C/Fortran code that is used internally by OpenCV tells me that it has not managed to open the video, which I know and is expected. I did not manage to filter this message yet.

Thank you in any case, it was quite a smart correction!

[hunminkim98]

Thanks for the really detailed feedback!
I've never gotten feedback like this before, so I'm very impressed and motivated.
I saw your feedback 5 days ago and wanted to get the desired result and show it to you, but unfortunately it didn't happen, so I decided to reply first to say thank you!

You told me about the model looking weird due to the lack of markers, so I increased the number of markers and rebuilt the model and got a much better result! Now I just need to lower the reprojection error and it's perfect!!! (For personal greed, I also want to do the camera calibration to the level you did, so I'll do it someday and send it to you!)

Below is a model created by adding more markers!

Oh and sorry for not answering when you asked if I zoomed in and clicked on the object points, I zoomed in as much as I could and tried to click on the same place in all cameras !
Regarding camera 3, I think it's probably a problem with the camera placement, I feel like openpose's detection is unstable if it's too close to the object, too far away, or maybe from behind, so I'm trying to find the optimal camera position. (If this doesn't work, I'm planning to increase the number of cameras you mentioned and build a bigger calibration cage!)

You've been so good with your feedback that I wanted to come back with a good result, but I thought I'd say thank you first. Thanks for the great feedback and motivation!!!
You must be really busy doing your post doc. Thanks for taking the time to reply, you're the best!!!

[davidpagnon]

This is the kind of feedback that encourages me a lot, thank you :)
It looks quite good!

Just out of curiosity, how did you detect these additional markers? Did you train a model with DeepLabCut? Or did you use another marker-based system?
I you wanted to keep it simple, in terms of camera system and implementation, you could add a certain weight to the angles corresponding to a neutral pose.

Also, where did you take the muscles from? From Rajagopal or Lai model?

[hunminkim98]

First, I didn't train the marker addition model myself, but I can use a pre-trained model from OpenCap repository (I should have trained the model myself, but I'm still learning).
This model used an LSTM network and models 77 and 93 from the marker augmenter repository also trained on 1475 hours of motion capture data. (version 0.3)

Next, the muscles for the model were taken from Lai!

Finally, I'm not sure what you mean by adding a specific weight to the angles corresponding to the neutral pose.
The way I understood it, I was under the impression that you could increase the importance for a particular camera during the camera calibration process... but I think I misunderstood. neutral pose means T-pose for static?
If you provide me explaining with example, it would be a lot of thanks to me .

Thank you !!

[davidpagnon]

I see, thanks for the information! Interesting way of combining OpenCap and Pose2Sim :)

What I mean is something like this:

In the same way that you can ask the system to try and match the position of a marker, you can ask it to match a dedicated position, such as 0° lumbar extension and 0° subtalar angle. When scaling, since you use a static pose you can add more of these constraints.

[hunminkim98]

Thank you so much!!!
It made perfect sense to me thanks to your kind explanation!
Thank you so much for helping me so much😊
I guess all that's left is for me to keep trying for good results!
I'll come back to you when I have something new, thanks!