Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Obtaining a 3D model using RealSense D405 #13588

Open
edwardp4 opened this issue Dec 9, 2024 · 11 comments
Open

Obtaining a 3D model using RealSense D405 #13588

edwardp4 opened this issue Dec 9, 2024 · 11 comments

Comments

@edwardp4
Copy link

edwardp4 commented Dec 9, 2024

| Camera Model | D405
| Firmware Version | 5.16.0.1
| Operating System & Version | Win 11
| Platform | PC |
| SDK Version | 2.0
| Language | python/C++/matlab
| Segment | Robot

I am working on a project that aims to scan a stationary object (the camera moves around it) and be able to perform dimensional analysis on the object. I have an Intel RealSense D405 camera and have been able to get single frame scans with depth data. What I need however, is an actual 3D model of my object, but it seems there is no one free available software for actively building a 3D point cloud while scanning. I have SDK 2.0 downloaded by that seems to only give live feed and not an accumulating point cloud like I need.

I tried using MATLAB, and am currently trying to now use Python for processing active depth data to build a 3D point cloud but am having a lot of trouble. Looking for any help in using the D405 to actively build a point cloud model in 3D to get dimensional data.

@MartyG-RealSense
Copy link
Collaborator

MartyG-RealSense commented Dec 10, 2024

Hi @edwardp4 The RealSense SDK has an example program called rs-kinfu that can progressively build up a pointcloud with a single RealSense camera in real-time by moving the camera around the object, though it is in C++ language rather than Python. The program can export the finished pointcloud as a .ply pointcloud data file.

https://github.com/IntelRealSense/librealsense/tree/master/wrappers/opencv/kinfu

@edwardp4
Copy link
Author

edwardp4 commented Dec 27, 2024

Hi @MartyG-RealSense thank you for pointing me in the right direction. I've taken some considerable time to install all necessary components and link many dependencies and libraries to finally get the rs-kinfu example to compile and work.

However, I have now run into another wall and after a lot of attempting at troubleshooting and fixes, I can't seem to figure out what the problem is. I've attached images of what I am working with, including the setup, the rs-imshow capture from the camera, and the resulting rs-kinfu output. For some reason the pointcloud appears to be many scattered lines from a focal point, but is generally nonsensical.

I was wondering if you could help me with what is going on. Thank you in advance I appreciate your help.

Quick edit: I've made the rs-pointcloud program work (last image), and it clearly outputs accurate points (i rotated it while streaming). So I am still unsure as to why rs-kinfu is not creating nor stitching these point clouds accurately.

Another edit: I also wonder if kinfu relies on any sensors from other D400 series cameras that the D405 does not have which is why it is not performing as expected? If so what camera should I consider using instead?

Screenshot 2024-12-27 140914
Screenshot 2024-12-27 154243
Screenshot 2024-12-27 154314
Screenshot 2024-12-27 154459
Screenshot 2024-12-27 164004

@MartyG-RealSense
Copy link
Collaborator

Are you moving the D405 camera around when using rs-kinfu? rs-kinfu works by building up the pointcloud image progressively as the camera is moved around using a process called 'frame fusion', instead of capturing the entire pointcloud immediately like rs-pointcloud does.

D405 does not have an infrared pattern projector component, so it is more difficult to depth-sense objects with a plain, low-detail surface. rs-imshow seems to be able to provide a good depth image of the orange object though, so it does not seem as though it is the lack of a pattern to project onto the object surface that is the problem.

You could try changing rs-kinfu's Visual Preset camera configuration to see whether a different preset provides better results. #13540 (comment) describes how to do this.

@edwardp4
Copy link
Author

Yes I was moving the camera around. I tried changing a lot of filters and preset modes within rs-kinfu with no luck. However the code to me suggests that it takes in some sort of inertial movement data that would require an internal motion sensor that I’m pretty sure the D405 does not have. Do other D400 series cameras have sensors that allow for tracking intrinsic movement?

@MartyG-RealSense
Copy link
Collaborator

The D405 is not equipped with an IMU motion sensor. However, the D415 and D435 models, which work with rs-kinfu, do not have an IMU either. The rs-kinfu application only makes use of depth data.

The models equipped with an IMU are D435i, D435if, D455, D455f, D456 and D457.

@edwardp4
Copy link
Author

Interesting. Do you have any idea as to why my rs-kinfu results are so illogical? It just seems like a bunch of scattered lines coming from a focal point.

@MartyG-RealSense
Copy link
Collaborator

I have not previously seen rs-kinfu used with D405. So there is the possibility that it could be an issue specific to that model. For example, D405 uses a different scale value for its depth than the other 400 Series models. D405 uses a scale of 0.01 and the rest of the range uses a scale of 0.001.

You could try changing line 206 from this:

depth_scale = dpt.get_depth_scale();

to this:

depth_scale = 0.001;

So that the program uses a fixed value of 0.001 for its scale instead of retrieving 0.01 from the camera hardware.

@edwardp4
Copy link
Author

edwardp4 commented Dec 28, 2024

that seems to actually have solved it, I'm now able to see the object. However, it is a little bit finicky and it seems like the camera needs to move extremely slow. Movements in general seem to cause blurring, and eventually the model just resets. It also seems to be struggling to actually process the depth information because although the objects are perpendicular to the surface, in the scan it has it skewed and leaning. Overall, seems a little tedious, with skewed results. Maybe a more powerful PC helps this?

image
image

So what I'm wondering, as the project involves take accurate measurements of the model, would you recommend I try to stitch individual captures from rs-pointcloud instead of using rs-kinfu? Or do you think kinfu can be tuned to be more accurate.

@MartyG-RealSense
Copy link
Collaborator

MartyG-RealSense commented Dec 28, 2024

rs-kinfu is known to have slow performance. A more powerful PC could help.

The program applies three post-processing filters, which impose a processing burden on the CPU because they are calculated on the computer and not on the camera hardware. You can disable these by commenting them out on lines 257-259. Start with the Spatial filter first as that has the heaviest burden, and then comment out the other two if performance does not noticeably improve.

@edwardp4
Copy link
Author

I tested even using some other filters, but ultimately it seems that it reduces the quality to exclude those default 3 filters. I think that for the purpose of taking accurate dimensions, kinfu is likely not the way to go due to the high processing it takes and how blurry and skewed the results are, at least with the D405 model.

Before I move on to trying to stitch individual captures using rs-pointcloud, do you know how accurate the motion sensors on the D435i are, and whether or not the IMU on it will be useful in generating an accurate model of the object?

@MartyG-RealSense
Copy link
Collaborator

RealSense IMU motion data can be 'noisy' and prone to fluctuation due to the IMU's sensitivity. So your plan for stitching individual captures sounds like a better option.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants