Tracking position and pose of a moving drone using L151 #13635
Replies: 12 comments 11 replies
-
|
Hi @ar-ramchandra Thanks very much for your questions! I would advise using RealSense 400 Series cameras instead of L515, as the L515 model is now retired and is no longer supported in recent versions of the RealSense SDK software.
https://www.intelrealsense.com/everdrone/
It would not be necessary to sync the shutters of the multiple cameras in this particular application.
https://docs.wpilib.org/en/stable/docs/software/vision-processing/apriltag/apriltag-intro.html
Alternatively, there are some non-Intel open source Apriltag detection projects that may be able to work with D455 / D456, such as apriltag_ros |
Beta Was this translation helpful? Give feedback.
-
|
@MartyG-RealSense thank you so very much for the quick and detailed response. My plan is to have the cameras static instead of mounting them on drones to get the X,y z of drones in the overlapping FOV of the cameras... Reading some of your previous posts, I noticed that the real sense cameras have a way of precicely estimating the X,y,z of objects from a function called get_indices or something along those lines.. If I use that function with a combination of camera excentrics, would I be able to achieve mapping the drones position ? These are small drones so they are incapable of carrying cameras Does rs2 allow calibration between cameras to do this or I would be better off with opencv triangulation ? I'm so heartbroken to read that the l515 cameras are EOL... such a wonderful piece of tech ! I wish I had gotten into this earlier and used some of these tech.. |
Beta Was this translation helpful? Give feedback.
-
|
Although L515 are retired, they are still available from the official RealSense store on a 'while stocks last' basis . The most recent version of the RealSense SDK that is recommended for use with L515 though is '2.50.0'. However, they are not suited for use outdoors. https://store.intelrealsense.com/buy-intel-realsense-lidar-camera-l515.html I am not aware of a method for obtaining the position of an object observed by the camera using intrinsics and extrinsics. A way to get the 3D 'pose' of an object without using a T265 is to use OpenCV's SolvePNP algorithm described at #10233 (comment) If you have an Nvidia Jetson computing board then object pose can also be obtained with a RealSense camera using ROS and Docker. https://github.com/pauloabelha/Deep_Object_Pose A RealSense model that may fit on a small drone is D405, as it is lightweight, small and square. Its depth sensing works best at close range though (up to 0.5 meters), but it can depth-sense up to a couple of meters from the camera. https://store.intelrealsense.com/buy-intel-realsense-depth-camera-d405.html |
Beta Was this translation helpful? Give feedback.
-
|
Hello @MartyG-RealSense thank you for the quick response. Forgive me for not describing it clearly... the aim is to us rs2 for position fixing and use April tags for pose, rs2.rs2_deproject_pixel_to_point() I have followed some advice previously given through this medium post; If I have 4 cameras(a,b,c, d) for redundancy to ensure no flying drone is obstructed;
|
Beta Was this translation helpful? Give feedback.
-
|
You can transform multiple coordinate sets to have a single common 3D position and rotation using the rs2_transform_point_to_point() instruction, as described at the C++ link below. https://dev.intelrealsense.com/docs/api-how-to#get-and-apply-depth-to-color-extrinsics An example of using this technique is 'stitching' together multiple 3D pointclouds from different views to create a combined 360 degree image. The section of official documentation linked to below also has information about rs2_transform_point_to_point() https://dev.intelrealsense.com/docs/projection-in-intel-realsense-sdk-20#extrinsic-camera-parameters |
Beta Was this translation helpful? Give feedback.
-
|
Hello @MartyG-RealSense thank you so very much for pointing me in the perfect direction ! You are awesome ! I see this issue too where you have described it in detail ! Thank you so much ! Is there a way to compute the excentric parameters of the cameras ? I've been trying to find how to do it with real sense itself (with an assumption that since rs made the devices they know best)... all I can find is a bunch of theory of what excentrics are and how to do it with openCV. Also if the drones had downward facing lights from led strips while flying, in low light conditions, which is why I wanted to stick with the l515 since it has the ability to perform in low light conditions. I will consider adding a ir cut filter on the lights because I read that ir can interfere with the LIDAR used by the system. We will be using it only in an indoor environment and not outdoors. Do u recommend I stick with the l515 or use d4xx ? @MartyG-RealSense I really appreciate your help, this has been driving me crazy for weeks now until I finally discovered this discussion and decided to ask here, it was the best decision I have made so far ! |
Beta Was this translation helpful? Give feedback.
-
|
You are very welcome. I'm pleased that I could be of help! The extrinsics of a particular camera can be retrieved from it with the instruction "get_extrinsics_to, which is demonstrated in the 'get-and-apply-depth-to-color-extrinsics' link above. In RealSense, extrinsics describe the real-world distance and rotation angle between two particular sensors on the camera (such as between depth and RGB). The intrinsic and extrinsic values are unique to each individual camera due to the manufacturing process at the factory. You can also retrieve a full listing of a camera's intrinsics and extrinsic values using the rs-enumerate-devices tool's calibration information mode by launching it with the command below.
RealSense 400 Series cameras (except for the D405 model) can also operate in low-light and dark / night conditions thanks to the infrared emitter component they are equipped with, which casts an infrared light source that can illuminate the camera's view of the observed scene whilst being invisible to the human eye. An example of this is using it in a medical ward to observe humans at night with a clear, bright image whilst not casting any visible light that would disturb the patients in the beds. If your preference is to use L515 then there is no obvious reason why they could not be used to observe drones indoors in the same way that a 400 Series camera can. However, if the views of the multiple L515s are overlapping then you may need to use the L515's specific form of multiple-camera hardware sync in order to negate possible interference caused by the overlap. https://dev.intelrealsense.com/docs/lidar-camera-l515-multi-camera-setup The 400 Series models do not experience interference from one another when their views overlap. You should also bear in mind whether you are likely to need additional L515 cameras in future, in case they become unavailable later. |
Beta Was this translation helpful? Give feedback.
-
|
@MartyG-RealSense thank you so much for the wonderful explanations I appreciate it a lot. You are truly an expert ! My only regret is that I did not reach out to you earlier ! I did read about the same link you shared previously, the reason I insit on using the l515 is that it has a nice range of up to 9m with awesome accuracy... Also would the presence of leds at the bottom of the drone cause any issues with camera ? Could you help me understand what issues I may face and how to overcome them ? |
Beta Was this translation helpful? Give feedback.
-
|
L515 has a range of 9 meters, but its best accuracy is up to around 3 meters and then accuracy declines beyond that distance. The 400 Series models have a 10 meters range. In that range, the D45x class of cameras (D455, D456, D457) have the best accuracy at long distance, having the same accuracy at 6 meters from the camera that other models have at only 3 meters. D45x type cameras can depth-sense as far as 20 meters, though accuracy will decline beyond the 6 meter point. 400 Series cameras do not have problems with depth-sensing in the presence of LED light. The L515 guide at the link below demonstrates that this camera can be used in the presence of LED light, though you may need to set the 'Low Ambient Light' camera configuration preset otherwise performnce could be reduced. https://www.intelrealsense.com/optimizing-the-lidar-camera-l515-range/ |
Beta Was this translation helpful? Give feedback.
-
|
Thank you so very much for your guidance ! This is amazing information ! I will be sure to keep this in mind... Am I understanding this correctly; The D45x, D43x seem to mention their error as <2% at 4 m, this translates to 8 CM of error (400 * 0.02). While the l515 has an error of 5 to 14 mm through 0.25 - 9m, if the error is max at 9m it would only be 1.4 cm... So the l515 is almost 8 times more precise... am I misunderstanding something ? Also would the l515 support the use of passive reflective ir spheres cause interference with the LiDAR? Like the one in below image;
|
Beta Was this translation helpful? Give feedback.
-
|
I do not have an exact comparison of L515 vs 400 Series accuracy. However, one of the notable features of L515 is that it can accurately analyze 23 million depth points per second. https://www.intelrealsense.com/lidar-camera-l515/ D45x type cameras have an accuracy of '<2% at 4 m' and D43x have an accuracy of '<2% at 2 m', reflecting how D45x cameras have better accuracy over long distance. However, these figures do not convert well to working out accuracy at a specific distance (1 m, 2 m, 3 m, etc). The best way to think about it is 'D45x has half the error at a particular distance that D43x has'. There is a math formula at the link below for calculating error over distance (called RMS Error) at the link below if you are interested, though it only applies to the 400 Series models. You can also obtain an automated calculation of RMS Error by using the SDK's Depth Quality Tool so that you do not have to do the calculations yourself. Regarding L515 and reflective objects, it can have problems with those. Using the 'Short Range' preset to reduce the camera's Laser Power and Receiver gain values can help to reduce the negative effects of reflections. Having said that, the 400 Series also has difficulty analyzing highly reflective surfaces for depth information. I suspect though that small reflective balls like that would not have a large impact on the depth image. At worst, there might be small black holes on the depth image that correspond to the position and size of the balls. |
Beta Was this translation helpful? Give feedback.
-
|
Would reducing those parameters have an effect on the accuracy of the camera ? I came across this paper where they used the l515 and similar reflective balls, they seem to be getting results which leads me to believe that this maybe an option .. Does intel have any recommendations for markers ? |
Beta Was this translation helpful? Give feedback.
-
|
Reducing the parameters should not have a negative effect. Often, using the Short Range preset provides a noticable improvement when there are problems in an image when using the camera's default preset of Max Range. Apriltag markers can work well with L515, like in the Intel video below where Apriltags are used to calibrate nine L515 cameras together. |
Beta Was this translation helpful? Give feedback.
-
|
Wow that looks really impressive, could you give me some more details about the excentric and intrinsic calibration of these cameras and how to do it ? Should I be using the sdk or the viewer gui to get these params for multiple cameras more easily? Is it just the regular checkerboard thing? Or do I have to buy any equipment to obtain these numbers ? |
Beta Was this translation helpful? Give feedback.
-
|
L515 does not have a calibration tool. Intel's original press release for the L515's launch stated that the canera retains its depth accuracy throughout its lifespan without the need for calibration. |
Beta Was this translation helpful? Give feedback.
-
|
Wow that is so cool... rs2::pipeline pipe; Is there an object that needs to be in both frames to get the excentrics ? |
Beta Was this translation helpful? Give feedback.
-
|
There does not need to be an object in front of the camera to get the extrinsics. |
Beta Was this translation helpful? Give feedback.
-
|
Oh ! That's great ! So we don't have to place an object in the overlapping fov of two cameras to obtain the excentric parameters to convert between one cameras coordinates system to another ? |
Beta Was this translation helpful? Give feedback.
-
|
You do not need anything in front of the camera to obtain the extrinsics when using a single camera. When obtaining the extrinsics of multiple cameras though, the cameras should be placed around the edges of a flat chessboard image to enable the positions of the cameras relative to each other to be calculated. An example of doing this is Intel's Python box_dimensioner_multicam box volume measurement example program. When the program is run, it automatically calculates the camera extrinsics relative to each other and only after that asks for an object to be placed on the chessboard in view of the cameras.
|
Beta Was this translation helpful? Give feedback.
-
|
Yes I did see this before and it's awesome ! In that use case the cameras seems to be placed close to the object and the checkerboard, if I were to place cameras in the corners of rooms near the ceiling, would we still be able to do this ? |
Beta Was this translation helpful? Give feedback.
-
|
box_dimensioner_multicam is particularly senstitive to distance from the board, and has an ideal distance for camera placement of around 0.5 meters from the 6x9 inch board. I think if you wanted to place the cameras high up, you might have to increase the size of the chessboard so that the squares are much larger so the cameras can still see them from high up. box_dimensioner_multicamcan be customized for larger boards by increasing the values of board height, width and square size parameters in its script. |
Beta Was this translation helpful? Give feedback.
-
|
Awesome ! So the idea is that I place the cameras high, looking down on a common fov, synced with shutters, and then use a large checkerboard in the common fov, compute excentrics of cameras in relation to each other, and then place markers or April tags on objects, get their position coordinates based on one camera using rs2.rs2_deproject_pixel_to_point(), then convert one cameras coordinate system to another cameras view to have a common global coordinate systems using point_to_point and camera excentrics. So to calculate excentrics, I place the checker board and then use the gui to obtain excentrics ? Is this also an option ? And then continue with rs2 python.. Could you confirm that the rs2 version that is supported with l515 supports all of these features ? Also what are considered realistic ranges in real world applications, would it actually be able to detect object ranges at up to 6m at sub 2cm accuracy? |
Beta Was this translation helpful? Give feedback.
-
|
Your plan sounds reasonable and achievable. Whilst L515 can depth sense up to 9 m, accuracy starts to reduce after 3 to 4 meters. You could likely obtain a good depth measurement at 6 meters. I don't know what the level of accuracy would be at that distance though. Most RS2 functions that are supported for the 400 Series cameras are also supported for L515, including rs2_deproject_pixel_to_point() and rs2_transform_point_to_point() |
Beta Was this translation helpful? Give feedback.
-
Hi im going to purchase 4 realsense cameras to tract the position of drones flying around in the view of these cameras.
My idea is to get 3d position (x,y,z) in meters of multiple drones by using these camears in realtime.
Here are some questions to help me understand the amount of work required for me to have a working system.
Any advice / guidence is deeply appreciated !
Any advice is deeply appreciated.
Beta Was this translation helpful? Give feedback.
All reactions