Self-driving cars require an understanding of important edge case scenarios that can occur in driving scenes. Scene3D is a neural network which addresses this challenge by processesing raw images and predicting the true per-pixel metric 3D scene coordinates of all pixels in the input image, allowing for a complete 3D reconstruction of the driving scene. The outputs of Scene3D can then be used to detect the object-ness of every pixel through methods such as 3D Voxel Analysis or Stixels, to detect all static, movable, and moving objects in the scene, irrespective of what the object is. This type of classification-agnostic obstacle detection is an essential component in robustly dealing with 'long-tail' edge case scenarios. Scene3D is part of the AutoSeg Foundation Model which forms the core of the vision-pipeline of the Autoware Autonomous Highway Pilot System