Are we on the right path for surface reconstruction?

[yuedajiong]

我们是否走在正确的表面重建路上？ GS本质是放松了“深度/法线/表面/形态/物理实在”的约束，追求了一种2.5D的“视觉上很象”“立体上未必象”“但能够支持观察点立体移动”的准重构。 除了2023-11的法国大神的SuGaR，还有2023-12的新加坡国立的大神的NeuSG，等等，都视图在做到表面的重构。SuGaR的方法个人感觉还是很重，从多变量正态分布采样开始，虽然有利用深度图的一些加速策略，到多正则项。 NeuSG的做法，看起来要轻量级一些。 当然，既然要重构mesh，最好就是达到商业产品可以用的“精美”的质量，形态准确，细节丰富。 一边学习一边反思：GS为啥快，如果最终目标是real 3D mesh而不是2.5D的视觉很象，我们从GS学习啥？和直接mesh-fit到底为啥不容易收敛, GS是因为先放弃了mesh的边/面的约束，让点自由的增山和移动吗？还是用GS点来表示本身就更合适？还是直接用梯度来驱动这些操作很有效？mesh也支持顶点的增删和移动呢？ 等等等等问题， 同理，SDF/UDF等等那么多方向，该从GS核心学习什么？ 所以，假设要沿GS往下走，GS++方向是这样各种加正则项呢，还是增加对一些学习的参数项目，特别关于depth, normals还有强相关的opacity的处理逻辑。 “一个更纯粹更原生的GS++ for mesh”应该是什么样子？

chatgpt translated:
Are we on the right path for surface reconstruction? The essence of GS lies in relaxing constraints such as "depth/normal/surface/form/physical reality," pursuing a quasi-reconstruction that is 2.5D—visually similar, not necessarily volumetrically similar, yet capable of supporting stereoscopic movement of the observation point. In addition to SuGaR from the genius in France in November 2023, there is also NeuSG from the genius at the National University of Singapore in December 2023, and so on—all attempting to achieve surface reconstruction. Personally, SuGaR's method feels quite complex, starting from sampling a multivariate normal distribution, although there are some acceleration strategies using depth maps, leading to multiple regularization terms. NeuSG's approach seems to be more lightweight.

Of course, since we are reconstructing the mesh, it is preferable to achieve "exquisite" quality suitable for commercial products—accurate form with rich details. While learning, there is ongoing reflection: why is GS fast? If the ultimate goal is a real 3D mesh rather than a visually similar 2.5D, what do we learn from GS? Why is it not easy to converge by directly mesh-fitting? Did GS give up constraints on mesh edges/faces, allowing points to freely increase and move? Is it more appropriate to represent GS points themselves, or is it effective to drive these operations directly with gradients? Does the mesh also support adding, deleting, and moving vertices?

And so on, and similarly in various directions such as SDF/UDF—what should we learn from the core of GS? Assuming we continue down the GS path, should GS++ involve adding various regularization terms, or should it increase support for some learning parameters, especially regarding the logic of handling depth, normals, and strongly related opacity? What would a "purer and more native GS++ for mesh" look like?

[yuedajiong]

NeuSG的问题，是为了得到normals（GS在避免的），又引入了NeuS，感觉也还是太重了。
The issue with NeuSG is that, in order to obtain normals (which GS avoids), it introduces NeuS, and it still feels too heavy.