real-case: resin toy wizard reconstruction. all better quality cases are at the end sections! In Chinese: 高斯溅射这条路上提取表面/mesh目前最靠谱的路。好效果的在最后部分。

[yuedajiong]

coarse density:
(the best way to reconstruct surface based on GS, so far. )

one of inputs:

8000 iteations: (from 7000)

sugarmesh_3Dgs7000_densityestim02_sdfnorm02_level03_decim1000000.zip

15000 iterations: (still no refine step)

some better quality cases are in the end sections !!!!!!!!!!!!!!!!!!!!!!!!!!!!!

[Anttwo]

Hello @yuedajiong ,

I'm happy to see SuGaR applied to more datasets!
Is this mesh the output of the coarse density script, or the refinement script?

I suppose this is the coarse mesh, so its colors are just the vertex colors output by the poisson algorithm, which are quite bad.
The refinement really helps to output a better looking mesh and a really nice texture (see images below).

Also, for rendering the mesh with a traditional texture, a good practice to get something close to your training images is to put an Emission shader to the texture, or to remove all light sources and just use a Ambient lighting with intensity 1.

For centered objects such as figures, SuGaR should work really well, even if the object has many details. I showcased two robots reconstructed like this in the presentation video, but I also tried other similar figures and the results is generally very nice. Here are some examples of meshes, with a traditional UV texture:

[yuedajiong]

@Anttwo great-master:

coarse, and density-mode, not sdf-mode.

I am training and adjusting now.

I will update once generated better results, minutes or hours.

[Anttwo]

You're right, density-mode actually works better for this kind of scene.
But after the coarse extraction, the final refinement phase (train_refined.py) helps to smooth the mesh and get a good-looking texture.

I can't wait to see your final result! 😃

[yuedajiong]

@Anttwo
Great-master, in many lucrative applications of stereo/surface reconstruction, precision is crucial for specific subjects such as objects or human. The algorithm supports the masks to control non-focused objects, making this functionality essential.

please refer to: graphdeco-inria/gaussian-splatting#541

I will merge that logic into your code, tomorrow, and show result to you.

[yuedajiong]

@Anttwo

For the surface reconstruction of everyday-visible objects, no specific adjustments are made for particular objects and data. I have tried almost all well-known non-GS approaches, but the quality of the reconstruction results is quite unsatisfactory. I have also attempted some GS approaches, but they cannot even reconstruct a very basic mesh.
Your method currently yields relatively better results, but there are still many areas that need improvement:

1. the mask for non-focused objects;
2. for many continuous components/surfaces of an independent object, the constructed points and patches are still somewhat discrete, and the depth information is inaccurate.
3. It seems that some surface smoothing is applied in the loss function, but in my resin wizard reconstruction, the surrounding background appears to be smoothed in patches, while the central human part is not optimized.
4. the normals still need optimization. we can see, in my example, the wizard is the inside of the non-watertight-surface object.

(still on adjusting ...)

[Anttwo]

Thank you for your feedback!

Indeed, I'm quite surprised by your results, as I experimented with many custom datasets on my side, and the output mesh is generally good, even with very detailed objects like a Telecom tower (I will put some qualitative results on the webpagge).
In particular, I did not experience this discrete "patch effect" that you have.

May I ask more details about your dataset:
Did you use the COLMAP script to generate the camera poses?
What is the scale of your scene?
Do you have blurry images on your dataset, or variations in the focal length?

Concerning normals, it is generally just a matter of convention during reconstruction ; Multiplying by -1 the normals should do the job (but for many softwares like Blender, it doesn't change anything).

Looking forward to your reply!

Edit: Here is a test I just made on a new scene which is, just like yours, a figure centered in the scene (actually a little challenging because of the strong black color, and strong specularity on the figure). Looking at your scene, you should be able to get a similar quality:

Depth (of the mesh):

Hybrid representation (Mesh + Gaussians):

Mesh with color texture:

[yuedajiong]

@Anttwo

I am checking my steps, including data-capturing，data-processing and training, especial data-processing.

I also believe and hope to reconstruct my data into a beautiful result similar to yours.

my data(video), my images(240 frames, all frames)
https://github.com/Anttwo/SuGaR/assets/52232153/b3705640-db59-408c-9c55-b576964f6c06

I will try your data such as leego firstly. doing ...
in fact, in GS and other algorithms, I can reconsutct out similar quality to those papers from public dataset.

[cdcseacave]

@yuedajiong the model vibrates during acquisition, so it becomes a dynamic scene

[yuedajiong]

@cdcseacave Thanks.
I am purchasing an electric turntable(below, 12$) and will retest it after stability smooth shooting.

[yuedajiong]

I used a lion-resin-toy, and almost perfect colmap poses, the mesh output is very good.

@Anttwo

[yuedajiong]

better one, close to make-money level (if clean garbage):

[yuedajiong]

@Anttwo

the mask logic in algorithm native(GS and SuGaR parts) is necessary.

[ygtxr1997]

better one, close to make-money level (if clean garbage):

Amazing results! Can you share your training input video of this lion toy?

[yuedajiong]

@ygtxr1997
KEEEEEEEEY:

1. MUST HAVE: accurate SfM result<!!!>, masked background<!!>, enough views <!>.

2. better to have: 均匀漫反射光线，买HAVE个直播卖货的旋转台拍摄。

[miaowu99]

@yuedajiong hi，I am facing the same problem as you. May I ask where the specific modifications are needed to incorporate mask or transparency information into the training？

[yuedajiong]

@miaowu99

1. use any 2d segmentation (likes SAM, gbremove, ...) to segment your input images, and fill backckground in white or black; and rest steps are same. (the way, simple)
2. segment and construct rgba as input, modifiy GS and GS render (including foreward and backward), donot render any gauss point if it mapped to 2D and channel-A(in rgbA) is transparent. (many guys will implement this 'mask' logic later.)

[miaowu99]

@yuedajiong Actually, I did 1, but the reconstructed mesh has a lot of noise background, which leads to poor quality of the foreground. Also, I found that GS was originally able to handle black backgrounds, but the process of generating mesh in Sugar does not have a background concept. It seems that we can only consider adding masks during the training process. Have you tried it? Also, can we communicate through QQ? I have sent an email to you.
GS points：

SuGaR mesh：

[yuedajiong]

@miaowu99
兄弟，GS算法原理，决定了这种背景噪音不好消除，本质是：
假设你关注的是中间这个玩具，你拍摄的时候，无论周边是白色，黑色，还是其他背景。
在SfM这一步，它都尝试尽力去照特征点来对齐，因为周边一大片，总有些看起来均匀的背景色，但实际有一些不同的像素。
假设你分割了 背景填充为纯黑纯白，然后去做SfM，假设就背景部分一个特征点都找不到，那这个问题只是推迟到后面GS重构部分。
在重构部分，没有mask，代码一眼可见； 同样，缺省的代码的channel是3是rgb也不支持rgba。 所以，本质没有mask。
那对于GS来说，干净的黑或白，也是它要重构的区域。 由于GS并没有准确的depth计算，从随机或者sfm的sparse points开始，总有一些点，在有限的视角下，哪些点depth不对（对某个视角欺骗人很像起了作用，但并不是物理上真正有的点），由于GS采用scale来放大了点，一个GS点带代表视觉上一大片，对于均匀的背景部分，所以就更容易一个大片（你只看点的时候就是稀疏的点，如果你渲染了也就是scale起作用），就是一点代表一片。

如果你要做的彻底，以对象重构为目标，做到赚钱的效果：用mask或者rgba； 如果你只是想稍微好一点，你可以用各种rule去把核心对象周边的点，给clean掉（比如远离点中心，比如点很稀疏），如果你不放心，你可以一边 clean一边refine train。

就是SuGaR作者，这个博士帅哥，他都计划要要做mask. (我自己实现的是rgba，修改cuda代码）
你要么等等，要么自己去实现。

[yuedajiong]

RGBA as mask，大概：
1）数据准备，对于A分量（我强制输入要么0要么1）；还有GS python部分，都好说，不难，掠过。
2）CUDA部分：
要把channle定义修改为4
然后foward沿途修改，对于A透明度部分，对于完全不透明就是需要忽略的背景，算到2D投影那里，都不要算那些渲染点。
对应的backward部分，我也还没有去推导和实现。 估计到春节，我会折腾。

[tianxiaguixin002]

要实现精准的mesh和贴图估计，应该试试隐式sdf烘焙的方案，如bakedsdf这种，3d gauss做精准的mesh目前还比不过隐式sdf (neus/volsdf类)的方案。

[yangqing-yq]

RGBA as mask，大概： 1）数据准备，对于A分量（我强制输入要么0要么1）；还有GS python部分，都好说，不难，掠过。 2）CUDA部分： 要把channle定义修改为4 然后foward沿途修改，对于A透明度部分，对于完全不透明就是需要忽略的背景，算到2D投影那里，都不要算那些渲染点。 对应的backward部分，我也还没有去推导和实现。 估计到春节，我会折腾。

请问你实现的带mask refine 的代码可以share吗？

[yuedajiong]

@yangqing-yq
search 'alpha' in render code (render with mask)

#https://github.com/fudan-zvg/4d-gaussian-splatting 2310.10642 Real-time Photorealistic Dynamic Scene Representation and Rendering with 4D Gaussian Splatting

#https://github.com/VITA-Group/4DGen 2312.17225 4DGen: Grounded 4D Content Generation with Spatial-temporal Consistency