refined mesh for zero123 setup

[stas-polukeev]

Hi!
I would like to know is it possible to apply the same logic as shown in readme in the Magic3D section, but for training with zero123?
In the given examples there is a text prompt, but is it possible to use an image? for example, can i first do: python launch.py --config configs/zero123.yaml and then after it's done somehow refine the trained nerf? And if it's possible, what would be the recomended way to do this?
thanks in advance!

[bennyguo]

Hi! It's definitely possible but I haven't tried yet. Theoretically you just need to copy the magic3d-refine-sd configuration and change the camera settings (including model.radius) to align with those in the zero123 configuration. Then you can run the refinement by passing the zero123 checkpoint path to system.geometry_convert_from.

[stas-polukeev]

@bennyguo I'll try that, thanks!

[stas-polukeev]

@bennyguo Hi! I've tried your solution, and changed the parameters of the camera, after that I launched this comand:

python launch.py --config=configs/magic3d-refine-sd_custom.yaml --train exp_root_dir="/data/test1/8"
system.geometry_convert_from=/data/test1/8/zero123/[64,\ 128]_8_rgba.png_prog0@20230705-115356/ckpts/last.ckpt

here magic3d-refine-sd_custom.yaml is the default magic3d-refine config with the changed camera parameters, but it gives the following error:
TypeError: Magic3D.Config.init() got an unexpected keyword argument 'freq'

Do you have any suggestions on how to solve it?
Thanks in advance!

[VincentHancoder]

Hi, I also get the same result as yours. Have you already solved the error? Or if you have any suggestion to write the copy of magic3d-refine-sd.yaml in a correct format?

[stas-polukeev]

@VincentHancoder hi! No! Unfortunately, it’s still unsolved. It seems that you just need to match parameters of the magic3D config with the consideration of the zero123 config. I didn’t have time to do such a deep dive into the code to rewrite the yaml file, there are some mismatches in the parameters that i do not now how to fix. If you, by any chance make it work, please notify me here!

[VincentHancoder]

Sure, I will share my copy format if it can work. But, I don't think it's easy for me as well. If you have any approach, you can share it here, too.

[stas-polukeev]

@VincentHancoder i started with changing camera params as was originally proposed, but found out that there are other mismatches. I tried adding smth from zero123 cfg also, but that solved one issue and produced another. May be if one looks a little deeper into the code, it will be more clear what to change, as I haven’t used this repo extensively.