RealFusion reproduction

[bennyguo]

No description provided.

[jaidevshriram]

Would love to see this! I'm currently looking to reimplement Dreambooth3D (https://dreambooth3d.github.io/), which also requires a textual inversion pipeline like RealFusion. Do you have any pointers on how best to integrate that with threestudio. Excited to use threestudio as a starting point! Thanks!

[lukemelas]

Hello! RealFusion author here. I'm happy to answer any questions and help with the implementation if you need help on any specific points.

And thanks for starting this library, it looks great. Excited to contribute to it.

[bennyguo]

@jaidevshriram Here're some tips on implementing a DreamBooth3D system:
Stage1

• Make a scripts/ directory. Add a DreamBooth training script in scripts/
• The training steps are the same as DreamFusionSystem, remember to point pretrained_model_name_or_path of the guidance to the path of the trained DreamBooth model.
• Export rendered images as well as camera parameters (maybe in NeRF-Synthetic format?) at the end of training for stage 2.

Stage 2

• Create a script in scripts/ to perform image-image translation using rendered images from stage 1.

Stage 3

• Using the DreamBooth training script again on the original images and I2I translated images.
• Write a multi-view dataset for the I2I translated images. Each batch contains multiple of these images along with the camera parameters. I'm not sure whether DreamBooth3D uses random-sampled cameras at this stage. If it does, a simple workaround would be to return a mix of I2I cameras and random cameras, and for random cameras you may return dummy blank images so that everything could be batched together.
• The training steps are almost the same as DreamFusionSystem, except that you should add a reconstruction loss.

Let me know if you have any problems. Looking forward to the DreamBooth3D implementation!

[bennyguo]

@lukemelas Hi! Thank you very much! We're currently working on reproducing RealFusion, very inspiring work! The paper is very well written, and I'll let you know if we have some problems with specific details.

[lukemelas]

Wonderful, and thank you for the kind words!

[jaidevshriram]

@bennyguo Thank you for the incredibly detailed response! Appreciate it.

[jaidevshriram]

Hey, just to follow up on this - do you have any tips on how best to organise a dreambooth pipeline. I've currently implemented a custom dreambooth3d system that modifies the dreamfusion system. However, it would be nice to replicate some of the dreambooth ideas w VSD guidance / fantastica3D / magic3D.

[bennyguo]

@jaidevshriram Great to hear this! Is your Dreambooth3D pipeline reproducing the paper results? I think threestudio has a very flexible structure that allows you to easily change any part of the system, for example the geometry type and guidance type. You could refer to prolificdreamer*.py (there're three of them) to see how I use the same system across 3 stages but enable different geometry and guidance at each stage.