Hyperparameters for NeRF training

[yashbhalgat]

Hi @Tom94, thank you for this wonderful piece of work -- I am sure this is useful to many researchers (like me)! I have implemented the hash encoding in PyTorch and the convergence speed is quite amazing (although not as fast as your C++/CUDA impl). I was able to train a 'HashNeRF' model for hotdog, chair and few other blender objects and the model converges 50x faster than NeRF in my implementation.

Although, I am not able to get as good results for the ficus and drums objects. I see some cloudy artifacts in the final rendering. Could you please share the hyperparameters that you used for the NeRF training of these objects?

My hyperparameters

I am using an Adam optimizer with:

• learning rate: 0.01 (tried 0.001, 0.0005)
• learning rate schedule: lr ^ (global_step / 10000)
• base resolution: 16
• max resolution: I tried 512, 1024, 4096
• weight decay for MLP: 1e-6 (no weight decay for embeddings)
• eps for embeddings: 1e-15

[Tom94]

Hi there, apologies for the delay in responding.

The hyperparameters you're listing seem to match ours. I suspect that our raymarching strategy of skipping empty space helps us avoid floaters more easily (by importance sampling the loss near surfaces). Other than that, the two implementations should behave the same.

It would probably be good for us to add an option to disable empty-space skipping to verify this. Much of the needed code is already there -- it just needs to be wired up appropriately.

[yashbhalgat]

Thanks @Tom94 for the details! I do have Importance Sampling in my code already. After some debugging, it seems that tuning the hashtable size and/or the finest resolution helps mitigate the floaters problem.

On a side note, I have open-sourced my pure PyTorch implementation here: https://github.com/yashbhalgat/HashNeRF-pytorch :)

[yangze0930]

Hi @Tom94 , thanks for your work! Have you figured out how much the adaptive raymarching strategy will help?

[Tom94]

Responding to a separate question made me realize another piece to this puzzle: this NeRF algorithm uses randomized background colors in transparent regions, which also helps reduce floaters.

Using a solid background color, the model can minimize its loss by simply predicting that background color, rather than transparency (zero density). In practice, it'll learn a mix of them, which can manifest as floaters.

By randomizing the background colors, the model is forced to learn zero density to let the randomized colors "shine through".

[pwais]

@Tom94 Could this be an opportunity to try a BARF-style annealing of the encoding / hash features to see if it can help mitigate floaters? Your observation here is really interesting... While clearly instant-ngp has shown that annealing & classic Nerf sampling is not necessary/efficient for many cases (thus your huge reduction in SGD iters even compared to meta-learned approaches), I've found that the floaters in instant-ngp can be worse than in some other approaches. It's interesting to hear about the impact of background colors.

Another question: instead of presuming a background color, can you just explicitly sample on the unit sphere for some large radius? Perhaps at the start of training? Perhaps restrict the encoding / features (BARF-style) to force the model to only learn low-frequency stuff for the background in the first few iters. Then maybe no background color is needed, and you don't need a separate background model.

[JoanCharmant]

Floaters are from collisions between one empty and one filled voxel. Since collisions only happen in the higher levels where the hash table is smaller than the universe of coordinates, but the lower levels are clean, there should be a way to give precedence to the lower levels.

If a point is contained in an empty low resolution voxel, none of the high resolution voxels contained in the same volume can possibly have anything in them. If they do it's because they are sharing their feature vector with another voxel somewhere else, and they should be ignored. Is there such a heuristic already in place? Using the occupancy maps perhaps?

[pwais]

Well the floaters I'm trying to reference are larger ones that tend to follow a single ray. Those are typical of all NeRF models, hashed features or not.

It would be cool to view the hash collisions in the GUI. I imagine they're pretty rare though?