Really would recommend watching the video on this: https://www.matthewtancik.com/nerf. The difference between NeRF and prior models are pretty significant.
rcorona
Something I'm curious about is how one might formulate a generalizable NeRF that may be used across scenes.
As I understand it, NeRFs are still "single-use" in the sense that you need to train a new network for each scene.
I wonder if perhaps there could be components of the network which could be applied across scenes in order to leverage learning priors from the data.
wongquijote
One thing that helps the volumetric 3D representation part of NERF (which uses a MLP) converge is to use positional encoded input which will help embed high dimensional features into the network inputs.
GarciaEricS
It's very cool that there is no machine learning involved with this, it is pure optimization. However, it seems to me that that may actually end up hurting this system's practicality for real-time uses? Running an optimization algorithm takes a good bit of time, especially with feasibility constraints, so I wonder if a machine learning model which already has precomputed weights, would be able to perform faster should it be trained properly. Perhaps the model would need to be so large to be useful that it ends up being slower overall.
Really would recommend watching the video on this: https://www.matthewtancik.com/nerf. The difference between NeRF and prior models are pretty significant.
Something I'm curious about is how one might formulate a generalizable NeRF that may be used across scenes. As I understand it, NeRFs are still "single-use" in the sense that you need to train a new network for each scene. I wonder if perhaps there could be components of the network which could be applied across scenes in order to leverage learning priors from the data.
One thing that helps the volumetric 3D representation part of NERF (which uses a MLP) converge is to use positional encoded input which will help embed high dimensional features into the network inputs.
It's very cool that there is no machine learning involved with this, it is pure optimization. However, it seems to me that that may actually end up hurting this system's practicality for real-time uses? Running an optimization algorithm takes a good bit of time, especially with feasibility constraints, so I wonder if a machine learning model which already has precomputed weights, would be able to perform faster should it be trained properly. Perhaps the model would need to be so large to be useful that it ends up being slower overall.