gaussian-splatting
pytorch-CycleGAN-and-pix2pix
gaussian-splatting | pytorch-CycleGAN-and-pix2pix | |
---|---|---|
7 | 10 | |
11,639 | 22,070 | |
8.1% | - | |
8.7 | 2.5 | |
9 days ago | 6 days ago | |
Python | Python | |
GNU General Public License v3.0 or later | GNU General Public License v3.0 or later |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
gaussian-splatting
-
Show HN: Gaussian Splat renderer in VR with Unity
Chris' post doesn't really give much background info, so here's what's going on here and why it's awesome.
Real-time 3D rendering has historically been based on rasterisation of polygons. This has brought us a long way and has a lot of advantages, but making photorealistic scenes takes a lot of work from the artist. You can scan real objects like photogrammetry and then convert to high poly meshes, but photogrammetry rigs are pro-level tools, and the assets won't render at real time speeds. Unreal 5 introduced Nanite which is a very advanced LoD algorithm and that helps a lot, but again, we seem to be hitting the limits of what can be done with polygon based rendering.
3D Gaussian Splats is a new AI based technique that lets you render in real-time photorealistic 3D scenes that were captured with only a few photos taken using normal cameras. It replaces polygon based rendering with radiance fields.
https://repo-sam.inria.fr/fungraph/3d-gaussian-splatting/
3DGS uses several advanced techniques:
1. A 3D point cloud is estimated by using "structure in motion" techniques.
2. The points are turned into "3D gaussians", which are sort of floating blobs of light where each one has a position, opacity and a covariance matrix defined using "spherical harmonics" (no me neither). They're ellipsoids so can be thought of as spheres that are stretched and rotated.
3. Rendering is done via a form of ray-tracing in which the 3D Gaussians are projected to the 2D screen (into "splats"), sorted so transparency works and then rasterized on the fly using custom shaders.
The neural network isn't actually used at rendering time, so GPUs can render the scene nice and fast.
In terms of what it can do the technique might be similar to Unreal's Nanite. Both are designed for static scenes. Whilst 3D Gaussians can be moved around on the fly, so the scene can be changed in principle, none of the existing animation, game engines or artwork packages know what to do without polygons. But this sort of thing could be used to rapidly create VR worlds based on only videos taken from different angles, which seems useful.
-
Bad accuracy after model training, Can someone help me ?
the repo which I used is : https://github.com/graphdeco-inria/gaussian-splatting
- The initial work to get the gaussian-splatting training code working on AMD/ROCm has been done
- Future Tech SD VR, 3D modeling, Movies, and Video Game Creation (Paper / Videos included)
-
Show HN: Real-Time 3D Gaussian Splatting in WebGL
Really cool, I am also working on a port of gaussian-splatting [0] but to WebGPU.
Like all the other implementations I have seen so far, this also makes the same mistake when projecting the ellipsoids in a perspective: First you calculate the covariance in 3D and then project that to 2D [1]. This approach only works with parallel / orthographic projections and applying it to perspectives leads to incorrect results. That is because perspective projections have two additional effects:
- Parallax movements (that is the view plane moves parallel to the ellipsoids) change the shape of the projected ellipse. E.g. a sphere only appears circular when in center of the view, once it moves to the edges it becomes stretched into an ellipse. This effect is manually counter balanced by this matrix I believe [2].
- Rotating an ellipse can change the position it appears at, or in other words creates additional translation. This effect is zero if the ellipse has one of its three axes pointing straight at the view (parallel to the normal of the view plane). But, if it is rotated 45°, then the tip of the ellipse that is closer to the view plane becomes larger through the perspective while the other end becomes smaller. Put together, this slightly shifts the center of the appearance away from the projected center of the ellipsoid.
- Conic sections can not only result in ellipses but also parabola and hyperbola. This however is an edge case that only happens when the ellipsoid intersects with the view plane and can probably be ignored as one would clip away such ellipsoids anyway.
The last two effects are not accounted for in these calculations in any of the implementations I have seen so far. What would be correct to do instead? Do not calculate the 3D covariance. Instead calculate the bounding cone around the ellipsoid which has its vertex at the camera position (perspective origin). Then intersect that with the view plane and the resulting conic section is guaranteed to be the correct contour of the perspective projection of the ellipsoid.
[0]: https://github.com/graphdeco-inria/gaussian-splatting
pytorch-CycleGAN-and-pix2pix
-
List of AI-Models
Click to Learn more...
- I want an A.I. to learn my art style so I can keep making art in my art style despite not having the time to do it.
-
I'm looking for an AI Art generator from images
pix2pix (https://github.com/junyanz/pytorch-CycleGAN-and-pix2pix) - This is a PyTorch implementation of the pix2pix algorithm for image-to-image translation. Given a set of images, the model can learn to generate a new image from a different domain that is similar to the input image.
-
Seamless textures with SD and PBR maps with a pix2pix cGAN
Using junyanz/pytorch-CycleGAN-and-pix2pix as a basis for pix2pix, I applied the same blending method to fix seams. It essentially takes an input image and generates an output. The results depend on the paired training data. In this case, each map (height, roughness, etc.) is a separate checkpoint and had to be trained on paired training data with the diffuse as the input and the respective map as the output.
- IA art
- Segmentation and clasification with UNET
-
Trying to understand PatchGAN discriminator
Code for https://arxiv.org/abs/1611.07004 found: https://github.com/junyanz/pytorch-CycleGAN-and-pix2pix
-
I made a 3d topographic map based on my recent civ6 game
pix2pix algorithm is used for translating Civ6Maps to heightmaps. Synthesized terrain was rendered in blender.
-
This Wojak Does Not Exist
https://github.com/junyanz/pytorch-CycleGAN-and-pix2pix
-
Training a neural net to generate Wojaks
I'm working on creating a face-to-wojak model using PyTorch CycleGan/Pix2Pix [0] and found some of my outputs to be outrageous yet somehow relatable. People are into it so thought I'd share on HN
[0] https://github.com/junyanz/pytorch-CycleGAN-and-pix2pix
What are some alternatives?
pix2pixHD - Synthesizing and manipulating 2048x1024 images with conditional GANs
splat - WebGL 3D Gaussian Splat Viewer
generative-inpainting-pytorch - A PyTorch reimplementation for paper Generative Image Inpainting with Contextual Attention (https://arxiv.org/abs/1801.07892)
pytorch-grad-cam - Advanced AI Explainability for computer vision. Support for CNNs, Vision Transformers, Classification, Object detection, Segmentation, Image similarity and more.
Deep-Fakes
AnimeGAN - Generating Anime Images by Implementing Deep Convolutional Generative Adversarial Networks paper
PaddleGAN - PaddlePaddle GAN library, including lots of interesting applications like First-Order motion transfer, Wav2Lip, picture repair, image editing, photo2cartoon, image style transfer, GPEN, and so on.
Anime-face-generation-DCGAN-webapp - A port of my Anime face generation using Pytorch into a Webapp
DeepMosaics - Automatically remove the mosaics in images and videos, or add mosaics to them.
SimSwap - An arbitrary face-swapping framework on images and videos with one single trained model!
anycost-gan - [CVPR 2021] Anycost GANs for Interactive Image Synthesis and Editing