MoveNet - Next-Generation Pose Detection

MoveNet is an ultra-fast and accurate model that detects 17 keypoints of a body. The model is offered on TF Hub with two variants (Lightning and Thunder). Lightning is intended for latency-critical applications, while Thunder is intended for applications that require high accuracy. Both models run faster than real time (30+ FPS) on most modern desktops, laptops, and phones, which proves crucial for live fitness, sports, and health applications. This is achieved by running the model completely client-side, in the browser using TensorFlow.js with no server calls needed after the initial page load and no dependencies to install.

Human pose estimation has come a long way in the last five years, but surprisingly has not surfaced in many applications just yet. This is because more focus has been placed on making pose models larger and more accurate, rather than doing the engineering work to make them fast and deployable everywhere. The mission is to design and optimize a model that leverages the best aspects of state-of-the-art architectures, while keeping inference times as low as possible. The result is a model that can deliver accurate keypoints across a wide variety of poses, environments, and hardware setups.

More information:

https://blog.tensorflow.org/2021/05/next-generation-pose-detection-with-movenet-and-tensorflowjs.html

AI Creates 3D Holograms for VR

A new method from MIT creates holograms for virtual reality, 3D printing and other potential applications in real-time. The team designed a convolutional neural network, a processing technique that uses a chain of trainable tensors to roughly mimic how humans process visual information. Training a neural network typically requires a large, high-quality dataset, which did not previously exist for 3D holograms. The team built a custom database of 4,000 pairs of computer-generated images. Each pair matched a picture (including color and depth information for each pixel) with its corresponding hologram.

To create the holograms in the new database, the researchers used scenes with complex and variable shapes and colors, with the depth of pixels distributed evenly from the background to the foreground, and with a new set of physics-based calculations to handle occlusion. That approach resulted in photorealistic training data. Next, the algorithm got to work. By learning from each image pair, the tensor network tweaked the parameters of its own calculations, successively enhancing its ability to create holograms.

More information:

https://bigthink.com/technology-innovation/3d-holograms

Realistic VR Oil Painting

A new application called Vermillion aims to bring oil painting to VR, with realistic painting mechanics and web browser integration. There are a few key features that are interesting for Vermillion and position it as a realistic oil painting sim for VR. First, it has traditional analog color mixing, which basically just means you can take different pigments and mix them together on your palette to find the ideal tone. It also has an integrated web browser that can stand next to your easel while you paint. This allows you to pull up painting tutorials, like those by the famous Bob Ross, and paint along to with ease. It is a great feature that has a very Infinite Office feel to it.

The other big feature is Vermillion’s ability to simulate wet-on-wet paint, essentially allowing you to move colors around and blur strokes after you have painted them onto the canvas, just like real life. You can also use pressure to change the thickness and application of your strokes as well, just as you would with real painting. Vermillion is not the only painting and art-focused app we have seen pop up lately. Painting VR, available on App Lab for Quest, is similar in premise but a bit more focused on providing a sandbox environment for you to muck around in. Vermillion plans to launch in Early Access in the Summer for PC VR on Steam, with plans for a Rift Oculus Store launch to follow and a Quest port in hot pursuit.

More information:

https://uploadvr.com/vermillion-brings-realistic-oil-painting-to-vr