Facebook Builds New AI That Predicts Future Process Steps and Aid in Guiding Activity

This might be intriguing, especially given Facebook's continuing development of augmented reality devices.

The Social Network has announced a new machine learning technique called 'Anticipative Video Transformer (AVT)' that may predict future activities in a visual interpretation-based process.

As seen in this example, the new process may examine an activity and then predict what action will follow as a result.

According to Facebook, this might have a variety of applications:

“AVT could be especially useful for applications such as an AR “action coach” or an AI assistant, by prompting someone that they may be about to make a mistake in completing a task or by reacting ahead of time with a helpful prompt for the next step in a task. For example, AVT could warn someone that the pan they’re about to pick up is hot, based on the person’s previous interactions with the pan.”

That appears like something out of a science fiction film, allowing for whole new smart home applications. Moreover, in the context of AR glasses, this might provide various useful tips to assist individuals in doing different tasks at home or at work.

“We train the model to predict future actions and features using three losses. First, we classify the features in the last frame of a video clip in order to predict labeled future action; second, we regress the intermediate frame feature to the features of the succeeding frames, which trains the model to predict what comes next; third, we train the model to classify intermediate actions. We’ve shown that by jointly optimizing the three losses, our model predicts future actions 10 percent to 30 percent better than models trained only with bidirectional attention.”

It's not something Facebook plans to implement immediately, but the potential is huge, and it might someday enable all kinds of new ways of guiding user behaviors and eliminating errors by anticipating future steps.

Facebook presents the example of replacing a car tire, with AR glasses guiding you in the appropriate direction. It may also serve as a reminder for your daily routines, based on visual assessments of where you are and what you're doing.

The possibilities are truly unlimited, and when you consider how Google Glass turned into a major tool in industrial workspaces by offering in-view pointers and instructions for technical applications, Facebook's wearable AR devices have even more potential.

It's still a long way from being a consumer-facing product, but the project highlights Facebook's ongoing AI development and speaks to the expanding capabilities that'll likely be included into a future stage of its AR glasses initiatives.

More information on Facebook's Anticipative Video Transformer (AVT) process may be found here.