Generative AI video startup Tavus raises $18M to bring face and voice cloning to any app

Tavus, a four-year-old generative AI startup that helps companies create digital “replicas” of individuals for automated personalized video campaigns, has confirmed a fresh $18 million in funding and revealed that it’s opening its platform for third-parties to integrate its technology into their own software.

Reports emerged back in August that Tavus had raised “about $18 million,” but details were scant. The company has now confirmed to TechCrunch that it has indeed raised $18 million in a Series A round led by Scale Venture Partners — an early-stage VC that has previously backed the likes of Box, HubSpot, and DocuSign. Other notable investors include Sequoia, which led Tavus’s $6.1 million seed round last year, which participated alongside Y Combinator (YC) and HubSpot.

Video takes center stage

The generative AI movement is best exemplified by text-based search engines like ChatGPT and text-to-image models such as DALL-E, which OpenAI is in the midst of combining into a single all-singing platform. But if the past few months have been anything to go by, generative AI could be on the cusp of another minor revolution, with video taking center stage.

OpenAI recently debuted Sora, a text-to-video model that could transform the creative industry as we know it. But it’s far from the only player in town, with tech giants such as Google working on similar tooling for several years, not to mention a slew of startups that have raised sizeable chunks of VC change over the past year for various realizations of how generative AI might intersect with video.

Tavus, for its part, works with its clients to create replicas of individuals through voice and face cloning. The idea is that sales and marketing teams can use Tavus to send personalized videos to prospects at scale, or maybe a product team can create individualized walkthrough videos for onboarding new customers — all via simple text-based prompts that leverage the previously created digital replica. And by integrating Tavus with third-party systems such as Salesforce or Mailchimp, companies can automate much of this — for instance, a customer that completes an online form requesting further information on a product can be emailed a video instantly, with a sales rep addressing the prospect by name and explaining the next steps.

Tavus has managed to secure some fairly big-name customers in its short life so far, including Salesforce and Facebook’s parent Meta, which co-founder and CEO Hassaan Raza said are using the platform to upsell to their respective B2B customers through personalized demo videos.

Tavus as a platform

So far, Tavus has been served via a SaaS app, through which customers create their own AI video templates. The onboarding process requires an individual, such as the CEO or sales executive, to record a 15-minute video based on a script provided by Tavus.

This is then used to train the AI, after which the user goes to a web-editor and selects which parts of the video they wish to personalize by defining the variables — such as location, executive name, company, or product. By tying Tavus into their CRM system, companies can tweak each of these variables to suit a particular customer segment, such as those who have expressed an interest in a particular product. 

Companies can create hundreds of these replicas with different personnel involved, replete with different backgrounds for different target markets.

Through the in-app editor, it’s possible to generate any number of different scripts to attach to each use-case — without having to re-record any of the original video.

While this core SaaS product isn’t going away, Tavus is today lifting the lid on a new turbo-charged version of its technology alongside the first instalment of a suite of developer APIs that allow third-parties to integrate Tavus into their own applications.

Replicate

The first facet of Tavus’s new developer platform to arrive is its “replica API,” which is all about creating “photo-realistic” digital replicas replete with text-to-video generation. With this, a company can replicate a person (e.g. head of marketing or CEO) using a new proprietary model created by Tavus dubbed “Phoenix” which is based on a deep learning method called neural radiance field (NeRF). This can generate a 3D construct of a person from 2D images in just a couple of minutes

“It essentially allows you to create entire videos with just two minutes of training data, which is a big leap forward from how we were previously doing the personalization at scale,” Raza told TechCrunch. “And so now all you have to do is record two minutes of training data, and it’ll create a full replica of you. And once you have replica, you can make as many videos as you want — from one, two, or a thousand scripts.”

The inaugural replica API leans on the entire functionality of the Phoenix model, and captures an individual’s facial motions including cheeks, nose, eyebrows, and lips.

“Moving your entire face drives realism, naturalness and quality — when you talk, your face expresses emotion beyond your lips moving,” Raza explained. “If you want to generate an entire video from a script — where you are speaking, one that looks natural and is incredibly high quality — you’d want to use the replica API.”

However, Tavus is also developing a number of additional APIs, including one specifically for lip-syncing; one for dubbing; and one for running mass, personalized video campaigns.

The lip-sync API will have a “lower entry cost,” according to Raza, and is better for situations where a “high-degree of quality and realism is not necessary.”

The dubbing API, meanwhile, also uses the lip-sync model but includes multi-language voice cloning too, meaning a monolinguist user can send out video campaigns in any number of languages using their own voice. In this instance, given that most of the video will remain the same, the API enables simple replacement of lip-movements to align with the different sounds coming from the user’s mouth. This could prove useful for the creators of a video-editing software suite, for example, where they wish to enable their users to add lip-syncing, editing, and dubbing to their videos.

And then the video campaign API basically bundles the replica API alongside a swathe of additional tooling — such as hosting, variable mapping, thumbnails, and analytics — for those looking to launch large-scale video campaigns.

“We’re bringing the ability for any developer to provide a end-to-end video campaign experience out of the box, within their own solutions,” Raza said. “Whereas the replica and lip-sync APIs are more ‘model-as-a-service,’ the campaign API gives you tools to build an AI video campaign platform easily.”

Raza remained coy on some of the early users are of the Tavus platform, but he did say that it’s “working with one of the largest video platforms” for customer engagement. “They’re looking to bring this to their millions of customers that are already using their platform to create video on a daily basis,” Raza said.

Deepfake dilemma

Instinctively, platforms such as Tavus are ripe for misuse — after all, what’s stopping anyone from uploading a pre-existing video to create a digital replica? Deepfakes are indeed a growing concern in the burgeoning AI movement, but Raza says that they have checks in place to avert chicanery. For instance, when a user submits their two minutes of training footage, they also have to submit a specific verbal consent statement which is then aligned to the audio in the training footage to ensure there is a match.

“We run these checks automatically, and then do a human check for every replica that makes it through the automated checks to ensure safety,” Raza said.

It’s easy to see how that might work with Tavus as a standalone SaaS app, but now that it’s a platform accessed by any number of companies via an API, who is in control of verification then? Well, as it turns out, Tavus is — the company wants to keep its hands on the verification wheel, even when it’s merely providing the engine for third-party developers.

“We run the same checks, and assume responsibility for verifications with [the] API as well,” Raza continued.

Extending reality

While OpenAI has become almost the public face of generative AI, there is more than enough room for different players bringing something different to the mix. Indeed, while DALL-E and OpenAI’s recently released Sora model are mostly about helping people create visuals from text prompts, Raza says Tavus’s raison d’être is more about “extending” a person’s own reality.

“We see a future where everyone wants to have a digital replica of themselves, they control that and they have full authority over that,” Raza said. “And it’s gonna be important that it actually ends up capturing more and more of your personality, more and more of your gestures and traits. That’s how we see things going forward — there will be the models that create things that don’t exist, and then there’ll be the models that extend your reality.”

With $18 million in the bank, Raza said that the recent cash injection will be used to “fuel the fire that’s already burning” at Tavus towers.

“We’re an AI research company, so we want to be able to continue development on newer models like Phoenix,” Raza said. “But then also just sustain our growth, we’ve had a tonne of demand continuously. And we want to be able to continuously hire on our machine learning and engineering teams to support our developer and SaaS customers.”