Facebook’s AI Model Copies Text Styles and Edits Text in Images

The past few years have taught us that our faces, voices, and lips can be copied and replicated with artificial intelligence. Now, an AI model created by Facebook researchers can imitate, edit, and replace handwritten and scene text using just a single word in an image.

Facebook unveiled TextStyleBrush, an AI research project, on Friday. The company points out that while most AI systems can replicate and replace text for well-defined and specialized tasks, TextStyleBrush is different because it can reproduce text in both handwriting and real-world scenes. Doing this is a lot more difficult for an AI model because of the different text options and nuances involved.

“It means understanding unlimited text styles for not just different typography and calligraphy, but also for different transformations, like rotations, curved text, and deformations that happen between paper and pen when handwriting; background clutter; and image noise,” Facebook explained in a news announcement.

TextStyleBrush is the first self-supervised AI model that can replicate and replace text in handwritten and real-world scenes by using just a single example word from an image, according to Facebook. It displayed high accuracy in automated tests and user studies for any type of text, the company said, adding that it intended to submit the work to a peer-reviewed journal.

The AI model works in a way that is similar to the way the style brush tools used in word processors do. In this case, the method is applied to text aesthetics in images. However, Facebook researchers decided to forgo traditional methods, which include defining specific parameters such as typeface or target style supervision, with TextStyleBrush.

“[W]e take a more holistic training approach and disentangle the content of a text image from all aspects of its appearance of the entire word box,” Facebook said. “The representation of the overall appearance can then be applied as one-shot-transfer without retraining on the novel source style samples.”

TextStyleBrush isn’t without its limitations, though. Facebook said these include working with text written in metallic objects or text that has characters in different colors.

While at first glance this AI model seems like a cool and efficient way to copy and replace text, it also shouldn’t be seen as just an innocent tool. One of the first things that I thought of, for instance, was how TextStyleBrush could be used to trick others or even commit crimes.

Facebook acknowledges that deepfake text attacks are a problem, and said it hoped that by openly publishing its research on TextStyleBrush, it could encourage new research to preempt these attacks. The company said that if AI researchers and practitioners can build out this technology before bad actors do, they can “learn to better detect this new style of deepfakes and build robust systems to combat them.”

Update 6/12/2021 9:30 p.m. ET: Embedded a link with an example of TextStyleBrush.