CHENNAI, India — A 20-minute voice recording and a few pictures of a person’s face are all Senthil Nayagam needs to make eerily convincing fake campaign videos in any language.
Political parties in India spent an estimated $16 billion in the world’s largest election that culminated Tuesday, when Prime Minister Narendra Modi claimed victory despite a weakened mandate.
In what has been called India’s first “artificial intelligence election,” significant campaign funds were spent in hiring artists such as Nayagam, chief executive of the Chennai-based start-up Muonium AI, to deploy deepfakes promoting or discrediting candidates that spread like wildfire across social media.
These videos use artificial intelligence to generate believable but false depictions of real people saying or doing just about anything creators want, from a video of Modi dancing to a Bollywood song to a politician who died in 2018 being resurrected to endorse his friend’s run for office.
Experts say that what unfolded in India in the past several weeks is just a preview of the chaos that could upend electoral processes in this year of consequential global elections, including in the U.S. in November. While deepfakes are not new, advances in AI technologies mean they can now be produced at an unprecedented scale, with higher quality and accessibility.“What you’re seeing now is the use of AI tools to be able to take on hundreds of different issues and create lots of different messages around those pieces of conflict at scale,” said Josh Lawson, director of AI and democracy at Aspen Digital.
“It just dilutes the information environment,” he said.
Nayagam is known mainly for his AI work for Bollywood and TV commercials. When political parties he declined to identify asked him for help in their campaigning, he was open to it — up to a point.
“Anything which is unethical, we want to avoid it,” he said. “I don’t want to do what is very controversial.”
That includes, he said, requests to alter something said in Parliament and to create fake telephonic conversations. “We have solidly refused them,” Nayagam said.
He also tries to deal with campaign managers rather than politicians directly, and even then he chooses to work with only a few. “I prefer to say no over being sorry for it,” he said.
Nayagam’s worries stem from a wave of arrests over deepfakes of politicians such as Home Affairs Minister Amit Shah, a top Modi aide. During the election, a video of Shah giving a speech was altered to falsely depict him promoting policy proposals that are unpopular with India’s minority groups.
Though Modi had reshared the deepfake of himself dancing, calling it “truly a delight,” he said the video of Shah was part of a “conspiracy to create tension in society.”
At a soundproof room where he records his clients’ voices, Nayagam showed NBC News reporters a deepfake he made of influential politician Muthuvel Karunanidhi, in his signature yellow scarf and oversized sunglasses, congratulating his friend T.R. Baalu on the launch of his memoir in January. Karunanidhi died in 2018.Baalu, 82, a parliamentary candidate from the Indian state of Tamil Nadu, where Karunanidhi was chief minister, won his race on Tuesday.
Nayagam also recorded the voice of an NBC News reporter for 20 minutes, even asking her to sing to capture different phonemes and tones. A photographer from his team then took pictures of her in front of a green screen. The audio and pictures were used to make a “model” that would then be used to generate the final deepfake.
In a matter of hours, Nayagam’s team produced a video that showed the reporter speaking fluently in Italian and Hindi, morphed her face into his daughter’s and made her appear as if speaking from the White House.
“Literally you have been cloned,” he said. “It’s like we have a seed and we can plant it anywhere and it can grow into any kind of tree or jungle.”
AI tasks that used to take days are only going to get faster, Nayagam said: “We will go into minutes and seconds in the future.”
Lawson said there is a risk people could lose sight of what’s reality. “So instead of voting based on facts and verifying what I understand, I am going to vote based on my vibes and intuition,” he said. Worried U.S. lawmakers have taken steps to guard the democratic process against AI, with at least 11 states banning election interference through the use of deepfakes.
Lawmakers in California, home to many of the world’s largest AI companies, are considering a number of AI-related bills, including one that would ban “materially deceptive” deepfakes related to elections in the 120 days before polling.
The lawmakers said they were concerned by AI-generated robocalls New Hampshire Democrats received in January in which a voice mimicking President Joe Biden’s encouraged them not to vote in the state’s primary. Bipartisan legislation banning the use of AI to falsely depict federal candidates has stalled in both the House and Senate and is unlikely to pass before the November election.
Stringent regulations are needed to tackle AI-powered disinformation during elections, said Raman Jit Singh Chima, the Asia-Pacific policy director at Access Now, a U.S.-based nonprofit focusing on digital civil rights.
The Big Tech platforms where the content is ultimately being shared also have a responsibility in moderating it, he said.
“You need proactive disclosure and regulatory measures on political candidates and parties,” he said.
While it is unclear to what extent AI affected voter perception or influenced results in India, the U.S. and other countries can draw lessons that could be crucial in protecting the integrity of their own elections, Lawson said.
Gathering data over the next few months on how the technology ultimately impacted the Indian election could give that critical insight.
“We need to do that right now,” he said.
