Experts stumped by ‘ad Turing test’ /

A panel of experienced marketing professionals struggled to tell ads made by students from those generated by AI in a competition pitched as the first ‘ad Turing test’.

The panel of 16 brand and advertising experts – which included R/GA’s global CCO, Tiffany Rolfe, and Brandtech Group CCO, Mark D’Arcy – were shown 10 poster ads, all created from the same brief for a fictional energy drink.

Seven of the ads were made by creative teams from advertising schools and three were made by industry professionals with AI tools. But the judges were only able to identify which was which with 57% accuracy – little better than guesswork.

Noah Brier, who founded the Brxnd (pronounced ‘brand’) organisation in 2022 to explore the intersection between AI and marketing, organised the competition and announced the results at the inaugural Brxnd Conference in New York on 16 May.

Brier also asked the 300 people who attended his conference to take the ad Turing test, and they were only able to tell the difference between human and machine creativity with 53% accuracy.

Asked whether he was surprised by the results, Brier told Contagious: ‘I didn't have any idea what to expect, to be honest. When we started recruiting teams in December, the technology available to solve these problems was totally different than what was available in April.’

In addition to submitting them to the Turing test, Brier gave the ads created for the contest to System1, which measures people’s emotional reactions to ads to gauge their effectiveness. The Turing test ads created using AI tools scored an average of 1.83 on System1’s rating system, which was just below the score for the average US print ad, at 1.9.

The teams who used AI were allowed to work with multiple AI systems to create their ads but were banned from editing or otherwise altering the output generated by the machines, although Brier had no way of ensuring compliance.

One team was disqualified from the ad Turing test because they admitted (from the outset) that they used AI to generate assets for their ad but then put them together themselves. However, Brier still gave that ad to System1 to measure, and it received the highest score of any entry – human or machine – at 2.8. (This ad was not included in the average System1 score mentioned above.)

The competition was named after the Turing test, which was proposed by mathematician and computer scientist Alan Turing in a 1950 paper as a way to gauge machine intelligence through a three-player game, in which a human asks a computer and another human to answer questions and then tries to determine which is which based only on their written responses. 

Brier described the ad Turing test to Contagious as a ‘fun’ and ‘interesting’ experiment but was careful to acknowledge its limitations. For one, he said, the AI ads were put together by professionals who had decades of experience making campaigns.

Brier told Contagious that he does not think AI is going to replace human creativity but described it as ‘the most amazing creative accelerant I've ever experienced’. He said that he formed Brxnd to help push the conversation around AI and marketing forward.

Asked if he had been similarly enthused about the impact of blockchain technology and the metaverse on advertising and creativity, Brier responded: ‘Honestly, I'm still kind of confused by what the metaverse is. I thought blockchain was interesting, but I could never wrap my head around the use cases and often argued with people about whether they were just describing a database or not.’

Brier previously co-founded Percolate, a content marketing platform that raised $106.5m in funding before it was acquired by Seismic, a sales platform, in 2019.

If you would like to take the Ad Turing Test, you can do so, here.