subscribe Support our award-winning journalism. The Premium package (digital only) is R30 for the first month and thereafter you pay R129 p/m now ad-free for all subscribers.
Subscribe now
Picture: 123RF/pitinan
Picture: 123RF/pitinan

Two prominent former Google researchers, one of whom was a co-inventor of the “transformer” artificial intelligence architecture that paved the way for the generative AI boom, announced on Thursday they had started a new AI company based in Tokyo.

Sakana AI, founded by David Ha and Llion Jones, declined to disclose funding details.

Jones is the fifth author on Google’s 2017 research paper “Attention is all you need”, which introduced the “transformer” deep-learning architecture that went on to become the basis for chatbot ChatGPT and the current race to develop products powered by generative AI. Ha was previously the head of research at Stability AI and a Google Brain researcher.

After the explosive paper came out, advances in generative AI foundation models have centred on making the “transformer”-based models larger and larger. Instead of doing that, Sakana AI will focus on creating new architectures for foundation models, Jones said.

“Rather than building one huge model that sucks all this data, our approach could be using a large number of smaller models, each with their own unique advantage and smaller data set, and having these models communicate and work with each other to solve a problem,” said Ha, though he clarified this was just an idea.

All the authors on the “Attention Is All You Need” paper have now left Google. The authors' new ventures have attracted millions in funding from venture investors, including Noam Shazeer, who is running AI chatbot start-up Character.AI, and Aidan Gomez, who founded large language model start-up Cohere.

Reuters 

subscribe Support our award-winning journalism. The Premium package (digital only) is R30 for the first month and thereafter you pay R129 p/m now ad-free for all subscribers.
Subscribe now

Would you like to comment on this article?
Sign up (it's quick and free) or sign in now.

Speech Bubbles

Please read our Comment Policy before commenting.