NTT R&D Forum2023　Special session2：Natured Inspired Intelligence and a New Paradigm for LLM

The speaker begins by expressing gratitude for the opportunity to speak at the NTT R&D Forum and introduces Sakana AI, an AI R&D company co-founded with his friend, Lion Jones. The name 'Sakana' is derived from the Japanese word for fish, symbolizing the collective intelligence and independent thinking that the company embodies. The speaker highlights the company's focus on nature-inspired research, such as collective intelligence and evolution, and the importance of non-conformity in pursuing innovative AI solutions. The speaker's background includes studying engineering science with a focus on neural networks at the University of Toronto and later working on Wall Street. He then transitioned to Google's Brain Team, where he contributed to generative AI research, including projects on multi-agent systems, evolution, and language models. The talk aims to cover Sakana AI's team, technical vision, and the rationale for starting the company in Japan.

The speaker discusses his skepticism about the current approach to AI development, particularly the belief that scaling up existing models will lead to strong AI. He advocates for a new type of foundation model inspired by complex adaptive systems, which are more adaptive and integrated with their environment. The speaker critiques the energy inefficiency and rigidity of large language models, suggesting that they are prone to security attacks and require constant fine-tuning. He references his own research, including a paper on 'collective intelligence for deep learning,' which explores nature-inspired systems. The speaker also mentions his work on 'weight agnostic neuron networks,' which demonstrated that neural network architectures can function effectively even with randomized weights, challenging the prevailing paradigm of AI model training.

The speaker explains the decision to establish Sakana AI in Japan, noting the country's growing interest in AI and the potential for developing a strong AI R&D ecosystem. He expresses concern about the current AI field being too Western or China-centric and emphasizes the importance of Japan, as a democratic country, becoming a leader in AI. The speaker believes that Japan's economic strength and cultural support for AI provide a solid foundation for creating a global AI ecosystem. He also addresses the challenge of attracting AI talent to Japan, countering the notion that it is difficult by arguing that Japan's appeal makes it easier to draw top talent. Since the company's launch, they have received numerous resumes from qualified candidates worldwide, indicating a strong interest in joining their efforts.

The speaker, known for his work on the Transformer model, discusses his research on character-level language modeling. He explains the concept and its advantages, such as not requiring a predefined vocabulary and potentially being more efficient for languages like Japanese. Despite the increased computational demands, character-level models can handle out-of-vocabulary words better and may be more suitable for morphologically rich languages. The speaker highlights his early work on character-level models for question answering and how pre-training these models improved their performance. He also touches on the cultural bias in AI research towards English and the need for Japanese companies to embrace character-level modeling given its benefits for the Japanese language.

The speaker delves into the history and impact of language model pre-training, starting with his 2016 paper that suggested its use for character-based models. He discusses how pre-training has become a standard practice, particularly with the success of Transformer models. The speaker's work showed significant improvements in language modeling performance when using character-level models, leading to a substantial reduction in bits per character. He emphasizes the importance of these findings, which were pivotal in the development of large-scale language models like GPT. Despite the success of word-level models, the speaker maintains that character-level modeling is a more logical and efficient approach, especially for languages with complex character systems.

The speaker addresses the limitations of word-level language models, particularly their inability to process characters within words. He uses examples, such as the difficulty of spelling words in reverse, to illustrate the challenges these models face. The speaker argues that character-level models can overcome these issues, as they can spell and manipulate characters more accurately. He also points out that large language models, despite their impressive capabilities, still struggle with spelling in image generation tasks when using word-level tokenization. The speaker concludes by advocating for a shift to character-level modeling, which he believes is not only simpler but also a better fit for languages like Japanese and potentially others with character-based writing systems.

In the final segment, the speaker engages in a discussion about the importance of language diversity in AI, particularly the Japanese language's role in developing AI models. He agrees with the idea that understanding Japanese can provide a different perspective on problem-solving, which is valuable for AI development. The speaker also touches on the potential for character-level modeling to be applied to other languages, such as Korean and Chinese, and the benefits of multilingual training for AI models. He concludes by suggesting that a universal language structure may exist, as evidenced by AI's ability to translate between language pairs it was not explicitly trained on, indicating a deeper understanding of language at a core level.