Building on this work, today we’re introducing PaLM 2, our next generation language model. PaLM 2 is a state-of-the-art language model with improved multilingual, reasoning and coding capabilities.
Over the past decade, Google has played a leading role in the major advancements in the field of AI. Their groundbreaking work in foundation models has set the industry standard and paved the way for AI-powered products that billions of people use every day. As Google continues to responsibly push the boundaries of AI, there is immense potential for transformative applications in areas like healthcare and human creativity.
Through years of AI development, Google has discovered that scaling up neural networks can unlock a wealth of possibilities. However, they have also learned that simply making models bigger does not always lead to better performance. Research creativity is crucial in building exceptional models. Recent advancements in model architecture and training techniques have enabled Google to unlock multimodality, recognize the significance of human feedback, and develop models more efficiently than ever before. These advancements serve as powerful building blocks as Google strives to advance the state of AI and create models that truly benefit people in their daily lives.
Introducing PaLM 2, Google’s next-generation language model, built upon their previous work. PaLM 2 represents a state-of-the-art language model with enhanced multilingual capabilities, reasoning abilities, and coding proficiency.
Multilinguality: PaLM 2 has undergone extensive training on multilingual text encompassing over 100 languages. This has significantly improved its capacity to comprehend, generate, and translate nuanced text across a wide range of languages, including idioms, poems, and riddles—a challenging problem to tackle. PaLM 2 has also demonstrated proficiency in advanced language exams at the “mastery” level.
Reasoning: PaLM 2’s comprehensive dataset includes scientific papers and web pages containing mathematical expressions. As a result, it exhibits improved capabilities in logic, common sense reasoning, and mathematics.
Coding: PaLM 2 was pre-trained on a vast amount of publicly available source code datasets. This means it excels at popular programming languages such as Python and JavaScript, while also having the ability to generate specialized code in languages like Prolog, Fortran, and Verilog.
PaLM 2 is not only more capable but also faster and more efficient than its predecessors. It comes in various sizes, namely Gecko, Otter, Bison, and Unicorn, allowing for easy deployment across a wide range of use cases. Gecko, being lightweight, can even function on mobile devices and support great interactive applications on-device, even offline. This versatility enables PaLM 2 to be tailored to support entire classes of products in multiple ways, ultimately benefiting a larger audience.
PaLM 2 powers over 25 Google products and features, as announced at I/O. It brings advanced AI capabilities directly into these products, serving consumers, developers, and enterprises of all sizes worldwide. Examples include expanding Bard to new languages, enabling coding updates, enhancing workspace features in Gmail, Google Docs, and Google Sheets for improved writing and organization, and introducing Med-PaLM 2 for medical knowledge extraction and synthesis. Additionally, Sec-PaLM focuses on security use cases to aid in cybersecurity analysis.
Google has also introduced the PaLM API for developers and customers, allowing access to the PaLM 2 model. Furthermore, Gemini, Google’s upcoming model, is being developed to be multimodal, highly efficient in tool and API integrations, and designed to enable future innovations such as memory and planning. Gemini, currently in training, exhibits never-before-seen multimodal capabilities. Once fine-tuned and rigorously tested for safety, Gemini will be available in various sizes and capabilities, similar to PaLM 2, to ensure widespread deployment across different products, applications, and devices for the benefit of all.
Google’s PaLM 2 model, while not providing detailed technical information about its training process, offers several notable improvements in capabilities. The focus has shifted from the size of the model to its functionality, and Google emphasizes that parameter count is not a reliable measure of a model’s capabilities.
PaLM 2 demonstrates advancements in common sense reasoning, mathematics, and logic. It has been trained on extensive math and science texts, including mathematical expressions, enabling it to handle math questions and puzzles, reason through problems, and provide diagrams. This addresses a challenge faced by large language models in handling mathematical queries without relying on external plugins.
Improved support for writing and debugging code is another key feature of PaLM 2. The model was trained on 20 programming languages, including JavaScript, Python, Prolog, Verilog, and Fortran. PaLM 2 serves as the foundation for Codey, Google’s specialized model for coding and debugging, which is being launched alongside PaLM 2.
PaLM 2 boasts enhanced multilingual capabilities, having been trained on a corpus that covers over 100 languages. It excels at multilingual tasks, including nuanced phrasing in different languages.
Google presents PaLM as a family of models, encompassing Codey, Med-PaLM 2 (focused on medical knowledge), Sec-PaLM (for security use cases), and a smaller version of PaLM 2 that can run on smartphones. The smaller model has potential applications in privacy-centric use cases. While the processing speed is not extremely fast at 20 tokens per second, it may be suitable for certain scenarios. Google did not disclose the specific phone used for testing.
Overall, Google’s PaLM 2 introduces significant advancements in capabilities, particularly in areas such as multilingual understanding, mathematics, logic, and coding.
