Summary of ChatGPT's One-year Anniversary: Are Open-Source Large Language Models Catching up?

Many individuals have become reliant on ChatGPT as a personal assistant, but concerns about its unpredictability, such as website issues and company turmoil, have encouraged the search for alternatives. ChatGPT, released at the end of 2022, caused a significant shift in AI, demonstrated by its ability to answer human questions accurately across various tasks. Interest in Large Language Models (LLMs) has surged, with new LLMs emerging frequently, especially from startups focusing on open-source alternatives.

Introduction

ChatGPT made a splash in the AI community by providing helpful, safe, and detailed answers, attracting 100 million users within two months of its launch. Concerns about its closed-source nature and access control by private companies have led to an increased focus on open-source LLMs.

Background

Training Modes

All LLMs rely on large-scale self-supervised pre-training, with data coming from various internet sources. Fine-tuning adapts pre-trained LLMs to specific downstream tasks, with instructive fine-tuning becoming increasingly popular.

Task Domains and Evaluation

Evaluating LLMs is an active research area, with various benchmarks being developed to assess capabilities in general knowledge, reasoning, and specific applications.

Open-source LLMs vs ChatGPT

General Capabilities

Open-source LLMs like Llama-2-70B have shown promising results, indicating a closing gap with proprietary models such as GPT-4, despite the latter's superior performance.

Agent Capabilities

Open-source LLMs have begun to surpass GPT-3.5-turbo in agent capabilities, thanks to specialized fine-tuning and pre-training.

Logical Reasoning Abilities

Open-source models like WizardCoder and WizardMath have improved logical reasoning by using enhanced instructive fine-tuning.

Modeling Long Context Capabilities

While GPT-3.5-turbo remains a strong performer in tasks involving long contexts, models like Llama-2-long are making strides in this area.

Application-Specific Capabilities

Open-source LLMs are starting to outperform GPT-3.5-turbo in specific applications like mental health analysis and radiology reports.

Moving Towards Trustworthy AI

Efforts to reduce illusions and improve safety in LLMs are vital for building trust, with GPT models demonstrating safer and more ethical behavior due to RLHF processes.

Discussion

LLM Development Trends

The release of ChatGPT shifted the focus of NLP research, leading to releases like Google's Bard and Anthropic's Claude. The success of these models is largely attributed to RLHF.

Best Practices for Open-source LLMs

Developing LLMs involves complex processes like data preparation, model architecture design, and training, with the community recognizing several best practices for each stage.

Challenges and Potential Problems

Issues such as data contamination during pre-training and the closed development of alignment techniques pose challenges to the progress of LLMs.

Conclusion

This survey offers insights and potential directions for open-source LLMs, indicating they are closing the gap with proprietary models like ChatGPT and sparking further research and development.

Reference: ChatGPT's One-year Anniversary: Are Open-Source Large Language Models Catching up?