current thoughts on large language models
October 11, 2023.
The Big Data of the the early 2000's is just data today. Big is subjective, similar to the intelligence of AI and the largeness of language models. For LLMs, this is crucial because we know that specific behaviors only emerges at a certain scale. The concept of "this doesn't work now, therefor it will never work" doesn't apply to a field where the underlying axioms change. For language models, the most capable model serves as the current axiom which means that perspective needs to change from "this doesn't work" to "this doesn't work yet". This adaption is unfamiliar but is an important one to hold close if you work in language modelling research.
As a response to this, one can get ahead of the scaling curve by documenting experiment failures because of a lack of insufficient "intelligence". Failure should only be considered as a function of scale and experiments should be made easy to return to once scaling has progressed. This will help build a strong intuition for what models are capable of at a given scale, where abilities emerge and the relationship between such emergence.
Scaling isn't an unconditional solution
In the end, we've only trained a model to predict the next token in a sequence and while this captures remarkable capabilities, it doesn't make a model helpful, safe, communicative, ... . Post-training techniques are necessary.
For dialogue systems, a nifty hack is to frame the prompt so that the next token automatically becomes an answer.
Q: When did the deep learning revolution start?
A:
Here, the model picks of just after the A:, answering the question without needing to be specifically modeled to. While simple, this hack demonstrates how doors open when working with natural language as a base medium; supporting long-established linguistic work from Chomsky and Minsky.
Historically, we've gone through an evolution of solutions when it comes to mapping textual input to desired output. BERT was designed to be used as a backbone, similar to ResNet, combined with specific output heads for different tasks; appending a linear layer for classification tasks. Then, T5 introduced the text-to-text approach where the same model (loss function, hyperparams , etc) could be used across a diverse set of tasks. The task specification was made part of the input prompt and models were trained to respond accordingly
cola sentence: the course is jumping well
not acceptable.
but this isn't how humans talk, it's feels forced and unnatural. Modern instruction finetuning takes the approach that seemed most obvious to begin with; just tell the model what you want it to do!
Is the following sentence acceptable? "The course is jumping well"
It is not acceptable.
It seems a bit stupid that this wasn't the approach to begin with but my understanding is that researchers didn't believe models could understand this type of language pre-2020.
Instruction tuning has proven effective in language model alignment and it's a vital part of any dialogue LLM you encounter, unfortunately it has inherent limitations holding it back - supervised learning. When a instruction has a unique solution, all is well
2 + 3
5
but consider the following coding task
Implement logistic regression with gradient descent in Python.
?
What should the target be here? There are obviously multiple ways to perform this task; do you prefer object oriented programming, what is your naming convention, do you use functional programming? The task no longer has a objective solution which is difficult to handle when the training objective is maximum likelihood. Again I find myself coming back to what I read in the Llama 2 paper
At the outset of the project, many among us expressed a preference for supervised annotation, attracted by its denser signal.
It's clear that researchers are attracted to supervised learning, but what we've come to find out is that it ultimately becomes a bottleneck to teaching models abstract and creative behavior. This is where RLHF steps in as the final tuning procedure. The language model is no longer fine-tuned but rather pitted against a reward model which steers the LM in a desired direction, while still being able to explore trajectories previously undiscovered ny humans
In addition, during annotation, the model has the potential to venture into writing trajectories that even the best annotators may not chart.