A model’s training examples reflect choices about collection, selection, and representation. For a supervised task, labels also define the distinctions the model is being asked to learn.
Consider whether the examples cover the situations in which the system will be used. A collection can be large while leaving important cases underrepresented. Inconsistent labels can make the intended task unclear even when the inputs themselves are plentiful.
When evaluating a system, ask how the task and data relate to the real use case. The amount of data is one part of the story; its relevance, quality, and coverage are others. Those questions help connect model performance with the work it is expected to support.
Bring the idea into a day.
Consider a collection of sample labels that covers only easy cases. An unfamiliar input may expose a boundary the examples never explained.
Another angle on the story.
Treat an example as part of the instruction. Its omissions and boundaries can influence a result as much as the words that describe the task.
- Ask which situations the examples cover.
- Look for a clear labeling rule.
- Connect the training task with the intended use.
Follow a related question
State the initial audience or scope.
A default that explains its boundaryCheck the image’s actual display size.
One picture, several useful sizesKeep learning
Related background to continue exploring this subject.
Google: an introduction to language models NIST: AI risk management framework
