What is the main difference between pre-training and fine- tuning in Transformers?
Answer options
A
None of the options given
B
Both are done simultaneously
C
Pre-training is on a large corpus and fine-tuning is task-
D
specific
E
Fine-tuning is done without labeled data
F
Pre-training uses smaller models
Correct answer: Pre-training is on a large corpus and fine-tuning is task-, specific
Explanation
Pre-training involves training a Transformer on a large, general corpus (e.g. web text) to learn broad language representations. Fine-tuning then adapts those representations to a specific downstream task using a smaller task-specific labeled dataset. Options [2] and [3] form the complete answer: 'Pre-training is on a large corpus and fine-tuning is task-specific.'