I am working through a data pipeline question related to image-text and video-text projects.
The first collection run is usually manageable. The harder part is keeping the dataset useful after the model has already been trained once. Sources change, duplicates accumulate, media disappears, and new examples may not be comparable with the original data.
I am currently thinking about a simple refresh process:
- keep a versioned manifest for every dataset release
- store the source timestamp and basic metadata
- compare new records against previous samples
- check that the image and text still belong together
- keep a small fixed evaluation set for comparison
- review failed or uncertain records instead of retrying forever
For people who have worked through fast.ai courses or practical computer-vision projects, what has been your approach to dataset refreshes?
Do you prefer rebuilding a dataset from scratch, adding incremental batches, or maintaining a fixed core set plus a rotating set of new examples? I would also be interested in how you balance data quality with the time and cost of repeated collection.
I am looking for practical experiences rather than a perfect architecture.