Title: How do you handle large image dataset collection and refresh?
I’m working on a computer vision data workflow and have been thinking about the challenges of keeping a large image dataset useful over time.
The initial collection is usually manageable. The harder part comes later:
- some image URLs stop working
- the same image appears across multiple sources
- metadata becomes inconsistent
- regional versions of the same page may return different results
- new content needs to be added without changing the dataset distribution too much
For those who have built larger image datasets, how do you usually handle refreshes?
Do you keep the original responses and create new dataset versions, or update the existing records in place? I’m also interested in how people manage deduplication, source tracking, and image quality checks before training.
For location-sensitive projects, do you keep country or region information as part of the dataset metadata?
I’d appreciate any practical advice, tools, or lessons learned from real projects.