Hi all — sharing my end-of-course project since a few pieces of it might be useful to others working on “real data” pipelines (the data part turned out harder than the model part, which feels very much in the spirit of what we learned).
The project
A price intelligence dashboard for consumer electronics across 4 markets (US/DE/JP/BR): daily price collection, a classifier for product condition/category, and a small forecasting model for price trend.
The part nobody warns you about: the data
Three lessons from the collection layer:
1. Geo-vantage matters. The same product URL serves different prices and availability per visitor country. I collect each market from in-country residential IPs (I use Thordata for the proxy layer — country/city targeting via their gateway, rotating residential runs about $0.65/GB which fits a hobby budget). Without this, my “multi-market” dataset was secretly 4 copies of the US catalog.
2. Collection provenance is a feature. I tag every record with (market, collection date, IP country). Half my debugging sessions ended with “ah, this anomaly is just Germany being Germany.” Keep the metadata.
3. Dedup before you train, not after you’re confused. Perceptual hashing saved me from training on thousands of near-identical product shots.
The model parts
- Classifier: product category + condition, using the fast.ai vision learner with transfer learning. Nothing exotic — the interesting bit was class imbalance across markets, handled with upsampling in the dataloader.
- Forecasting: tabular price history → next-week price, gradient boosting. Beat a naive baseline by a useful margin; the scraped features (competitor price deltas) mattered more than the time features.
Results
- Collection: ~6k product records across 4 markets over 6 weeks
- Classifier: 94% accuracy on held-out products (leakage-checked across product IDs)
- The dashboard is a Streamlit app polling the SQLite backend
What I’d do differently
- Start the proxy layer on day one — my first two weeks of data got thrown away after I realized the geo-vantage problem
- Log block rates from the start as a dataset quality signal
Happy to share code structure for anyone doing something similar. And if anyone’s collecting multi-market data for their own project, happy to share the collection-layer config too.