A consumer goods price observatory rests on three pillars: a shared product reference base, regular price collection in stores or online, and strict quality control. Crowdsourcing and automation complement each other. The first brings field coverage, the second brings regularity. Without a solid data structure, neither produces usable information.
Start with the product reference base
The first difficulty is not collecting a price but knowing which product it refers to. The same bottle of oil appears under different names, sizes and codes depending on the retailer. You need a reference base that identifies every item uniquely: barcode where available, brand, size, unit of measure and category.
Next, define a tracked basket: a limited number of everyday goods that represent common purchases. A small, well-described basket beats a huge catalogue that is poorly normalised.
- One unique identifier per product, based on the barcode if possible
- Normalised size and unit to compare per kilo or per litre
- Hierarchical categories for aggregating results
- List of outlets with geographic area and retailer type
Crowdsourcing: covering the field with contributors
Crowdsourcing means collecting prices from consumers or paid collectors through a simple mobile app: scan the barcode, photograph the shelf label, enter the price and location. It is an effective way to cover neighbourhood shops, markets and areas with little online visibility.
Its limit is quality. Typing errors, duplicates and unreliable contributions are unavoidable. Plan for a proof photo, consistency rules, cross-validation between several contributors and a reliability score for each one.
Automation: regularity and volume
For retailers that publish prices online, automated collection allows regular readings with no manual effort. It does require respecting the sites' terms of use, applicable law and reasonable request frequency. Promotional leaflets can also be read using text recognition.
The two sources complement each other: automation provides rhythm and history, the field provides the reality of physical shelves. The challenge is to merge these streams into one model with date, place, shelf price, any promotional price and source.
- Separate regular price from promotional price
- Keep the source and date of every reading
- Automatically flag outlier values
- Archive history instead of overwriting the old price
Quality, publication and uses
Trust in an observatory depends on its methodology, which should be published: scope, frequency, cleaning rules and known limits. It is better to show a cautious trend than a precise but fragile figure, and to say clearly when coverage is insufficient.
Uses vary: consumer information, monitoring for retailers and manufacturers, analysis for institutions or journalists. Each audience expects a different level of detail, which well-designed dashboards and exports can serve.
Frequently asked questions
How do you compare prices of products in different sizes?
By normalising the unit of measure, for example price per kilo or per litre, and linking each item to a unique identifier in a product reference base.
Is crowdsourcing reliable enough for an observatory?
Yes, provided you require proof, cross-check several contributions and measure contributor reliability. Without controls, errors pile up quickly.
Is automated price collection allowed?
It depends on the websites' terms of use and local law. Check both before collecting and limit request frequency. If in doubt, seek legal advice.