Configurator scraping: how to monitor prices that only appear after configuration

Some prices never appear on a page. A lease rate, an insurance premium or a telecom bundle often only shows up once someone has configured the product: duration, mileage, options, address. Until then, the website shows a "from" price at best.

That is exactly why standard price monitoring tools run into trouble in these markets. If you want to benchmark them, you have to do what a customer does: walk through the configurator.

What is configurator scraping?

Configurator scraping is the automated collection of prices by completing an online configurator the way a customer would. The scraper selects options, such as contract duration or mileage, submits them and records the resulting price together with the configuration it belongs to. The result is price data for specific, comparable configurations rather than a headline price.

Why can't a standard scraper handle a configurator?

A standard scraper reads a price that is already on a page. In a configurator, the price is built dynamically: it depends on the combination of options chosen, on the logic behind the configurator and, often, on stock, campaigns and subsidy schemes that change over time.

That creates three problems. The number of possible combinations is practically unlimited. The price only appears once the right steps have been completed. And a small change in the flow, such as a new option or an extra step, can break a scraper without anyone noticing. In automotive leasing, even small configuration differences can lead to large price differences, so errors are not just a technical issue: they distort the benchmark.

Which markets depend on configurators?

The pattern shows up wherever the price is the result of choices rather than a fixed label:

• Automotive leasing: contract duration, mileage, trim level, options and tax category together determine the lease rate.
• Telecom: availability and price often depend on the address, the bundle and the handset combination.
• Insurance: premiums come out of a quote flow and depend on the profile and product details entered.
• Travel and hospitality: prices depend on dates, number of guests and room or package type.
• Made-to-order B2B products: the specification defines the product, and therefore the price.

How do you decide which configurations to monitor?

Trying to monitor everything is the most common mistake. Every extra option multiplies the number of combinations, and you end up with a large dataset without commercial focus. Start from the decisions the data has to support.

Choose a reference set: the brands, models or propositions that matter, the contract structures that dominate the market and the configurations that drive most of your volume. In leasing that could be a limited set of duration and mileage combinations per model. In telecom it could be a set of bundles checked at a selection of addresses.

Then separate sticker prices from market prices. A competitor's "from" price often belongs to a very specific configuration, built for visibility and lead generation. The configurations that are actually sold, and the deals a competitor pushes hardest in campaigns, say more about the real competitive position.

What makes configurator data reliable?

Configurator data is only useful if every price can be traced back to the exact configuration behind it. That takes a few habits:

• Store every price together with all selected options and the time of collection.
• Validate each run: check that the configurator was completed, that the price is plausible and that the selected options match the requested ones.
• Detect changes in the flow, such as new steps or renamed options, before they turn into gaps in your data.
• Repeat measurements on a fixed schedule, because campaigns and temporary offers move prices.
• Keep a person in the loop for exceptions that automation cannot judge.

How do you compare configured prices fairly?

Two configured prices are only comparable if the configurations are equivalent. A lease with a higher mileage allowance or a more extensive trim level is not the same proposition as a cheaper one. Define which attributes must match and which may differ within a tolerance, then normalise the price to a common basis, such as the total monthly cost of an equivalent package. We describe this approach in more detail in Matching without a barcode: How to compare prices when there's no EAN.

Which configurators can't be monitored?

Monitoring only works for prices that are publicly available. If a configurator requires a login, or if the price depends on personal data that cannot be simulated, it is usually not suitable. If you are unsure whether your competitors' websites qualify, the quickest way to find out is to look at the specific configurator.

Build it yourself, or have someone else own it?

Deciding which configurations matter for your business is something you are well placed to do. What is harder to sustain is the engineering: building scrapers that complete configurators, keeping them working when the flow changes, and matching the results to your own propositions month after month.

At Competify, this is what we do for markets such as automotive leasing, hospitality, telecom and insurance. We agree with you which configurations matter, build and maintain the scrapers, match and validate the data, and deliver it through a dashboard, API or file transfer. If you would rather not own that part yourself, please plan an introduction meeting.