The Deduplication Problem: One Tender, Five Portals

Open a handful of tender monitoring tools side by side and search for the same large infrastructure contract, and there is a good chance the identical opportunity shows up four or five times, each with a slightly different title, a different reference number, and a different publication date. This is not a bug in any one system. It is a structural feature of how public procurement gets published, and it is one of the least glamorous but most consequential problems in building a usable tender feed.

For a procurement team, duplicates are not a minor annoyance. They eat review time, distort pipeline counts, and make it easy to miss that two listings pointing at the same underlying contract have different deadline information, one of which is outdated.

Why the same tender ends up everywhere

A single public contract can legitimately appear on multiple official channels at once. A national contracting authority publishes the notice on its own domestic portal, because local law requires it. If the contract value crosses the relevant EU threshold, the same authority is also required to publish on TED, often through a translated or reformatted notice generated automatically from the national system. Sector-specific platforms, development bank notice boards, and commercial aggregators that scrape public sources then pick up the notice a third, fourth, and fifth time, sometimes republishing it with their own internal reference number and a re-formatted summary.

Each of these publications is legitimate. None of them is wrong to exist. The problem is entirely on the consumption side: a supplier trying to build one clean pipeline out of many overlapping, independently formatted sources.

Why simple matching does not work

The obvious first approach, matching on the tender's official reference number, sounds reliable and mostly is not. National portals often use one numbering convention, TED assigns its own separate publication number to the same notice, and aggregators frequently generate their own internal ID that has no relationship to either. Title matching fares even worse: the same contract might read as one exact phrase in the source language on the national portal and a loosely translated paraphrase on TED, with an aggregator's version trimmed or reworded again for its own formatting.

Date mismatches compound the problem. A notice might be initially published on the national portal, amended a week later with an extended deadline, and picked up by an aggregator at either the original or the amended version, with no reliable signal indicating which is current.

What identity resolution actually involves

Cleaning this up requires treating deduplication as an identity resolution problem rather than a simple string-matching one. That means combining several weaker signals into one confident judgement rather than relying on any single field. Contracting authority name and location, normalised to account for spelling variation and administrative naming differences, forms one anchor. Estimated contract value, when present across multiple listings, is a strong signal when it matches closely, even if reference numbers differ entirely. Publication and deadline dates, allowing for the translation and processing lag between a notice appearing on a national portal and the same notice appearing on TED days later, help establish a timeline rather than a single fixed point.

Semantic similarity between descriptions matters more than exact text matching, since two independently translated or summarised versions of the same scope of work will rarely share identical wording but will describe the same underlying deliverables, quantities, and location in recognisably similar terms. Combining these signals, rather than depending on any one of them being perfectly clean, is what allows a system to say with real confidence that two listings are the same underlying opportunity rather than two coincidentally similar ones.

The cost of getting it wrong in either direction

Deduplication carries risk on both sides. Merge too aggressively, treating genuinely different contracts as duplicates because they share a buyer and a similar value, and a supplier silently loses visibility into real opportunities. Merge too conservatively, and the feed fills with near-identical listings that waste review time and inflate pipeline metrics without adding real signal. Neither failure mode is acceptable for a team relying on the feed to make bid or no-bid decisions quickly.

The amendment problem sits on top of this. When a genuine duplicate is correctly identified, the system also needs to determine which version carries the current deadline and specification, and surface that version rather than an average or, worse, the stalest one.

A clean feed, not five overlapping ones

Building and maintaining reliable deduplication across dozens of national portals, TED, and third-party aggregators is exactly the kind of unglamorous infrastructure work that determines whether a tender monitoring tool is actually usable day to day. TRINTA reads what a company sells and surfaces matched tenders from official sources daily, resolved down to one entry per real opportunity, so a procurement team reviews one clean pipeline instead of untangling five overlapping ones by hand.