
Deduplicating 12,000 job postings taught me the same job is posted five times
I built a job search API that aggregates live postings by keyword and location, and the first version...
Tag archive

I built a job search API that aggregates live postings by keyword and location, and the first version...
A provider directory that ingests clinic listings from four feeds will hand you the same clinic four...
TL;DR: In a catalogue import, treat a supplier image as content before treating it as a filename....
Sam K., Samuel Kim, [email protected]. Deciding what's the same thing is harder and more important than it looks.
Alerts that fire twice for the same event are worse than alerts that never fire at all. The first...
I run a Job Postings Search API (https://x402.freeq.one/tools/jobs.html) that queries live postings...
Short answer: implement notification deduplication as a server-owned, stable event-ID contract,...
The Frontier: What to Fetch Next The crawl frontier is a priority queue of URLs, not a...
Short answer: read image metadata, hash the bytes, and check both values against a processed-image...
How I added an LLM second-opinion layer to a RAG deduplication gate, using a two-threshold review band to clear embedding false positives without blocking valid training tasks.
The same policy document uploaded three times with minor edits crowds out other results and confuses retrieval. How to detect and handle near-duplicat
Exact duplicates are easy to catch. Near-duplicates — the same example paraphrased — quietly bias a fine-tune and are far more common than people expe