A doppelgänger is a biologically unrelated look-alike or double, of a living person.
In fiction and mythology, a doppelgänger is often portrayed as a ghostly or paranormal phenomenon and usually seen as a harbinger of bad luck. Other traditions and stories equate a doppelgänger with an evil twin. In modern times, the term twin stranger is occasionally used.
On our mission to crawl the web we often find doppelgängers, URLs that we already visited ; or web pages, images, audios, videos, embedded content. Anything we archive in fact.
This an implementation of an url agnostic deduplication server using ScyllaDB.
This project is a WIP for now.