How to Deduplicate BibTeX Entries?

I'm writing my dissertation recently. My dissertation is a combination of several publications from my PhD career. Therefore, part of my dissertation writing process involves copy-pasting the papers together into a single document.

Like any good academics, I typeset my publications with LaTeX, and use BibTeX to incorporate citations into the documents. My collection of bibliographies is fairly ad hoc: during each writing project, I search for related work to cite in my paper. Unlike most others, I create a separate bib file for each BibTeX entry named after the citation key. For example, I would have bib/ndn-tlv.bib for a BibTeX entry named "ndn-tlv", and bib/Mininet.bib for the "Mininet" entry. This allows me to find available citation keys with a quick glance over the bib/ directory. My build process then concatenates these small bib files into ref.bib as an input to BibTeX.

My dissertation combines all my publications, and thus needs a union of BibTeX entries from those combinations. To make this union, I can copy all these single-entry bib files into the same directory. If two previous papers cited the same reference, their bib files should have the same name, and only one copy would be left in the combined directory.

Except that the above assumption is true only if I cited the same reference with the same citation key. And so I discovered a citation appearing twice in my dissertation:

duplicate references in dissertation