fly in the ointment

Something (especially a minor thing) which ruins or spoils everything else, or makes it less pleasant; a nuisance or problem; a disagreeable or unpleasant detail.