Hilchos Lashon Hara chapter 7, section 2

One may not believe lashon hara even if the speaker said it in front of the victim, for he did not admit it was true. How much more so if the speaker merely says he would be willing to say the story before the victim. People often err in this. Even if the victim keeps silent and even if he normally doesn’t keep silent, perhaps this time he controlled himself and decided not to argue, or perhaps he understood that the listeners would certainly believe the speaker more than himself. Because in such circumstances people generally believe the speaker even if the victim denies the story a hundred times.