Shemiras Halashon Halachos

Hilchos Rechilus, chapter 6, section 1

One may not believe the rechilus even if it was spoken in front of many people.

Hilchos Rechilus, chapter 6, section 2

The same applies if he said in front of the person spoken of: "You said such and such about so and so." Even if the person kept silent as he said it and it is not usually his nature to keep quiet about other matters, this is no proof that the story is true.