Hilchos Lashon Hara chapter 8, section 7

Know too, that if the person is muchzak in a town as being wicked due to other sins which permit one to disparage him, the law is the same. (See required conditions in Beér Mayim Chayim). What is muchzak? If the people of the town agree that he is wicked without any doubt (due to constant evil reports about him concerning adultery and suchlike, things whose prohibition is known by every Jew). However, if there is only a rumor about him one may not rely on it to disparage him, heaven forefend, or even believe it in one’s heart as explained above in chapter 7. (I was very afraid to mention this leniency because tale bearers who hear anything bad of someone will immediately presume that he is muchzak as wicked and disparage him, relying on this book. Nonetheless, I did not omit this information for as the Gemara Bava Basra 89b cites from Hoshea 14:10 – The ways of the Lord are straight and the righteous walk in them and transgressors stumble in them.)