Hilchos Lashon Hara chapter 6, section 6

This applies if they were not talking lashon hara when he first sat with them. But if they were already speaking lashon hara or if he could leave and does not bother to do so, or if he knew these people always talk lashon hara, he is considered a sinner as they are even if he doesn’t not like what they are saying and does nothing to encourage them for he transgressed the sages injunction to keep far away from unsuitable speech.
If he has intent to enjoy their words his sin is too heavy to bear and he is inscribed above in the book of remembrances as a wicked person and a baal lashon hara. For we find in Pirkei d’Rabbi Eliezer that Rabbi Eliezer the Great ordered his son Hurkenos as follows: My son, do not sit with a group that speaks evil of their fellow men for when the words rise above they are written in a book and all present there are written beneath the name of an evil group and baalei lashon hara. Therefore one must keep far away from such people.