Not explicitly. But, in my experience, that is usually the implication when people say that.

These are Nazis. If you aren’t fighting them, then you’re allowing them to spread their hate.