What happens when the weaker brother/Pharisee is one of/all of your elders? What happens when you get an elder's visit for going to a football match, the cinema, drinking alcohol, etc? This is not a situtation beyond the bounds of reality in some quarters (particularly in some brethren churches).
If an elder must be apt to teach and he is coming down on the flock in matters of indifference, then it's clear he doesn't get the gospel, justification, etc. It is HE who should face discipline, not the guy/gal who was reported for drinking a Baby Sham. If this freedom quenching is a collective spirit in the eldership and not the views of a maverick, you are left with only one option, leave.
It was for freedom that Christ set you free. Stand firm then, and do not allow yourself to be burdened by a yolk of slavery.