I think it makes sense to withhold voting from validators removed in the event of being tombstoned or jailed. But I agree that those knocked from the active set through no fault of their own should have their vote count in some way.
Alternatively, could there be some conditional type logic added to the voting portion of the software for delegates. That way its kind of like a ballot, and those delegates who are active enough to participate in governance can still stake with the validator of their choosing IF they are in the active set, or otherwise shift their vote to adopt that of another validator IF the validator they are staked with is not in the active set. That way delegates are more incentivized to actively participate in governance to make sure their vote counts one way or the other, and it prevents against the type of attack @JesseLivermore described.
Idk how technically feasible something like this would be or if it would be worthwhile, but popped into my mind reading through some of these comments.