It matters that they might not vote and perhaps better candidates get elected in their absence.
But even if the best of the best were elected in their absence this would have a large bad effect.
Having a lot of people perceive themselves disenfranchised is very bad , even if it is a false perception.
It would be a better idea for the greatest number of people possible be very well informed and motivated to vote.