I don't recall anyone ever commenting on my SS number except when I applied for my F:L drivers License in 1977 and the examiner said that I was from New Mexico, which I am not, but that is where I had my first job with a real paycheck.
This is a non-issue.