They "fled" because Government was setting themself up as the "go-to" organization for any and all assistance. It had trace to do with race. Conservatives support self sufficiency and limited Government, while embracing charity and personal assistance. Government, led by Democrats and liberal Republicans put the gas peddle to the floor in trying to "help" anyone it could, regardless the circumstances that led to how/why any help is needed in the 1st place. Programs for unwed mothers facilitated more out of wedlock births, just to name one example. Increasing unemployment benefits, yet another
Blacks left the GOP not because of any racial twist, but because the left began championing the need for more Government services, which in turn made it ok for worse and worse decision making and judgements by individuals. As The GOP kept trying to rain in the poor decision making, the left found a gold mine of PR, to lay the claim of how the Republicans don't care about the poor, and in particular poor blacks. Goldwater, had very little to do with it, IMHO
In other words, the GOP allowed Democrats to with the PR battle, despite the fact that there are far more racists in the Democrat ranks. One need only look to the NAACP and folks like LaRaza for a pelethora of examples to that fact.