I am a 6ft tall, 180 lb, reasonably fit male. I also have what I think of as a pretty attuned danger detector about neighborhoods, which has less to do with poverty and more to do with gang markings, and general upkeep of the neighborhood and property. For example, parts of Hyde Park in Chicago are poor, but feels quite safe.
On the other hand, there are parts of Baltimore (and Chicago) that I would be uncomfortable walking alone during the day, much less at 3AM. Anyone who feels otherwise either knows the neighborhood well enough to know it is safe despite the warning signs, or is simply oblivious to the warning signs.