I don’t think you have the right to “feel safe.” I do think that you should have a reasonable expectation of “being safe.” While subtle, they are different things.
People can be made to feel unsafe for all sorts of stupid reasons. People can be afraid of men, or blacks, or Muslims, or basically anything.
Your rights end when they are in others’ rights… right to exist if nothing else.
You must be safe… but you have no right to “feel” safe.