A very good question and a controversial subject indeed. And I can totally understand your concern. My answer is my own personal take on the subject & not intended as the final word on anything.
First let me preface my response by noting that Eisenhower is preseident when I am born, so that means I'm from a generation raised in a different era when the level of public civility and citizenship has a different context than today ( in other words, I'm a product of the values I'm raised with
).
I'm not a professional photographer and I do not sell my work for profit. My images are taken in the public forum & most frequently on public property. I'm out there practicing the art of photography as an individual amongst individuals. As I note, when I do candids, I "feel" what's going on & I hope that my subjects, when/if they might ever encounter my images will understand the context in which I shoot them ( which is artistic and respectful ). If someone should object to their image appearing in one of my galleries, they need only notify me & I would remove the image. This is the practice amongst photographers on PBase where I keep my galleries. This hasn't happened to me yet, & I really do not anticipate a problem. ( that is one reason why I do not cloak my identity or my source gallery here on TUG or any other board I belong to).
Once again, this is simply my take of a subject that some people do indeed feel prickly about. But I maintain my own context as a citizen & I keep doing what I love, which is photography. It's also my own opinion that society today is more intemperate and intolerant and this sometimes leads to misunderstandings which would not happen if folks learned and adhere to the art of dialog which allows lots of room for disagreement, but actually functions to create understanding. In other words, I'd take the image down, but make certain that the individual understood my true motive.
Another thing that folks should be aware of when photographing in public is that you may indeed be asked not to take any photographs when on private property ( a restaurant/store/shop/musuem etc. ). Sometimes signs are posted, other times it's best to inquire. Some venues are actually conducive to photography & it's allowed. No etched-in-stone rules here, you just keep heads up and maintain a respectful demeanor so as to avoid misunderstandings.
Thanks for your thoughtful question icconections. I've not given the final answer, just my own approach from doing this for several years.
Barry