First, set aside the legal situation. The law's job is to produce reasonable answers which we can use to get on with it. The critic's job is to unravel some of the complexity of the situation, to clarify what is. I am, in addition to being a crazy old man shouting at my hallucinations, a critic. My interest here is in our attitudes.
Consider the cathedral of Notre Dame de Paris. It was made, let us simplify for the purpose of discussion, by a bunch of masons. If you write a novel and describe the building, the hand of the mason is at most gently present in your book. If you paint a scene including the cathedral, the mason is more present. If you photograph the same scene, most will agree that the work of the mason is indeed present in the picture. The picture contains a literal copy, in some sense, of the mason's labors.
You could argue at this point that the picture is a collaboration between the photographer and the mason. Few would refuse to wrap themselves in that mantle, were it offered them.
Nobody sensible would propose that the descendants of the masons are owed royalties. Neither would anyone suggest that the mason's contribution is irrelevant. And,
I venture to suggest that almost nobody would propose 'screw those guys, this us all about me'. Our attitude toward the fellows who designed and built the church is positive and charitable. We hold them in some regard, with some respect. If we think of them at all, anyways.
Suddenly, when a rude rent-a-cop rolls up, the self-styled street photographer holds quite a different attitude. You might think that a mall's architect sucks, or that the designer who made the latest Nike logo sucks. The rent-a-cop is certainly rude and ill informed about the law. None of these things ought to change our attitude about who has what rights. Rights, whether there legal sort, or the sort where one is owed a modicum of regard, of respect, ought to be more or less immutable, surely.
Now, suddenly, our legalistic rights become huffily asserted. Not usually to the security guard, no, we scurried away from him. Later, on internet forums, we wax strident. We become smug assholes, falling back on the letter of the law (or rather what we have collectively decided, with the legal expertise of a forum, what the letter of the law is).
The warm feelings of respect have gone away somewhere, what remains is disdain, anger, and a wonderfully shallow sense of entitlement. What is left is a resentful sense that everyone else involved ought to be grateful that we are here with our big camera, and a bunch of weak-sauce arguments about why they ought to be glad. Social media! Exposure!
Maybe the masons just aren't comfortable having their work photographed. It is, frankly, none if your goddamned business why they'd prefer you photographing elsewhere. And perhaps that doesn't mean you shouldn't photograph. I carry on, myself. But how dare you get on your bully pulpit and berate them for it.
If photography is to matter at all, we should have some respect for what's in front of the lens. It's not just a bunch of shit to exploit in order to build up our social media following. Like Soylent Green, it's people.