When a libertarian says that the most basic individual right is the right not to be aggressed against, a clever interlocutor may accuse the libertarian of begging the question, of stuffing the rabbit into the hat. The trick, the critic will say, is in the word aggress: libertarians allegedly rig the game by restricting the category of aggression to only the actions they disapprove of, thereby institutionalizing many corrupt activities.
For example, If Jones tells Smith to get off land to which Jones has legal title, is it really clear that Smith is in the wrong and Jones is in the right? The critic will offer a counter-narrative: it's considered Jones's land because the political system arbitrarily defines property rights in a certain way. It might have defined rights differently so that Smith could walk on the land as wishes. So why not see Jones as the aggressor against Smith?
If the libertarian responded that Jones transformed the hitherto unowned parcel by mixing his labor with it, perhaps by clearing and fencing it, the critic might respond that Jones's act constituted aggression because, unlike yesterday and the day before, no one now may step on the land without Jones's permission. Jones, in other words, restricts everyone else's freedom. Who's right and who's wrong would depend on one's point of view.
This case against libertarian property rights implies that land has never been unowned because it has always been owned by humanity in common. Such a position was taken most famously by Henry George. While George did not oppose individuals' use of parcels of land, he said that users ought to have to pay land rent to the community, the true owners. This was George's "single tax." Murray Rothbard rebutted George's case in both its moral and economic dimensions. (See also Rothbard's Power and Market.)
If the point of rights theory is to enable human beings to flourish as they live side by side peacefully and cooperatively in society, then any theory that regards land and other scarce resources as jointly owned by all of humanity is in for problems. The moral is the practical. So imagine the impracticality of determining how a piece of land is to be used if everyone is to have a say in the matter. Yet if human beings are to prosper, decisions about how to use scarce resources are crucial. No one is infallible or has a monopoly of wisdom about the "best" use of resources, but we have the next best thing: the market and its price system. The market provides indispensable signals about ever-changing supplies and consumer preferences. Ludwig von Mises and F. A. Hayek made their marks as great economists by, among other things, showing that market prices are the only things we have to relieve, insofar as possible, our ignorance about how scarce resources can be used best to serve everyone's welfare. Private property and free markets expand rather than contract the public's access to resources.
The critic of libertarianism may listen and nod but continue to insist that we have no objective way to tell who is the aggressor: Smith or Jones. But maybe we do.
Life is not an abstraction. Individual people are beings who live day to day through the pursuit of projects, which usually involve the cooperation of others. Since we are physical beings, that pursuit requires control over things, including land, and therefore noninterference by other people. How could we live and plan long term if our activities could be interfered with and the fruits of our efforts could be appropriated by others? I take for granted that each person is a self-owner because denial of this principle collapses in absurdity. Lincoln wrote that "if slavery isn't wrong, nothing is wrong." Abolitionists called slave owners "man-stealers." If self-ownership isn't right, then nothing is right.
The principle of nonaggression is universal: you may not interfere with me, and I may not interfere with you. Liberty for all means no one is aggressed against. Society should be based on consent and cooperation.
In the story above, if we assume Jones acquired the land justly through homesteading, purchase, or gift, then the land is part of his project, and Smith's trespass constitutes interference with Jones's life. (Of course, trespass can be trivial, and methods of prevention or redress would have to be proportional to the offense. Put bluntly, Jones can't shoot Smith merely for setting foot on his land.)
Yes, in a physical sense, Jones's ownership "interferes" with Smith's freedom, although not his ability to live as a human being (except perhaps in an emergency). But human action is never merely physical. Justice is relevant. The same physical act can be just or unjust depending on the circumstances.
I think this demonstrates that the libertarian case does not pack its conclusions into its definition of aggression. Hard cases of course can arise, but generally we can determine who is the rightful owner and who is wrongfully interfering.
Finally, I have not tried to sort out the case of ownership clouded by historical injustice, namely, theft. What to do about this is a complicated matter, in part because of the variety of cases, on which I claim no particular wisdom. Those who wish to delve into the problem can begin by looking at what Rothbard had to say in The Ethics of Liberty.