"Against the Grain" by James C. Scott first got me agreeing along those lines. But I just read an article in The Guardian (link at end) that points out that pre-ag man wiped out most of the megafauna, which is perhaps what drove us to adopt agriculture. So maybe we were doomed from the moment we evolved big brains and could so dominate and widen our ecological niche. C'est la vie!