The Legal Theory Bookworm recommends The Fourteenth Amendment and the Privileges and Immunities of American Citizenship by Kurt T. Lash. Here is a description:
This book presents the history behind a revolution in American liberty: the 1868 addition of the Privileges or Immunities Clause of the Fourteenth Amendment. This exhaustively researched book follows the evolution in public understanding of "the privileges and immunities of citizens of the United States," from the early years of the Constitution to the critical national election of 1866. For the first 92 years of our nation's history, nothing in the American Constitution prevented states from abridging freedom of speech, prohibiting the free exercise of religion, or denying the right of peaceful assembly. The suppression of freedom in the southern states convinced the Reconstruction Congress and the supporters of the Union to add an amendment forcing the states to respect the rights announced in the first eight amendments. But rather than eradicate state autonomy altogether, the people embraced the Fourteenth Amendment that expanded the protections of the Bill of Rights and preserved the Constitution's original commitment to federalism and the principle of limited national power.
And from the reviews:
"There has been a great deal of commentary on the original meaning of the Privileges or Immunities Clause of the Fourteenth Amendment, but that meaning has remained uncertain and contested. Kurt Lash's approach to the question has been to do something not previously attempted by historians or legal scholars: read all the available literature by antebellum and Reconstruction-era contemporaries on the subject. As a result he has gone far toward producing a definitive account of the Clause's history, and shown how contemporaries understood the "rights" contained in that Clause as sharply distinct from those contained in the Privileges and Immunities Clause of Article IV of the Constitution. Building on that distinction, Lash has fashioned a compelling argument as to how the original meaning of the Privileges or Immunities Clause should be understood." G. Edward White, David and Mary Harrison Distinguished Professor and University Professor, University of Virginia School of Law
"Kurt Lash is a rare combination - a scholar with profound legal knowledge, the courage to tackle big questions, and the ability to understand historical change. The Fourteenth Amendment and the Privileges Immunities of American Citizenship instantly takes its place as essential reading for anyone who thinks or writes about the sources, evolution, and application of the Fourteenth Amendment, America's second constitution." Garrett Epps, University of Baltimore and correspondent for The Atlantic
"The Fourteenth Amendment and the Privileges and Immunities of American Citizenship is a deep and important book. It will surely revolutionize our understanding of the original public meaning of the enigmatic "privileges or immunities" clause of the Fourteenth Amendment. Lash argues that language that seems opaque to contemporary readers was a phrase of art, which establishes a clear ceiling and floor on the meaning of the clause. His claims are sure to be controversial, but they demand answers. If you want to understand the Fourteenth Amendment, you will need to read this book." Lawrence Solum, Georgetown Law