When longtime Washington correspondent Jesse Holland was covering the inauguration of President Barack Obama he was considering the political importance of the first African American elected as Chief Executive taking residence in the White House.

Holland, a historian as well as a journalist, became interested in the first African American residents of the White House: the slaves owned by U.S. presidents.

Even before the White House was built, slaves were serving not only on the plantations owned by many of the Founders, but also as servants on the first President’s domestic staff.

“When we think about George Washington and Thomas Jefferson, we think of the Father of the Country; we think about the writer of the Declaration of Independence. But we don’t think about them as Virginia aristocrats, which is what they were,” Holland tells Steve Fast. “The way that they could keep themselves in the upper echelon of Virginia was holding slaves.”

Of the first twelve U.S. presidents, only John Adams and John Quincy Adams were not slave owners.

“This was because they were Quakers, and they didn’t believe in slavery,” Holland says, adding that both of those Presidents paid the domestic staff out of their own pockets.

“One of the things that I discovered while working with this book is that most of these presidents knew morally that slavery was wrong,” Holland says. “They also knew that to divest themselves of slaves would be a financial hit to them.”

In his book “The Invisibles: The Untold Story of African American Slaves in the White House” Holland tells the story of prominent slaves including Oney Judge, Joseph Fossett and Paul Jennings.

Listen to the interview: Jesse Holland on The Steve Fast Show

Follow Steve Fast on Twitter @SteveFastShow