After it was completed, it became the home of Mrs. Lee and her two small children. On April 27, 1831, Mrs. Lee married Edward Baker, a young lawyer, who later became a close friend of Abraham Lincoln in Springfield, where the Bakers moved in August, 1835.  The next year they sold the mansion to Orange Heaton.  Moving to California in 1852 and then to the new state of Oregon in 1860, Edward Baker was elected U.S. Senator.  At Lincoln’s inauguration in March, 1861, Baker, known for his oratory skills, made the shortest public speech of his career with these words: “Fellow citizens, I introduce to you Abraham Lincoln, the President-elect of the United States.”  With the outbreak of the Civil War, he organized the California Regiment and was commissioned its Colonel.  He was killed October 21, 1861 during the battle of Ball’s Run, Virginia.

Charles Drury Hodges bought the Lee-Baker mansion in March, 1850 from Orange Heaton.  A lawyer and merchant, Hodges served as county judge, circuit judge, representative in the U.S. Congress and Illinois state senator during his career.  About 1854 Judge Hodges added a second story to the east part of the original Lee house, incorporating it into a two-story Italianate style wing.  With its wide front veranda and its yard enclosed with a wrought iron fence (both now gone), the imposing Victorian structure was a Carrollton showpiece. After Judge Hodges’ death in 1884, his family continued to make it their home until Mrs. Hodges died in 1899.  The following year their son Beverly C. Hodges converted the family home into the Hodges Office Building.  He ended the 70-year ownership of the house by the family when he deeded the property to Dr. N.D. Vedder in 1921. 

The house gradually deteriorated into the 1970s, until its very existence was threatened. Thanks to historical preservationists, the house was saved and added to the National Historic Register in 1980. Since then, it has become home to the  Greene County Historical and Genealogical Society and, if the stories are to be believed -- a number of resident ghosts. A number of eerie reports have circulated about the old mansion over the years, including sightings of a child who has been seen in one of the bedrooms, another child that vanishes just inside of the kitchen door, bright lights, weird sounds, footsteps, a mysterious voice that has been heard throughout the house, loud knocks that come in threes -- and much more!