It was added to the National Register of Historic Places listings in Smith County in 1976, and has been a Recorded Texas Historic Landmark since 1962. The Daughters of the American Revolution designated it an Historic Site in 2010.

Named Bonnie Castle by original owner Samuel Gallatin Smith, the Greek Revival cottage was built in 1859 on a parcel of 9 acres in Tyler. Smith joined the Confederate States Army in 1861 and sold Bonnie Castle to local Tyler teacher F.N. Gary, who never resided in the house. After the 1862 Capture of New Orleans, Gary made the house available to numerous families of refugees fleeing the Union Army.

Gary sold the house to retired physician Samuel A. Goodman in 1866. In 1872, Goodman's physician son William J. Goodman purchased the house for $3,000 and moved in with his wife Priscilla Gaston Goodman. William and Priscilla raised four children in the house, and in 1880 added a second floor to the structure. Victorian double galleries were also added on all sides of the house.

The Goodman daughter Sallie continued to live in the home with her husband James Hutcheson LeGrand after their 1893 marriage. The couple was active in local civic affairs, providing their home as a gather place for several charitable and social events.