The oldest part of the house was begun in 1820 when a saddler by the name of Harris Sandifer built a small shop for his harness making. Frank Wingfield bought the property and and enlarged the original cottage. Mr. Wingfield sold the house to Isaiah Tucker Irvin II who enlarged it and it was his widow who brought it to its present magnificence.

The full daylight English basement on which the upper two stories sit was typical of the Washington houses in the ante-bellum era. The wiid in this house is all hand planed virgin pine. Since the house was called "The Old Castle" it was felt that painting would be expensive and difficult but the resin in the old pine had so solidified that the painting went swiftly and smoothly. Young Charles Irvin who lived here with his family was the aide who assisted General Toombs in his escape from arresting Union soldiers, staying with him until he saw the General safely aboard ship for England.

The flowers and shrubs were originally planted by Mrs. Oliver Dyson, and have been a delight for visitors and townspeople alike.