The Dickie family built it in 1832. In 1898 it burned, and was replaced by Valley View. It is seated on the Castle Creek tract that Captain William Dickie surveyed and patented in 1750.Dickie had acquired the patent of 2,700 acres from King George II, the land running from the top the mountain at Massies Mill down to the Edge of the village. Dickie sold some this land, but retained the 790 acres around Castle Creek.
In 1752, his son James seated the tract and used the plentiful timber on his land to build a sturdy log cabin. Dickie followed the custom of the frontier country to build the log cabin from the great trees that stood on all sides. It apparently stood for eighty years. William Lewis Dickie assumed the management of the plantation in 1836 under a deed of trust made by his father. He began the production of apples at Castle Hall and began the transition from tobacco to apples. In 1832, William Dickie built Castle Hall, the first family mansion, to replace the log building. It was a three and a half story house. In 1898 was destroyed by a fire along with many family old furnishings and heirlooms. However, the family portraits of Reverend Adam Dickie, rector of Drysdale Parish in King and Queen County, and his wife Patricia were saved and restored, and now hang at Valley View. Today, a solitary chimney-end wall of the brick house still stands to mark its site.