One source has it that Sir Robert Fairlie of that Ilk built the present castle in 1521, the family having held the lands since the fourteenth century. In around 1656-1660 the last of the Fairlie family sold the castle and barony to the Boyle family in the person of David, first Earl of Glasgow.