This structure dates from the 1700s when it was used as a tavern, inn and stable for travelers. There was a large farm and mills on the site. Stuart's Brook, a tributary of the Scarborough River ran through the property. This structure is located on Route 1 in Scarborough just before Saco. This brick building replaced the original structure in 1815.

The tavern was originally owned by Captain Mulberry Milliken who was in the shipping industry with Richard King. Captain Benjamin Milliken took over the running of the tavern from his father. The tavern business began to diminish after 1840 when the railroad between Portland and Boston began running. The house was bought about 1875 by Silas Lary. The house is still owned by the Lary family.