John Stavers, who came to Portsmouth from England, built the Earl of Halifax Tavern in 1766, later renamed Pitt Tavern. It was more than a place for food and lodging; its third floor was designed specifically as a meeting place for the St. John’s chapter of the Masons, and it was a terminus for the Flying Coach stageline, which Stavers’ brother, Bartholomew, operated.

During the American Revolution, Stavers’ loyalty to the colonial cause was questioned. Fears of him being a Tory were exacerbated by the fact that Bartholomew returned to England rather than support independence. The Masons moved their meetings to a different location, and stories of mobs attacking the tavern have since entered local folklore. Most notoriously, Mark Noble, in protest of Stavers’ supposed loyalties (or was it Noble’s own unpaid tavern bill?) attempted to chop down the tavern sign. Stavers sent his enslaved man, James, to stop the destruction, which James did by hitting Mark Noble over the head with an ax handle, injuring him non-fatally. Since James was considered nothing more than a piece of property without his own agency, Stavers was held responsible for his actions and was thrown in jail for a short time.