Looking at subsequent census records doesn't really help but I would say definitely either the large building on the north side of the road or the one nearly opposite on the south side. Looking at them on street view I think the one on the south looks more like it might have been an inn but could be either.