Your guess of there being a Workhouse at the address seems to be correct.

Westbury-on-Severn Workhouse
In 1789-90, a parish workhouse was erected on the west side of the High Street at the north of Westbury.
http://www.workhouses.org.uk/index.html?WestburyOnSevern/WestburyOnSevern.shtml