Hmmm, yes that 1851 census is a bugbear! Everything else in that census record fits perfectly, except where he was supposedly born. Maybe he didn't really know where he was born because of the circumstances of his birth? For example his mother possibly not on the scene after his birth (early death?), or if only his father brought him up, and/or the possibility he was illegitimate and/or Seymour Colchester being his step father rather than biological father.

The plot thickens! Come on you Sherlock Holmes's out there, help me out :)