Since Castle Garden was in New York City it is highly unlikely that a Southern Sympathizer would have enterd through that port. If he came to fight for the South I suggest looking at a southern port, such as Charleston, Mobile, New Orleans or the Texas ports.
Dewey