I personally believe in hard facts!
Thomas Charles Burley is definitely James Lee's son and I believe William 'Lee' Burley is also his son.
As you may know there is an ongoing transcription of the Parish Records on this site. It might be just a case of waiting until one of the other Burley children are married and that record has been transcribed before you find out if all the children belong to James.