My heart breaks for this poor baby! To me he appears to be a creature that has never known the comfort and love of a human bond, and trusts no one. Having said that, I am optimistic that you will be able to win him over by doing just as you are but being very cautious and backing off when his protests escalate - he needs to know that he has some bit of control over his environment. As for taking off the collar, I would consider continuing to do that, just let someone else put the collar back on so that you are only associated with the positive removal and freedom from the collar.
Time, patience, and healing are what he needs...
Time, patience, and healing are what he needs...