My friend with Deeky boy had to make the very same decision last year, he was made homeless and Deeky went down south to his mum and brothers house, Andy was and still is heart broken and like you doesn't think it'd be fair on Deeky uprooting him after being away so long, Deeky has a great life with them and Andy takes stregnth knowing its him mum and brother looking after him...
personally I wouldn't go visit either as I'd feel the same about the dog being confused and me being upset...
I really feel for you, you will miss him loads, try to keep your chin up, be strong for Boomer xx