You say "make sense" - well to me - no it doesn't. It doesn't make sense to have a puppy live in the way you expect it to live. My advice would be to re-home it to somebody who can give him the life he should have.

Toilet training can be more difficult with a small breed dog, but it sounds like your guy is doing well in that he is using the pads quite well.

Many young dogs don't feel comfortable pooing outside their own space (usually a garden but in your case on the pads) until they get to 8-12 months old.

The outside World is not as safe a place for them and having to squat and put themselves in a vulnerable position takes quite a bit of time for a lot of young dogs.

He will more than likely grow into it so don't give up hope.

Ignorance Of Radicals
I think the only ignorant one is yourself, a living, breathing animal, that should be a much loved member of the family left on their own for hours on end, sometimes visited by friends, but more often not.
Thank goodness I am at great pains to weed such as yourself out when I am looking for families for my puppies. Some of them are left, but not for that length of time.My families put their dogs in daycare if there is a situation where they are going to be left, a member of the family comes in, or get a sitter
Feel so very sorry for your poor puppy, what a life to be condemned to, shut up in solitary all day with no socialisation or interaction apart from 2 days, then silence and nothing for 5 days.

It does all sound very chaotic for the little chap - I was going to avoid this thread but just wanted to agree with the advice given the OP. We are not radicals on this site, we are dog lovers who care about other people's dogs, not just our own. My biggest concern of all for this little guy is the apparent lack of socialisation. It is so important especially for single dogs to meet other dogs as soon as possible. I personally prefer to start this process at 8 weeks, taking the dog to the pub in a carrier and getting him used to people and the outside world. You can use training pads or newspaper to put the pup down to go to the loo without fear of picking up any nasty diseases. I did this with Hal from 8 weeks, and it paid off dividends.

I really hope that the OP considers our advice carefully instead of writing us off as radicals!

you cant toilet train if you dont spend the time doing it
it is unfair on the puppy imo but i hate leaving mine for more than an hour a day
the best i would say you can hope for is to continue using the pads just be aware with no training / socialization you could end up with behavior problems further down the line

That is a very long time to leave an adult dog without a toilet break or walk to break up the boredom, and I would never advocate leaving a young puppy for the length of time you do. Sorry. Pups need lots of interaction and companionship. To effectively toilet train a pup you need to be with him to ensure access to the outdoors and you need to be there to reward the behaviour you want.
If I were you, I would look at employing a dog walker or seek out a friend or responsible person to spend time with your puppy when you can't be there. Otherwise it probably would be for the best if you found a new home for your puppy where he will receive the training, exercise and companionship all dogs deserve.