Minor bad service from UPS

I've been having some problems with something I ordered from a different company, so basically I got the first item shipped to my house but it was defective and the company replaced it but then I had to call UPS to come pick it up and re-deliver the new one. In the last 2 weeks I have had UPS deliver to my house 3 times. I've had the same driver each time. Today I answered the door, he said "Am I done delivering here now?" I just replied with "Um, maybe..." because his comment surprised me. I signed for the package and as he was leaving he said, "Well hopefully I won't need to come back AGAIN."

He wasn't joking and he sounded like he was actually annoyed with me. Wtf? Maybe I'm being sensitive but that just seemed completely unnecessary to say that to me. Is it really that big of a deal to come to the same house multiple times?