He had a purpose, no right

I met an intruder on the stairs yesterday.
'You don't live in these halls, do you?' I asked him, pointing at the flyers in his hand.
'No,' he said. He slips in every week to spread folders for some pizza-delivery-service.
'And you're not allowed in here, are you?' I pressed on.
He shrugged.
'Then would you please leave the building?' I blushed while asking. It's not that I mind that he leaves slippery pieces of paper on the floor – the next morning the cleaning ladies always pick them up – and in a way I was even glad he had the flyers on him. At least he had a purpose being in my halls, other than nicking my stuff. What I don't like is the fact that, apparently, anybody can get in without a key.
'No,' he answered. He was not very impressed.
'You're a student, right?' he asked.
'Yes,' I replied. For a moment I didn't know what to do next. Then he just walked further up the stairs, ignoring my kind request.
I dithered, perhaps I even said 'hey...', but when he was out of sight I felt weak and drooped off into my room.
Should I've gone after him and made a scene? But he was only here for the flyers. And nobody in my halls gives a damn about the flyers. They would just find it absurd that I was making such a big fuss of it. On the other hand, he had no right to be there, and should have left when I asked him to. He was being rude to me. That's enough reason to throw him out, right?

Why did I not find myself able to make an intruder leave my house? Why would I rather curl back in my room than stand my grounds? And whatever happened to my home advantage?