Hagrid looked at Harry with warmth and respect blazing in his eyes, but Harry, instead of feeling pleased and proud, felt quite sure there had been a horrible mistake. A wizard? Him? How could he possibly be? He'd spent his life being clouted by Dudley, and bullied by Aunt Petunia and Uncle Vernon; if he was really a wizard, why hadn't they been turned into warty toads every time they'd tried to lock him in his cupboard? If he'd once defeated the greatest sorcerer in the world, how come Dudley had always been able to kick him around like a football?
"Hagrid," he said quietly, "I think you must have made a mistake. I don't think I can be a wizard."

Harry Potter and the Philosopher's Stone - Chapter 4 : The Keeper of the Keys
See this quote in the Pensive -