Should speed limits be scrapped?

I am a very good driver. I have never had so much as a scratch in the eight years I've been driving. I do, however, know folk who have had accidents and they have all said the same thing "I was checking how fast I was going". I consider myself a good driver because I'm sensible. I'll cruise at 90mph on an empty dual carriageway, but I wont go above 30 in a busy town centre. This is not because of speed restriction, but due to the fact I don't want to hurt anyone/get into trouble. Even if I did speed, I would only get punished if I was seen by a police officer. If he sees me driving dangerously, he'd pull me anyway regardless of speed restriction. Why do we have these restrictions? They cause more harm than good. For anyone who disagrees with me, you could probably agree that Jeremy Clarkson is the authority on driving. He made this point himself on the last episode of Top Gear. I'm glad I'm not alone in realising this stupidity. What are your thoughts?