How do you let go?

I find myself doing this more than I'd like. I hold grudges against myself for dumb things I've done or said or the lack of action when I should have known better. I torment myself with this and even things that I've resolved will haunt me and I am left battle worn in my mind. I've always found it hard to let friendships end on a sour note and I am trying to realize that I can't make anyone stay my friend after I've damaged it. But my problem is letting it all go and moving on. What do you do? I know my actions aren't helping anyone, least of all me so I'm looking for a new way to handle things. After all, the definition of insanity is doing the same thing and expecting different results. This problem has driven me to the edge of insanity. I'm open to suggestions.