Everybody has, at least once in their lifetime, said something they regret, and it is usually done out of self defence. Sometimes they can make peace with the other person, sometimes they can't. They way I see it, if you are truly sorry for what's happened then that's what matters. Self forgiveness is everything when it's too late to apologise.
Your brother would not want you to suffer because of that one moment. Your hurt is his hurt, and I believe he has already forgiven you. What I am about to say is not based on religion. Find the right time and space, be peaceful and apologise to him. It's never too late. He hears you. Like I said earlier, you need to forgive yourself too.
You are not an awful person and you do not deserve the worst in life. Everybody makes mistakes. It's what we learn from those mistakes that matters.