Do you have any options of blocking people or moderating comments on your blog? I don't know many blog sites, maybe some of the biggest ones before but now I rarely look because I felt insecure at my own words and what others would think of them. I also had "somebody" directly abusing me and it hurt a lot - managed to work out the same little language mistakes that me ex-wife made (this was when I was with her), that it was an old friend of hers that was very materialistic and telling her elsewhere to leave me for somebody rich and supposedly better. The comments hurt a lot.
It got to the same point as on any forum I was ever on before, and I left. Even without knowing the exact content of your blogs and these comments, it seems a form of online bullying and as not warranted, there should be means to do something about it. Blocking if there are options, moderating, or make a complaint to the website that these comments are happening and you are not comfortable with it. You shouldn't be made to feel like this, but don't give up on doing something you enjoy