I had the same problem, and each time it was sebus or an in-grown hair. I don't think there is anything to be worried about, but when in doubt, it is always better to seek for medical advice.