Post visibility for replies

Just had someone tell me my post visibility was set to public, on a reply to a followers-only post (when it should be followers-only as well). Does this matter? And if so, is it possible to fix? Generally, is there a way to control the visibility/privacy of replies?