I'm more familiar with the phrase "be afraid of [something]", but is "be afraid to [verb] also common? Can I use "be afraid of leaving" here instead?
Great question. Both "be afraid of verb+ing" and "be afraid to verb" are common and correct. Traditionally we use "be afraid of verb+ing" when a person feels afraid in general but "be afraid to verb" is used when a person feels afraid in a particular circumstance. In the case of this clip, since it is a particular circumstance "be afraid to" is used. I advise you to remember that native speakers might not always follow this rule between the two different forms. A native speaker might as well say "don't be afraid to make mistakes", in the same situation which another would say "don't be afraid of making mistakes". So many times, in general conversation between native speakers there is no difference between the two forms. Ex: I'm afraid to say what I really think. = I'm afraid of saying what I really think. Ex: He's afraid of losing his independence. = He's afraid to lose his independence.