Can I say "I guess all (of) the worst part is..." instead of "I guess the worst part of all is…"?
No, in this case you cannot interchange. Saying that something is "the worst part of all" means that everything has been bad but you're going to tell us the particular part that was the worst out of everything else. It is essentially a comparative statement, comparing the worst part to the others. Whereas saying "all of the worst part is" sounds as though you're going to speak about all areas of the worst part in general. There is no comparison or reference to the other parts. So the nuance is different. Ex: The worst part is that I'm so tired. Ex: All of the worst part was tiring.