Is "hard to listen" skipped version of "it's hard to listen"? Can I always skip "it's" in front of the adjective?
Yes, that's correct. The speaker has omitted the "it's" in front of "hard to listen to." It's common to omit words like "it," "it's," and articles like "the" or "a/an" while speaking, but you should generally include them anyway. In some cases, it's okay to omit the article at the beginning of sentences, for the purpose of conciseness. Ex: Paper's by the front door. = The paper's by the front door. Ex: Pizza's here. = The pizza's here.