Can I say "a way too big project" instead? Is it different from "way too big a project"?
The speaker here means "way too big [of] a project," with the "of" being omitted while speaking. The speaker has used this expression to emphasize how difficult it was to handle the project. "A way too big project" is grammatically incorrect. If you want to use this grammar pattern, you can remember it as "too [adjective] of a [noun]." Ex: There was too big of a skill gap between the two soccer players. Ex: It was way too big [of] a burden for the single mother to raise three children by herself.