How to deal with edge pawn rush

I always lose games if the opponent immediately edge pawn rush. 
For example, if opponent ignore center completely and immediately go a7, b6, h7, g6, then start rushing with c5, f5. 
Everything is always defended I usually need to lose a minor piece such as a knight or bishop to get out of the bind. 
If I castle I would immediately lose, do I leave my kind in center for these type of side pawn rushes?

If your opponent is attacking on the flanks, control the center. The pawn pushes also make their king weaker too so you can also attack

Don't waste time reacting to every pawn push. Instead, develop pieces quickly (knights, bishops, and castle when appropriate).