Nonlinearity Detection Using Penalization-Based Principle