{"componentChunkName":"component---src-templates-post-template-js","path":"/posts/datascience/cross-validation-and-regularization","result":{"data":{"markdownRemark":{"id":"a7d5b84c-6f27-5f34-b836-929e8711c637","html":"

The train-test Split

\n\n

Recipe for Successful Generalization

\n
    \n
  1. Split data into training(90%) and testing set(10%)
    2. \n
  2. Use only training data when training, use cross validation to test generalization(do not look at testing data!)
    3. \n
  3. Commit to the model and train once more using only training data
    4. \n
  4. Test the model using testing data. If accuracy is not acceptable, return to step 2
    5. \n
  5. Train on all available data and ship it
    6. \n
\n

Regularization

\n

Parametrically controlling the model complexity\nNaive Solution: Find the best value of theta which uses less than B features. Unfortunately, this is an NP hard combinatorial search problem.

\n\n

Standardization: Ensure each dimension has the same scale and is centered around 0\nIntercept Terms: Typically doesn’t regularize the intercept term

\n

Ridge Regression

\n

Model: Y = X * theta\nLoss: Squared Loss\nRegularization: L2 regularization\nObjective Function Squared Loss + added penalty\n\n \n \n \n \n \n \n \n \n

\n

Note that there is always a unique optimal parameter vector for Ridge Regression!

\n

Lasso Regression

\n

Model: Y = X * theta\nLoss: Squared Loss\nRegularization: L2 regularization\nObjective Function Average Squared Loss + added penalty\n\n \n \n \n \n \n \n \n \n

\n

Note that there is NO closed form solution for the optimal parameter vector for LASSO so we must use numerical methods like gradient descent

\n

\n \n