Description
1. Linear Regression
In LinearRegression\linearRegression.py, fill in the missing lines in the python code to estimate 𝛽, using (1) closed-form solution, (2) batch gradient descent, and (3) stochastic gradient descent.
- Report the learned weights and MSE (Mean Square Error) in the test dataset for each version, are they the same and why?
- Apply z-score normalization for each feature x and report whether the normalization affect 𝛽 and MSE (Mean Square Error) in the test dataset, for all three versions of the algorithm, and why?
2. Logistic Regression and Model Selection
In LogisticRegression\LogisticRegression.py, fill in the missing lines in the python code to estimate 𝛽, using (1) batch gradient descent and (2) Newton Raphson method.
- Report the learned weights and accuracy (Mean Square Error) in the test dataset for each version, are they the same and why? Discuss the pros and cons of the two methods.
- Similar to linear regression, regularization can be added to logistic regression.
Consider the new objective function as:
𝑝
𝐽(𝛽) = −∑(𝑦𝑖𝒙𝑇𝑖 𝛽 − log(1 + exp{𝒙𝑖𝑇 𝛽}))/n + 𝜆∑𝛽𝑗2
𝑖 𝑗=0
where n is the total number of data points in the training dataset and p is the
dimensionality of attributes. Please compute its first derivative 𝜕𝐽(𝛽) and
𝜕𝛽𝑗
implement a regularized batch gradient descent algorithm accordingly. Discuss how the regularization is affecting the training loss (in terms of average log likelihood) and test accuracy based on the experimental results.
