Description
2 Problem 2 (10 points)
Regularized risk minimization: Modify the Matlab code for “polyreg.m” such that it learns a multivariate regression function f : R100 →R, where the basis functions are of the form
k
f(x;θ) = Xθixi
i=1
The data-set is available in “problem2.mat”. As before, the x variable contains {x1,…,xN} and the y variable contains their scalar labels {y1,…,yN}.
Use an l2 loss function to penalize the complexity of the model, e.g. minimize the risk
Use two-fold cross validation (as in Problem 1) to find the best value for λ. Include a plot showing training and testing risk across various choices of λ. A reasonable range for this data set would be from λ = 0 to λ = 1000. Also, mark the λ which minimizes the testing error on the data set.
What do you notice about the training and testing error?
3 Problem 3 (10 points)
Logistic Squashing Function. The logistic squashing function is given by g(z) = 1/(1 + exp(−z)). Show that it satisfies the property g(−z) = 1−g(z). Also show that its inverse is given by g−1(y) = ln(y/(1 − y).
4 Problem 4 (20 points)
Logistic Regression: Implement a linear logistic regression algorithm for binary classification in Matlab using gradient descent. Your code should accept a dataset {(x1,y1),…,(xN,yN)} where xi ∈Rd and yi ∈{0,1} and find a parameter vector θ∈Rd for the classification function
which minimizes the empirical risk with logistic loss
.
Since you are using gradient descent, you will have to specify the step size η and the tolerance . Pick reasonable values for η and to then use your code to learn a classification function for the dataset in “dataset4.mat”. Type “load dataset4” and you will have the variables X (input vectors) and Y (binary labels) in your Matlab environment which contain the dataset.
Show any derivations you need to make for this algorithm.
Use the whole data set as training. Show with figures the resulting linear decision boundary on the 2D X data. Show the binary classification error and the empirical risk you obtained throughout the run from random initialization until convergence. Note the number of iterations needed for your choice of η and .
