Description
Assignment policy:
1. Cite all sources of code that you hand in that are not your original work. You can putthe citation into comments in your program. For example, if you find and use code found on a web site, include a comment that says, for example:
# the following code is from https://www.quackit.com/python/tutorial/python_ hello_world.cfm.
Use the complete URL so that the marker can check the source.
2. When you upload your project to GitLab, make sure your repository is set to private. Any repository set to public or internal is visible to other students in the class, and counts as academic misconduct.
4. Discuss and share ideas with other programmers as much as you like, but make surethat when you write your code that it is your own. A good rule of thumb is to wait 20 minutes after talking with somebody before writing your code. If you exchange code with another student, write code while discussing it with a fellow student, or copy code from another person’s console, then this code is not yours.
5. Collaborative coding is strictly prohibited. Your assignment submission must be strictlyyour code. Discussing anything beyond assignment requirements and ideas is a strictly forbidden form of collaboration. This includes sharing code, discussing code itself, or modelling code after another student’s algorithm. You can not use (even with citation) another student’s code.
Instructions:
The goal of this assignment is to give you some practice working with neural nets. The assignment consists of three parts, corresponding to three different data sets. You’ll be experimenting with variations of a neural net to solve classification problems, and writing up your results in a report.
All data sets and starter code files are included in the Assignment4.zip file provided on D2L.
Package management: It is recommended that you use a python virtual environment to manage any external packages you use for the assignment. Instructions for how to set this up can be found at https://docs.python.org/3/tutorial/venv. html. The following packages will be needed for Assignment 4:
numpy – scientific computing library for python (see https://numpy.org/install/) matplotlib – allows easy plotting of data and depiction of images idx2numpy – useful for reading the MNIST image files
Part 1: MNIST Dataset: The home page of the MNIST database is http: //yann.lecun.com/exdb/mnist/. The data set is divided into a set of 60,000 training examples and a set of 10,000 test examples, each consisting of separate files for the images and their labels. Details on the file format can be found at the bottom of the MNIST web page, and will also be covered in tutorial. Note that Nielsen’s textbook trains the net on only 50,000 samples, but you should use all 60,000.
Part 2: notMNIST Dataset: The machine learning community is a bit sick of seeing MNIST digits pop up everywhere, so they created a similar dataset and named it notMNIST. Created by Yaroslav Bulatov, a research engineer previously at Google and now at OpenAI, notMNIST is designed to look like the classic MNIST dataset, but less ‘clean’ and extremely ‘cute’. The images are still 28×28 and there are also 10 labels, representing letters ‘A’ to ‘J’. The homepage for the dataset is http://yaroslavvb.blogspot.com/2011/09/notmnist-dataset.html.
Your task for this dataset is the same as for the MNIST. In this case, however, your starter code hyperparameters will yield an initial accuracy of roughly 60% on the test data. Follow the same instructions as above to reach an accuracy of over 90%. Save your final trained net with the filename “part2.pkl”, and include this in your repository.
In addition to this information, include a short paragraph comparing your results from this dataset to your results from the original MNIST dataset. Note that even though the datasets are similar, it is much harder to get a high accuracy for notMNIST. Why do you think this is the case? Include a brief explanation.
Part 3: Coronary Heart Disease Dataset: The file heart.csv contains data taken from https://web.stanford.edu/~hastie/ElemStatLearn//. Each entry lists a series of observed health factors, followed by a boolean indicating the presence of coronary heart disease. Your goal is to use the first 9 variables to predict the predict the last. Useful information on the dataset can be found in heartmeta.txt.
You will need to load the data into feature vectors yourself, and divide it into training and testing sets. When converting to feature vectors, you will need to convert family history to a boolean, and rescale the age to have maximum value 1. All other variables should be converted to z-scores using the mean and standard deviation calculated from the dataset. These values are provided in heartmeta.txt so you can check your results, but you should calculate them yourself in your code.
Part 1: 98% or higher
Part 2: 93% or higher
Part 3: 75% or higher
Version control: As in the previous assignments, you will be using either GitLab or GitHub to maintain version control and to share your final project with the TAs. Your assignment should be kept in a repository titled CPSC 501 A4. As you develop your code, make sure to use proper version control practices, making regular commits with descriptive messages. This includes keeping a local copy of your assignment (including the git repository) on your own computer, just in case 🙂
Report: Create a written PDF report that discusses the variations you tried and their performances. For each neural net you train, you need to keep track of
1. the hyperparameters used to train the net (number of layers, number of neuronsin each layer, epochs, step size, batch size, activation function)
2. its performance on training data after each epoch
3. the time required to train the net
Make sure all of these factors are included in your report. Note that you will need to add the timing code yourself to track how long the net.SGD() function takes.
Rubric (100 pts total):
Version control: Used Git properly, making multiple small commits with informative messages (5 pts)
Part 1:
– Input is correctly handled, including construction of feature vectors and division into training/testing data. (10 pts)
– At least three neural nets are described, including the one given in the starter code. Description in the report is clear, with appropriate code excerpts. Choice of hyperparameters is well justified and leads to improvements in performance. (10 pts)
– Final neural net has accuracy at least 95% (5 pts)
– Three misclassified MNIST images are included, along with both the incorrect outputs and the correct labels from the dataset (5 pts)
– (Bonus) Final neural net has accuracy at least 98% (5 pts)
Part 2:
– Input is correctly handled, including construction of feature vectors and division into training/testing data. (10 pts)
– At least three neural nets are described, including the one given in the starter code. Description in the report is clear, with appropriate code excerpts. Choice of hyperparameters is well justified and leads to improvements in performance. (10 pts)
– Final neural net has accuracy at least 90% (5 pts)
– Comparison is made to MNIST dataset, with reasonable conclusions drawn (5 pts)
– (Bonus) Final neural net has accuracy at least 93% (5 pts)
Part 3:
– Input is correctly handled, including construction of feature vectors and division into training/testing data. (10 pts)
– At least three neural nets are described, including the one given in the starter code. Description in the report is clear, with appropriate code excerpts. Choice of hyperparameters is well justified and leads to improvements in performance. (15 pts)
– Final neural net has accuracy at least 72% (5 pts)
– (Bonus) Final neural net has accuracy at least 75% (5 pts)
Logistics: Clear, working instructions on how to access GitLab/GitHub project, submitted as part of the PDF report. (5 pts)
author: [email protected]
