Introduction

The goal of the mini-project is to predict the presence of a species (such as a tree/shrub/etc) from presence-only observations. This is useful when trying to mangage populations of species, reducing impacts of invasive species, and preserving endangered species. Predicting habitats in which it is present and grows quickly is important for managing it effectively.

The goal is to predict the distribution from _{presence-only} data. This is a machine learning problem but it is different from the standard classification/regression setting. It is an example of a presence-only dataset. An example of what actual presence data looks like can be seen on EDDMapS. The data generally consists of positive samples only. Most observations are reported by volunteers. People generally do not report when they do not observe the plant and therefore there are no (or very few) negative samples.

Presence-only data is more common in machine learning than you may expect. Positive samples are often easier to collect than negative ones. Some obvious settings that are similar are predicting the presence of insects, pests, and diseases. A lot of businesses have access to positive samples (their customers) and have little access to negative samples (people who are not their customers). Models that deal with presence-only data are also common in other areas of machine learning, such as natural language modeling and (inverse) reinforcement learning.

In this project, we will use New England’s landscape. It is divided into rectangular cells _L. Each cell is identified by a unique _cellid, and for each cell we know its latitude and longitude. The training data is the number of observations _nl reported that have been reported for each cell _{l ∈L}. It is important to note that there may be significant population in a cell, but no-one has reported an observation from there. There are 19 biologically-relevant predictors available from WorldClim which can be used to generalize the observation from the current cells.

The goal if this project is to predict observed populations in cells that are not a part of the training set. Biologists instead usually care about predicting the population in a cell, or its suitability for the species, but ground truth data on that is difficult to gather. The project has two following similar sub-problems.

1. A synthetic subproblem. The population in each cell is computed from a simulated model which is based on a real plant species. The synthetic problem makes is possible to have ground truth data and describe exactly the process that generates the observations. The observations in this synthetic model are based only on a subset of the 19 predictors; there are no other considerations.
2. A real subproblem. This is a real dataset derived from EDDMaps for a mystery plant species in New England. There may be no perfect fit for this model. Other features in addition to the ones provided may be useful when fitting the model. Spatial considerations, such as the presence of the species in nearby-cells, may also play a role in this part of the project.

Objective and evaluation

The goal is to predict, for each cell in the landscape not included in the training set, number of observations in the test set. The output should be a _number for each cell not included in the training set.

Datasets

The following files are available:

1. _csv: Landscape data: cellid, latitude, longitude, bio-features
2. _csv: Presence-only data for the synthetic problem. It contains cell id, and the number of reports (_freq)
3. _csv: Presence-only data for the real problem. It contains cell id, and the number of reports (_freq)

The test sets are, of course, secret for now.

Landscape

The landscape dataset contains the list of all cells, their coordinates, and the biologically-relevant variables. The dataset is available as the _{landscape.csv} file:

landscape <- read_csv(“landscape.csv”, col_types = cols(cellid = col_integer())) limits <- function(x){c(min(x), max(x))} limits_lat <- limits(landscape$lat) limits_long <- limits(landscape$long)

If you would like to see how the landscape data was generated, please see the _{landscape.csv} file.

Description of the variables is at: http://worldclim.org/bioclim. Briefly, their meaning is a follows.

Lets plot the landscape data to get some sense of what it looks like. This code plots some of the climatic variables and the outline of the states. This is the variable _bio13:

And this is _bio6:

Synthetic (Fake) Presence Data

The code in this section demonstrates how the synthetic data is generated. The underlying presence data generated by this code is _different from the data that is provided. The purpose of this code is to give you some information that you can use to design your algorithms.

The synthetic presence data uses only features _bio2, _bio3, _bio5, _bio8 and _bio16. The dependence on these features may be wither linear or non-linear. These features are used to generate true _prevalence of the species in the landscape for each. Think of this as the number of all specimen for each cell.

The synthetic observation data is sampled based on the prevalence of the species in each cell. The procedure corresponds to sampling each specimen with some small fixed probability. That mean that cells that have higher prevalence are likely to have more observations. This a common assumption with species models. We can plot the individual observations (since the observations are on top of each other, they are “jittered”).

The results are aggregated by the cell to make it more convenient working with them.

7

6

5

4

3

2

1

Finally, about half of the observed cell are included in the training data (provided). The other half is retained for the test set (unavailable).

Training

7

6

5

4

3

2

1

Real Reported Presence Data

The real dataset is generated similarly. We take the reported observations, determine which cell they it should be included in, and then aggregate the counts by cell.

Training

Submission and Evaluation

The goal is to predicted the number of observations for each cell in the landscape that matches the test as closely as possible. The evaluation metric is as follows. Each correctly-predicted observation earns 1 point, while each incorrectly-predicted observation _{loses 0.3} points. For examples, lets say that cell 1 in the test set has 5 observations. If you predict 3 then you gain 3 points, and if you predict 7 you lose 0.6 (2*0.3) points. The predicted counts can be real numbers, not just integers.

You should submit the following files:

1. A very _short document (Rmd, Jupyter, Latex, PDF) describing the method(s) used and results (1-2 pages including plots)
2. csv CSV file with two columns: cellid and pred_freq for cells that are NOT in the training set synthetic_train.csv
3. csv: CSV file with two columns: cellid and pred_freq for cells that are NOT in the training set real_train.csv

Observations in Test Sets

The number of observations in the test sets are:

• Synthetic: 5013
• Real: 1981

Example Evaluation

Lets say I figure out the precise coefficients _β for the synthetic problem. I can then generate predictions directly from the estimated prevalence data. I also need to remove all cells that are in the training data from the prediction.

The score can now be computed by joining the results with the test set and computing the difference. The statement below also filters out any extraneous columns, cells that are not in the landscape, and cells from the training set. It assumes that all missing predictions are 0.

## [1] 96.7

Simple Analysis and Ideas

Do not forget about feature transformations.

You can try any technique that you like. Feel free to focus on either the real or the synthetic problem. There are many more considerations that are important when it comes to the real data sets. Some examples:

• The density of the data is subject to density of the human population.
• The assumption that there are more observations when the plant is more prevalent may not be true. If it is growing everywhere nobody bothers to report it.
• There are spatial effects due to how the plant spreads.
• Other features in addition to the ones provided may be useful. You are on your own where to get them. To get started, think of using some of the methods that we have covered. Can you use them in a problem like this? Perhaps you may want to think about the expressing the likelihood of this dataset and base your method on that.

If you would like to explore some of the literature that has been written on the topic, I recommend these papers as a good starting point:

• Phillips, S. J., Dudik, M., & Schapire, R. E. (2004). A maximum entropy approach to species distribution modeling. In International Conference on Machine Learning (ICML).
• Royle, J. A., Chandler, R. B., Yackulic, C., & Nichols, J. D. (2012). Likelihood analysis of species occurrence probability from presence-only data for modelling species distributions. Methods in Ecology and Evolution, 3, 545–554.
• Hastie, T., & Fithian, W. (2013). Inference from presence-only data; the ongoing controversy. Ecography, 36(8), 864–867.
• Fithian, W., & Hastie, T. (2013). Finite-sample equivalence in statistical models for presence-only data. Annals of Applied Statistics, 7(4), 1917–1939.
• Renner, I. W., & Warton, D. I. (2013). Equivalence of MAXENT and Poisson Point Process Models for Species Distribution Modeling in Ecology. Biometrics, 69(1), 274–281.

The post Mini-project: Predicting Distribution of a Species appeared first on Mantutor.

It is mentioned in Chapter 7 of ISL that a cubic regression spline with one knot at ξ can be obtained using a basis of the form x, x², x³, [x − ξ]³₊, where [x − ξ]³₊ = (x − ξ)³ if x > ξ and equals 0 otherwise. We will now show that a function of the form

f(x) = β₀ + β₁x + β₂x² + β₃x³ + β₄[x − ξ]₊³

is indeed a cubic regression spline, regardless of the values of β₀,β₁,β₂, β₃,β₄.

1. Find a cubic polynomial

f₁(x) = a₁ + b₁x + c₁x² + d₁x³

such that f(x) = f₁(x) for all x ≤ ξ. Express a₁,b₁,c₁,d₁ in terms of β₀,β₁,β₂,β₃,β₄.

2. Find a cubic polynomialf₂(x) = a₂ + b₂x + c₂x² + d₂x³

such that f(x) = f₂(x) for all x > ξ. Express a₂,b₂,c₂,d₂ in terms of β₀,β₁,β₂,β₃,β₄. We have now established that f(x) is a piecewise polynomial.

3. Show that f₁(ξ) = f₂(ξ). That is, f(x) is continuous at ξ.

Problem 2 [25%]

Use linear, cubic, and natural regression splines investigated Chapter 7 of ISL to the Auto data set. Is there evidence for non-linear relationships in this data set? Create some informative plots to justify your answer.

Problem 3 [25%]

You will now derive the Bayesian connection to the lasso as discussed in Section 6.2.2. of ISL.

1. Suppose that y_i = β₀ + ^Pp_j=1 x_ijβ_j + _i where ₁,…,_n are independent and identically distributed from a normal distribution N(0,1). Write out the likelihood for the data as a function of values β.
2. Assume that the prior for β : β₁,…,β_p is that they are independent and identically distributed according to a Laplace distribution with mean zero and variance c. Write out the posterior for β in this setting using Bayes theorem.
3. Argue that the lasso estimate is the value of β with maximal probability under this posterior distribution. Compute log of the probability in order to make this point. Hint: The denominator (= the probability of data) can be ignored in computing the maximum probability.
4. Suppose that ₁,…,_n are independent and identically distributed according to the Laplace distribution.

What are the maximum likelihood/MAP estimates of β_i under this assumption? Hint: See https: //en.wikipedia.org/wiki/Least_absolute_deviations

1

Problem 4 [25%]

Based on a true story, according to: The Drunkard’s Walk: How Randomness Rules Our Lives, Leonard Mlodinow

Suppose that you applied for a life insurance and underwent a physical exam. The bad news is that your application was rejected because you tested positive for HIV. The test’s sensitivity is 99.7% and specificity is 98.5% [https://en.wikipedia.org/wiki/Diagnosis_of_HIV/AIDS#Accuracy_of_HIV_testing]. However, after studying the CDC website, you find that in your ethnic group (age, gender, race, …) only one in 10,000 people is infected. What is the probability that you actually have HIV?

2

The post Assignment 5 CS 750/850 Machine Learning appeared first on Mantutor.

What are the advantages and disadvantages of very flexible (vs less flexible) approach for regression or classification?

1. When would be a more flexible approach preferable?
2. What about a less-flexible approach?

Problem 2 [33%]

Install and learn to use R (https://www.r-project.org/) or Python, read the labs in Chapter 2 of the textbook. We recommend that you use R Notebooks of RStudio to typeset homeworks. Jupyter is a comparable tool for Python. Use Python or another tool (like MATLAB or Julia) if you have some experience and you will not need help from the TA/instructor. Then:

1. Download the advertising dataset (Advertising.csv) from http://www-bcf.usc.edu/~gareth/ISL/data. html and load it into R/Python (use function read.csv() in R or Pandas in Python)
2. What are the minimum, maximum, and mean value of each feature? (in R use function summary() and or range())
3. Produce a scatterplot matrix of all variables (in R use function pairs())
4. Produce a histogram of TV advertising (in R use function hist())

Problem 3 [34%]

Describe some real-life applications for machine learning.

1. Describe one real-life application in which classification combined with prediction may be useful. Describe the response and predictors.
2. Describe one real-life application in which classification combined with inference may be useful. Describe the response and predictors.
3. Describe one real-life application in which regression combined with prediction may be useful. Describe the response and predictors.
4. Describe one real-life application in which regression combined with inference may be useful. Describe the response and predictors.

Optional Problem O3 [39%]

This problem can be substituted for Problem 3 above, for 5 points extra credit. At most one of the problems 3 and O3 will be considered.

Read sections 1.2, 1.2.1, 1.2.2 in [Bishop, C. M. (2006). Pattern Recognition and Machine Learning] and solve Exercise 1.5.

1

Hints

1. An easy way to launch help for any function in R, such as summary, is to execute: > ?summary
2. See http://rmarkdown.rstudio.com/pdf_document_format.html for how to generate a PDF from an R notebook in R-studio. You will also need to install L^ATEXwhich you can get from https://www. latex-project.org/get/
3. For more advanced (and prettier?) plotting capabilities, see the package ggplot: http://ggplot2.

tidyverse.org/ and https://github.com/rstudio/cheatsheets/raw/master/data-visualization-2.1.pdf

4. If you think you may struggle with R, consider signing up for MATH 759, a 1-credit online introduction to R.

The post Assignment 0 CS 750/850 Machine Learning appeared first on Mantutor.

In this exercise, we will predict the number of applications received using the other variables in the College (ISLR::College) data set.

1. Fit a linear model using least squares on the training set, and report the test error obtained.
2. Use best subset selection with cross-validation. Report the test error obtained.
3. Fit a ridge regression model on the training set, with λ chosen by cross-validation.
4. Fit a lasso model on the training set, with λ chosen by cross-validation. Report the test error obtained, along with the number of non-zero coefficient estimates.
5. Briefly comment on the results obtained. How accurately can we predict the number of college applications received? Is there much difference among the test errors resulting from these approaches?

Problem 2 [25%]

We will try to predict per capita crime rate in the Boston dataset.

1. Try out best subset selection, the lasso, ridge regression, and PCR on this problem. Present and discuss results for the approaches that you consider.
2. Propose a model (or set of models) that seem to perform well on this data set, and justify your answer. Make sure that you are evaluating model performance using validation set error, cross-validation, or some other reasonable alternative, as opposed to using training error.

Problem 3 [25%]

Suppose we have a linear regression problem with P features. We estimate the coefficients in the linear regression model by minimizing the RSS for the first p features:

n                            p                2

                                                            X                                X

yi − β0 −                βjxij

i=1                              j=1

where p ≤ P. For parts (1) through (5), indicate which of i. through v. is correct. Briefly justify your answer.

1. As we increase p from 1 to P, the training RSS will typically:
2. Remain constant. ii. Steadily increase. iii. Steadily decrease. iv. Increase initially, and then eventually start decreasing in an inverted U shape.
3. Decrease initially, and then eventually start increasing in a U shape.

1

2. Repeat (1) for test MSE.
3. Repeat (1) for squared bias.
4. Repeat (1) for variance.
5. Repeat (1) for the irreducible error (Bayes error).

Problem 4 [25%]

Suppose we estimate the regression coefficients in a linear regression model by minimizing

n                            p                2                               p

                                           X                               X                                                  X

_{ i} − β₀ −    β_jx_ij subject to     |β_j| ≤ s y

i=1                              j=1                                                 j=1

for a particular value of s. For parts (1) through (5), indicate which of i. through v. is correct. Justify your answer.

1. As we increase s from 0, the training RSS will typically:
2. Remain constant. ii. Steadily increase. iii. Steadily decrease. iv. Increase initially, and then eventually start decreasing in an inverted U shape.
3. Decrease initially, and then eventually start increasing in a U shape.
4. Repeat (1) for test RSS.
5. Repeat (1) for (squared) bias.
6. Repeat (1) for variance.
7. Repeat (1) for the irreducible error (Bayes error).

Problem O4 [30%]

This problem can be substituted for Problem 4 above, for up to 5 points extra credit. The better score from problems 4 and O4 will be considered.

Solve Exercise 3.6 in [Bishop, C. M. (2006). Pattern Recognition and Machine Learning].

The post Assignment 4 CS 750/850 Machine Learning appeared first on Mantutor.

Suppose that I collected data for a group of machine learning students from last year. For each student, I have a feature X₁ = hours studied for the class every week, X₂ = overall GPA, and Y = whether the student receives an A. We fit a logistic regression model and produce estimated coefficients, β^ˆ₀ = −6,β^ˆ₁ = −0.1,β^ˆ₂ = 1.0.

1. Estimate the probability of getting an A for a student who studies for 40h and has an undergrad GPA of 2.0
2. By how much would the student in part 1 need to improve their GPA or adjust time studied to have a 90% chance of getting an A in the class? Is that likely?

Problem 2 [25%]

Consider a classification problem with two classes T (true) and F (false). Then, suppose that you have the following four prediction models:

• T: The classifier predicts T for each instance (always)
• F: The classifier predicts F for each instance (always)
• C: The classifier predicts the correct label always (100% accuracy)
• W: The classifier predicts the wrong label always (0% accuracy)

You also have a test set with 60% instances labeled T and 40% instances labeled F. Now, compute the following statistics for each one of your algorithms:

Some of the rows above may be the same.

Problem O2 [30%]

This problem can be substituted for Problem 2 above, for up to 5 points extra credit. The better score from problems 2 and O2 will be considered.

Solve Exercise 3.4 in [Bishop, C. M. (2006). Pattern Recognition and Machine Learning].

1

Problem 3 [25%]

In this problem, you will derive the bias-variance decomposition of MSE as described in Eq. (2.7) in ISL. Let f be the true model, f^ˆ be the estimated model. Consider fixed instance x₀ with the label y₀ = f(x₀). For simplicity, assume that Var[] = 0, in which case the decomposition becomes:

                                      ²

− f

E^h(y₀                      ^ˆ(x₀))²ⁱ = Var[f^ˆ(x₀)]+_E[f(x₀) − f^ˆ(x₀)]_ .

{z            }

|              {z               }                 Variance      Bias test MSE Prove that this equality holds.

Hints:

1. You may find the following decomposition of variance helpful:
2. This link could be useful: https://en.wikipedia.org/wiki/Variance#Basic_properties

Problem 4 [25%]

Please help me. I wrote the following code that computes the MSE, bias, and variance for a test point.

## MSE:                           2869.61343086216

## Bias^2:                       2681.61281912074

## Var:                             208.889568601581

## Bias^2+Var: 2890.50238772232

I expected that the MSE would be equal to Biasˆ2 + Variance, but that does not seem to be the case. The MSE is 2402.515 and Biasˆ2 + Variance is 2428.706. Was my assumption wrong or is there a bug in my code? Is it a problem that I am computing the expectation only over 10 trials?

Hint: If you are using Python and need help with this problem, please come to see me (Marek).

The post Assignment 3 CS 750/850 Machine Learning appeared first on Mantutor.

This problem examines the use and assumptions of LDA and QDA. We will be using the dataset Default from ISLR.

1. Split the data into a training set (70%) and a test set (30%). Then compare the classification error of LDA, QDA, and logistic regression when predicting default as a function of features of your choice. Which method appears to work best?
2. Report the confusion table for each classification method. Make sure to label which dimension is the predicted class and which one is the true class. What do you observe?
3. Are the LDA assumptions satisfied when predicting default as a function of balance only (i.e default ~ balance)? You can use qqnorm and qqline to examine whether the conditional class distributions are normally distributed. Also examine standard deviations of the class distributions. Are the QDA assumptions satisfied?
4. Would you ever want to use LDA in place of QDA even when you suspect that some of the assumptions are violated (e.g. different conditional standard deviations) for LDA?

Hint: Check out TidyVerse for a collection of packages that can help with data manipulation. And see the Rstudio cheatsheets for a convenient and concise reference to the methods. This is entirely optional!

Problem 2 [30%]

Using the MNIST dataset, fit classification models in order to predict the digit 1 (vs all others).

1. Compare the classification error for each one of these methods:
2. Logistic regression
3. K-NN with 2 reasonable choices of k
4. LDA
5. Explore at least one transformation of the features (predictors), such as considering their combinations, and run the methods from part 1 on the data.
6. Which one of the methods works the best?

Make sure to split the data into a training set and a test set. No need to run on the entire dataset; a subsample of say 10000 datapoints is OK.

Hint: There is a file in the gitlab repository: assignments/mnist_simple.Rmd which you can use as a starting point. If you are using Python, please checkout this package. If you have trouble getting started, please do not hesitate to ask the instructor or the TAs or Piazza for help.

1

Problem O2 [35%]

This problem can be substituted for Problem 2 above, for up to 5 points extra credit. The better score from problems 2 and O2 will be considered.

Solve Exercises 1.11 and 1.13 in [Bishop, C. M. (2006). Pattern Recognition and Machine Learning].

Problem 3 [20%]

Logistic regression uses the logistic function to predict class probabilities:

eβ₀+β₁X

p(X) =

1 + eβ₀+β₁X

This is equivalent to assuming a linear model for the prediction of the log-odds:

 p(X)

log               = β₀ + β₁X

1 −p(X)

Using algebraic manipulation, prove that these two expressions are identical. See Section 4.3 in ISLR and equations (4.2) & (4.3) for more context.

Problem 4 [20%]

This problem examines the differences between LDA and QDA.

1. For an arbitrary training set, would you expect for LDA or QDA to work better on the training set?
2. If the Bayes decision boundary between the two classes is linear, would you expect LDA or QDA to work better on the training set? What about the test set?
3. As the sample size increases, do you expect the prediction accuracy of QDA with respect to LDA increase or decrease
4. True or False: Even if the Bayesian decision boundary for a given problem is linear, we will probably achieve a superior test error rate using QDA rather than LDA because QDA is more flexible and can model a linear decision boundary. Justify your answer.

The post Assignment 2 CS 750/850 Machine Learning appeared first on Mantutor.

Feel free to achieve the results using commands other than the ones mentioned. R, especially, shines when it comes to processing and visualization of structured data, but you would not be able to tell from the ISL book. It uses the oldest and simplest (and ugliest) subset of R. I recommend checking out dplyr for data processing [3,4] and GGPlot [1,4] for plotting.

Problem 1 [25%]

In this exercise you will create some simulated data and will fit simple linear regression models to it. Make sure to use set.seed(1) [P: np.random.seed(1)] prior to starting part (1) to ensure consistent results.

1. Using the rnorm() [P: np.random.normal] function, create a vector, x, containing 100 observations drawn from a√ N(0,3) distribution (Normal distribution with the mean 0 and the standard deviation

3). This represents a feature, X.

2. Using the rnorm() function, create a vector, eps, containing 100 observations drawn from a√ N(0,0.5)

distribution i.e. a normal distribution with mean zero and standard deviation             0.5.

3. Using x and eps, generate a vector y according to the model Y :

Y = −2 + 0.6X +

What is the length (number of elements) of y? What are the values of β₀,β₁ in the equation above (intercept and slope)?

4. Create a scatterplot displaying the relationship between x and y. Comment on what you observe. [P: see [2]]
5. Fit a least squares linear model to predict y using x. Comment on the model obtained. How do β^ˆ₀,β^ˆ₁ compare to β₀,β₁?
6. Display the least squares line on the scatterplot obtained in 4.
7. Now fit a polynomial regression model that predicts y using x and x². Is there evidence that the quadratic term improves the model fit? Explain your answer.

Optional Problem O1 [30%]

This problem can be substituted for Problem 1 above, for up to 5 points extra credit. At most one of the problems 1 and O1 will be considered.

Read Chapter 1 and solve Exercises 1.6 and 1.10 in [Bishop, C. M. (2006). Pattern Recognition and Machine Learning].

Problem 2 [25%]

Read through Section 2.3 in ISL. Load the Auto data set and make sure to remove missing values from the data. Then answer the following questions:

1. Which predictors are quantitative and which ones are qualitative?

1

2. What is the range, mean, and standard deviation of each predictor? Use range() [pandas.DataFrame.min and max] function.
3. Investigate the predictors graphically using plots. Create plots highlighting relationships between predictors. See [1] for a ggplot cheatsheet.
4. Compute the matrix of correlations between variables using the function cor() [P: pandas.DataFrame.corr]. Exclude the name variable.
5. Use the lm() function to perform a multiple linear regression with mpg as the response. [P: using rpy package is acceptable] Exclude name as a predictor, since it is qualitative. Briefly comment on the output: What is the relationship between the predictors? What does the coefficient for year variable suggest?
6. Use the symbols * and : to fit linear regression models with interaction effects. What do you observe?√
7. Try a few different transformations of variables, such as log(X), X, X². What do you observe?

Problem 3 [25%]

Using equation (3.4) in ISL, argue that in the case of simple linear regression, the least squares line always passes through the point (¯x,y¯).

Problem 4 [25%]

It is claimed in the ISL book that in the case of simple linear regression of Y onto X, the R² statistic (3.17) is equal to the square of the correlation between X and Y (3.18). Prove that this is the case. For simplicity, you may assume that x¯ = ¯y = 0.

References

Each reference is a link. Please open the PDF in a viewer if it is not working on the website.

1. R GGPlot cheat sheet
2. Python Pandas data visualization
3. R For Data Science
4. Cheatsheets

2

The post Assignment 1 CS 750/850 Machine Learning appeared first on Mantutor.