Description
Data Mining
1 WORKING WITH A TEXT CORPUS – NLP
(50 marks total)
There are 10 files in this corpus.
HINTS:
- Refer back to the NLP Practical (week 7) for tools to handle text data.
- Don’t forget to remove stop words before performing any text analysis, as you did in the NLP
Practical.
Your mission is to analyse the data using scikit-learn, nltk and TextBlob. First, compute the following statistics for each book:
- polarity
- subjectivity
- word count (without stop words) in the book
- most frequent term (word) in the book
- normalised frequency of most frequent word (normalised by the word count)
- term frequency of the most frequent word in the book
- inverse document frequency of the most frequent word in the book
- TF-IDF for the most frequent word in the book
In your report, include a table in which you report all these statistics, clearly indicating which statistics go with which book.
Also indicate which TF-IDF formula you use (see Lecture notes from week 7 about TF-IDF).
(40 marks for computing the above statistics for all 10 books, 5 marks per statistic)
Next, mine the data for patterns. The books can be labelled by author (Potter and Kipling). Can you detect any patterns in any of the statistics between the two authors? Can you use the patterns to train a classifier that will accurately determine which author wrote a given book? In your report, clearly describe the pattern(s) you have identified, which technique(s) you applied to find the pattern(s), how you trained a classifier, which classifier you used and your classification results.
(10 marks for explanation)
2 WORKING WITH AN IMAGE DATA SET – IMAGE PROCESSING
(50 marks total)
The 5 raw images which you downloaded are shown in Figure 1. Your mission is to analyse the image data using scikit-image, as follows:
- (5 marks) Generate a greyscale version of each image.
- (5 marks) Generate a black-and-white version of each image.
- (5 marks) Detect edges in each image.
- (5 marks) Detect contours in each image.
- (10 marks) Detect the green in each image
- (10 marks) Detect straight lines in each image using the Hough transform.
In your report, explain your analysis and how you reached your answer to the question above. For each of the 6 images, describe clearly which technique(s) you applied and show the results for each type of analysis (a-f). (10 marks for explanation)
A-39 B-26 C-15
A-23 B-51 C-44
examples:
(2e) greens (A-39) (2f) straight lines (B-51)
Figure 1: Raw images (top and middle rows) and example answers (bottom row)
