Description
- Download and unzip “Project4_sentences.zip” and “Project4_code.zip” files.
A set of sentences is given in the file “sentences.txt”. Each sentence is a line in the file. Create the feature vector by writing a program that applies the following text mining techniques to this set of sentences.
- Tokenize sentences
- Remove punctuation and special characters
- Remove numbers
- Convert upper-case to lower-case
- Remove stop words. A set of stop words is provided in the file “txt”
- Perform stemming. Use the Porter stemming code provided in the file “txt”
- Combine stemmed words.
- Extract most frequent words.
Provide the feature vector in your report.
- Using the feature vector generated in first task, write a program that generates the Term Document Matrix (TDM) for ALL the sentences in “txt”, similar to TDM below. Example TDM
| Keyword set | anonymous | identify | car | … |
| Sentence 1 | 1 | 4 | 3 | … |
| Sentence 2 | 2 | 0 | 1 | … |
| ….. | … | … | … | … |
| Sentence 20 | 2 | 0 | 0 | … |
- Provide the TDM in your report.
- For each of the text mining steps (A to H), explain why they are used, and what sort of information is lost while applying each of the text-mining steps.
- Write a program implementing the clustering algorithm of your choice (WTA or FCAN). Apply that algorithm to TDM to group similar sentences together.
- How many clusters/topics have you identified?
- What drives the dimensionality of TDM? What can you do to reduce that dimensionality? Does the order of data being fed to algorithm matter?
- Show and comment the results.
