[SOLVED] COMP472-Assignment 3 Covid-19 Fact Checking

Description

In this project, you will build a Naive Bayes bag-of-Word (NB-BOW) approach to determine if a tweet contains a verifiable factual claim. Then, you will compare the output of the NB model to a Word2Vec LSTM approach given to you (see Section 3).

1         The Dataset

Download the Assignment 3 dataset available on Moodle. This dataset, created by [Alam et al., 2020], contains a collection of tweets related to Covid-19 collected from March 9–10, 2020 and March 20–25, 2020. These tweets have been labeled for 7 different questions, such as:

Q1       Does the tweet contain a verifiable factual claim?

Q2       To what extent does the tweet appear to contain false information?

…

Q6      Is the tweet harmful for society and why?

For example, an instance of the dataset contains:

In this assignment, we will only use the classification of the 1st question (q1 label) whose labels are binary (yes/no):

1. yes – the tweet contains a verifiable factual claim
2. no – the tweet does not contain a verifiable factual claim

The dataset is already split into a training set of 400 instances and a test set of 55 instances.

2         The Naive Bayes Classifier (NB-BOW)

You will code a Multinomial Naive Bayes classifier yourself.

2.1        Parameters

Your Naive Bayes Classifier should use the following parameters:

Vocabulary: First fold the training set in lower case, then build a list of all words appearing in the training set. This list will constitute your vocabulary V which will be used as features. To identify the words, tokenise the tweets based on spaces only, use the words as features, and word frequencies as feature values.

Experiment with 2 versions of the model:

Original Vocabulary : one model where all words appearing in the training set are used as features. Let’s call this model NB-BOW-OV.

Filtered Vocabulary : a second model where you filter out the words that appear only once in the training set, so V contains only the words that appear at least 2 times in the training set. Let’s call this model NB-BOW-FV.

Smoothing: to smooth, use additive smoothing (add-δ) with δ = 0.01.

Log: To avoid arithmetic underflow, work in log₁₀ space.

2.2        Output

For both your models (NB-BOW-OV & NB-BOW-FV), your program should create 2 output files: a trace file (see Section 2.2.1) and one overall evaluation file (see Section 2.2.2).

2.2.1        Trace Files

Given a test set, your program should create trace files called trace NB-BOW-OV.txt and trace NB-BOW-FV.txt,

The trace file should contain:

1. the tweet ID as indicated in the test file, followed by 2 spaces
2. the most likely class as determined by your model (i.e. the label yes, no), followed by 2 spaces
3. the score of the most likely class (in scientific notation), followed by 2 spaces
4. the correct class as indicated in the test file, followed by 2 spaces
5. the label correct or wrong (depending on the case), followed by a carriage return.

For example the file trace NB-BOW-O.txt could contain:

2.2.2        Overall Evaluation Files

In addition to the trace file, create text files called eval NB-BOW-OV.txt and eval NB-BOW-FV.txt summarising the performance of the model with the initial test set given on Moodle. The file should indicate the model’s:

1. accuracy (Acc), carriage return
2. per-class precision (yes-P, no-P) separated by 2 spaces, then a carriage return
3. per-class recall (yes-R, no-R) separated by 2 spaces, then a carriage return,
4. per-class F1-measure (yes-F, no-F) separated by 2 spaces, then a carriage return, For example the file eval NB-BOW-OV.txt could contain:

2.3        Programming Environment

You must use Python 3.8. In addition, you must use GitHub (make sure your project is private while developing).

3         The LSTM Classifier (LSTM-W2V)

Your wonderful TAs have implemented a classifier for the same task using an LSTM and Word2Vec embeddings (LSTM-W2V). This code generates the output files (see Section 4.1) for you, so it can be used to compare the performance of your NB-BOW approach.

1. Download the embedding #6 from the http://vectors.nlpl.eu/repository. Note that the download is 606MB.
2. Download the code available at https://gitlab.com/Feasinde/lstm-for-covid-disinformation.
3. Place both downloads in the same folder.
4. Run the code and generate the corresponding output files. Note that the code may take a good 5 to 10 minutes to run.