This assignment counts for 10% of the course grade.
Assignments turned in after the deadline but before April 8 are subject to a 30% grade penalty.
In this assignment you will write a naive Bayes classifier to identify hotel reviews as either true or fake, and either positive or negative. You will be using the word tokens as features for classification. The assignment will be graded based on the performance of your classifiers, that is how well they perform on unseen test data compared to the performance of a reference classifier.
A set of training and development data will be made available as a compressed ZIP archive on Blackboard. The uncompressed archive will have the following files:
train-labeled.txt containing labeled training data
with a single training instance (hotel review) per line (total 960 lines).
The first 3 tokens in each line are:
True or Fake
Pos or Neg
dev-text.txt with unlabeled development data,
containing just the unique identifier followed by the text of the
review (total 320 lines).
dev-key.txt with the corresponding labels
for the development data, to serve as an answer key.
You will write two programs: nblearn.py will learn a
naive Bayes model from the training data, and
nbclassify.py will use the model to classify new data. If
using Python 3, you will name your programs nblearn3.py
and nbclassify3.py. The learning program will be invoked
in the following way:
> python nblearn.py /path/to/input
The argument is a single file containing the training data; the program
will learn a naive Bayes model, and write the model parameters to a
file called nbmodel.txt. The format of the model is up to
you, but it should follow the following guidelines:
nbclassify.py to successfully label new data.
The classification program will be invoked in the following way:
> python nbclassify.py /path/to/input
The argument is a single file containing the test data file; the program
will read the parameters of a naive Bayes model from the file
nbmodel.txt, classify each entry in the test data, and
write the results to a text file called nboutput.txt in
the same format as the answer key.
All submissions will be completed through Vocareum; please consult the instructions for how to use Vocareum.
Multiple submissions are allowed; only the final submission will be graded. Each time you submit, a submission script trains your model on the training data, runs your classifier on the development data, and reports the results. Do not include the data in your submission: the submission script reads the data from a central directory, not from your personal directory. You are encouraged to submit early and often in order to iron out any problems, especially issues with the format of the final output.
The performance of you classifier will be measured automatically; failure to format your output correctly may result in very low scores, which will not be changed.
For full credit, make sure to submit your assignment well before the deadline. The time of submission recorded by the system is the time used for determining late penalties. If your submission is received late, whatever the reason (including equipment failure and network latencies or outages), it will incur a late penalty.
If you have any issues with Vocareum with regards to logging in, submission, code not executing properly, etc., please contact Siddharth.
After the due date, we will train your model on a combination of the training and development data, run your classifier on unseen test data, and compute the F1 score of your output compared to a reference annotation for each of the four classes (true, fake, positive, and negative). Your grade will be based on the performance of your classifier. We will calculate the mean of the four F1 scores and scale it to the performance of a naive Bayes classifier developed by the instructional staff (so if that classifier has F1=0.8, then a score of 0.8 will receive a full credit, and a score of 0.72 will receive 90% credit).