CSCI 544 — Applied Natural Language Processing

Spring 2021

Synopsis

This course covers both fundamental and cutting-edge topics in Natural Language Processing (NLP) and provides students with hands-on experience in NLP applications.

This graduate course is intended for:

• students who want to understand current NLP technologies
• students interested in building NLP applications
• students interested in applications of NLP like sentiment analysis, dialogue systems, question answering systems, among others

Recommended preparation: Proficiency in programming, algorithms and data structures, basic knowledge of linear algebra and machine learning.

Related Courses

This course is part of USC’s curriculum in natural language processing. There is a sister course, CSCI 662 Advanced Natural Language Processing, which covers complementary (and advanced) material and is intended for PhD students (or students who want to continue to a PhD program).

Course Objectives

Students in the course will learn to perform the following:

1. Read technical literature in Natural Language Processing (including original research articles) and answer questions about such readings.
2. Implement language processing algorithms and test them on natural language data.
3. Solve language processing problems and explain the reasoning behind their solutions.

Coursework and Grading

• (10%) Reading quizzes: Weekly web-based quizzes relating to reading assignments.
• (40%) Coding assignments: A series of programming exercises spread throughout the semester.
• (ungraded) Written assignments: A set of written problem sets, designed as preparation for the midterm and final exams.
• (24%) Midterm exams: Two exams taken during the semester, with questions similar to those from the homework assignments and quizzes.
• (16%) Written paper: NLP prototype design proposal, presenting the design of a novel, original NLP application.
• (5%) Research article presentation: A presentation of an original research article to the entire class (in pairs).
• (5%) Final exam: An on-line, take home comprehensive exam taken during the final exam period.

Grading scale

The following scale is used for determining final grades (note that A is the highest grade given by USC).

• A: 92%, A–: 90%, B+: 87%, B: 82%, B–: 80%, C+: 77%, C: 72%, C–: 70%, D+: 67%, D: 62%, D–: 60%

Grade challenges

• Grades on an assignment or exam will only be changed if there is an error in grading. Grading errors are simple mistakes made on the part of the graders, and not differences in interpretation of a question or answer. A student who wishes to challenge a grade must identify the grading error before asking for a grade change.
• Students are welcome to discuss any aspect of the assignments and exams, but there will be no negotiation on grades, and no changes other than the correction of grading errors.

Late Policy

• There will be a penalty for turning in homework late; the penalty will vary by assignment. In no instance will credit be given for a homework assignment submitted after the solution has been discussed in class.
• Reading quizzes are intended to make sure that students are prepared for class, so no late quizzes will be accepted.
  • One exception: students who registered late will be allowed to make up the quizzes they missed due to late registration. All make-up quizzes will be taken after the add/drop date, in the week of February 8–12.
• Homework assignments will not be accepted by email. If there are technical or other issues with the submission system you should write to us and we will work to fix these issues, but do not send homework by email just because you weren’t able to submit it through the system.
• Assignments are usually due at the end of the day. We may not be available to solve issues close to the deadline, so you should plan on submitting your homework early, even if only as a draft. Multiple submissions are generally allowed, and the last submission will be graded. Quizzes may only be submitted once.
• Students are responsible for on-time submission. When submitting your assignments, please allow time to fix issues such as your computer freezing or crashing, network latencies or outages, and so on. If your assignment is submitted late because of a technical issue, it will be penalized as late.
• Students who are not able to make an assignment or exam deadline due to an emergency (for example, a medical emergency) or a conflict (for example, a job interview) must inform me as soon as the issue arises, in order to make alternate arrangements. If you wait until after the deadline, or the grades are returned, or the end of the course, you will not be allowed to make up the work.

  Note that in general, conflicts with exams or deadlines for another class do not qualify for making alternate arrangements. Working and studying for multiple classes is an expectation from university students.

Communication

Please use the class discussion boards on Piazza for questions and issues regarding homework assignments and the course in general. This way, the entire class can participate and see the questions and answers. Email should be reserved for communication of a personal nature. If we receive questions by email where the response could be helpful for the class, we may ask you to repost the question on the discussion boards.

Any special requests must be submitted in writing.

Resources

• Blackboard (reading quizzes, distribution of written assignments, written paper, final exam)
• Piazza (class announcements, class discussion boards)
• Vocareum (coding assignments)
• Crowdmark (midterm exams)

Recommended textbooks

The course does not have a textbook. Required readings will be specified in the schedule below as the course progresses, and will include a combination of select textbook chapters as well as original research articles. The links below give access to the full text of several textbooks; these are useful for general background on Natural Language Processing, and to supplement some of the materials taught in class. Any chapters that are required will be detailed in the schedule; otherwise, these texts are not required.

• Dan Jurafsky and James H. Martin, Speech and Language Processing (3rd edition draft, 2020)
• Christopher D. Manning, Prabhakar Raghavan and Hinrich Schütze, Introduction to Information Retrieval (2008) (select chapters)
• Yoav Goldberg, Neural Network Methods for Natural Language Processing (2017) (This link provides free access to the PDF of the full book when accessed from the USC network)
• Jacob Eisenstein, Natural Language Processing (free version, 2018) or the published version (2019)
• Emily M. Bender, Linguistic Fundamentals for Natural Language Processing: 100 Essentials from Morphology and Syntax (2013) (This link provides free access to the PDF of the full book when accessed from the USC network)

Schedule

Topics listed in the schedule are tentative and subject to change.

Due dates for the coding assignments and written paper will be announced when the assignments are handed out.

Optional readings are useful for the class, but will not appear on the quizzes. Optional readings will be added as the term progresses.

• Week 1
  • January 18: No class (Martin Luther King’s birthday)
  • January 20: Introduction and basic concepts
    • First coding assignment available; due February 2
• Week 2
  • January 25: Corpora, words, and lemmatization
    • Optional: Jurafsky and Martin, Speech and Language Processing (3rd edition draft), Chapter 2: Regular Expressions, Text Normalization, and Edit Distance.
    • Optional: Porter (1980): An algorithm for suffix stripping
    • Second coding assignment available; due February 9
  • Reading Quiz due January 26:
    • Manning, Raghavan and Schutze, Introduction to information retrieval, Chapter 13: Text classification and Naive Bayes.
    • Jurafsky and Martin, Speech and Language Processing (3rd edition draft), Chapter 5: Logistic Regression.
  • January 27: Text classification and naïve Bayes
    • Optional: Manning, Raghavan and Schutze, Introduction to information retrieval, Chapter 8: Evaluation in information retrieval.
    • Optional: Jurafsky and Martin, Speech and Language Processing (3rd edition draft), Chapter 4: Naive Bayes Classification and Sentiment.
    • Written homework 1 out
• Week 3
  • February 1: Generative and discriminative models; logistic regression; how to read a research article
  • Reading Quiz due February 2:
    • Hal Daumé III, A Course in Machine Learning (v. 0.99 draft), Chapter 4: The Perceptron.
    • Manning, Raghavan and Schutze, Introduction to Information Retrieval, Chapter 14: Vector space classification, sections 14.4–14.6 (other sections optional).
  • February 3: The perceptron; linear classifiers
    • Written homework 2 out
• Week 4
  • February 8: Part-of-speech tagging, hidden Markov models
    • Optional: Jurafsky and Martin, Speech and Language Processing (3rd edition draft), Chapter 8: Part-of-Speech Tagging.
    • Optional: Jurafsky and Martin, Speech and Language Processing (3rd edition draft), Appendix A: Hidden Markov Models.
    • Written homework 3 out
  • Reading Quiz due February 9:
    • Brill (1992): A Simple Rule-Based Part of Speech Tagger
    • Ratnaparkhi (1996): A Maximum Entropy Model for Part-Of-Speech Tagging
  • February 10: Brill and maximum entropy taggers
    • Optional: Brill (1995): Transformation-Based Error-Driven Learning and Natural Language Processing: A Case Study in Part-of-Speech Tagging.
    • Optional: Roche and Schabes (1995): Deterministic Part-Of-Speech Tagging With Finite State Transducers.
    • Third coding assignment available; due February 23
    • Fourth coding assignment available; due March 2
• Week 5
  • February 15: No class (President’s Day)
  • Reading Quiz due February 16:
    • Young (1996): A review of large-vocabulary continuous-speech recognition. IEEE Signal Processing Magazine 13(5): 45–57. This link should allow free download when accessed from the USC network. From outside the network, it can be accessed by signing into the USC libraries website with your USC NetID credentials.
  • February 17: Language modeling, speech recognition
    • Optional: Jurafsky and Martin, Speech and Language Processing (3rd edition draft), Chapter 3: N-gram Language Models.
    • Optional: Jurafsky and Martin, Speech and Language Processing (3rd edition draft), Chapter 25: Phonetics.
    • Optional: Jurafsky and Martin, Speech and Language Processing (3rd edition draft), Chapter 26: Automatic Speech Recognition and Text-to-Speech.
    • Written homework 4 out
• Week 6
  • February 22: Statistical machine translation
  • Reading Quiz due February 23:
    • Lafferty, McCallum and Pereira (2001): Conditional Random Fields: Probabilistic Models for Segmenting and Labeling Sequence Data
  • February 24: Discriminative sequence labeling, Unsupervised sequence labeling
    • Optional: Grenager, Klein, and Manning (2005): Unsupervised Learning of Field Segmentation Models for Information Extraction..
    • Optional: Hearst (1997): TextTiling: Segmenting Text into Multi-paragraph Subtopic Passages.
• Week 7
  • Reading Quiz due February 28:
    • Mikolov, Yih and Zweig (2013): Linguistic Regularities in Continuous Space Word Representations
    • Sutskever, Vinyals and Le (2014): Sequence to Sequence Learning with Neural Networks
  • March 1: Neural language models, word embeddings
    • Optional: Jurafsky and Martin, Speech and Language Processing (3rd edition draft), Chapter 7: Neural Networks and Neural Language Models and Chapter 9: Deep Learning Architectures for Sequence Processing.
    • Optional: Goldberg, Neural Network Methods for Natural Language Processing (2017), Chapter 9: Language Modeling and Chapter 14: Recurrent Neural Networks: Modeling Sequences and Stacks. (This link provides free access to the PDF of the full book when accessed from the USC network)
  • March 3: Sequence-to-sequence models, neural machine translation
    • Optional: Jurafsky and Martin, Speech and Language Processing (3rd edition draft), Chapter 11: Machine Translation and Encoder-Decoder Models.
    • Fifth coding assignment available; due March 23
• Week 8
  • Reading Quiz due March 7:
    • Levy, Goldberg and Dagan (2015): Improving Distributional Similarity with Lessons Learned from Word Embeddings.
  • March 8: Word and sub-word embeddings; transformers; BERT; guest presentation by Akshat Choube: multimodal fusion
    • Optional: Jurafsky and Martin, Speech and Language Processing (3rd edition draft), Chapter 6: Vector Semantics and Embeddings.
    • Optional: Levy and Goldberg (2014): Neural Word Embedding as Implicit Matrix Factorization.
    • Optional: Vaswani et al. (2017): Attention Is All You Need.
    • Optional: Devlin, Chang, Lee, and Toutanova (2019): BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding.
    • Optional: Choube and Soleymani (2020): Punchline Detection using Context-Aware Hierarchical Multimodal Fusion.
  • March 10: First midterm exam due by 11:59 PM (Pacific time)
    • Class does not meet live on exam days; instructional staff will check the class message boards on Piazza every 1–2 hours between 8 AM and 8 PM (Pacific Time) to answer questions.
• Week 9
  • Reading Quiz due March 14:
    • Charniak (1997): Statistical Techniques for Natural Language Parsing
    • Jurafsky and Martin, Speech and Language Processing (3rd edition draft), Chapter 14: Dependency Parsing.
  • March 15: Natural language syntax, probabilistic parsing
    • Optional: Jurafsky and Martin, Speech and Language Processing (3rd edition draft), Chapter 12: Constituency Grammars.
    • Optional: Jurafsky and Martin, Speech and Language Processing (3rd edition draft), Chapter 13: Constituency Parsing.
    • Optional: Jurafsky and Martin, Speech and Language Processing (3rd edition draft), Appendix C: Statistical Constituency Parsing.
    • Written homework 5 out
  • March 17: Top-down chart parsing, dependency parsing
• Week 10
  • Reading Quiz due March 21:
    • Levin, Pieraccini and Eckert (2000): A Stochastic Model of Human-Machine Interaction for Learning Dialog Strategies. This link should allow free download when accessed from the USC network. From outside the network, it can be accessed by signing into the USC libraries website with your USC NetID credentials. Alternative link, may or may not work
  • March 22: Dialogue and conversational agents
    • Optional: Jurafsky and Martin, Speech and Language Processing (3rd edition draft), Chapter 24: Chatbots and Dialog Systems.
    • Optional: Lee and Stent (2016): Task Lineages: Dialog State Tracking for Flexible Interaction (Sigdial 2016 best paper; a video of the talk is also available)
    • Written homework 6 out
  • March 24: More on dialogue
• Week 11
  • Reading Quiz due March 28:
    • Gildea and Palmer (2002): The Necessity of Parsing for Predicate Argument Recognition
  • March 29 Semantics: role labeling, word semantics
    • Optional: Jurafsky and Martin, Speech and Language Processing (3rd edition draft), Chapter 19: Semantic Role Labeling.
  • March 31 Experiment design; annotation and evaluation; reference
    • Optional: Passonneau and Carpenter (2014): The Benefits of a Model of Annotation.
• Week 12
  • April 5: Second midterm exam due by 11:59 PM (Pacific time)
    • Class does not meet live on exam days; instructional staff will check the class message boards on Piazza every 1–2 hours between 8 AM and 8 PM (Pacific Time) to answer questions.
  • April 7: No class (Wellness Day)
• Weeks 13–16
  • April 11: Written paper due
  • April 12, 14, 19, 21, 26: Student article presentations (see detailed schedule)
  • April 28: Reference, generation, summarization, and final thoughts.
• May 10: Final Exam due by 11:59 PM (Pacific time)

Statement on Academic Conduct and Support Systems

Academic Conduct:

Plagiarism – presenting someone else’s ideas as your own, either verbatim or recast in your own words – is a serious academic offense with serious consequences. Please familiarize yourself with the discussion of plagiarism in SCampus in Part B, Section 11, “Behavior Violating University Standards” https://policy.usc.edu/scampus-part-b/. Other forms of academic dishonesty are equally unacceptable. See additional information in SCampus and university policies on scientific misconduct, http://policy.usc.edu/scientific-misconduct.

Support Systems:

Student Health Counseling Services – (213) 740-7711 – 24/7 on call
engemannshc.usc.edu/counseling
Free and confidential mental health treatment for students, including short-term psychotherapy, group counseling, stress fitness workshops, and crisis intervention.

National Suicide Prevention Lifeline – 1 (800) 273-8255 – 24/7 on call
http://www.suicidepreventionlifeline.org
Free and confidential emotional support to people in suicidal crisis or emotional distress 24 hours a day, 7 days a week.

Relationship and Sexual Violence Prevention Services (RSVP) – (213) 740-4900 – 24/7 on call
https://engemannshc.usc.edu/rsvp/
Free and confidential therapy services, workshops, and training for situations related to gender-based harm.

Office of Equity and Diversity (OED)/Title IX – (213) 740-5086
https://equity.usc.edu/,  http://titleix.usc.edu/
Information about how to get help or help a survivor of harassment or discrimination, rights of protected classes, reporting options, and additional resources for students, faculty, staff, visitors, and applicants. The university prohibits discrimination or harassment based on the following protected characteristics: race, color, national origin, ancestry, religion, sex, gender, gender identity, gender expression, sexual orientation, age, physical disability, medical condition, mental disability, marital status, pregnancy, veteran status, genetic information, and any other characteristic which may be specified in applicable laws and governmental regulations.

Bias Assessment Response and Support – (213) 740-2421
https://studentaffairs.usc.edu/bias-assessment-response-support/
Avenue to report incidents of bias, hate crimes, and microaggressions for appropriate investigation and response.

The Office of Disability Services and Programs – (213) 740-0776
http://dsp.usc.edu
Support and accommodations for students with disabilities. Services include assistance in providing readers/notetakers/interpreters, special accommodations for test taking needs, assistance with architectural barriers, assistive technology, and support for individual needs.

USC Support and Advocacy – (213) 821-4710
https://studentaffairs.usc.edu/ssa/
Assists students and families in resolving complex personal, financial, and academic issues adversely affecting their success as a student.

Diversity at USC – (213) 740-2101
https://diversity.usc.edu/
Information on events, programs and training, the Provost’s Diversity and Inclusion Council, Diversity Liaisons for each academic school, chronology, participation, and various resources for students.

USC Emergency – UPC: (213) 740-4321, HSC: (323) 442-1000 – 24/7 on call
http://dps.usc.edu/,  http://emergency.usc.edu
Emergency assistance and avenue to report a crime. Latest updates regarding safety, including ways in which instruction will be continued if an officially declared emergency makes travel to campus infeasible.

USC Department of Public Safety – UPC: (213) 740-6000, HSC: (323) 442-1200 – 24/7 on call
http://dps.usc.edu
Non-emergency assistance or information.

USC Viterbi Honor Code

The Code was developed by Viterbi students, and its text is as follows:

Engineering enables and empowers our ambitions and is integral to our identities. In the Viterbi community, accountability is reflected in all our endeavors.

• Engineering+ Integrity.
• Engineering+ Responsibility.
• Engineering+ Community.
• Think good. Do better. Be great.

These are the pillars we stand upon as we address the challenges of society and enrich lives.

Academic Integrity Violations

All coding and writing must be done individually (unless instructed otherwise), and not copied from other students. Copying or plagiarism is grounds for failure of an assignment, or in serious cases failure of the course.

Use of the internet or other outside resources to find solutions to homework problems is considered cheating.

Please read my note on academic integrity.