CSCI 544 — Applied Natural Language Processing

Fall 2020

My intention is to teach this course in a partly asynchronous mode. This means:

• Many of the lectures will be pre-recorded; when lectures are pre-recorded, class meeting times will be for discussion and collaborative problem solving.
• Some activities (for example, student presentations) will be conducted in a synchronous fashion at class meeting times (details in the schedule).
• Most of the interaction will be over message boards.
• Office hours will be by appointment only, in order to enable the flexibility required by students being in different time zones.

All of the above is subject to change as the semester progresses and we see what works and what doesn’t. My main reason for trying asynchronous teaching with pre-recorded lectures is that last semester, once we moved online, student participation in the lectures dropped considerably. However, if this doesn’t work out, we may switch to synchronous lectures.

Synopsis

This course covers both fundamental and cutting-edge topics in Natural Language Processing (NLP) and provides students with hands-on experience in NLP applications.

This graduate course is intended for:

• students who want to understand current NLP technologies
• students interested in building NLP applications
• students interested in applications of NLP like sentiment analysis, dialogue systems, question answering systems, among others

Recommended preparation: Proficiency in programming, algorithms and data structures, basic knowledge of linear algebra and machine learning.

Related Courses

This course is part of USC’s curriculum in natural language processing. There is a sister course, CSCI 662 Advanced Natural Language Processing, which covers complementary (and advanced) material and is intended for PhD students (or students who want to continue to a PhD program).

Course Objectives

Students in the course will learn to perform the following:

1. Read technical literature in Natural Language Processing (including original research articles) and answer questions about such readings.
2. Implement language processing algorithms and test them on natural language data.
3. Solve language processing problems and explain the reasoning behind their solutions.

Coursework and Grading

The following are the types of assignments anticipated for the course; since the course is still being worked out, these are subject to change. An exact breakdown of assignments, including grading percentages, will be announced together with the course schedule.

• (10%) Reading quizzes: Weekly web-based quizzes relating to reading assignments.
• (40%) Coding assignments: A series of programming exercises spread throughout the semester.
• (ungraded) Written assignments: A set of written problem sets, designed as preparation for the midterm and final exams.
• (20%) Midterm exam: One or more exams taken during the semester, with questions similar to those from the homework assignments and quizzes.
• (5%) Research article presentation: A presentation of an original research article to the entire class (in pairs).
• (20%) Final paper: NLP prototype design proposal, presenting the design of a novel, original NLP application.
• (5%) Final exam: An on-line, take home comprehensive exam taken during the final exam period.

Grading scale

The following scale is used for determining final grades (note that A is the highest grade given by USC).

• A: 92%, A–: 90%, B+: 87%, B: 82%, B–: 80%, C+: 77%, C: 72%, C–: 70%, D+: 67%, D: 62%, D–: 60%

Grade challenges

• Grades on an assignment or exam will only be changed if there is an error in grading. Grading errors are simple mistakes made on the part of the graders, and not differences in interpretation of a question or answer. A student who wishes to challenge a grade must identify the grading error before asking for a grade change.
• Students are welcome to discuss any aspect of the assignments and exams, but there will be no negotiation on grades, and no changes other than the correction of grading errors.

Late Policy

• There will be a penalty for turning in homework late; the penalty will vary by assignment. In no instance will credit be given for a homework assignment submitted after the solution has been discussed in class.
• Reading quizzes are intended to make sure that students are prepared for class, so no late quizzes will be accepted. (One exception: students who registered late will be allowed to make up the quizzes they missed due to late registration.)
• Homework assignments will not be accepted by email. If there are technical or other issues with the submission system you should write to us and we will work to fix these issues, but do not send homework by email just because you weren’t able to submit it through the system.
• Assignments are usually due at the end of the day. We may not be available to solve issues close to the deadline, so you should plan on submitting your homework early, even if only as a draft. Multiple submissions are generally allowed, and the last submission will be graded. Quizzes may only be submitted once.
• Students are responsible for on-time submission. When submitting your assignments, please allow time to fix issues such as your computer freezing or crashing, network latencies or outages, and so on. If your assignment is submitted late because of a technical issue, it will be penalized as late.
• Students who are not able to make an assignment or exam deadline due to an emergency (for example, a medical emergency) or a conflict (for example, a job interview) must inform me as soon as the issue arises, in order to make alternate arrangements. If you wait until after the deadline, or the grades are returned, or the end of the course, you will not be allowed to make up the work.

  Note that in general, conflicts with exams or deadlines for another class do not qualify for making alternate arrangements. Working and studying for multiple classes is an expectation from university students.

Communication

Please use the class discussion boards on Piazza for questions and issues regarding homework assignments and the course in general. This way, the entire class can participate and see the questions and answers. Email should be reserved for communication of a personal nature. If we receive questions by email where the response could be helpful for the class, we may ask you to repost the question on the discussion boards.

Any special requests must be submitted in writing.

Resources

• Blackboard (reading quizzes, class announcements, distribution of written assignments, final exam)
• Piazza (class discussion boards)
• Vocareum (coding assignments)
• Crowdmark (midterm exam, final paper)

Recommended textbooks

The course does not have a textbook. Required readings will be specified in the schedule below as the course progresses, and will include a combination of select textbook chapters as well as original research articles. The links below give access to the full text of several textbooks; these are useful for general background on Natural Language Processing, and to supplement some of the materials taught in class. Any chapters that are required will be detailed in the schedule; otherwise, these texts are not required.

• Dan Jurafsky and James H. Martin, Speech and Language Processing (3rd ed. draft, 2019)
• Christopher D. Manning, Prabhakar Raghavan and Hinrich Schütze, Introduction to Information Retrieval (2008) (select chapters)
• Yoav Goldberg, Neural Network Methods for Natural Language Processing (2017) (This link provides free access to the PDF of the full book when accessed from the USC network)
• Jacob Eisenstein, Natural Language Processing (free version, 2018) or the published version (2019)
• Emily M. Bender, Linguistic Fundamentals for Natural Language Processing: 100 Essentials from Morphology and Syntax (2013) (This link provides free access to the PDF of the full book when accessed from the USC network)

Schedule

Topics listed in the schedule are tentative and subject to change.

Details (readings, assignments) will be added as the term progresses.

Optional readings are useful for the class, but will not appear on the quizzes.

• August 25: Introduction and basic concepts
  • First coding assignment available; due September 7.
• August 27: Corpora, words, and lemmatization
  • Second coding assignment available; due September 14.
  • Optional: Jurafsky and Martin, Speech and Language Processing (3rd edition draft), Chapter 2: Regular Expressions, Text Normalization, and Edit Distance.
• Reading Quiz due August 31:
  • Manning, Raghavan and Schutze, Introduction to information retrieval, Chapter 13: Text classification and Naive Bayes.
  • Jurafsky and Martin, Speech and Language Processing (3rd edition draft), Chapter 5: Logistic Regression.
• September 1: Text classification and naïve Bayes
  • Optional: Manning, Raghavan and Schutze, Introduction to information retrieval, Chapter 8: Evaluation in information retrieval.
  • Optional: Jurafsky and Martin, Speech and Language Processing (3rd edition draft), Chapter 4: Naive Bayes Classification and Sentiment.
• September 3: Generative and discriminative models; logistic regression
• Reading Quiz due September 7:
  • Hal Daumé III, A Course in Machine Learning (v. 0.99 draft), Chapter 4: The Perceptron.
  • Manning, Raghavan and Schutze, Introduction to Information Retrieval, Chapter 14: Vector space classification, sections 14.4–14.6 (other sections optional).
• September 8: How to read a research article; the perceptron
• September 10: Linear classifiers
• Reading Quiz due September 14:
  • Brill (1992): A Simple Rule-Based Part of Speech Tagger
  • Ratnaparkhi (1996): A Maximum Entropy Model for Part-Of-Speech Tagging
• September 15: Part-of-speech tagging, hidden Markov models
  • Optional: Jurafsky and Martin, Speech and Language Processing (3rd edition draft), Chapter 8: Part-of-Speech Tagging.
  • Optional: Jurafsky and Martin, Speech and Language Processing (3rd edition draft), Chapter A: Hidden Markov Models.
  • Third coding assignment available; due September 28.
  • Fourth coding assignment available; due October 5.
• September 17: Coding assignment 3-4 roundup; Brill and maximum entropy taggers
• Reading Quiz due September 21:
  • Young (1996): A review of large-vocabulary continuous-speech recognition. IEEE Signal Processing Magazine 13(5): 45–57. This link should allow free download when accessed from the USC network. From outside the network, it can be accessed by signing into the USC libraries website with your USC NetID credentials.
• September 22: Language modeling, speech recognition
  • Optional: Jurafsky and Martin, Speech and Language Processing (3rd edition draft), Chapter 3: N-gram Language Models.
• September 24: Speech recognition (cont.), statistical machine translation
• Reading Quiz due September 28:
  • Charniak (1997): Statistical Techniques for Natural Language Parsing
  • Jurafsky and Martin, Speech and Language Processing (3rd edition draft), Chapter 15: Dependency Parsing.
• September 29: Natural language syntax, probabilistic parsing
  • Optional: Jurafsky and Martin, Speech and Language Processing (3rd edition draft), Chapter 12: Constituency Grammars.
  • Optional: Jurafsky and Martin, Speech and Language Processing (3rd edition draft), Chapter 13: Constituency Parsing.
  • Optional: Jurafsky and Martin, Speech and Language Processing (3rd edition draft), Chapter 14: Statistical Constituency Parsing.
• October 1: Top-down chart parsing, dependency parsing
• Reading Quiz due October 5:
  • Levin, Pieraccini and Eckert (2000): A Stochastic Model of Human-Machine Interaction for Learning Dialog Strategies.
• October 6: Dialogue and conversational agents
  • Optional: Jurafsky and Martin, Speech and Language Processing (3rd edition draft), Chapter 26: Dialog Systems and Chatbots.
• October 8: Statistical machine translation
  • Fifth coding assignment available, due October 26.
• Reading Quiz due October 12:
  • Gildea and Palmer (2002): The Necessity of Parsing for Predicate Argument Recognition
• October 13 Semantics: role labeling, word semantics
• October 15: Midterm exam due by 11:59 PM (Pacific time)
• Reading Quiz due October 19:
  • Mikolov, Yih and Zweig (2013): Linguistic Regularities in Continuous Space Word Representations
  • Sutskever, Vinyals and Le (2014): Sequence to Sequence Learning with Neural Networks
• October 20: Neural language models, word embeddings
  • Optional: Jurafsky and Martin, Speech and Language Processing (3rd edition draft), Chapter 7: Neural Networks and Neural Language Models and Chapter 9: Sequence Processing with Recurrent Networks.
  • Optional: Goldberg, Neural Network Methods for Natural Language Processing (2017), Chapter 9: Language Modeling and Chapter 14: Recurrent Neural Networks: Modeling Sequences and Stacks. (This link provides free access to the PDF of the full book when accessed from the USC network)
• October 22: Discriminative sequence labeling, sequence-to-sequence models, neural machine translation
  • Optional: Lafferty, McCallum and Pereira (2001): Conditional Random Fields: Probabilistic Models for Segmenting and Labeling Sequence Data
  • Optional: Jurafsky and Martin, Speech and Language Processing (3rd edition draft), Chapter 10: Encoder-Decoder Models, Attention, and Contextual Embeddings.
• Reading Quiz due October 26:
  • Levy, Goldberg and Dagan (2015): Improving Distributional Similarity with Lessons Learned from Word Embeddings.
• October 27: Word embeddings; transformers
  • Optional: Levy and Goldberg (2014): Neural Word Embedding as Implicit Matrix Factorization.
  • Optional: Vaswani et al. (2017): Attention Is All You Need.
• October 29: BERT; sub-word embeddings; unsupervised models
  • Optional: Devlin, Chang, Lee, and Toutanova (2019): BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding.
  • Optional: Grenager, Klein, and Manning (2005): Unsupervised Learning of Field Segmentation Models for Information Extraction..
  • Optional: Hearst (1997): TextTiling: Segmenting Text into Multi-paragraph Subtopic Passages.
• Reading Quiz due November 2:
  • Passonneau and Carpenter (2014): The Benefits of a Model of Annotation.
• November 3: Experiment design; annotation and evaluation
• November 5: Annotation; reference
• November 10, 12, 16, 17, 19: Student article presentations (see detailed schedule)
• November 24: No class meeting (moved to November 16 in the evening, to accommodate students in China and India)
• November 26: Final paper due (extended from November 24)
• December 8: Final Exam due by 6:30 PM (Pacific time)

Statement on Academic Conduct and Support Systems

Academic Conduct:

Plagiarism – presenting someone else’s ideas as your own, either verbatim or recast in your own words – is a serious academic offense with serious consequences. Please familiarize yourself with the discussion of plagiarism in SCampus in Part B, Section 11, “Behavior Violating University Standards” https://policy.usc.edu/scampus-part-b/. Other forms of academic dishonesty are equally unacceptable. See additional information in SCampus and university policies on scientific misconduct, http://policy.usc.edu/scientific-misconduct.

Support Systems:

Student Health Counseling Services – (213) 740-7711 – 24/7 on call
engemannshc.usc.edu/counseling
Free and confidential mental health treatment for students, including short-term psychotherapy, group counseling, stress fitness workshops, and crisis intervention.

National Suicide Prevention Lifeline – 1 (800) 273-8255 – 24/7 on call
http://www.suicidepreventionlifeline.org
Free and confidential emotional support to people in suicidal crisis or emotional distress 24 hours a day, 7 days a week.

Relationship and Sexual Violence Prevention Services (RSVP) – (213) 740-4900 – 24/7 on call
https://engemannshc.usc.edu/rsvp/
Free and confidential therapy services, workshops, and training for situations related to gender-based harm.

Office of Equity and Diversity (OED)/Title IX – (213) 740-5086
https://equity.usc.edu/,  http://titleix.usc.edu/
Information about how to get help or help a survivor of harassment or discrimination, rights of protected classes, reporting options, and additional resources for students, faculty, staff, visitors, and applicants. The university prohibits discrimination or harassment based on the following protected characteristics: race, color, national origin, ancestry, religion, sex, gender, gender identity, gender expression, sexual orientation, age, physical disability, medical condition, mental disability, marital status, pregnancy, veteran status, genetic information, and any other characteristic which may be specified in applicable laws and governmental regulations.

Bias Assessment Response and Support – (213) 740-2421
https://studentaffairs.usc.edu/bias-assessment-response-support/
Avenue to report incidents of bias, hate crimes, and microaggressions for appropriate investigation and response.

The Office of Disability Services and Programs – (213) 740-0776
http://dsp.usc.edu
Support and accommodations for students with disabilities. Services include assistance in providing readers/notetakers/interpreters, special accommodations for test taking needs, assistance with architectural barriers, assistive technology, and support for individual needs.

USC Support and Advocacy – (213) 821-4710
https://studentaffairs.usc.edu/ssa/
Assists students and families in resolving complex personal, financial, and academic issues adversely affecting their success as a student.

Diversity at USC – (213) 740-2101
https://diversity.usc.edu/
Information on events, programs and training, the Provost’s Diversity and Inclusion Council, Diversity Liaisons for each academic school, chronology, participation, and various resources for students.

USC Emergency – UPC: (213) 740-4321, HSC: (323) 442-1000 – 24/7 on call
http://dps.usc.edu/,  http://emergency.usc.edu
Emergency assistance and avenue to report a crime. Latest updates regarding safety, including ways in which instruction will be continued if an officially declared emergency makes travel to campus infeasible.

USC Department of Public Safety – UPC: (213) 740-6000, HSC: (323) 442-1200 – 24/7 on call
http://dps.usc.edu
Non-emergency assistance or information.

USC Viterbi Honor Code

The Code was developed by Viterbi students, and its text is as follows:

Engineering enables and empowers our ambitions and is integral to our identities. In the Viterbi community, accountability is reflected in all our endeavors.

• Engineering+ Integrity.
• Engineering+ Responsibility.
• Engineering+ Community.
• Think good. Do better. Be great.

These are the pillars we stand upon as we address the challenges of society and enrich lives.

Academic Integrity Violations

All coding and writing must be done individually (unless instructed otherwise), and not copied from other students. Copying or plagiarism is grounds for failure of an assignment, or in serious cases failure of the course.

Use of the internet or other outside resources to find solutions to homework problems is considered cheating.

Please read my note on academic integrity.