In this assignment, you'll implement a dialogue act classifier, combining it with the labeling model from HW4 to perform slot-filling and generation to make a complete dialogue system.
Whenever the instructions below say to "report" something, it should be reported in the README.md file that you submit.
Visit this GitHub Classroom link to create a Git repository for you, and clone it to your computer. Initially, it contains the following files:
file | description |
data.part{1,2,3}/train | training data |
data.part{1,2,3}/dev | development/validation data |
data.part{1,2,3}/test | test data |
layers.py | possibly useful neural network layers |
frames.py | possibly useful code for frames |
score_frames.py | compute various accuracies for frames |
score_types.py | compute various accuracies for frame types |
bleu.py | compute BLEU scores |
backend.py | database backend |
tokenizer.py | very simple tokenizer |
.words
: just words.bio
: BIO tagged (same format as HW4).types
: just frame types.frames
: just frames.delex
: delexicalizedtrain
, dev
, and test
in data.part1
is:
I'm going from Cambridge to the Stansted Airport . \t find_train
where \t
is a tab. In this example, I'm going from Cambridge to the Stansted Airport .
are the input words and find_train
is the correct output dialogue act.
Train the classifier on the training data. Report your accuracy (at each epoch, if there are epochs) on the dev set, which should reach at least 80%.1 You can use score_types.py
to compute accuracy, or you can just do it yourself.
I'm going from Cambridge to the Stansted Airport .
then the correct output would be
find_train ( train-departure = Cambridge ; train-destination = Stansted Airport )
The spaces are required (they'll make your life easier in Part 3). You can use the frames.Frame
class, which knows how to print itself in the right format.train
, dev
, and test
in data.part2
is
I'm going from Cambridge to the Stansted Airport . \t find_train ( train-departure = Cambridge ; train-destination = Stansted Airport )
Run your combined natural language "understanding" system on the dev and test data and report your score on both. To compute an accuracy score, run
python3 score_frames.py <your-dev-output> part2/dev.frames
python3 score_frames.py <your-test-output> part2/test.frames
The frame type accuracy score corresponds to the classification accuracy from Part 1. The argument F1 is analogous to the F1 score from HW4. The exact match score is simply the percentage of frames that are exactly correct. This score should be at least 68%1 on dev and 70% on test.3backend.py
exposes a function backend.backend(q)
. The argument q
is a Frame object representing a user query, and the return value is a list of Frame objects representing the computer's response.
You can run backend.py
from the command line to understand how it works. Here are some queries you can try:
python backend.py "find_restaurant ( restaurant-food = italian ; restaurant-pricerange = cheap )"
python backend.py "find_attraction ( attraction-pricerange = free ; attraction-type = museum ; attraction-area = east )"
python backend.py "find_hotel ( hotel-area = centre ; hotel-pricerange = expensive )"
python backend.py "find_train ( train-departure = Cambridge ; train-destination = Stansted Airport ; train-day = sunday ; train-arriveby = 11:00)"
Integrate your system with this backend to make an interactive system.3 Use input()
to read a line of text from the user, tokenizer.tokenize
to tokenize it, your system to convert it to a frame, and backend.backend
to generate results.part3/train.delex
contains delexicalized frames and delexicalized strings for the kinds of responses we want the computer to be able to generate. Train your machine translation system from HW2 (or the official HW2 solution) on this data to translate delexicalized frames to delexicalized strings.1part3/dev.frames
and part3/test.frames
. Compute the BLEU score against part3/dev.words
and part3/test.words
, respectively. The BLEU score should be at least 17% on both dev1 and test.3Could you recommend me an expensive hotel in the centre?
I would like to know of a museum that is free to attend.
Do you know of a cheap italian food restaurant?
I need a train from Cambridge to Stansted Airport arriving by 11:00 on Sunday.
Please read these submission instructions carefully.