CSE 40657/60657
Homework 2

Visit this GitHub Classroom link to create a Git repository for you, and clone it to your computer. Initially, it contains the following files:

The training data consists of a number of Star Wars scripts (Episodes 1–2, 4–7, Rogue One, Solo, Rebels S1E1–3). The dev and test data are Episode 3. For each dataset, there are three or four files: suffix zh-en means Chinese-English parallel text (tab-separated), zh means Chinese, en means (correct) English, and backstroke.en means the English of Backstroke of the West.

As distributed, model2.py implements the following model, which is a variant of direct IBM Model 2 with the $t$ and $a$ tables factored into smaller matrices: \begin{align} P(\mathbf{e} \mid \mathbf{f}) &= \prod_{i=1}^m \sum_{j=1}^n a(j \mid i) \, t(e_i \mid f_j) \\ t(e_i \mid f_j) &= \left[\operatorname{softmax} \mathbf{U} \mathbf{V}_{f_j} \right]_{e_i}. \\ a(j \mid i) &= \left[\operatorname{softmax} \mathbf{K} \mathbf{Q}_{i} \right]_j \\ \mathbf{Q} &= \begin{bmatrix}\mathbf{\bar Q}_{1,*} \\ \vdots \\ \mathbf{\bar Q}_{m,*}\end{bmatrix} \\ \mathbf{K} &= \begin{bmatrix}\mathbf{\bar K}_{1,*} \\ \vdots \\ \mathbf{\bar K}_{n,*}\end{bmatrix}. \end{align} where $\mathbf{\bar{Q}}_{i,*}$ and $\mathbf{\bar{K}}_{j,*}$ (for $i,j = 1, \ldots, 100$) are learnable parameter vectors that can be thought of as embeddings of the numbers 1 to 100.

In this assignment, you'll improve this model so that it translates better than Backstroke of the West. (Although, because the training data is so small, this is a challenging task, and it probably won't be that much better.)

You may reuse any code you've used in previous homework assignments, or even the solution or another student's code as long as you cite properly.

1. These are your first steps

1. Run python model2.py --train data/train.zh-en --dev data/dev.zh-en data/test.zh -o test.model2.en, which trains direct IBM Model 2 and then translates the test set into the file test.model2.en.1
2. Run python bleu.py data/test.backstroke.en data/test.en to compute a BLEU score for Backstroke of the West (higher is better, 1.0 is the maximum). Run python bleu.py test.model2.en data/test.en to compute a BLEU score for Model 2. Report both scores.1
3. After each epoch, the training prints out the translation of the first ten sentences of the dev set. Write down some observations about what's wrong with the translations.3 (Note: The badness of these translations isn't the fault of Brown et al., who never intended Model 2 to be used directly for translation, as we are doing here.)

2. I've made a lot of special modifications myself

1. Modify model2.Decoder.output() so that the weighted average is inside the softmax, as in equation (3.30).5
2. The train perplexity should reach about 60 and the dev perplexity about 120.1
3. Run the model and recompute the BLEU score.1 The BLEU score probably didn't get much better, but write down any qualitative differences you see in the dev translations.3

3. Now witness the power of this fully operational translation system

1. Modify the model further to improve its translations.7 See Section 3.5 for some recipes for how to do this. Most of your modifications will be in the Encoder and Decoder classes. You may use any of the classes/functions provided in layers.py, many of which have tricks tailored for small training data.
2. Train the model. Record the trainer's output1 and save the model using the --save option.1
3. Run the model and report the BLEU score. To get full credit, your score must be better than Backstroke of the West's.3 Write down some observations about what did or didn't improve.3

Submission

Please read these submission instructions carefully.

1. As often as you want, add and commit your submission files to the repository you created in the beginning.
2. Before submitting, the repository should contain:
   • All of the code that you wrote.
   • Your final saved model from Part 3.
   • A README.md file with
     • Instructions on how to build/run your code.
     • Your responses to all of the instructions/questions in the assignment.
3. To submit:
   • Push your work to GitHub and create a release in GitHub by clicking on "Releases" on the right-hand side, then "Create a new release" or "Draft a new release". Fill in "Tag version" and "Release title" with the part number(s) you're submitting and click "Publish Release".
   • If you submit the same part more than once, the grader will grade the latest release for that part.
   • For computing the late penalty, the submission time will be considered the commit time, not the release time.