# CSE 40657/60657 Homework 2

Due
2021/10/01 5pm
Points
30

In 2005, a blog post went viral that showed a bootleg copy of Star Wars: Episode III – Revenge of the Sith with its Chinese version translated (apparently by low-quality machine translation) back into an English movie called Star War: The Third Gathers – The Backstroke of the West. Can you do better?

Visit this GitHub Classroom link to create a Git repository for you, and clone it to your computer. Initially, it contains the following files:

 data/train.* training data data/dev.* development data data/test.* test data bleu.py evaluation script for translation layers.py some useful neural network layers model2.py direct IBM Model 2

The training data consists of a number of Star Wars scripts (Episodes 1–2, 4–7, Rogue One, Solo, Rebels S1E1–3). The dev and test data are Episode 3. For each dataset, there are three or four files: suffix zh-en means Chinese-English parallel text (tab-separated), zh means Chinese, en means (correct) English, and backstroke.en means the English of Backstroke of the West.

As distributed, model2.py implements the following model, which is a variant of direct IBM Model 2 with the $t$ and $a$ tables factored into smaller matrices: \begin{align} P(\mathbf{e} \mid \mathbf{f}) &= \prod_{i=1}^m \sum_{j=1}^n a(j \mid i) \, t(e_i \mid f_j) \\ t(e_i \mid f_j) &= \left[\operatorname{softmax} \mathbf{U} \mathbf{V}_{f_j} \right]_{e_i}. \\ a(j \mid i) &= \left[\operatorname{softmax} \mathbf{K} \mathbf{Q}_{i} \right]_j \\ \mathbf{Q} &= \begin{bmatrix}\mathbf{\bar Q}_{1,*} \\ \vdots \\ \mathbf{\bar Q}_{m,*}\end{bmatrix} \\ \mathbf{K} &= \begin{bmatrix}\mathbf{\bar K}_{1,*} \\ \vdots \\ \mathbf{\bar K}_{n,*}\end{bmatrix}. \end{align} where $\mathbf{\bar{Q}}_{i,*}$ and $\mathbf{\bar{K}}_{j,*}$ (for $i,j = 1, \ldots, 100$) are learnable parameter vectors that can be thought of as embeddings of the numbers 1 to 100.

In this assignment, you'll improve this model so that it translates better than Backstroke of the West. (Although, because the training data is so small, this is a challenging task, and it probably won't be that much better.)

You may reuse any code you've used in previous homework assignments, or even the solution or another student's code as long as you cite properly.

### 1. These are your first steps

1. Run python model2.py --train data/train.zh-en --dev data/dev.zh-en data/test.zh -o test.model2.en, which trains direct IBM Model 2 and then translates the test set into the file test.model2.en.1
2. Run python bleu.py data/test.backstroke.en data/test.en to compute a BLEU score for Backstroke of the West (higher is better, 1.0 is the maximum). Run python bleu.py test.model2.en data/test.en to compute a BLEU score for Model 2. Report both scores.1
3. After each epoch, the training prints out the translation of the first ten sentences of the dev set. Write down some observations about what's wrong with the translations.3 (Note: The badness of these translations isn't the fault of Brown et al., who never intended Model 2 to be used directly for translation, as we are doing here.)

### 2. I've made a lot of special modifications myself

1. Modify model2.Decoder.output() so that the weighted average is inside the softmax, as in equation (3.30).5
2. The train perplexity should reach about 60 and the dev perplexity about 120.1
3. Run the model and recompute the BLEU score.1 The BLEU score probably didn't get much better, but write down any qualitative differences you see in the dev translations.3

### 3. Now witness the power of this fully operational translation system

1. Modify the model further to improve its translations.7 See Section 3.5 for some recipes for how to do this. Most of your modifications will be in the Encoder and Decoder classes. You may use any of the classes/functions provided in layers.py, many of which have tricks tailored for small training data.
2. Train the model. Record the trainer's output1 and save the model using the --save option.1
3. Run the model and report the BLEU score. To get full credit, your score must be better than Backstroke of the West's.3 Write down some observations about what did or didn't improve.3