序列到序列的语言翻译模型代码(tensorflow)解析

序列到序列的RNN语言机器翻译模型的tensorflow代码解析。

0x00.前言

（原文发表在博客，欢迎访问

这份代码最开始在基于RNN的语言模型与机器翻译NMT看到。本着溯本求源的心态，我搜了一下代码，找到了Brok-Bucholtz/P4-Beta/language_translation/dlnd_language_translation.ipynb
。

（更完整版在Language Translation

因为工作中需要，所以要对其代码及神经网络结构有所了解。但是其中涉及了很多seq2seq函数，我没有接触过。所以这里对其进行一次代码分析。

0x01.代码解析

1.数据预处理

定义了两个函数text_to_ids解析文本，sentence_to_ids将一句话中的词转换为对应的id。

2.输入

定义函数model_inputs来创建tf.placeholder。

3.编码

RNN的编码层：

def encoding_layer(rnn_inputs, rnn_size, num_layers, keep_prob):
"""
 Create encoding layer
 :param rnn_inputs: Inputs for the RNN
 :param rnn_size: RNN Size
 :param num_layers: Number of layers
 :param keep_prob: Dropout keep probability
 :return: RNN state
 """
# TODO: Implement Function
# Encoder
# 首先创建多层lstm
    enc_cell = tf.contrib.rnn.MultiRNNCell([tf.contrib.rnn.BasicLSTMCell(rnn_size)] * num_layers)
# 加dropout层
    enc_cell = tf.contrib.rnn.DropoutWrapper(enc_cell, output_keep_prob=keep_prob)
# 动态rnn
    rnn, state = tf.nn.dynamic_rnn(enc_cell, rnn_inputs, dtype=tf.float32)

return state

4.解码-训练层

下面介绍几个其中用到的函数：

tf.contrib.seq2seq.simple_decoder_fn_train(encoder_state, name=None)在dynamic_rnn_decoder中使用的序列到序列模型的简单解码器。simple_decoder_fn_train是序列到序列模型的简单训练函数。当dynamic_rnn_decoder处于训练模式时应该使用它。
tf.contrib.seq2seq.dynamic_rnn_decoder(cell, decoder_fn, inputs=None, sequence_length=None, parallel_iterations=None, swap_memory=False, time_major=False, scope=None, name=None)由RNNCell和解码器功能规定的序列到序列模型的动态RNN(drnn)解码器。有训练和推理两种模式。

def decoding_layer_train(encoder_state, dec_cell, dec_embed_input, sequence_length, decoding_scope,
 output_fn, keep_prob):
"""
 Create a decoding layer for training
 :param encoder_state: Encoder State
 :param dec_cell: Decoder RNN Cell
 :param dec_embed_input: Decoder embedded input
 :param sequence_length: Sequence Length
 :param decoding_scope: TenorFlow Variable Scope for decoding
 :param output_fn: Function to apply the output layer
 :param keep_prob: Dropout keep probability
 :return: Train Logits
 """
# TODO: Implement Function
    train_decoder_fn = tf.contrib.seq2seq.simple_decoder_fn_train(encoder_state)
    train_pred, _, _ = tf.contrib.seq2seq.dynamic_rnn_decoder(
        dec_cell, train_decoder_fn, dec_embed_input, sequence_length, scope=decoding_scope)

# Apply output function
    train_logits =  output_fn(train_pred)
# dropout层
    train_logits = tf.nn.dropout(train_logits, keep_prob)
return train_logits

5.解码-推理层

tf.contrib.seq2seq.simple_decoder_fn_inference(output_fn, encoder_state, embeddings, start_of_sequence_id, end_of_sequence_id, maximum_length, num_decoder_symbols, dtype=tf.int32, name=None)在dynamic_rnn_decoder中使用的序列到序列模型的简单解码器
simple_decoder_fn_inference是序列到序列模型的简单推理函数。当dynamic_rnn_decoder处于推理模式时应该使用。

def decoding_layer_infer(encoder_state, dec_cell, dec_embeddings, start_of_sequence_id, end_of_sequence_id,
 maximum_length, vocab_size, decoding_scope, output_fn, keep_prob):
"""
 Create a decoding layer for inference
 :param encoder_state: Encoder state
 :param dec_cell: Decoder RNN Cell
 :param dec_embeddings: Decoder embeddings
 :param start_of_sequence_id: GO ID
 :param end_of_sequence_id: EOS Id
 :param maximum_length: The maximum allowed time steps to decode
 :param vocab_size: Size of vocabulary
 :param decoding_scope: TensorFlow Variable Scope for decoding
 :param output_fn: Function to apply the output layer
 :param keep_prob: Dropout keep probability
 :return: Inference Logits
 """
# TODO: Implement Function
    infer_decoder_fn = tf.contrib.seq2seq.simple_decoder_fn_inference(
        output_fn, encoder_state, dec_embeddings, start_of_sequence_id, end_of_sequence_id, 
        maximum_length, vocab_size)
    inference_logits, _, _ = tf.contrib.seq2seq.dynamic_rnn_decoder(dec_cell, infer_decoder_fn, scope=decoding_scope)
    inference_logits = tf.nn.dropout(inference_logits, keep_prob)
return inference_logits

6.构建解码层

引自原文：

构建解码层
- 实现decode_layer()来创建一个解码器RNN层。
- 使用rnn_size和num_layers创建RNN单元进行解码。
使用lambda创建输出函数，将其输入，logits转换为class logits。
- 使用您的decode_layer_train(encoder_state，dec_cell，dec_embed_input，sequence_length，decode_scope，output_fn，keep_prob)函数来获取训练分对数(logits)。
- 使用你的decode_layer_infer(encoder_state，dec_cell，dec_embeddings，start_of_sequence_id，end_of_sequence_id，maximum_length，vocab_size，decode_scope，output_fn，keep_prob)来获取推理分对数(logits)。
- (注意：您需要使用tf.variable_scope来在训练和推理之间共享变量。

def decoding_layer(dec_embed_input, dec_embeddings, encoder_state, vocab_size, sequence_length, rnn_size,
 num_layers, target_vocab_to_int, keep_prob):
"""
 Create decoding layer
 :param dec_embed_input: Decoder embedded input
 :param dec_embeddings: Decoder embeddings
 :param encoder_state: The encoded state
 :param vocab_size: Size of vocabulary
 :param sequence_length: Sequence Length
 :param rnn_size: RNN Size
 :param num_layers: Number of layers
 :param target_vocab_to_int: Dictionary to go from the target words to an id
 :param keep_prob: Dropout keep probability
 :return: Tuple of (Training Logits, Inference Logits)
 """
# Decoder RNNs
# 多层lstm
    dec_cell = tf.contrib.rnn.MultiRNNCell([tf.contrib.rnn.BasicLSTMCell(rnn_size)] * num_layers)

with tf.variable_scope("decoding") as decoding_scope:
# Output Layer
# 添加全连接层
        output_fn = lambda x: tf.contrib.layers.fully_connected(x, vocab_size, None, scope=decoding_scope)
# 解码-训练
        train_logits = decoding_layer_train(encoder_state, dec_cell, dec_embed_input, sequence_length, decoding_scope,
                         output_fn, keep_prob)    
with tf.variable_scope("decoding", reuse=True) as decoding_scope:
# 解码-推理 
        infer_logits = decoding_layer_infer(encoder_state, dec_cell, dec_embeddings, target_vocab_to_int['<GO>'], target_vocab_to_int['<EOS>'],
                         sequence_length, vocab_size, decoding_scope, output_fn, keep_prob)
return train_logits, infer_logits

7.构建神经网络

引自原文

将上述实现的功能应用于：
- 应用嵌入于解码的输入数据。
- 使用encoding_layer(rnn_inputs，rnn_size，num_layers，keep_prob)编码输入。
- 使用process_decoding_input(target_data，target_vocab_to_int，batch_size)函数处理目标数据。
- 应用嵌入于解码的目标数据。
- 使用decode_layer(dec_embed_input，dec_embeddings，encoder_state，vocab_size，sequence_length，rnn_size，num_layers，target_vocab_to_int，keep_prob)对编码输入进行解码。

tf.contrib.layers.embed_sequence将一系列符号(字符)映射到嵌入序列。典型的用例是在编码器和解码器之间重用嵌入。
tf.nn.embedding_lookup在嵌入张量列表中查找ids。

def seq2seq_model(input_data, target_data, keep_prob, batch_size, sequence_length, source_vocab_size, target_vocab_size,
 enc_embedding_size, dec_embedding_size, rnn_size, num_layers, target_vocab_to_int):
"""
 Build the Sequence-to-Sequence part of the neural network
 :param input_data: Input placeholder
 :param target_data: Target placeholder
 :param keep_prob: Dropout keep probability placeholder
 :param batch_size: Batch Size
 :param sequence_length: Sequence Length
 :param source_vocab_size: Source vocabulary size
 :param target_vocab_size: Target vocabulary size
 :param enc_embedding_size: Decoder embedding size
 :param dec_embedding_size: Encoder embedding size
 :param rnn_size: RNN Size
 :param num_layers: Number of layers
 :param target_vocab_to_int: Dictionary to go from the target words to an id
 :return: Tuple of (Training Logits, Inference Logits)
 """
# TODO: Implement Function
    enc_embed_input = tf.contrib.layers.embed_sequence(input_data, source_vocab_size, enc_embedding_size)
    encoder_state = encoding_layer(enc_embed_input, rnn_size, num_layers, keep_prob)
    target_data = process_decoding_input(target_data, target_vocab_to_int, batch_size)
#target_embed = tf.contrib.layers.embed_sequence(target_data, target_vocab_size, dec_embedding_size) 
    dec_embeddings = tf.Variable(tf.random_uniform([target_vocab_size, dec_embedding_size]))
    target_embed = tf.nn.embedding_lookup(dec_embeddings, target_data)
return decoding_layer(target_embed, dec_embeddings, encoder_state, target_vocab_size, sequence_length, rnn_size, \
              num_layers, target_vocab_to_int, keep_prob)

8.训练神经网络

定义参数
创建图
- tf.identity 返回与输入张量或值相同的维度与内容的张量。
- tf.contrib.seq2seq.sequence_loss 一序列分对数(logits)的加权交叉熵。
- tf.train.Optimizer.compute_gradients(loss, var_list=None, gate_gradients=1, aggregation_method=None, colocate_gradients_with_ops=False, grad_loss=None)计算var_list中变量的损失梯度。是minimize()的第一部分。
- tf.train.Optimizer.apply_gradients(grads_and_vars, global_step=None, name=None)将梯度应用于变量。是minimize()的第二部分。

train_graph = tf.Graph()
with train_graph.as_default():
    input_data, targets, lr, keep_prob = model_inputs()
    sequence_length = tf.placeholder_with_default(max_source_sentence_length, None, name='sequence_length')
    input_shape = tf.shape(input_data)

# 我们创建的模型
    train_logits, inference_logits = seq2seq_model(
        tf.reverse(input_data, [-1]), targets, keep_prob, batch_size, sequence_length, len(source_vocab_to_int), len(target_vocab_to_int),
        encoding_embedding_size, decoding_embedding_size, rnn_size, num_layers, target_vocab_to_int)

    tf.identity(inference_logits, 'logits')
with tf.name_scope("optimization"):
# Loss function
        cost = tf.contrib.seq2seq.sequence_loss(
            train_logits,
            targets,
            tf.ones([input_shape[0], sequence_length]))

# Optimizer
        optimizer = tf.train.AdamOptimizer(lr)

# Gradient Clipping
        gradients = optimizer.compute_gradients(cost)
        capped_gradients = [(tf.clip_by_value(grad, -1., 1.), var) for grad, var in gradients if grad is not None]
        train_op = optimizer.apply_gradients(capped_gradients)

训练

def get_accuracy(target, logits):
"""
 Calculate accuracy
 """
    max_seq = max(target.shape[1], logits.shape[1])
if max_seq - target.shape[1]:
        target = np.pad(
            target,
            [(0,0),(0,max_seq - target.shape[1])],
'constant')
if max_seq - logits.shape[1]:
        logits = np.pad(
            logits,
            [(0,0),(0,max_seq - logits.shape[1]), (0,0)],
'constant')

return np.mean(np.equal(target, np.argmax(logits, 2)))

train_source = source_int_text[batch_size:]
train_target = target_int_text[batch_size:]

valid_source = helper.pad_sentence_batch(source_int_text[:batch_size])
valid_target = helper.pad_sentence_batch(target_int_text[:batch_size])

with tf.Session(graph=train_graph) as sess:
    sess.run(tf.global_variables_initializer())

for epoch_i in range(epochs):
for batch_i, (source_batch, target_batch) in enumerate(
                helper.batch_data(train_source, train_target, batch_size)):
            start_time = time.time()

            _, loss = sess.run(
                [train_op, cost],
                {input_data: source_batch,
                 targets: target_batch,
                 lr: learning_rate,
                 sequence_length: target_batch.shape[1],
                 keep_prob: keep_probability})

            batch_train_logits = sess.run(
                inference_logits,
                {input_data: source_batch, keep_prob: 1.0})
            batch_valid_logits = sess.run(
                inference_logits,
                {input_data: valid_source, keep_prob: 1.0})

            train_acc = get_accuracy(target_batch, batch_train_logits)
            valid_acc = get_accuracy(np.array(valid_target), batch_valid_logits)
            end_time = time.time()
            print('Epoch {:>3} Batch {:>4}/{} - Train Accuracy: {:>6.3f}, Validation Accuracy: {:>6.3f}, Loss: {:>6.3f}'
                  .format(epoch_i, batch_i, len(source_int_text) // batch_size, train_acc, valid_acc, loss))

0x02.神经网络结构

代码中神经网络可以看作编码与解码两层结构，其中编码层是动态的多层lstm，解码层可分为训练和推理两部分，参数包含了编码层的state和另一个多层lstm。

seq2seq模型由编码与解码两部分构成，编码部分将输入编码为状态向量，解码部分将状态向量作为输入输出序列。下面是一个常见的seq2seq模型，其中编解码采用的神经网络是lstm(rnn)。（后续争取对seq2seq做一个总结

序列到序列的语言翻译模型代码(tensorflow)解析

0x03.参考

从Encoder到Decoder实现Seq2Seq模型

秒客网

序列到序列的语言翻译模型代码(tensorflow)解析

0x00.前言

0x01.代码解析

1.数据预处理

2.输入

3.编码

4.解码-训练层

5.解码-推理层

6.构建解码层

7.构建神经网络

8.训练神经网络

0x02.神经网络结构

0x03.参考

相关文章