Neural Collaborative Filtering (NCF)

May 24, 2020

---

Neural Collaborative Filtering is a network for recommendation systems and was one of the benchmarks in the MLPerf 0.5 submission. It is a small network with about 10 layers, however, a single training step (forward and backward propagation) invokes about 200 GPU CUDA kernels (depending on the batch size). The best way to understand a DL network and GPU performance is to understand every single CUDA kernel i.e. which layer of the network invoked the kernel, with what arguments (tensor shapes and datatypes) and in which direction (forward propagation or backward propagation).

In this blog post, I will explain every kernel used in the training of NCF. All the information in the tables below was obtained using NVidia's PyTorch Profiler, PyProf, on a Turing T4 GPU. The information below is only a subset of what is provided by PyProf. The code and instructions for obtaining a detailed profile are here. Note that Volta and Ampere GPUs will have slightly different kernel names e.g. volta_* as opposed to turing_*.

Model Architecture

Model Parameters

The parameters in the profiled code are

Users = 138493
Items = 26744
MF User Embedding = 64
MF Item Embedding = 64
MLP User Embedding = 128
MLP Item Embedding = 128
MLP Layers = [256, 128, 64]
Batch Size = 1048576

GPU Kernels

The tables below show the GPU kernels invoked in 1 training step. For every GPU kernel we show the direction (fprop, bprop), name of the layer, name of the operation, and the input tensor shapes / matrix dimensions for the operation. PyProf provides a lot of additional information for every GPU kernel e.g. grid dimensions, block dimensions, silicon time, datatypes, flops, bytes, tensor core usage and so on.

Kernels 1 through 38 correspond to forward propagation. Kernels 1, 2, 4 and 5 correspond to the four embedding lookups. Kernel 3 corresponds to the element wise multiplication (Hadamard product). Kernel 6 corresponds to the concatenation before the MLP layers. Kernels 7 through 18 correspond to the 3 Linear layers. Kernel 19 corresponds to the concatenation before the NeuMF layer. Kernels 20, 21 correspond to the final NeuMF Linear layer. Kernel 22 corresponds to a cast operation from float16 to float32 before the loss layer.

Kernels 23 through 36 correspond to the Binary Cross Entropy Loss calculation. Kernel 37 calculates the average loss. Kernel 38 corresponds to the static/dynamic loss scaling i.e. multiplication by a scalar.

The following kernels are invoked during back propagation. Kernels 42 through 44 correspond to the Loss layer (in float32). Kernel 45 converts the gradient from float32 to float16. Kernels 46 through 48 calculate the data gradient and weight gradient through the NeuMF Linear layer. Kernel 49 calculates the bias gradient. Kernels 50 through 67 calculate the gradient through the dropout layer, ReLU layer, data gradient, weight gradient, and bias gradient through the three Linear layers.

Back propagation through the Embedding layers invoke a lot of Thrust and CUB kernels. Kernels 68 through 97 correspond to the MLP Item Embedding layer.

Kernels 98 through 127 correspond to the MLP User Embedding layer.

Kernels 128, 129 correspond to the element wise multiplication.

Kernels 130 through 158 correspond to the MF Item Embedding layer.

Kernels 159 through 187 correspond to the MF User Embedding layer.

At the end of back propagation, we have all the weight gradients. Kernels 188 through 194 correspond to reversing the loss scaling and applying the Adam optimizer on all the weights and biases.