Model-Parallelism/03_FTRunInferenceOfTheGPT-J.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"./images/DLI_Header.png\" style=\"width: 400px;\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 3. Optimizing inference with NVIDIA FasterTransformer library \n",
    "\n",
    "In this lab, we are going to look at the NVIDIA FasterTransformer library and how it optimizes execution of large language models. We will use it to deploy GPT-J 6B initially using just a single GPU but afterwards taking advantage of its Tensor and Pipeline parallelism capabilities on multiple GPUs.  \n",
    "\n",
    "We will conclude this notebook by comparing the latency between our baseline implementation using Transformers library and the FasterTransformer Tensor and Pipeline parallel deployments. In the next notebook we will look at how to serve our FasterTransformer optimized model to your customers/users using Triton Inference Server. \n",
    "\n",
    "To summarize, in this notebook we will: \n",
    "* Review the features of NVIDIA FasterTransformer library. \n",
    "* Learn how to build the development environment including building FasterTransformer library. \n",
    "* Learn how to prepare a checkpoint of GPT-J model (or other Transformers based model) for inference with FasterTransformer. \n",
    "* Run inference of the model on a single GPU. \n",
    "* Extend the execution to multiple GPUs using Tensor Parallelism. \n",
    "* Profile the single and multi GPU pipelines to capture information about throughput and latency. \n",
    "\n",
    "**[3.1 NVIDIA FasterTransformer](#3.1)<br>** \n",
    "**[3.2 Overall Inference Pipeline with NVIDIA FasterTransformer](#3.2)<br>** \n",
    "**[3.3 Download and Build NVIDIA FasterTransformer library](#3.3)<br>** \n",
    "**[3.4 Download and prepare GPT-J checkpoint](#3.4)<br>** \n",
    "**[3.5 Convert weights into FT-friendly format for the inference](#3.5)<br>** \n",
    "**[3.6 GPT-J inference using C++ bindings](#3.6)<br>** \n",
    "&nbsp;&nbsp;&nbsp;&nbsp;[3.6.1 Inference on 1 GPU ](#3.4.1)<br> \n",
    "&nbsp;&nbsp;&nbsp;&nbsp;[3.6.2 Inference on 2 GPUs ](#3.4.2)<br> "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 3.1 NVIDIA FasterTransformer"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Introduction \n",
    "\n",
    "[NVIDIA’s FasterTransformer (FT)](https://github.com/NVIDIA/FasterTransformer) is a library implementing an accelerated engine for the inference of Transformer-based Neural Networks. FasterTransformer was developed with an emphasis on inference of large models also including support for multi GPU and multi node inference.  \n",
    "\n",
    "FasterTransformer contains an optimized implementation of the Transformer block that contains both the encoder and decoder components. FasterTransformer can be used to execute both encoder-decoder architectures like T5, as well as BERT-like (encoder only) or GPT-3 like (decoder only) models. It is written in C++/CUDA and relies on cuBLAS, cuBLASlt , cuSPARSELt libraries to provide the fastest computational pipelines on GPU. \n",
    "\n",
    "## Large language models \n",
    "\n",
    "GPT and GPT-J are both variants of the transformer architecture. They do not implement the encoder module or cross multi-head attention. They are decoder only models. They both use GeLU as the activation function. In 2020, OpenAI demonstrated that using a large language model trained in self supervised way on large volume of training data can significantly improve the capacity of GPT model ([refer to the paper for more details](https://arxiv.org/abs/2005.14165)). The largest GPT-3 variant, has 175 billion parameters, which consumes about 350 GBs even when represented in half-precision. Therefore putting such a model on a single GPU is impossible making multi-gpu or even multi-node deployment a necessity. To solve the challenges of latency and memory footprint, FasterTransformer provides high efficiency kernels, optimized for memory usage, and support for model parallelism. It also comes with a wide range of other features including: \n",
    "* Number of checkpoint converters:\n",
    "  * Huggingface\n",
    "  * Megatron\n",
    "  * Nemo Megatron\n",
    "  * TensorFlow\n",
    "* Support for a range of data types:\n",
    "  * FP32\n",
    "  * FP16\n",
    "  * INT8 (weights only, PTQ for batch size 1 and 2)\n",
    "* Advanced feature:\n",
    "  * Multi-GPU multi-node inference\n",
    "  * Dynamic random seed\n",
    "  * Stop tokens\n",
    "  * Beam search and sampling are both supported\n",
    "  * FP32, FP16 and INT8 inference\n",
    "* Frameworks\n",
    "  * TensorFlow\n",
    "  * PyTorch\n",
    "  * C++\n",
    "  * Triton backend\n",
    "  \n",
    " \n",
    "This section of the notebook discusses how FasterTransformer can be used for optimization of the GPT-J model. It explains the optimization workflow for both single and multi GPU deployments. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Tensor and Pipeline Parallelism \n",
    "\n",
    "Under the hood, FasterTransformer relies on MPI and NVIDIA NCCL to enable inter/intra node communication. Using this software stack, anyone can run huge Transformers in Tensor-Parallelism mode on multiple GPUs to reduce computational latency. At the same time, tensor parallelism and pipeline parallelism can be combined to execute large models with billions and trillions of parameters (which amount to terabytes of weights) in Multi-GPU and Multi-Node environments. \n",
    "\n",
    "We have discussed the techniques below in the lecture but let us revisit them before diving into the implementation detail: \n",
    "- Data Parallelism (DP) - is a technique used during the training process. Every GPU receives the same copy of the model but different data to process. The GPUs execute the forward pass in parallel and exchange the gradients during the backward pass allowing all the devices to make a synchronized weights update based on the average of the accumulated gradients. \n",
    "- Tensor Parallelism (TP) - is a technique used both during training and inference. Instead of splitting the data across multiple GPUs, selected layers of the model are distributed. If using Tensor Parallelism across 8 GPUs each layer affected/its tensor is split into 8 segments, each processed on a separate GPU in parallel. The results are gathered at the end of the step. \n",
    "- Pipeline Parallelism (PP) - similarly, this is a technique used both in training and inference. Here individual layers are not being split into pieces, instead they are sequentially distributed across multiple GPUs. E.g. if training a 10 layer deep neural network across 2 GPUs, the first five layers would be deployed on the first GPU and the rest on the second GPU. Each GPU is processing data sequentially and the second GPU needs to wait for results from the first GPU. \n",
    "\n",
    "The diagram below demonstrates the difference between Tensor and Pipeline parallelism. \n",
    "\n",
    "<div style=\"text-align:center\"><img src=\"./images/image3.png\" style=\"width: 1000px;\"></div>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Optimizations in FasterTransformer library \n",
    "\n",
    "  \n",
    "\n",
    "FT allows us to speed up the inference pipeline achieving lower latency and higher throughput compared to the common deep learning frameworks. Below are the key optimization techniques that allow FT to achieve its performance: \n",
    "1. <b>Layer Fusion</b></br> \n",
    "During the model pre-processing stage, certain layers can be combined to form individual execution kernels. This allows for considerable reduction in GPU memory bandwidth increasing mathematical density of our model, thus accelerating computation at the inference stage. For example, all operations in the multi-head attention block can be combined into a single kernel. \n",
    "2. <b>Autoregressive models: Keys/Values caching. </b></br> \n",
    "To prevent recomputing the previous keys and values, FT allocates a buffer to store them at each step. Although the cache consumes additional memory, FT can remove the cost of recomputing those metrices but also costs associated with memory buffer allocation, and the cost of concatenation. The diagram below illustrates the process:<br/> \n",
    "<div style=\"text-align:center\"> \n",
    "<img src=\"./images/KV_caching v2.PNG\" style=\"width: 50%;position:relative;\"><br/> \n",
    "<em>Keys/Values caching</em> \n",
    "</div> \n",
    "<br/><br/> \n",
    "3. <b>Memory optimization</b></br> \n",
    "Unlike traditional models like BERT, large transformer models have billions/trillions of parameters. For instance, GPT-3 175B uses 350 GB of memory even if we store in FP16 representation. FasterTransformer can reuse the memory buffer of decoder layers. Since GPt-3 is constructed of 96 layers, we only need to allocate 1/96 of the required memory. \n",
    "4. <b>Usage of MPI and NCCL to enable inter/intra node communication and support model parallelism. </b></br> \n",
    "FasterTransormer provides support for both tensor and pipeline parallelism. Tensor parallel in FasterTransformer follows the implementation of Megatron-LM. For both the self-attention block and feed forward block, FT splits the weights of the first matrix multiplication by row and splits the weights of the second matrix multiplication by column. This allows FT to execute each transformer block with just two reduction operations. For pipeline parallelism, FasterTransformer splits the whole batch of request into multiple micro batches and hides the bubble of communication. FasterTransformer will adjust the micro-batch size automatically for different cases. Users can adjust the model parallelism by modifying the gpt_config.ini file. We recommend using tensor parallel intra node and pipeline parallel inter node because tensor parallel implementation requires more NCCL communication. \n",
    "5. <b>MatMul kernel autotuning (GEMM Autotuning)</b></br> \n",
    "Matrix Multiplication is the most computationally intensive operation in transformer-based neural networks. FT uses functionality from CuBLAS and CuTLASS libraries to execute GEMM operations. It is important to know that MatMul operations can be executed in many different ways using different low-level algorithms on the “hardware” level depending on the properties of the metrices being processed. The GemmBatchedEx function implements the MatMul operation and has “cublasGemmAlgo_t”  as an input parameter. Using this parameter, we can choose different low-level algorithms executing this operation. The FasterTransformer library uses this parameter to do a real-time benchmark of all low-level algorithms and to choose the best one for the specific parameters of the model (size of the attention layers, number of attention heads, size of the hidden layer, …) and batch dimensions. Additionally, FT uses hardware-accelerated low-level functions for some parts of the network such as: __expf(), __shfl_xor_sync() \n",
    "<div style=\"text-align:center\"> \n",
    "<img src=\"./images/kernel autotuning.png\" style=\"width: 20%;position:relative;\"><br/> \n",
    "<em>Kernel autotuning</em> \n",
    "</div> \n",
    "<br/><br/> \n",
    "6. <b>Reduced precision inference</b></br> \n",
    "FT has kernels that support inference using low-precision input data in fp16 and int8. Both these regimes allow acceleration due to the reduction in data transfer and required memory. At the same time, int8 and fp16 computations can be hardware accelerated using TensorCores (available on all GPU architectures starting from Volta).  \n",
    "7. <b>Other optimizations include:</b></br> \n",
    "    - Efficient C++ implementation of BeamSearch  \n",
    "    - Optimized all-reduce implementation of eight-way Tensor Parallelism. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 3.2 Overall Inference Pipeline with NVIDIA FasterTransformer \n",
    "The diagram listed below lists all the steps involved in using FasterTransformer library to deploy large models to production. In the next section, we will go through them one at a time. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div style=\"text-align:center\">\n",
    "<img src=\"./images/FT_pipeline.PNG\" style=\"width: 70%\"/>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3.3 Download and Build NVIDIA FasterTransformer library\n",
    "\n",
    "### Step 1: Cloning FasterTransformer library from GitHub\n",
    "Execute the below command to clone the code repository. To ensure consistent student experience, we will use a specific branch."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Cloning into 'FasterTransformer'...\n",
      "remote: Enumerating objects: 6827, done.\u001b[K\n",
      "remote: Counting objects: 100% (6827/6827), done.\u001b[K\n",
      "remote: Compressing objects: 100% (1944/1944), done.\u001b[K\n",
      "remote: Total 6827 (delta 4862), reused 6673 (delta 4807), pack-reused 0\u001b[K\n",
      "Receiving objects: 100% (6827/6827), 67.91 MiB | 57.47 MiB/s, done.\n",
      "Resolving deltas: 100% (4862/4862), done.\n",
      "/dli/task/FasterTransformer\n",
      "Note: switching to '6b3fd4392831f972d48127e881a048567dd92811'.\n",
      "\n",
      "You are in 'detached HEAD' state. You can look around, make experimental\n",
      "changes and commit them, and you can discard any commits you make in this\n",
      "state without impacting any branches by switching back to a branch.\n",
      "\n",
      "If you want to create a new branch to retain commits you create, you may\n",
      "do so (now or later) by using -c with the switch command. Example:\n",
      "\n",
      "  git switch -c <new-branch-name>\n",
      "\n",
      "Or undo this operation with:\n",
      "\n",
      "  git switch -\n",
      "\n",
      "Turn off this advice by setting config variable advice.detachedHead to false\n",
      "\n",
      "HEAD is now at 6b3fd43 Update gpt_guide.md\n"
     ]
    }
   ],
   "source": [
    "!git clone https://github.com/NVIDIA/FasterTransformer.git\n",
    "%cd FasterTransformer\n",
    "!git checkout 6b3fd4392831f972d48127e881a048567dd92811"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 2: Building the FT library"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Install dependencies"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We should install additional libraries like `cmake` for build automation, and we also should install compression libraries \"zstd\" and \"libz-dev\" that will allow decompressing weights of the GPT-J 6b model. </br>\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Get:1 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64  InRelease [1581 B]\n",
      "Get:2 https://deb.nodesource.com/node_16.x focal InRelease [4583 B]            \u001b[0m\u001b[33m\u001b[33m\n",
      "Get:3 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64  Packages [923 kB]\n",
      "Get:4 http://security.ubuntu.com/ubuntu focal-security InRelease [114 kB]    \u001b[0m\u001b[33m\n",
      "Hit:5 http://archive.ubuntu.com/ubuntu focal InRelease                         \u001b[0m\n",
      "Get:6 https://deb.nodesource.com/node_16.x focal/main amd64 Packages [777 B]   \u001b[0m\n",
      "Get:7 http://archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB]        \u001b[0m\u001b[33m\n",
      "Get:8 http://archive.ubuntu.com/ubuntu focal-backports InRelease [108 kB][33m  \u001b[0m\u001b[33m\u001b[33m\u001b[33m\u001b[33m\n",
      "Get:9 http://security.ubuntu.com/ubuntu focal-security/universe amd64 Packages [1019 kB]\n",
      "Get:10 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 Packages [31.2 kB]\n",
      "Get:11 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages [1314 kB]\n",
      "Get:12 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 Packages [28.5 kB]\n",
      "Get:13 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 Packages [1998 kB]\n",
      "Get:14 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages [2545 kB]33m\u001b[33m\n",
      "Get:15 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 Packages [2141 kB]\n",
      "Get:16 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [3027 kB]\u001b[33m\n",
      "Get:17 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 Packages [28.6 kB]\n",
      "Get:18 http://archive.ubuntu.com/ubuntu focal-backports/main amd64 Packages [55.2 kB]\n",
      "Fetched 13.5 MB in 1s (11.5 MB/s)[33m                        \u001b[0m\u001b[33m\u001b[33m\u001b[33m\u001b[33m\n",
      "Reading package lists... Done\n",
      "Building dependency tree       \n",
      "Reading state information... Done\n",
      "202 packages can be upgraded. Run 'apt list --upgradable' to see them.\n",
      "\u001b[1;33mW: \u001b[0mTarget Packages (Packages) is configured multiple times in /etc/apt/sources.list:50 and /etc/apt/sources.list.d/cuda-ubuntu2004-x86_64.list:1\u001b[0m\n",
      "\u001b[1;33mW: \u001b[0mTarget Packages (Packages) is configured multiple times in /etc/apt/sources.list:50 and /etc/apt/sources.list.d/cuda-ubuntu2004-x86_64.list:1\u001b[0m\n",
      "Reading package lists... Done\n",
      "Building dependency tree       \n",
      "Reading state information... Done\n",
      "Note, selecting 'zlib1g-dev' instead of 'libz-dev'\n",
      "The following additional packages will be installed:\n",
      "  cmake-data libjsoncpp1 librhash0 libuv1 zlib1g\n",
      "Suggested packages:\n",
      "  cmake-doc ninja-build\n",
      "The following NEW packages will be installed:\n",
      "  cmake cmake-data libjsoncpp1 librhash0 libuv1 zlib1g-dev zstd\n",
      "The following packages will be upgraded:\n",
      "  xz-utils zlib1g\n",
      "2 upgraded, 7 newly installed, 0 to remove and 200 not upgraded.\n",
      "Need to get 6187 kB of archives.\n",
      "After this operation, 30.8 MB of additional disk space will be used.\n",
      "Get:1 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 zlib1g amd64 1:1.2.11.dfsg-2ubuntu1.5 [54.2 kB]\n",
      "Get:2 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 xz-utils amd64 5.2.4-1ubuntu1.1 [82.6 kB]\n",
      "Get:3 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libuv1 amd64 1.34.2-1ubuntu1.3 [80.8 kB]\n",
      "Get:4 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 cmake-data all 3.16.3-1ubuntu1.20.04.1 [1613 kB]\n",
      "Get:5 http://archive.ubuntu.com/ubuntu focal/main amd64 libjsoncpp1 amd64 1.7.4-3.1ubuntu2 [75.6 kB]\n",
      "Get:6 http://archive.ubuntu.com/ubuntu focal/main amd64 librhash0 amd64 1.3.9-1 [113 kB]\n",
      "Get:7 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 cmake amd64 3.16.3-1ubuntu1.20.04.1 [3668 kB]\n",
      "Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 zlib1g-dev amd64 1:1.2.11.dfsg-2ubuntu1.5 [155 kB]\n",
      "Get:9 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 zstd amd64 1.4.4+dfsg-3ubuntu0.1 [343 kB]\n",
      "Fetched 6187 kB in 1s (4725 kB/s)0m\u001b[33m\n",
      "W: Target Packages (Packages) is configured multiple times in /etc/apt/sources.list:50 and /etc/apt/sources.list.d/cuda-ubuntu2004-x86_64.list:1\n",
      "\n",
      "(Reading database ... 38211 files and directories currently installed.)\n",
      "Preparing to unpack .../zlib1g_1%3a1.2.11.dfsg-2ubuntu1.5_amd64.deb ...\n",
      "\u001b7\u001b[24;0f\u001b[42m\u001b[30mProgress: [  0%]\u001b[49m\u001b[39m [..........................................................] \u001b8\u001b7\u001b[24;0f\u001b[42m\u001b[30mProgress: [  3%]\u001b[49m\u001b[39m [#.........................................................] \u001b8Unpacking zlib1g:amd64 (1:1.2.11.dfsg-2ubuntu1.5) over (1:1.2.11.dfsg-2ubuntu1.2) ...\n",
      "\u001b7\u001b[24;0f\u001b[42m\u001b[30mProgress: [  5%]\u001b[49m\u001b[39m [###.......................................................] \u001b8Setting up zlib1g:amd64 (1:1.2.11.dfsg-2ubuntu1.5) ...\n",
      "(Reading database ... 38211 files and directories currently installed.)..................................] \u001b8\u001b7\u001b[24;0f\u001b[42m\u001b[30mProgress: [ 11%]\u001b[49m\u001b[39m [######....................................................] \u001b8(Reading database ... \n",
      "Preparing to unpack .../0-xz-utils_5.2.4-1ubuntu1.1_amd64.deb ...\n",
      "\u001b7\u001b[24;0f\u001b[42m\u001b[30mProgress: [ 14%]\u001b[49m\u001b[39m [#######...................................................] \u001b8Unpacking xz-utils (5.2.4-1ubuntu1.1) over (5.2.4-1ubuntu1) ...\n",
      "\u001b7\u001b[24;0f\u001b[42m\u001b[30mProgress: [ 16%]\u001b[49m\u001b[39m [#########.................................................] \u001b8Selecting previously unselected package libuv1:amd64.\n",
      "Preparing to unpack .../1-libuv1_1.34.2-1ubuntu1.3_amd64.deb ...\n",
      "\u001b7\u001b[24;0f\u001b[42m\u001b[30mProgress: [ 19%]\u001b[49m\u001b[39m [##########................................................] \u001b8Unpacking libuv1:amd64 (1.34.2-1ubuntu1.3) ...\n",
      "\u001b7\u001b[24;0f\u001b[42m\u001b[30mProgress: [ 22%]\u001b[49m\u001b[39m [############..............................................] \u001b8Selecting previously unselected package cmake-data.\n",
      "Preparing to unpack .../2-cmake-data_3.16.3-1ubuntu1.20.04.1_all.deb ...\n",
      "\u001b7\u001b[24;0f\u001b[42m\u001b[30mProgress: [ 24%]\u001b[49m\u001b[39m [##############............................................] \u001b8Unpacking cmake-data (3.16.3-1ubuntu1.20.04.1) ...\n",
      "\u001b7\u001b[24;0f\u001b[42m\u001b[30mProgress: [ 27%]\u001b[49m\u001b[39m [###############...........................................] \u001b8Selecting previously unselected package libjsoncpp1:amd64.\n",
      "Preparing to unpack .../3-libjsoncpp1_1.7.4-3.1ubuntu2_amd64.deb ...\n",
      "\u001b7\u001b[24;0f\u001b[42m\u001b[30mProgress: [ 30%]\u001b[49m\u001b[39m [#################.........................................] \u001b8Unpacking libjsoncpp1:amd64 (1.7.4-3.1ubuntu2) ...\n",
      "\u001b7\u001b[24;0f\u001b[42m\u001b[30mProgress: [ 32%]\u001b[49m\u001b[39m [##################........................................] \u001b8Selecting previously unselected package librhash0:amd64.\n",
      "Preparing to unpack .../4-librhash0_1.3.9-1_amd64.deb ...\n",
      "\u001b7\u001b[24;0f\u001b[42m\u001b[30mProgress: [ 35%]\u001b[49m\u001b[39m [####################......................................] \u001b8Unpacking librhash0:amd64 (1.3.9-1) ...\n",
      "\u001b7\u001b[24;0f\u001b[42m\u001b[30mProgress: [ 38%]\u001b[49m\u001b[39m [#####################.....................................] \u001b8Selecting previously unselected package cmake.\n",
      "Preparing to unpack .../5-cmake_3.16.3-1ubuntu1.20.04.1_amd64.deb ...\n",
      "\u001b7\u001b[24;0f\u001b[42m\u001b[30mProgress: [ 41%]\u001b[49m\u001b[39m [#######################...................................] \u001b8Unpacking cmake (3.16.3-1ubuntu1.20.04.1) ...\n",
      "\u001b7\u001b[24;0f\u001b[42m\u001b[30mProgress: [ 43%]\u001b[49m\u001b[39m [#########################.................................] \u001b8Selecting previously unselected package zlib1g-dev:amd64.\n",
      "Preparing to unpack .../6-zlib1g-dev_1%3a1.2.11.dfsg-2ubuntu1.5_amd64.deb ...\n",
      "\u001b7\u001b[24;0f\u001b[42m\u001b[30mProgress: [ 46%]\u001b[49m\u001b[39m [##########################................................] \u001b8Unpacking zlib1g-dev:amd64 (1:1.2.11.dfsg-2ubuntu1.5) ...\n",
      "\u001b7\u001b[24;0f\u001b[42m\u001b[30mProgress: [ 49%]\u001b[49m\u001b[39m [############################..............................] \u001b8Selecting previously unselected package zstd.\n",
      "Preparing to unpack .../7-zstd_1.4.4+dfsg-3ubuntu0.1_amd64.deb ...\n",
      "\u001b7\u001b[24;0f\u001b[42m\u001b[30mProgress: [ 51%]\u001b[49m\u001b[39m [#############################.............................] \u001b8Unpacking zstd (1.4.4+dfsg-3ubuntu0.1) ...\n",
      "\u001b7\u001b[24;0f\u001b[42m\u001b[30mProgress: [ 54%]\u001b[49m\u001b[39m [###############################...........................] \u001b8Setting up libuv1:amd64 (1.34.2-1ubuntu1.3) ...\n",
      "\u001b7\u001b[24;0f\u001b[42m\u001b[30mProgress: [ 57%]\u001b[49m\u001b[39m [################################..........................] \u001b8\u001b7\u001b[24;0f\u001b[42m\u001b[30mProgress: [ 59%]\u001b[49m\u001b[39m [##################################........................] \u001b8Setting up xz-utils (5.2.4-1ubuntu1.1) ...\n",
      "\u001b7\u001b[24;0f\u001b[42m\u001b[30mProgress: [ 62%]\u001b[49m\u001b[39m [####################################......................] \u001b8\u001b7\u001b[24;0f\u001b[42m\u001b[30mProgress: [ 65%]\u001b[49m\u001b[39m [#####################################.....................] \u001b8Setting up zlib1g-dev:amd64 (1:1.2.11.dfsg-2ubuntu1.5) ...\n",
      "\u001b7\u001b[24;0f\u001b[42m\u001b[30mProgress: [ 68%]\u001b[49m\u001b[39m [#######################################...................] \u001b8\u001b7\u001b[24;0f\u001b[42m\u001b[30mProgress: [ 70%]\u001b[49m\u001b[39m [########################################..................] \u001b8Setting up librhash0:amd64 (1.3.9-1) ...\n",
      "\u001b7\u001b[24;0f\u001b[42m\u001b[30mProgress: [ 73%]\u001b[49m\u001b[39m [##########################################................] \u001b8\u001b7\u001b[24;0f\u001b[42m\u001b[30mProgress: [ 76%]\u001b[49m\u001b[39m [###########################################...............] \u001b8Setting up cmake-data (3.16.3-1ubuntu1.20.04.1) ...\n",
      "\u001b7\u001b[24;0f\u001b[42m\u001b[30mProgress: [ 78%]\u001b[49m\u001b[39m [#############################################.............] \u001b8\u001b7\u001b[24;0f\u001b[42m\u001b[30mProgress: [ 81%]\u001b[49m\u001b[39m [###############################################...........] \u001b8Setting up zstd (1.4.4+dfsg-3ubuntu0.1) ...\n",
      "\u001b7\u001b[24;0f\u001b[42m\u001b[30mProgress: [ 84%]\u001b[49m\u001b[39m [################################################..........] \u001b8\u001b7\u001b[24;0f\u001b[42m\u001b[30mProgress: [ 86%]\u001b[49m\u001b[39m [##################################################........] \u001b8Setting up libjsoncpp1:amd64 (1.7.4-3.1ubuntu2) ...\n",
      "\u001b7\u001b[24;0f\u001b[42m\u001b[30mProgress: [ 89%]\u001b[49m\u001b[39m [###################################################.......] \u001b8\u001b7\u001b[24;0f\u001b[42m\u001b[30mProgress: [ 92%]\u001b[49m\u001b[39m [#####################################################.....] \u001b8Setting up cmake (3.16.3-1ubuntu1.20.04.1) ...\n",
      "\u001b7\u001b[24;0f\u001b[42m\u001b[30mProgress: [ 95%]\u001b[49m\u001b[39m [######################################################....] \u001b8\u001b7\u001b[24;0f\u001b[42m\u001b[30mProgress: [ 97%]\u001b[49m\u001b[39m [########################################################..] \u001b8Processing triggers for libc-bin (2.31-0ubuntu9.2) ...\n",
      "\n",
      "\u001b7\u001b[0;24r\u001b8\u001b[1A\u001b[J\u001b[1;33mW: \u001b[0mTarget Packages (Packages) is configured multiple times in /etc/apt/sources.list:50 and /etc/apt/sources.list.d/cuda-ubuntu2004-x86_64.list:1\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "!apt update\n",
    "!DEBIAN_FRONTEND=noninteractive apt install -y cmake xz-utils zstd libz-dev"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Build library"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We downloaded the FasterTransformer code from Github. We will build this library manually as well as additional scripts that allow us to convert pre-trained files of the GPT-J into FT binary format. Let us create a `build` directory that will be used as the main directory for the built process. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/dli/task/FasterTransformer/build\n"
     ]
    }
   ],
   "source": [
    "!mkdir -p build\n",
    "%cd build "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let us build the library with the arg `-DBUILD_MULTI_GPU=ON ` that includes in the build pipeline multi-GPU cases like `GPT-J`. Please note, this step can take several minutes. Feel free to take a short break as you wait for the build process to complete."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "scrolled": true,
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Submodule '3rdparty/Megatron-LM' (https://github.com/NVIDIA/Megatron-LM.git) registered for path '../3rdparty/Megatron-LM'\n",
      "Submodule 'examples/tensorflow/bert/tensorflow_bert/bert' (https://github.com/google-research/bert.git) registered for path '../examples/tensorflow/bert/tensorflow_bert/bert'\n",
      "Cloning into '/dli/task/FasterTransformer/3rdparty/Megatron-LM'...\n",
      "Cloning into '/dli/task/FasterTransformer/examples/tensorflow/bert/tensorflow_bert/bert'...\n",
      "Submodule path '../3rdparty/Megatron-LM': checked out '90e0a0dd08159e1c95f4f9d99bb8687f327d36c3'\n",
      "Submodule path '../examples/tensorflow/bert/tensorflow_bert/bert': checked out 'eedf5716ce1268e56f0a50264a88cafad334ac61'\n",
      "-- The CXX compiler identification is GNU 9.3.0\n",
      "-- The CUDA compiler identification is NVIDIA 11.6.55\n",
      "-- Check for working CXX compiler: /usr/bin/c++\n",
      "-- Check for working CXX compiler: /usr/bin/c++ -- works\n",
      "-- Detecting CXX compiler ABI info\n",
      "-- Detecting CXX compiler ABI info - done\n",
      "-- Detecting CXX compile features\n",
      "-- Detecting CXX compile features - done\n",
      "-- Check for working CUDA compiler: /usr/local/cuda/bin/nvcc\n",
      "-- Check for working CUDA compiler: /usr/local/cuda/bin/nvcc -- works\n",
      "-- Detecting CUDA compiler ABI info\n",
      "-- Detecting CUDA compiler ABI info - done\n",
      "-- Looking for C++ include pthread.h\n",
      "-- Looking for C++ include pthread.h - found\n",
      "-- Performing Test CMAKE_HAVE_LIBC_PTHREAD\n",
      "-- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Failed\n",
      "-- Looking for pthread_create in pthreads\n",
      "-- Looking for pthread_create in pthreads - not found\n",
      "-- Looking for pthread_create in pthread\n",
      "-- Looking for pthread_create in pthread - found\n",
      "-- Found Threads: TRUE  \n",
      "-- Found CUDA: /usr/local/cuda (found suitable version \"11.6\", minimum required is \"10.2\") \n",
      "CUDA_VERSION 11 is greater or equal than 11, enable -DENABLE_BF16 flag\n",
      "-- Add DBUILD_MULTI_GPU, requires MPI and NCCL\n",
      "-- Found MPI_CXX: /opt/hpcx/ompi/lib/libmpi.so (found version \"3.1\") \n",
      "-- Found MPI: TRUE (found version \"3.1\")  \n",
      "-- Found NCCL: /usr/include  \n",
      "-- Determining NCCL version from /usr/include/nccl.h...\n",
      "-- Looking for NCCL_VERSION_CODE\n",
      "-- Looking for NCCL_VERSION_CODE - not found\n",
      "-- Found NCCL (include: /usr/include, library: /usr/lib/x86_64-linux-gnu/libnccl.so.2.11.4)\n",
      "-- Assign GPU architecture (sm=70,75,80,86)\n",
      "-- COMMON_HEADER_DIRS: /dli/task/FasterTransformer;/usr/local/cuda/include\n",
      "-- Configuring done\n",
      "-- Generating done\n",
      "-- Build files have been written to: /dli/task/FasterTransformer/build\n",
      "\u001b[35m\u001b[1mScanning dependencies of target cublasAlgoMap\u001b[0m\n",
      "\u001b[35m\u001b[1mScanning dependencies of target calibrate_quantize_weight_kernels\u001b[0m\n",
      "\u001b[35m\u001b[1mScanning dependencies of target nccl_utils\u001b[0m\n",
      "\u001b[35m\u001b[1mScanning dependencies of target matrix_vector_multiplication\u001b[0m\n",
      "\u001b[35m\u001b[1mScanning dependencies of target memory_utils\u001b[0m\n",
      "\u001b[35m\u001b[1mScanning dependencies of target custom_ar_kernels\u001b[0m\n",
      "\u001b[35m\u001b[1mScanning dependencies of target sampling_topk_kernels\u001b[0m\n",
      "\u001b[35m\u001b[1mScanning dependencies of target sampling_topp_kernels\u001b[0m\n",
      "\u001b[35m\u001b[1mScanning dependencies of target beam_search_topk_kernels\u001b[0m\n",
      "\u001b[35m\u001b[1mScanning dependencies of target sampling_penalty_kernels\u001b[0m\n",
      "\u001b[35m\u001b[1mScanning dependencies of target gpt_kernels\u001b[0m\n",
      "\u001b[35m\u001b[1mScanning dependencies of target beam_search_penalty_kernels\u001b[0m\n",
      "\u001b[35m\u001b[1mScanning dependencies of target activation_kernels\u001b[0m\n",
      "\u001b[35m\u001b[1mScanning dependencies of target unfused_attention_kernels\u001b[0m\n",
      "\u001b[35m\u001b[1mScanning dependencies of target stop_criteria\u001b[0m\n",
      "\u001b[35m\u001b[1mScanning dependencies of target ban_bad_words\u001b[0m\n",
      "\u001b[35m\u001b[1mScanning dependencies of target decoder_masked_multihead_attention\u001b[0m\n",
      "\u001b[35m\u001b[1mScanning dependencies of target logprob_kernels\u001b[0m\n",
      "\u001b[35m\u001b[1mScanning dependencies of target bert_preprocess_kernels\u001b[0m\n",
      "\u001b[35m\u001b[1mScanning dependencies of target layernorm_kernels\u001b[0m\n",
      "\u001b[35m\u001b[1mScanning dependencies of target add_residual_kernels\u001b[0m\n",
      "\u001b[35m\u001b[1mScanning dependencies of target online_softmax_beamsearch_kernels\u001b[0m\n",
      "\u001b[35m\u001b[1mScanning dependencies of target decoding_kernels\u001b[0m\n",
      "\u001b[35m\u001b[1mScanning dependencies of target word_list\u001b[0m\n",
      "\u001b[35m\u001b[1mScanning dependencies of target quantization_int8_kernels\u001b[0m\n",
      "\u001b[35m\u001b[1mScanning dependencies of target activation_int8_kernels\u001b[0m\n",
      "\u001b[35m\u001b[1mScanning dependencies of target layout_transformer_int8_kernels\u001b[0m\n",
      "\u001b[35m\u001b[1mScanning dependencies of target transpose_int8_kernels\u001b[0m\n",
      "\u001b[35m\u001b[1mScanning dependencies of target trt_fused_multi_head_attention\u001b[0m\n",
      "\u001b[35m\u001b[1mScanning dependencies of target layernorm_int8_kernels\u001b[0m\n",
      "\u001b[35m\u001b[1mScanning dependencies of target softmax_int8_kernels\u001b[0m\n",
      "\u001b[35m\u001b[1mScanning dependencies of target matrix_transpose_kernels\u001b[0m\n",
      "[  1%] \u001b[32mBuilding CUDA object src/fastertransformer/kernels/CMakeFiles/matrix_vector_multiplication.dir/matrix_vector_multiplication.cu.o\u001b[0m\n",
      "[  1%] \u001b[32mBuilding CUDA object src/fastertransformer/kernels/CMakeFiles/calibrate_quantize_weight_kernels.dir/calibrate_quantize_weight_kernels.cu.o\u001b[0m\n",
      "[  1%] \u001b[32mBuilding CUDA object src/fastertransformer/utils/CMakeFiles/memory_utils.dir/memory_utils.cu.o\u001b[0m\n",
      "[  1%] \u001b[32mBuilding CUDA object src/fastertransformer/kernels/CMakeFiles/custom_ar_kernels.dir/custom_ar_kernels.cu.o\u001b[0m\n",
      "[  1%] \u001b[32mBuilding CUDA object src/fastertransformer/kernels/CMakeFiles/sampling_penalty_kernels.dir/sampling_penalty_kernels.cu.o\u001b[0m\n",
      "[  1%] \u001b[32mBuilding CUDA object src/fastertransformer/kernels/CMakeFiles/activation_kernels.dir/activation_kernels.cu.o\u001b[0m\n",
      "[  1%] \u001b[32mBuilding CUDA object src/fastertransformer/kernels/CMakeFiles/gpt_kernels.dir/gpt_kernels.cu.o\u001b[0m\n",
      "[  1%] \u001b[32mBuilding CUDA object src/fastertransformer/kernels/CMakeFiles/stop_criteria.dir/stop_criteria_kernels.cu.o\u001b[0m\n",
      "[  2%] \u001b[32mBuilding CUDA object src/fastertransformer/kernels/CMakeFiles/beam_search_penalty_kernels.dir/beam_search_penalty_kernels.cu.o\u001b[0m\n",
      "[  2%] \u001b[32mBuilding CUDA object src/fastertransformer/kernels/CMakeFiles/unfused_attention_kernels.dir/unfused_attention_kernels.cu.o\u001b[0m\n",
      "[  2%] \u001b[32mBuilding CUDA object src/fastertransformer/kernels/CMakeFiles/ban_bad_words.dir/ban_bad_words.cu.o\u001b[0m\n",
      "[  2%] \u001b[32mBuilding CUDA object src/fastertransformer/kernels/CMakeFiles/decoder_masked_multihead_attention.dir/decoder_masked_multihead_attention.cu.o\u001b[0m\n",
      "[  2%] \u001b[32mBuilding CUDA object src/fastertransformer/kernels/CMakeFiles/bert_preprocess_kernels.dir/bert_preprocess_kernels.cu.o\u001b[0m\n",
      "[  2%] \u001b[32mBuilding CUDA object src/fastertransformer/kernels/CMakeFiles/layernorm_kernels.dir/layernorm_kernels.cu.o\u001b[0m\n",
      "[  2%] \u001b[32mBuilding CUDA object src/fastertransformer/kernels/CMakeFiles/add_residual_kernels.dir/add_residual_kernels.cu.o\u001b[0m\n",
      "[  4%] \u001b[32mBuilding CUDA object src/fastertransformer/kernels/CMakeFiles/quantization_int8_kernels.dir/quantization_int8_kernels.cu.o\u001b[0m\n",
      "[  4%] \u001b[32mBuilding CUDA object src/fastertransformer/kernels/CMakeFiles/layout_transformer_int8_kernels.dir/layout_transformer_int8_kernels.cu.o\u001b[0m\n",
      "[  4%] \u001b[32mBuilding CUDA object src/fastertransformer/kernels/CMakeFiles/activation_int8_kernels.dir/activation_int8_kernels.cu.o\u001b[0m\n",
      "[  4%] \u001b[32mBuilding CUDA object src/fastertransformer/kernels/CMakeFiles/transpose_int8_kernels.dir/transpose_int8_kernels.cu.o\u001b[0m\n",
      "[  4%] \u001b[32mBuilding CUDA object src/fastertransformer/kernels/CMakeFiles/decoding_kernels.dir/decoding_kernels.cu.o\u001b[0m\n",
      "[  4%] \u001b[32mBuilding CUDA object src/fastertransformer/kernels/CMakeFiles/softmax_int8_kernels.dir/softmax_int8_kernels.cu.o\u001b[0m\n",
      "[  4%] \u001b[32mBuilding CUDA object src/fastertransformer/kernels/CMakeFiles/matrix_transpose_kernels.dir/matrix_transpose_kernels.cu.o\u001b[0m\n",
      "[  4%] \u001b[32mBuilding CUDA object src/fastertransformer/kernels/CMakeFiles/layernorm_int8_kernels.dir/layernorm_int8_kernels.cu.o\u001b[0m\n",
      "[  4%] \u001b[32mBuilding CUDA object src/fastertransformer/kernels/CMakeFiles/beam_search_topk_kernels.dir/beam_search_topk_kernels.cu.o\u001b[0m\n",
      "[  4%] \u001b[32mBuilding CUDA object src/fastertransformer/kernels/CMakeFiles/sampling_topk_kernels.dir/sampling_topk_kernels.cu.o\u001b[0m\n",
      "[  6%] \u001b[32mBuilding CUDA object src/fastertransformer/kernels/CMakeFiles/sampling_topp_kernels.dir/sampling_topp_kernels.cu.o\u001b[0m\n",
      "[  6%] \u001b[32mBuilding CXX object src/fastertransformer/utils/CMakeFiles/nccl_utils.dir/nccl_utils.cc.o\u001b[0m\n",
      "[  6%] \u001b[32mBuilding CXX object src/fastertransformer/utils/CMakeFiles/cublasAlgoMap.dir/cublasAlgoMap.cc.o\u001b[0m\n",
      "[  6%] \u001b[32mBuilding CUDA object src/fastertransformer/kernels/CMakeFiles/logprob_kernels.dir/logprob_kernels.cu.o\u001b[0m\n",
      "[  6%] \u001b[32mBuilding CUDA object src/fastertransformer/kernels/CMakeFiles/online_softmax_beamsearch_kernels.dir/online_softmax_beamsearch_kernels.cu.o\u001b[0m\n",
      "[  6%] \u001b[32mBuilding CXX object src/fastertransformer/utils/CMakeFiles/word_list.dir/word_list.cc.o\u001b[0m\n",
      "[  7%] \u001b[32mBuilding CXX object 3rdparty/trt_fused_multihead_attention/CMakeFiles/trt_fused_multi_head_attention.dir/cudaDriverWrapper.cpp.o\u001b[0m\n",
      "[  7%] \u001b[32mBuilding CXX object 3rdparty/trt_fused_multihead_attention/CMakeFiles/trt_fused_multi_head_attention.dir/fused_mha_with_relPosBias_fp16_128_32_kernel.sm75.cpp.o\u001b[0m\n",
      "\u001b[01m\u001b[0m\u001b[01m/dli/task/FasterTransformer/src/fastertransformer/kernels/decoding_kernels.cu(390)\u001b[0m: \u001b[01;35mwarning\u001b[0m #550-D: variable \u001b[01m\"selected_beam_index\"\u001b[0m was set but never used\n",
      "\n",
      "\u001b[01m\u001b[0m\u001b[01m/dli/task/FasterTransformer/src/fastertransformer/kernels/matrix_vector_multiplication.cu(271)\u001b[0m: \u001b[01;35mwarning\u001b[0m #226-D: invalid format string conversion\n",
      "\n",
      "\u001b[01m\u001b[0m\u001b[01m/dli/task/FasterTransformer/src/fastertransformer/kernels/matrix_vector_multiplication.cu(142)\u001b[0m: \u001b[01;35mwarning\u001b[0m #177-D: variable \u001b[01m\"input_val\"\u001b[0m was declared but never referenced\n",
      "          detected during instantiation of \u001b[01m\"void fastertransformer::int8WeightPerChannelLdkMultiplicationLauncher(const int8_t *, const T *, const float *, T *, int, int, int, cudaStream_t) [with T=float]\"\u001b[0m \u001b[32m\n",
      "(325): here\u001b[0m\n",
      "\n",
      "\u001b[01m\u001b[0m\u001b[01m/dli/task/FasterTransformer/src/fastertransformer/kernels/matrix_vector_multiplication.cu(205)\u001b[0m: \u001b[01;35mwarning\u001b[0m #177-D: variable \u001b[01m\"input_val\"\u001b[0m was declared but never referenced\n",
      "          detected during instantiation of \u001b[01m\"void fastertransformer::int8WeightPerChannelLdkMultiplicationLauncher(const int8_t *, const T *, const float *, T *, int, int, int, cudaStream_t) [with T=float]\"\u001b[0m \u001b[32m\n",
      "(325): here\u001b[0m\n",
      "\n",
      "[  7%] \u001b[32m\u001b[1mLinking CUDA device code CMakeFiles/nccl_utils.dir/cmake_device_link.o\u001b[0m\n",
      "[  7%] \u001b[32m\u001b[1mLinking CUDA device code CMakeFiles/word_list.dir/cmake_device_link.o\u001b[0m\n",
      "[  7%] \u001b[32m\u001b[1mLinking CXX static library ../../../lib/libnccl_utils.a\u001b[0m\n",
      "[  7%] Built target nccl_utils\n",
      "[  7%] \u001b[32m\u001b[1mLinking CXX static library ../../../lib/libword_list.a\u001b[0m\n",
      "\u001b[35m\u001b[1mScanning dependencies of target longformer_kernels\u001b[0m\n",
      "[  7%] \u001b[32mBuilding CUDA object src/fastertransformer/kernels/CMakeFiles/longformer_kernels.dir/longformer_kernels.cu.o\u001b[0m\n",
      "[  7%] Built target word_list\n",
      "\u001b[35m\u001b[1mScanning dependencies of target add_bias_transpose_kernels\u001b[0m\n",
      "[  8%] \u001b[32mBuilding CUDA object src/fastertransformer/kernels/CMakeFiles/add_bias_transpose_kernels.dir/add_bias_transpose_kernels.cu.o\u001b[0m\n",
      "[  8%] \u001b[32mBuilding CXX object 3rdparty/trt_fused_multihead_attention/CMakeFiles/trt_fused_multi_head_attention.dir/fused_mha_with_relPosBias_fp16_128_32_kernel.sm80.cpp.o\u001b[0m\n",
      "[  8%] \u001b[32m\u001b[1mLinking CUDA device code CMakeFiles/cublasAlgoMap.dir/cmake_device_link.o\u001b[0m\n",
      "[  8%] \u001b[32m\u001b[1mLinking CXX static library ../../../lib/libcublasAlgoMap.a\u001b[0m\n",
      "[  8%] Built target cublasAlgoMap\n",
      "\u001b[35m\u001b[1mScanning dependencies of target nvtx_utils\u001b[0m\n",
      "[  8%] \u001b[32mBuilding CXX object src/fastertransformer/utils/CMakeFiles/nvtx_utils.dir/nvtx_utils.cc.o\u001b[0m\n",
      "\u001b[01m\u001b[0m\u001b[01m/dli/task/FasterTransformer/src/fastertransformer/kernels/sampling_topp_kernels.cu(532)\u001b[0m: \u001b[01;35mwarning\u001b[0m #177-D: variable \u001b[01m\"KEYS_PER_LDG\"\u001b[0m was declared but never referenced\n",
      "\n",
      "\u001b[01m\u001b[0m\u001b[01m/dli/task/FasterTransformer/src/fastertransformer/kernels/decoding_kernels.cu(390)\u001b[0m: \u001b[01;35mwarning\u001b[0m #550-D: variable \u001b[01m\"selected_beam_index\"\u001b[0m was set but never used\n",
      "\n",
      "[  8%] \u001b[32m\u001b[1mLinking CUDA device code CMakeFiles/nvtx_utils.dir/cmake_device_link.o\u001b[0m\n",
      "\u001b[01m\u001b[0m\u001b[01m/dli/task/FasterTransformer/src/fastertransformer/kernels/matrix_vector_multiplication.cu(271)\u001b[0m: \u001b[01;35mwarning\u001b[0m #226-D: invalid format string conversion\n",
      "\n",
      "\u001b[01m\u001b[0m\u001b[01m/dli/task/FasterTransformer/src/fastertransformer/kernels/matrix_vector_multiplication.cu(142)\u001b[0m: \u001b[01;35mwarning\u001b[0m #177-D: variable \u001b[01m\"input_val\"\u001b[0m was declared but never referenced\n",
      "          detected during instantiation of \u001b[01m\"void fastertransformer::int8WeightPerChannelLdkMultiplicationLauncher(const int8_t *, const T *, const float *, T *, int, int, int, cudaStream_t) [with T=float]\"\u001b[0m \u001b[32m\n",
      "(325): here\u001b[0m\n",
      "\n",
      "\u001b[01m\u001b[0m\u001b[01m/dli/task/FasterTransformer/src/fastertransformer/kernels/matrix_vector_multiplication.cu(205)\u001b[0m: \u001b[01;35mwarning\u001b[0m #177-D: variable \u001b[01m\"input_val\"\u001b[0m was declared but never referenced\n",
      "          detected during instantiation of \u001b[01m\"void fastertransformer::int8WeightPerChannelLdkMultiplicationLauncher(const int8_t *, const T *, const float *, T *, int, int, int, cudaStream_t) [with T=float]\"\u001b[0m \u001b[32m\n",
      "(325): here\u001b[0m\n",
      "\n",
      "[  9%] \u001b[32m\u001b[1mLinking CXX static library ../../../lib/libnvtx_utils.a\u001b[0m\n",
      "[  9%] Built target nvtx_utils\n",
      "[  9%] \u001b[32mBuilding CXX object 3rdparty/trt_fused_multihead_attention/CMakeFiles/trt_fused_multi_head_attention.dir/fused_mha_with_relPosBias_fp16_128_32_kernel.sm86.cpp.o\u001b[0m\n",
      "[  9%] \u001b[32mBuilding CXX object 3rdparty/trt_fused_multihead_attention/CMakeFiles/trt_fused_multi_head_attention.dir/fused_mha_with_relPosBias_fp16_256_32_kernel.sm75.cpp.o\u001b[0m\n",
      "[  9%] \u001b[32m\u001b[1mLinking CUDA device code CMakeFiles/quantization_int8_kernels.dir/cmake_device_link.o\u001b[0m\n",
      "[ 10%] \u001b[32m\u001b[1mLinking CUDA device code CMakeFiles/matrix_transpose_kernels.dir/cmake_device_link.o\u001b[0m\n",
      "[ 10%] \u001b[32m\u001b[1mLinking CUDA static library ../../../lib/libquantization_int8_kernels.a\u001b[0m\n",
      "[ 10%] \u001b[32m\u001b[1mLinking CUDA static library ../../../lib/libmatrix_transpose_kernels.a\u001b[0m\n",
      "[ 11%] \u001b[32mBuilding CXX object 3rdparty/trt_fused_multihead_attention/CMakeFiles/trt_fused_multi_head_attention.dir/fused_mha_with_relPosBias_fp16_256_32_kernel.sm80.cpp.o\u001b[0m\n",
      "[ 11%] Built target quantization_int8_kernels\n",
      "[ 11%] Built target matrix_transpose_kernels\n",
      "[ 11%] \u001b[32mBuilding CXX object 3rdparty/trt_fused_multihead_attention/CMakeFiles/trt_fused_multi_head_attention.dir/fused_mha_with_relPosBias_fp16_256_32_kernel.sm86.cpp.o\u001b[0m\n",
      "\u001b[35m\u001b[1mScanning dependencies of target gemm_func\u001b[0m\n",
      "[ 11%] \u001b[32mBuilding CXX object src/fastertransformer/utils/gemm_test/CMakeFiles/gemm_func.dir/gemm_func.cc.o\u001b[0m\n",
      "\u001b[35m\u001b[1mScanning dependencies of target encoder_igemm_func\u001b[0m\n",
      "[ 11%] \u001b[32m\u001b[1mLinking CUDA device code CMakeFiles/transpose_int8_kernels.dir/cmake_device_link.o\u001b[0m\n",
      "\u001b[01m\u001b[0m\u001b[01m/dli/task/FasterTransformer/src/fastertransformer/kernels/decoding_kernels.cu(390)\u001b[0m: \u001b[01;35mwarning\u001b[0m #550-D: variable \u001b[01m\"selected_beam_index\"\u001b[0m was set but never used\n",
      "\n",
      "[ 11%] \u001b[32mBuilding CXX object src/fastertransformer/utils/gemm_test/CMakeFiles/encoder_igemm_func.dir/encoder_igemm_func.cc.o\u001b[0m\n",
      "[ 11%] \u001b[32m\u001b[1mLinking CUDA static library ../../../lib/libtranspose_int8_kernels.a\u001b[0m\n",
      "[ 11%] \u001b[32m\u001b[1mLinking CUDA device code CMakeFiles/layout_transformer_int8_kernels.dir/cmake_device_link.o\u001b[0m\n",
      "[ 11%] Built target transpose_int8_kernels\n",
      "\u001b[35m\u001b[1mScanning dependencies of target vit_kernels\u001b[0m\n",
      "[ 11%] \u001b[32mBuilding CUDA object src/fastertransformer/kernels/CMakeFiles/vit_kernels.dir/vit_kernels.cu.o\u001b[0m\n",
      "[ 11%] \u001b[32m\u001b[1mLinking CUDA static library ../../../lib/liblayout_transformer_int8_kernels.a\u001b[0m\n",
      "[ 11%] Built target layout_transformer_int8_kernels\n",
      "\u001b[35m\u001b[1mScanning dependencies of target xlnet_attention_kernels\u001b[0m\n",
      "[ 12%] \u001b[32mBuilding CUDA object src/fastertransformer/kernels/CMakeFiles/xlnet_attention_kernels.dir/xlnet_attention_kernels.cu.o\u001b[0m\n",
      "[ 13%] \u001b[32m\u001b[1mLinking CUDA device code CMakeFiles/activation_int8_kernels.dir/cmake_device_link.o\u001b[0m\n",
      "\u001b[01m\u001b[0m\u001b[01m/dli/task/FasterTransformer/src/fastertransformer/kernels/matrix_vector_multiplication.cu(271)\u001b[0m: \u001b[01;35mwarning\u001b[0m #226-D: invalid format string conversion\n",
      "\n",
      "\u001b[01m\u001b[0m\u001b[01m/dli/task/FasterTransformer/src/fastertransformer/kernels/matrix_vector_multiplication.cu(142)\u001b[0m: \u001b[01;35mwarning\u001b[0m #177-D: variable \u001b[01m\"input_val\"\u001b[0m was declared but never referenced\n",
      "          detected during instantiation of \u001b[01m\"void fastertransformer::int8WeightPerChannelLdkMultiplicationLauncher(const int8_t *, const T *, const float *, T *, int, int, int, cudaStream_t) [with T=float]\"\u001b[0m \u001b[32m\n",
      "(325): here\u001b[0m\n",
      "\n",
      "\u001b[01m\u001b[0m\u001b[01m/dli/task/FasterTransformer/src/fastertransformer/kernels/matrix_vector_multiplication.cu(205)\u001b[0m: \u001b[01;35mwarning\u001b[0m #177-D: variable \u001b[01m\"input_val\"\u001b[0m was declared but never referenced\n",
      "          detected during instantiation of \u001b[01m\"void fastertransformer::int8WeightPerChannelLdkMultiplicationLauncher(const int8_t *, const T *, const float *, T *, int, int, int, cudaStream_t) [with T=float]\"\u001b[0m \u001b[32m\n",
      "(325): here\u001b[0m\n",
      "\n",
      "[ 13%] \u001b[32m\u001b[1mLinking CUDA static library ../../../lib/libactivation_int8_kernels.a\u001b[0m\n",
      "[ 13%] Built target activation_int8_kernels\n",
      "\u001b[35m\u001b[1mScanning dependencies of target quantize_weight\u001b[0m\n",
      "[ 13%] \u001b[32mBuilding CUDA object src/fastertransformer/kernels/CMakeFiles/quantize_weight.dir/quantize_weight.cu.o\u001b[0m\n",
      "[ 13%] \u001b[32mBuilding CXX object 3rdparty/trt_fused_multihead_attention/CMakeFiles/trt_fused_multi_head_attention.dir/fused_mha_with_relPosBias_fp16_64_32_kernel.sm75.cpp.o\u001b[0m\n",
      "[ 13%] \u001b[32mBuilding CXX object 3rdparty/trt_fused_multihead_attention/CMakeFiles/trt_fused_multi_head_attention.dir/fused_mha_with_relPosBias_fp16_64_32_kernel.sm80.cpp.o\u001b[0m\n",
      "[ 13%] \u001b[32mBuilding CXX object 3rdparty/trt_fused_multihead_attention/CMakeFiles/trt_fused_multi_head_attention.dir/fused_mha_with_relPosBias_fp16_64_32_kernel.sm86.cpp.o\u001b[0m\n",
      "[ 14%] \u001b[32mBuilding CXX object 3rdparty/trt_fused_multihead_attention/CMakeFiles/trt_fused_multi_head_attention.dir/fused_mha_with_relPosBias_int8_256_32_kernel.sm75.cpp.o\u001b[0m\n",
      "[ 14%] \u001b[32m\u001b[1mLinking CUDA device code CMakeFiles/add_bias_transpose_kernels.dir/cmake_device_link.o\u001b[0m\n",
      "\u001b[01m\u001b[0m\u001b[01m/dli/task/FasterTransformer/src/fastertransformer/kernels/decoding_kernels.cu(390)\u001b[0m: \u001b[01;35mwarning\u001b[0m #550-D: variable \u001b[01m\"selected_beam_index\"\u001b[0m was set but never used\n",
      "\n",
      "[ 14%] \u001b[32m\u001b[1mLinking CUDA static library ../../../lib/libadd_bias_transpose_kernels.a\u001b[0m\n",
      "[ 14%] Built target add_bias_transpose_kernels\n",
      "\u001b[35m\u001b[1mScanning dependencies of target dequantize_kernels\u001b[0m\n",
      "[ 15%] \u001b[32mBuilding CUDA object src/fastertransformer/kernels/CMakeFiles/dequantize_kernels.dir/dequantize_kernels.cu.o\u001b[0m\n",
      "[ 15%] \u001b[32m\u001b[1mLinking CUDA device code CMakeFiles/gemm_func.dir/cmake_device_link.o\u001b[0m\n",
      "[ 16%] \u001b[32m\u001b[1mLinking CUDA device code CMakeFiles/encoder_igemm_func.dir/cmake_device_link.o\u001b[0m\n",
      "[ 16%] \u001b[32m\u001b[1mLinking CXX static library ../../../../lib/libgemm_func.a\u001b[0m\n",
      "[ 16%] \u001b[32mBuilding CXX object 3rdparty/trt_fused_multihead_attention/CMakeFiles/trt_fused_multi_head_attention.dir/fused_mha_with_relPosBias_int8_256_32_kernel.sm80.cpp.o\u001b[0m\n",
      "[ 16%] Built target gemm_func\n",
      "\u001b[35m\u001b[1mScanning dependencies of target gen_relative_pos_bias\u001b[0m\n",
      "[ 16%] \u001b[32mBuilding CUDA object src/fastertransformer/kernels/CMakeFiles/gen_relative_pos_bias.dir/gen_relative_pos_bias.cu.o\u001b[0m\n",
      "[ 16%] \u001b[32m\u001b[1mLinking CXX static library ../../../../lib/libencoder_igemm_func.a\u001b[0m\n",
      "[ 16%] Built target encoder_igemm_func\n",
      "\u001b[35m\u001b[1mScanning dependencies of target transform_mask_kernels\u001b[0m\n",
      "[ 16%] \u001b[32mBuilding CUDA object src/fastertransformer/kernels/CMakeFiles/transform_mask_kernels.dir/transform_mask_kernels.cu.o\u001b[0m\n",
      "\u001b[35m\u001b[1mScanning dependencies of target unfused_attention_int8_kernels\u001b[0m\n",
      "[ 16%] \u001b[32mBuilding CUDA object src/fastertransformer/kernels/CMakeFiles/unfused_attention_int8_kernels.dir/unfused_attention_int8_kernels.cu.o\u001b[0m\n",
      "[ 16%] \u001b[32m\u001b[1mLinking CUDA device code CMakeFiles/ban_bad_words.dir/cmake_device_link.o\u001b[0m\n",
      "\u001b[01m\u001b[0m\u001b[01m/dli/task/FasterTransformer/src/fastertransformer/kernels/matrix_vector_multiplication.cu(271)\u001b[0m: \u001b[01;35mwarning\u001b[0m #226-D: invalid format string conversion\n",
      "\n",
      "\u001b[01m\u001b[0m\u001b[01m/dli/task/FasterTransformer/src/fastertransformer/kernels/matrix_vector_multiplication.cu(142)\u001b[0m: \u001b[01;35mwarning\u001b[0m #177-D: variable \u001b[01m\"input_val\"\u001b[0m was declared but never referenced\n",
      "          detected during instantiation of \u001b[01m\"void fastertransformer::int8WeightPerChannelLdkMultiplicationLauncher(const int8_t *, const T *, const float *, T *, int, int, int, cudaStream_t) [with T=float]\"\u001b[0m \u001b[32m\n",
      "(325): here\u001b[0m\n",
      "\n",
      "\u001b[01m\u001b[0m\u001b[01m/dli/task/FasterTransformer/src/fastertransformer/kernels/matrix_vector_multiplication.cu(205)\u001b[0m: \u001b[01;35mwarning\u001b[0m #177-D: variable \u001b[01m\"input_val\"\u001b[0m was declared but never referenced\n",
      "          detected during instantiation of \u001b[01m\"void fastertransformer::int8WeightPerChannelLdkMultiplicationLauncher(const int8_t *, const T *, const float *, T *, int, int, int, cudaStream_t) [with T=float]\"\u001b[0m \u001b[32m\n",
      "(325): here\u001b[0m\n",
      "\n",
      "[ 16%] \u001b[32m\u001b[1mLinking CUDA static library ../../../lib/libban_bad_words.a\u001b[0m\n",
      "[ 16%] Built target ban_bad_words\n",
      "\u001b[35m\u001b[1mScanning dependencies of target xlnet_preprocess_kernels\u001b[0m\n",
      "[ 16%] \u001b[32mBuilding CUDA object src/fastertransformer/kernels/CMakeFiles/xlnet_preprocess_kernels.dir/xlnet_preprocess_kernels.cu.o\u001b[0m\n",
      "[ 16%] \u001b[32mBuilding CXX object 3rdparty/trt_fused_multihead_attention/CMakeFiles/trt_fused_multi_head_attention.dir/fused_mha_with_relPosBias_int8_64_32_kernel.sm75.cpp.o\u001b[0m\n",
      "[ 16%] \u001b[32mBuilding CXX object 3rdparty/trt_fused_multihead_attention/CMakeFiles/trt_fused_multi_head_attention.dir/fused_mha_with_relPosBias_int8_64_32_kernel.sm80.cpp.o\u001b[0m\n",
      "[ 16%] \u001b[32m\u001b[1mLinking CUDA device code CMakeFiles/stop_criteria.dir/cmake_device_link.o\u001b[0m\n",
      "[ 16%] \u001b[32m\u001b[1mLinking CUDA static library ../../../lib/libstop_criteria.a\u001b[0m\n",
      "[ 16%] Built target stop_criteria\n",
      "\u001b[35m\u001b[1mScanning dependencies of target reverse_roll_kernels\u001b[0m\n",
      "[ 16%] \u001b[32mBuilding CUDA object src/fastertransformer/kernels/CMakeFiles/reverse_roll_kernels.dir/reverse_roll_kernels.cu.o\u001b[0m\n",
      "[ 17%] \u001b[32mBuilding CXX object 3rdparty/trt_fused_multihead_attention/CMakeFiles/trt_fused_multi_head_attention.dir/fused_multihead_attention_fp16_128_64_kernel.sm75.cpp.o\u001b[0m\n",
      "[ 18%] \u001b[32m\u001b[1mLinking CUDA device code CMakeFiles/sampling_penalty_kernels.dir/cmake_device_link.o\u001b[0m\n",
      "[ 18%] \u001b[32mBuilding CXX object 3rdparty/trt_fused_multihead_attention/CMakeFiles/trt_fused_multi_head_attention.dir/fused_multihead_attention_fp16_128_64_kernel.sm80.cpp.o\u001b[0m\n",
      "[ 18%] \u001b[32m\u001b[1mLinking CUDA static library ../../../lib/libsampling_penalty_kernels.a\u001b[0m\n",
      "[ 18%] Built target sampling_penalty_kernels\n",
      "\u001b[35m\u001b[1mScanning dependencies of target cnpy\u001b[0m\n",
      "[ 18%] \u001b[32mBuilding CXX object examples/cpp/xlnet/CMakeFiles/cnpy.dir/cnpy.cpp.o\u001b[0m\n",
      "[ 18%] \u001b[32mBuilding CXX object 3rdparty/trt_fused_multihead_attention/CMakeFiles/trt_fused_multi_head_attention.dir/fused_multihead_attention_fp16_384_64_kernel.sm75.cpp.o\u001b[0m\n",
      "[ 18%] \u001b[32mBuilding CXX object 3rdparty/trt_fused_multihead_attention/CMakeFiles/trt_fused_multi_head_attention.dir/fused_multihead_attention_fp16_384_64_kernel.sm80.cpp.o\u001b[0m\n",
      "[ 18%] \u001b[32m\u001b[1mLinking CUDA device code CMakeFiles/activation_kernels.dir/cmake_device_link.o\u001b[0m\n",
      "[ 18%] \u001b[32m\u001b[1mLinking CUDA device code CMakeFiles/quantize_weight.dir/cmake_device_link.o\u001b[0m\n",
      "[ 18%] \u001b[32m\u001b[1mLinking CUDA static library ../../../lib/libactivation_kernels.a\u001b[0m\n",
      "[ 18%] \u001b[32m\u001b[1mLinking CUDA device code CMakeFiles/add_residual_kernels.dir/cmake_device_link.o\u001b[0m\n",
      "[ 19%] \u001b[32m\u001b[1mLinking CUDA static library ../../../lib/libquantize_weight.a\u001b[0m\n",
      "[ 19%] \u001b[32mBuilding CXX object 3rdparty/trt_fused_multihead_attention/CMakeFiles/trt_fused_multi_head_attention.dir/fused_multihead_attention_fp16_64_64_kernel.sm75.cpp.o\u001b[0m\n",
      "[ 19%] Built target activation_kernels\n",
      "[ 20%] \u001b[32mBuilding CXX object 3rdparty/trt_fused_multihead_attention/CMakeFiles/trt_fused_multi_head_attention.dir/fused_multihead_attention_fp16_64_64_kernel.sm80.cpp.o\u001b[0m\n",
      "[ 20%] Built target quantize_weight\n",
      "\u001b[35m\u001b[1mScanning dependencies of target gpt_example_utils\u001b[0m\n",
      "[ 21%] \u001b[32m\u001b[1mLinking CUDA static library ../../../lib/libadd_residual_kernels.a\u001b[0m\n",
      "[ 21%] \u001b[32mBuilding CXX object examples/cpp/multi_gpu_gpt/CMakeFiles/gpt_example_utils.dir/gpt_example_utils.cc.o\u001b[0m\n",
      "[ 21%] \u001b[32m\u001b[1mLinking CUDA device code CMakeFiles/bert_preprocess_kernels.dir/cmake_device_link.o\u001b[0m\n",
      "[ 21%] \u001b[32m\u001b[1mLinking CUDA device code CMakeFiles/calibrate_quantize_weight_kernels.dir/cmake_device_link.o\u001b[0m\n",
      "[ 21%] Built target add_residual_kernels\n",
      "[ 21%] \u001b[32mBuilding CXX object 3rdparty/trt_fused_multihead_attention/CMakeFiles/trt_fused_multi_head_attention.dir/fused_multihead_attention_fp16_96_64_kernel.sm75.cpp.o\u001b[0m\n",
      "[ 21%] \u001b[32mBuilding CXX object 3rdparty/trt_fused_multihead_attention/CMakeFiles/trt_fused_multi_head_attention.dir/fused_multihead_attention_fp16_96_64_kernel.sm80.cpp.o\u001b[0m\n",
      "[ 21%] \u001b[32m\u001b[1mLinking CUDA static library ../../../lib/libbert_preprocess_kernels.a\u001b[0m\n",
      "[ 21%] \u001b[32m\u001b[1mLinking CUDA static library ../../../lib/libcalibrate_quantize_weight_kernels.a\u001b[0m\n",
      "[ 21%] Built target bert_preprocess_kernels\n",
      "\u001b[35m\u001b[1mScanning dependencies of target cublasMMWrapper\u001b[0m\n",
      "[ 21%] Built target calibrate_quantize_weight_kernels\n",
      "\u001b[35m\u001b[1mScanning dependencies of target BaseSamplingLayer\u001b[0m\n",
      "\u001b[35m\u001b[1mScanning dependencies of target encoder_gemm_func\u001b[0m\n",
      "[ 21%] \u001b[32mBuilding CXX object 3rdparty/trt_fused_multihead_attention/CMakeFiles/trt_fused_multi_head_attention.dir/fused_multihead_attention_int8_128_64_kernel.sm75.cpp.o\u001b[0m\n",
      "[ 21%] \u001b[32mBuilding CXX object src/fastertransformer/utils/CMakeFiles/cublasMMWrapper.dir/cublasMMWrapper.cc.o\u001b[0m\n",
      "[ 21%] \u001b[32mBuilding CXX object src/fastertransformer/utils/gemm_test/CMakeFiles/encoder_gemm_func.dir/encoder_gemm_func.cc.o\u001b[0m\n",
      "[ 21%] \u001b[32mBuilding CXX object src/fastertransformer/layers/sampling_layers/CMakeFiles/BaseSamplingLayer.dir/BaseSamplingLayer.cc.o\u001b[0m\n",
      "\u001b[01m\u001b[K/dli/task/FasterTransformer/src/fastertransformer/kernels/custom_ar_kernels.cu:\u001b[m\u001b[K In function ‘\u001b[01m\u001b[Kvoid fastertransformer::kernelLaunchConfig(int&, int&, size_t, int, size_t)\u001b[m\u001b[K’:\n",
      "\u001b[01m\u001b[K/dli/task/FasterTransformer/src/fastertransformer/kernels/custom_ar_kernels.cu:303:10:\u001b[m\u001b[K \u001b[01;35m\u001b[Kwarning: \u001b[m\u001b[Kcomparison of integer expressions of different signedness: ‘\u001b[01m\u001b[Ksize_t\u001b[m\u001b[K’ {aka ‘\u001b[01m\u001b[Klong unsigned int\u001b[m\u001b[K’} and ‘\u001b[01m\u001b[Kint\u001b[m\u001b[K’ [\u001b[01;35m\u001b[K-Wsign-compare\u001b[m\u001b[K]\n",
      "  303 |     \u001b[01;35m\u001b[K        if (elts < (elts_per_th\u001b[m\u001b[Kread * DEFAULT_BLOCK_SIZE)) {  // local reduce\n",
      "      |          \u001b[01;35m\u001b[K^\u001b[m\u001b[K  \u001b[01;35m\u001b[K~~~~~~~~~~~~~~~~~~~~~~~\u001b[m\u001b[K\n",
      "[ 21%] \u001b[32mBuilding CXX object 3rdparty/trt_fused_multihead_attention/CMakeFiles/trt_fused_multi_head_attention.dir/fused_multihead_attention_int8_128_64_kernel.sm80.cpp.o\u001b[0m\n",
      "[ 21%] \u001b[32m\u001b[1mLinking CUDA device code CMakeFiles/decoding_kernels.dir/cmake_device_link.o\u001b[0m\n",
      "[ 21%] \u001b[32m\u001b[1mLinking CXX static library ../../../lib/libgpt_example_utils.a\u001b[0m\n",
      "[ 22%] \u001b[32mBuilding CXX object 3rdparty/trt_fused_multihead_attention/CMakeFiles/trt_fused_multi_head_attention.dir/fused_multihead_attention_int8_384_64_kernel.sm75.cpp.o\u001b[0m\n",
      "[ 22%] Built target gpt_example_utils\n",
      "[ 22%] \u001b[32m\u001b[1mLinking CUDA static library ../../../lib/libdecoding_kernels.a\u001b[0m\n",
      "\u001b[35m\u001b[1mScanning dependencies of target decoding_gemm_func\u001b[0m\n",
      "[ 22%] \u001b[32m\u001b[1mLinking CUDA device code CMakeFiles/custom_ar_kernels.dir/cmake_device_link.o\u001b[0m\n",
      "[ 22%] \u001b[32mBuilding CXX object src/fastertransformer/utils/gemm_test/CMakeFiles/decoding_gemm_func.dir/decoding_gemm_func.cc.o\u001b[0m\n",
      "[ 22%] Built target decoding_kernels\n",
      "\u001b[35m\u001b[1mScanning dependencies of target gpt_gemm_func\u001b[0m\n",
      "[ 22%] \u001b[32mBuilding CXX object 3rdparty/trt_fused_multihead_attention/CMakeFiles/trt_fused_multi_head_attention.dir/fused_multihead_attention_int8_384_64_kernel.sm80.cpp.o\u001b[0m\n",
      "[ 22%] \u001b[32mBuilding CXX object src/fastertransformer/utils/gemm_test/CMakeFiles/gpt_gemm_func.dir/gpt_gemm_func.cc.o\u001b[0m\n",
      "[ 23%] \u001b[32m\u001b[1mLinking CUDA static library ../../../lib/libcustom_ar_kernels.a\u001b[0m\n",
      "[ 23%] \u001b[32mBuilding CXX object 3rdparty/trt_fused_multihead_attention/CMakeFiles/trt_fused_multi_head_attention.dir/fused_multihead_attention_v2_fp16_128_64_kernel.sm70.cpp.o\u001b[0m\n",
      "[ 23%] Built target custom_ar_kernels\n",
      "\u001b[35m\u001b[1mScanning dependencies of target t5_gemm_func\u001b[0m\n",
      "[ 24%] \u001b[32mBuilding CXX object src/fastertransformer/utils/gemm_test/CMakeFiles/t5_gemm_func.dir/t5_gemm_func.cc.o\u001b[0m\n",
      "[ 24%] \u001b[32mBuilding CXX object 3rdparty/trt_fused_multihead_attention/CMakeFiles/trt_fused_multi_head_attention.dir/fused_multihead_attention_v2_fp16_128_64_kernel.sm75.cpp.o\u001b[0m\n",
      "/dli/task/FasterTransformer/src/fastertransformer/kernels/matrix_vector_multiplication.cu: In instantiation of ‘\u001b[01m\u001b[Kvoid fastertransformer::int8WeightPerChannelLdkMultiplicationLauncher(const int8_t*, const T*, const float*, T*, int, int, int, cudaStream_t) [with T = float; int8_t = signed char; cudaStream_t = CUstream_st*]\u001b[m\u001b[K’:\n",
      "\u001b[01m\u001b[K/dli/task/FasterTransformer/src/fastertransformer/kernels/matrix_vector_multiplication.cu:325:205:\u001b[m\u001b[K   required from here\n",
      "\u001b[01m\u001b[K/dli/task/FasterTransformer/src/fastertransformer/kernels/matrix_vector_multiplication.cu:269:8:\u001b[m\u001b[K \u001b[01;35m\u001b[Kwarning: \u001b[m\u001b[Kconversion lacks type at end of format [\u001b[01;35m\u001b[K-Wformat=\u001b[m\u001b[K]\n",
      "  269 |        \u001b[01;35m\u001b[K printf(\"[ERROR][int8WeightPerChannelLdkMultiplicationLauncher] (%d % %d != 0) || (%d % 4 !\u001b[m\u001b[K= 0).\\n\",\n",
      "      |        \u001b[01;35m\u001b[K^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\u001b[m\u001b[K\n",
      "\u001b[01m\u001b[K/dli/task/FasterTransformer/src/fastertransformer/kernels/matrix_vector_multiplication.cu:269:8:\u001b[m\u001b[K \u001b[01;35m\u001b[Kwarning: \u001b[m\u001b[Kunknown conversion type character ‘\u001b[01m\u001b[K \u001b[m\u001b[K’ in format [\u001b[01;35m\u001b[K-Wformat=\u001b[m\u001b[K]\n",
      "\u001b[01m\u001b[K/dli/task/FasterTransformer/src/fastertransformer/kernels/matrix_vector_multiplication.cu:288:26:\u001b[m\u001b[K \u001b[01;35m\u001b[Kwarning: \u001b[m\u001b[Kcomparison of integer expressions of different signedness: ‘\u001b[01m\u001b[Kunsigned int\u001b[m\u001b[K’ and ‘\u001b[01m\u001b[Kconst int\u001b[m\u001b[K’ [\u001b[01;35m\u001b[K-Wsign-compare\u001b[m\u001b[K]\n",
      "  288 |     whi\u001b[01;35m\u001b[Kle (block.x * 4 > k) \u001b[m\u001b[K{\n",
      "      |        \u001b[01;35m\u001b[K~~~~~~~~~~~~~~~~~~^~~\u001b[m\u001b[K\n",
      "/dli/task/FasterTransformer/src/fastertransformer/kernels/matrix_vector_multiplication.cu: In instantiation of ‘\u001b[01m\u001b[Kvoid fastertransformer::int8WeightPerChannelLdkMultiplicationLauncher(const int8_t*, const T*, const float*, T*, int, int, int, cudaStream_t) [with T = __half; int8_t = signed char; cudaStream_t = CUstream_st*]\u001b[m\u001b[K’:\n",
      "\u001b[01m\u001b[K/dli/task/FasterTransformer/src/fastertransformer/kernels/matrix_vector_multiplication.cu:334:203:\u001b[m\u001b[K   required from here\n",
      "\u001b[01m\u001b[K/dli/task/FasterTransformer/src/fastertransformer/kernels/matrix_vector_multiplication.cu:269:8:\u001b[m\u001b[K \u001b[01;35m\u001b[Kwarning: \u001b[m\u001b[Kconversion lacks type at end of format [\u001b[01;35m\u001b[K-Wformat=\u001b[m\u001b[K]\n",
      "  269 |        \u001b[01;35m\u001b[K printf(\"[ERROR][int8WeightPerChannelLdkMultiplicationLauncher] (%d % %d != 0) || (%d % 4 !\u001b[m\u001b[K= 0).\\n\",\n",
      "      |        \u001b[01;35m\u001b[K^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\u001b[m\u001b[K\n",
      "\u001b[01m\u001b[K/dli/task/FasterTransformer/src/fastertransformer/kernels/matrix_vector_multiplication.cu:269:8:\u001b[m\u001b[K \u001b[01;35m\u001b[Kwarning: \u001b[m\u001b[Kunknown conversion type character ‘\u001b[01m\u001b[K \u001b[m\u001b[K’ in format [\u001b[01;35m\u001b[K-Wformat=\u001b[m\u001b[K]\n",
      "\u001b[01m\u001b[K/dli/task/FasterTransformer/src/fastertransformer/kernels/matrix_vector_multiplication.cu:288:26:\u001b[m\u001b[K \u001b[01;35m\u001b[Kwarning: \u001b[m\u001b[Kcomparison of integer expressions of different signedness: ‘\u001b[01m\u001b[Kunsigned int\u001b[m\u001b[K’ and ‘\u001b[01m\u001b[Kconst int\u001b[m\u001b[K’ [\u001b[01;35m\u001b[K-Wsign-compare\u001b[m\u001b[K]\n",
      "  288 |     whi\u001b[01;35m\u001b[Kle (block.x * 4 > k) \u001b[m\u001b[K{\n",
      "      |        \u001b[01;35m\u001b[K~~~~~~~~~~~~~~~~~~^~~\u001b[m\u001b[K\n",
      "/dli/task/FasterTransformer/src/fastertransformer/kernels/matrix_vector_multiplication.cu: In instantiation of ‘\u001b[01m\u001b[Kvoid fastertransformer::int8WeightPerChannelLdkMultiplicationLauncher(const int8_t*, const T*, const float*, T*, int, int, int, cudaStream_t) [with T = __nv_bfloat16; int8_t = signed char; cudaStream_t = CUstream_st*]\u001b[m\u001b[K’:\n",
      "\u001b[01m\u001b[K/dli/task/FasterTransformer/src/fastertransformer/kernels/matrix_vector_multiplication.cu:344:221:\u001b[m\u001b[K   required from here\n",
      "\u001b[01m\u001b[K/dli/task/FasterTransformer/src/fastertransformer/kernels/matrix_vector_multiplication.cu:269:8:\u001b[m\u001b[K \u001b[01;35m\u001b[Kwarning: \u001b[m\u001b[Kconversion lacks type at end of format [\u001b[01;35m\u001b[K-Wformat=\u001b[m\u001b[K]\n",
      "  269 |        \u001b[01;35m\u001b[K printf(\"[ERROR][int8WeightPerChannelLdkMultiplicationLauncher] (%d % %d != 0) || (%d % 4 !\u001b[m\u001b[K= 0).\\n\",\n",
      "      |        \u001b[01;35m\u001b[K^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\u001b[m\u001b[K\n",
      "\u001b[01m\u001b[K/dli/task/FasterTransformer/src/fastertransformer/kernels/matrix_vector_multiplication.cu:269:8:\u001b[m\u001b[K \u001b[01;35m\u001b[Kwarning: \u001b[m\u001b[Kunknown conversion type character ‘\u001b[01m\u001b[K \u001b[m\u001b[K’ in format [\u001b[01;35m\u001b[K-Wformat=\u001b[m\u001b[K]\n",
      "\u001b[01m\u001b[K/dli/task/FasterTransformer/src/fastertransformer/kernels/matrix_vector_multiplication.cu:288:26:\u001b[m\u001b[K \u001b[01;35m\u001b[Kwarning: \u001b[m\u001b[Kcomparison of integer expressions of different signedness: ‘\u001b[01m\u001b[Kunsigned int\u001b[m\u001b[K’ and ‘\u001b[01m\u001b[Kconst int\u001b[m\u001b[K’ [\u001b[01;35m\u001b[K-Wsign-compare\u001b[m\u001b[K]\n",
      "  288 |     whi\u001b[01;35m\u001b[Kle (block.x * 4 > k) \u001b[m\u001b[K{\n",
      "      |        \u001b[01;35m\u001b[K~~~~~~~~~~~~~~~~~~^~~\u001b[m\u001b[K\n",
      "[ 24%] \u001b[32m\u001b[1mLinking CUDA device code CMakeFiles/xlnet_attention_kernels.dir/cmake_device_link.o\u001b[0m\n",
      "[ 24%] \u001b[32m\u001b[1mLinking CUDA static library ../../../lib/libxlnet_attention_kernels.a\u001b[0m\n",
      "[ 24%] Built target xlnet_attention_kernels\n",
      "\u001b[35m\u001b[1mScanning dependencies of target swin_igemm_func\u001b[0m\n",
      "[ 24%] \u001b[32mBuilding CXX object src/fastertransformer/utils/gemm_test/CMakeFiles/swin_igemm_func.dir/swin_igemm_func.cc.o\u001b[0m\n",
      "[ 24%] \u001b[32mBuilding CXX object 3rdparty/trt_fused_multihead_attention/CMakeFiles/trt_fused_multi_head_attention.dir/fused_multihead_attention_v2_fp16_128_64_kernel.sm80.cpp.o\u001b[0m\n",
      "[ 24%] \u001b[32m\u001b[1mLinking CUDA device code CMakeFiles/matrix_vector_multiplication.dir/cmake_device_link.o\u001b[0m\n",
      "[ 25%] \u001b[32m\u001b[1mLinking CUDA static library ../../../lib/libmatrix_vector_multiplication.a\u001b[0m\n",
      "[ 25%] Built target matrix_vector_multiplication\n",
      "\u001b[35m\u001b[1mScanning dependencies of target xlnet_gemm_func\u001b[0m\n",
      "[ 25%] \u001b[32mBuilding CXX object src/fastertransformer/utils/gemm_test/CMakeFiles/xlnet_gemm_func.dir/xlnet_gemm_func.cc.o\u001b[0m\n",
      "[ 26%] \u001b[32m\u001b[1mLinking CUDA device code CMakeFiles/encoder_gemm_func.dir/cmake_device_link.o\u001b[0m\n",
      "[ 27%] \u001b[32mBuilding CXX object 3rdparty/trt_fused_multihead_attention/CMakeFiles/trt_fused_multi_head_attention.dir/fused_multihead_attention_v2_fp16_128_64_kernel.sm86.cpp.o\u001b[0m\n",
      "\u001b[35m\u001b[1mScanning dependencies of target swin_gemm_func\u001b[0m\n",
      "[ 27%] \u001b[32m\u001b[1mLinking CUDA device code CMakeFiles/cublasMMWrapper.dir/cmake_device_link.o\u001b[0m\n",
      "[ 27%] \u001b[32m\u001b[1mLinking CXX static library ../../../../lib/libencoder_gemm_func.a\u001b[0m\n",
      "[ 27%] \u001b[32mBuilding CXX object src/fastertransformer/utils/gemm_test/CMakeFiles/swin_gemm_func.dir/swin_gemm_func.cc.o\u001b[0m\n",
      "[ 27%] Built target encoder_gemm_func\n",
      "[ 27%] \u001b[32mBuilding CXX object 3rdparty/trt_fused_multihead_attention/CMakeFiles/trt_fused_multi_head_attention.dir/fused_multihead_attention_v2_fp16_256_64_kernel.sm70.cpp.o\u001b[0m\n",
      "[ 27%] \u001b[32m\u001b[1mLinking CXX static library ../../../lib/libcublasMMWrapper.a\u001b[0m\n",
      "[ 28%] \u001b[32m\u001b[1mLinking CUDA device code CMakeFiles/gpt_kernels.dir/cmake_device_link.o\u001b[0m\n",
      "[ 28%] Built target cublasMMWrapper\n",
      "[ 28%] \u001b[32mBuilding CXX object 3rdparty/trt_fused_multihead_attention/CMakeFiles/trt_fused_multi_head_attention.dir/fused_multihead_attention_v2_fp16_256_64_kernel.sm75.cpp.o\u001b[0m\n",
      "[ 28%] \u001b[32m\u001b[1mLinking CUDA static library ../../../lib/libgpt_kernels.a\u001b[0m\n",
      "\u001b[35m\u001b[1mScanning dependencies of target cublasINT8MMWrapper\u001b[0m\n",
      "[ 28%] \u001b[32mBuilding CXX object src/fastertransformer/utils/CMakeFiles/cublasINT8MMWrapper.dir/cublasINT8MMWrapper.cc.o\u001b[0m\n",
      "[ 28%] Built target gpt_kernels\n",
      "[ 28%] \u001b[32mBuilding CXX object 3rdparty/trt_fused_multihead_attention/CMakeFiles/trt_fused_multi_head_attention.dir/fused_multihead_attention_v2_fp16_256_64_kernel.sm80.cpp.o\u001b[0m\n",
      "[ 29%] \u001b[32m\u001b[1mLinking CUDA device code CMakeFiles/softmax_int8_kernels.dir/cmake_device_link.o\u001b[0m\n",
      "[ 30%] \u001b[32mBuilding CXX object 3rdparty/trt_fused_multihead_attention/CMakeFiles/trt_fused_multi_head_attention.dir/fused_multihead_attention_v2_fp16_256_64_kernel.sm86.cpp.o\u001b[0m\n",
      "[ 31%] \u001b[32m\u001b[1mLinking CUDA device code CMakeFiles/decoding_gemm_func.dir/cmake_device_link.o\u001b[0m\n",
      "[ 31%] \u001b[32m\u001b[1mLinking CUDA static library ../../../lib/libsoftmax_int8_kernels.a\u001b[0m\n",
      "[ 31%] \u001b[32mBuilding CXX object 3rdparty/trt_fused_multihead_attention/CMakeFiles/trt_fused_multi_head_attention.dir/fused_multihead_attention_v2_fp16_384_64_kernel.sm70.cpp.o\u001b[0m\n",
      "[ 31%] Built target softmax_int8_kernels\n",
      "[ 31%] \u001b[32m\u001b[1mLinking CXX static library ../../../../lib/libdecoding_gemm_func.a\u001b[0m\n",
      "[ 31%] \u001b[32mBuilding CXX object 3rdparty/trt_fused_multihead_attention/CMakeFiles/trt_fused_multi_head_attention.dir/fused_multihead_attention_v2_fp16_384_64_kernel.sm75.cpp.o\u001b[0m\n",
      "[ 31%] Built target decoding_gemm_func\n",
      "[ 31%] \u001b[32mBuilding CXX object 3rdparty/trt_fused_multihead_attention/CMakeFiles/trt_fused_multi_head_attention.dir/fused_multihead_attention_v2_fp16_384_64_kernel.sm80.cpp.o\u001b[0m\n",
      "[ 31%] \u001b[32mBuilding CXX object 3rdparty/trt_fused_multihead_attention/CMakeFiles/trt_fused_multi_head_attention.dir/fused_multihead_attention_v2_fp16_384_64_kernel.sm86.cpp.o\u001b[0m\n",
      "[ 31%] \u001b[32m\u001b[1mLinking CUDA device code CMakeFiles/gpt_gemm_func.dir/cmake_device_link.o\u001b[0m\n",
      "[ 31%] \u001b[32m\u001b[1mLinking CUDA device code CMakeFiles/swin_igemm_func.dir/cmake_device_link.o\u001b[0m\n",
      "[ 32%] \u001b[32mBuilding CXX object 3rdparty/trt_fused_multihead_attention/CMakeFiles/trt_fused_multi_head_attention.dir/fused_multihead_attention_v2_fp16_64_64_kernel.sm70.cpp.o\u001b[0m\n",
      "[ 32%] \u001b[32m\u001b[1mLinking CXX static library ../../../../lib/libgpt_gemm_func.a\u001b[0m\n",
      "[ 32%] \u001b[32mBuilding CXX object 3rdparty/trt_fused_multihead_attention/CMakeFiles/trt_fused_multi_head_attention.dir/fused_multihead_attention_v2_fp16_64_64_kernel.sm75.cpp.o\u001b[0m\n",
      "[ 32%] \u001b[32m\u001b[1mLinking CXX static library ../../../../lib/libswin_igemm_func.a\u001b[0m\n",
      "[ 32%] Built target gpt_gemm_func\n",
      "[ 32%] \u001b[32mBuilding CXX object 3rdparty/trt_fused_multihead_attention/CMakeFiles/trt_fused_multi_head_attention.dir/fused_multihead_attention_v2_fp16_64_64_kernel.sm80.cpp.o\u001b[0m\n",
      "[ 32%] Built target swin_igemm_func\n",
      "[ 32%] \u001b[32mBuilding CXX object 3rdparty/trt_fused_multihead_attention/CMakeFiles/trt_fused_multi_head_attention.dir/fused_multihead_attention_v2_fp16_64_64_kernel.sm86.cpp.o\u001b[0m\n",
      "[ 32%] \u001b[32mBuilding CXX object 3rdparty/trt_fused_multihead_attention/CMakeFiles/trt_fused_multi_head_attention.dir/fused_multihead_attention_v2_fp16_96_64_kernel.sm70.cpp.o\u001b[0m\n",
      "[ 33%] \u001b[32mBuilding CXX object 3rdparty/trt_fused_multihead_attention/CMakeFiles/trt_fused_multi_head_attention.dir/fused_multihead_attention_v2_fp16_96_64_kernel.sm75.cpp.o\u001b[0m\n",
      "[ 33%] \u001b[32mBuilding CXX object 3rdparty/trt_fused_multihead_attention/CMakeFiles/trt_fused_multi_head_attention.dir/fused_multihead_attention_v2_fp16_96_64_kernel.sm80.cpp.o\u001b[0m\n",
      "[ 33%] \u001b[32mBuilding CXX object 3rdparty/trt_fused_multihead_attention/CMakeFiles/trt_fused_multi_head_attention.dir/fused_multihead_attention_v2_fp16_96_64_kernel.sm86.cpp.o\u001b[0m\n",
      "[ 33%] \u001b[32mBuilding CXX object 3rdparty/trt_fused_multihead_attention/CMakeFiles/trt_fused_multi_head_attention.dir/fused_multihead_attention_v2_int8_128_64_kernel.sm72.cpp.o\u001b[0m\n",
      "[ 33%] \u001b[32m\u001b[1mLinking CUDA device code CMakeFiles/t5_gemm_func.dir/cmake_device_link.o\u001b[0m\n",
      "[ 34%] \u001b[32mBuilding CXX object 3rdparty/trt_fused_multihead_attention/CMakeFiles/trt_fused_multi_head_attention.dir/fused_multihead_attention_v2_int8_128_64_kernel.sm75.cpp.o\u001b[0m\n",
      "[ 34%] \u001b[32mBuilding CXX object 3rdparty/trt_fused_multihead_attention/CMakeFiles/trt_fused_multi_head_attention.dir/fused_multihead_attention_v2_int8_128_64_kernel.sm80.cpp.o\u001b[0m\n",
      "[ 34%] \u001b[32mBuilding CXX object 3rdparty/trt_fused_multihead_attention/CMakeFiles/trt_fused_multi_head_attention.dir/fused_multihead_attention_v2_int8_128_64_kernel.sm86.cpp.o\u001b[0m\n",
      "[ 34%] \u001b[32m\u001b[1mLinking CXX static library ../../../../lib/libt5_gemm_func.a\u001b[0m\n",
      "[ 35%] \u001b[32m\u001b[1mLinking CUDA device code CMakeFiles/BaseSamplingLayer.dir/cmake_device_link.o\u001b[0m\n",
      "[ 35%] Built target t5_gemm_func\n",
      "[ 35%] \u001b[32mBuilding CXX object 3rdparty/trt_fused_multihead_attention/CMakeFiles/trt_fused_multi_head_attention.dir/fused_multihead_attention_v2_int8_192_64_kernel.sm72.cpp.o\u001b[0m\n",
      "[ 35%] \u001b[32m\u001b[1mLinking CXX static library ../../../../lib/libBaseSamplingLayer.a\u001b[0m\n",
      "[ 35%] Built target BaseSamplingLayer\n",
      "[ 36%] \u001b[32m\u001b[1mLinking CUDA device code CMakeFiles/xlnet_gemm_func.dir/cmake_device_link.o\u001b[0m\n",
      "[ 36%] \u001b[32mBuilding CXX object 3rdparty/trt_fused_multihead_attention/CMakeFiles/trt_fused_multi_head_attention.dir/fused_multihead_attention_v2_int8_192_64_kernel.sm75.cpp.o\u001b[0m\n",
      "[ 37%] \u001b[32m\u001b[1mLinking CUDA device code CMakeFiles/cublasINT8MMWrapper.dir/cmake_device_link.o\u001b[0m\n",
      "[ 37%] \u001b[32m\u001b[1mLinking CUDA device code CMakeFiles/vit_kernels.dir/cmake_device_link.o\u001b[0m\n",
      "[ 38%] \u001b[32m\u001b[1mLinking CUDA device code CMakeFiles/swin_gemm_func.dir/cmake_device_link.o\u001b[0m\n",
      "[ 38%] \u001b[32m\u001b[1mLinking CXX static library ../../../../lib/libxlnet_gemm_func.a\u001b[0m\n",
      "[ 39%] \u001b[32mBuilding CXX object 3rdparty/trt_fused_multihead_attention/CMakeFiles/trt_fused_multi_head_attention.dir/fused_multihead_attention_v2_int8_192_64_kernel.sm80.cpp.o\u001b[0m\n",
      "[ 39%] \u001b[32m\u001b[1mLinking CXX static library ../../../lib/libcublasINT8MMWrapper.a\u001b[0m\n",
      "[ 39%] Built target xlnet_gemm_func\n",
      "[ 40%] \u001b[32mBuilding CXX object 3rdparty/trt_fused_multihead_attention/CMakeFiles/trt_fused_multi_head_attention.dir/fused_multihead_attention_v2_int8_192_64_kernel.sm86.cpp.o\u001b[0m\n",
      "[ 40%] \u001b[32m\u001b[1mLinking CUDA static library ../../../lib/libvit_kernels.a\u001b[0m\n",
      "[ 40%] Built target cublasINT8MMWrapper\n",
      "[ 40%] \u001b[32m\u001b[1mLinking CXX static library ../../../../lib/libswin_gemm_func.a\u001b[0m\n",
      "[ 40%] \u001b[32mBuilding CXX object 3rdparty/trt_fused_multihead_attention/CMakeFiles/trt_fused_multi_head_attention.dir/fused_multihead_attention_v2_int8_256_64_kernel.sm72.cpp.o\u001b[0m\n",
      "[ 40%] Built target vit_kernels\n",
      "[ 40%] Built target swin_gemm_func\n",
      "[ 40%] \u001b[32mBuilding CXX object 3rdparty/trt_fused_multihead_attention/CMakeFiles/trt_fused_multi_head_attention.dir/fused_multihead_attention_v2_int8_256_64_kernel.sm75.cpp.o\u001b[0m\n",
      "[ 40%] \u001b[32mBuilding CXX object 3rdparty/trt_fused_multihead_attention/CMakeFiles/trt_fused_multi_head_attention.dir/fused_multihead_attention_v2_int8_256_64_kernel.sm80.cpp.o\u001b[0m\n",
      "[ 41%] \u001b[32mBuilding CXX object 3rdparty/trt_fused_multihead_attention/CMakeFiles/trt_fused_multi_head_attention.dir/fused_multihead_attention_v2_int8_256_64_kernel.sm86.cpp.o\u001b[0m\n",
      "[ 41%] \u001b[32mBuilding CXX object 3rdparty/trt_fused_multihead_attention/CMakeFiles/trt_fused_multi_head_attention.dir/fused_multihead_attention_v2_int8_384_64_kernel.sm72.cpp.o\u001b[0m\n",
      "[ 41%] \u001b[32mBuilding CXX object 3rdparty/trt_fused_multihead_attention/CMakeFiles/trt_fused_multi_head_attention.dir/fused_multihead_attention_v2_int8_384_64_kernel.sm75.cpp.o\u001b[0m\n",
      "[ 41%] \u001b[32mBuilding CXX object 3rdparty/trt_fused_multihead_attention/CMakeFiles/trt_fused_multi_head_attention.dir/fused_multihead_attention_v2_int8_384_64_kernel.sm80.cpp.o\u001b[0m\n",
      "[ 41%] \u001b[32m\u001b[1mLinking CUDA device code CMakeFiles/beam_search_penalty_kernels.dir/cmake_device_link.o\u001b[0m\n",
      "[ 41%] \u001b[32mBuilding CXX object 3rdparty/trt_fused_multihead_attention/CMakeFiles/trt_fused_multi_head_attention.dir/fused_multihead_attention_v2_int8_384_64_kernel.sm86.cpp.o\u001b[0m\n",
      "[ 41%] \u001b[32m\u001b[1mLinking CUDA static library ../../../lib/libbeam_search_penalty_kernels.a\u001b[0m\n",
      "[ 42%] \u001b[32mBuilding CXX object 3rdparty/trt_fused_multihead_attention/CMakeFiles/trt_fused_multi_head_attention.dir/fused_multihead_attention_v2_int8_64_64_kernel.sm75.cpp.o\u001b[0m\n",
      "[ 42%] Built target beam_search_penalty_kernels\n",
      "[ 42%] \u001b[32mBuilding CXX object 3rdparty/trt_fused_multihead_attention/CMakeFiles/trt_fused_multi_head_attention.dir/fused_multihead_attention_v2_int8_64_64_kernel.sm80.cpp.o\u001b[0m\n",
      "[ 42%] \u001b[32mBuilding CUDA object 3rdparty/trt_fused_multihead_attention/CMakeFiles/trt_fused_multi_head_attention.dir/qkvToContext.cu.o\u001b[0m\n",
      "\u001b[35m\u001b[1mScanning dependencies of target BaseBeamSearchLayer\u001b[0m\n",
      "[ 42%] \u001b[32mBuilding CUDA object src/fastertransformer/layers/beam_search_layers/CMakeFiles/BaseBeamSearchLayer.dir/BaseBeamSearchLayer.cu.o\u001b[0m\n",
      "[ 42%] \u001b[32m\u001b[1mLinking CUDA device code CMakeFiles/gen_relative_pos_bias.dir/cmake_device_link.o\u001b[0m\n",
      "[ 43%] \u001b[32m\u001b[1mLinking CUDA device code CMakeFiles/transform_mask_kernels.dir/cmake_device_link.o\u001b[0m\n",
      "[ 43%] \u001b[32m\u001b[1mLinking CUDA static library ../../../lib/libgen_relative_pos_bias.a\u001b[0m\n",
      "[ 43%] Built target gen_relative_pos_bias\n",
      "[ 43%] \u001b[32m\u001b[1mLinking CUDA static library ../../../lib/libtransform_mask_kernels.a\u001b[0m\n",
      "[ 43%] \u001b[32m\u001b[1mLinking CUDA device code CMakeFiles/xlnet_preprocess_kernels.dir/cmake_device_link.o\u001b[0m\n",
      "[ 43%] Built target transform_mask_kernels\n",
      "[ 44%] \u001b[32m\u001b[1mLinking CUDA static library ../../../lib/libxlnet_preprocess_kernels.a\u001b[0m\n",
      "[ 44%] Built target xlnet_preprocess_kernels\n",
      "[ 45%] \u001b[32m\u001b[1mLinking CXX static library ../../../lib/libcnpy.a\u001b[0m\n",
      "[ 45%] Built target cnpy\n",
      "[ 45%] \u001b[32m\u001b[1mLinking CUDA device code CMakeFiles/logprob_kernels.dir/cmake_device_link.o\u001b[0m\n",
      "[ 46%] \u001b[32m\u001b[1mLinking CUDA static library ../../../lib/liblogprob_kernels.a\u001b[0m\n",
      "[ 46%] Built target logprob_kernels\n",
      "[ 46%] \u001b[32m\u001b[1mLinking CUDA device code CMakeFiles/reverse_roll_kernels.dir/cmake_device_link.o\u001b[0m\n",
      "[ 46%] \u001b[32m\u001b[1mLinking CUDA static library ../../../lib/libreverse_roll_kernels.a\u001b[0m\n",
      "[ 46%] Built target reverse_roll_kernels\n",
      "[ 46%] \u001b[32m\u001b[1mLinking CUDA device code CMakeFiles/memory_utils.dir/cmake_device_link.o\u001b[0m\n",
      "[ 46%] \u001b[32m\u001b[1mLinking CUDA static library ../../../lib/libmemory_utils.a\u001b[0m\n",
      "[ 46%] Built target memory_utils\n",
      "\u001b[35m\u001b[1mScanning dependencies of target ParallelGptDecoderLayerWeight\u001b[0m\n",
      "\u001b[35m\u001b[1mScanning dependencies of target gemm\u001b[0m\n",
      "\u001b[35m\u001b[1mScanning dependencies of target custom_ar_comm\u001b[0m\n",
      "\u001b[35m\u001b[1mScanning dependencies of target GptJDecoderLayerWeight\u001b[0m\n",
      "\u001b[35m\u001b[1mScanning dependencies of target xlnet_gemm\u001b[0m\n",
      "\u001b[35m\u001b[1mScanning dependencies of target FfnLayerINT8\u001b[0m\n",
      "\u001b[35m\u001b[1mScanning dependencies of target t5_gemm\u001b[0m\n",
      "\u001b[35m\u001b[1mScanning dependencies of target gpt_gemm\u001b[0m\n",
      "\u001b[35m\u001b[1mScanning dependencies of target vit_gemm\u001b[0m\n",
      "\u001b[35m\u001b[1mScanning dependencies of target swin_gemm\u001b[0m\n",
      "\u001b[35m\u001b[1mScanning dependencies of target decoding_gemm\u001b[0m\n",
      "\u001b[35m\u001b[1mScanning dependencies of target bert_gemm\u001b[0m\n",
      "\u001b[35m\u001b[1mScanning dependencies of target XlnetAttentionLayer\u001b[0m\n",
      "\u001b[35m\u001b[1mScanning dependencies of target test_logprob_kernels\u001b[0m\n",
      "\u001b[35m\u001b[1mScanning dependencies of target FfnLayer\u001b[0m\n",
      "[ 47%] \u001b[32mBuilding CUDA object tests/unittests/CMakeFiles/test_logprob_kernels.dir/test_logprob_kernels.cu.o\u001b[0m\n",
      "[ 47%] \u001b[32mBuilding CXX object src/fastertransformer/models/multi_gpu_gpt/CMakeFiles/ParallelGptDecoderLayerWeight.dir/ParallelGptDecoderLayerWeight.cc.o\u001b[0m\n",
      "[ 48%] \u001b[32mBuilding CXX object src/fastertransformer/utils/CMakeFiles/custom_ar_comm.dir/custom_ar_comm.cc.o\u001b[0m\n",
      "[ 48%] \u001b[32mBuilding CXX object src/fastertransformer/models/t5/CMakeFiles/t5_gemm.dir/t5_gemm.cc.o\u001b[0m\n",
      "[ 49%] \u001b[32mBuilding CXX object src/fastertransformer/models/xlnet/CMakeFiles/xlnet_gemm.dir/xlnet_gemm.cc.o\u001b[0m\n",
      "[ 49%] \u001b[32mBuilding CXX object src/fastertransformer/utils/CMakeFiles/gemm.dir/gemm.cc.o\u001b[0m\n",
      "[ 49%] \u001b[32mBuilding CXX object src/fastertransformer/models/gptj/CMakeFiles/GptJDecoderLayerWeight.dir/GptJDecoderLayerWeight.cc.o\u001b[0m\n",
      "[ 49%] \u001b[32mBuilding CXX object src/fastertransformer/layers/CMakeFiles/FfnLayerINT8.dir/FfnLayerINT8.cc.o\u001b[0m\n",
      "[ 49%] \u001b[32mBuilding CXX object src/fastertransformer/models/decoding/CMakeFiles/decoding_gemm.dir/decoding_gemm.cc.o\u001b[0m\n",
      "[ 49%] \u001b[32mBuilding CXX object src/fastertransformer/models/swin/CMakeFiles/swin_gemm.dir/swin_gemm.cc.o\u001b[0m\n",
      "[ 49%] \u001b[32mBuilding CXX object src/fastertransformer/models/vit/CMakeFiles/vit_gemm.dir/vit_gemm.cc.o\u001b[0m\n",
      "[ 49%] \u001b[32mBuilding CXX object src/fastertransformer/models/bert/CMakeFiles/bert_gemm.dir/bert_gemm.cc.o\u001b[0m\n",
      "[ 50%] \u001b[32mBuilding CXX object src/fastertransformer/layers/CMakeFiles/FfnLayer.dir/FfnLayer.cc.o\u001b[0m\n",
      "[ 50%] \u001b[32mBuilding CXX object src/fastertransformer/models/multi_gpu_gpt/CMakeFiles/gpt_gemm.dir/gpt_gemm.cc.o\u001b[0m\n",
      "[ 51%] \u001b[32mBuilding CXX object src/fastertransformer/layers/xlnet_attention_layers/CMakeFiles/XlnetAttentionLayer.dir/XlnetAttentionLayer.cc.o\u001b[0m\n",
      "[ 51%] \u001b[32m\u001b[1mLinking CUDA device code CMakeFiles/layernorm_int8_kernels.dir/cmake_device_link.o\u001b[0m\n",
      "[ 51%] \u001b[32m\u001b[1mLinking CUDA static library ../../../lib/liblayernorm_int8_kernels.a\u001b[0m\n",
      "[ 51%] Built target layernorm_int8_kernels\n",
      "[ 51%] \u001b[32m\u001b[1mLinking CUDA device code CMakeFiles/dequantize_kernels.dir/cmake_device_link.o\u001b[0m\n",
      "[ 51%] \u001b[32m\u001b[1mLinking CUDA static library ../../../lib/libdequantize_kernels.a\u001b[0m\n",
      "[ 51%] Built target dequantize_kernels\n",
      "[ 52%] \u001b[32m\u001b[1mLinking CXX executable ../../../../bin/vit_gemm\u001b[0m\n",
      "[ 52%] Built target vit_gemm\n",
      "[ 52%] \u001b[32m\u001b[1mLinking CXX executable ../../../../bin/bert_gemm\u001b[0m\n",
      "[ 52%] \u001b[32m\u001b[1mLinking CXX executable ../../../../bin/swin_gemm\u001b[0m\n",
      "[ 52%] \u001b[32m\u001b[1mLinking CXX executable ../../../../bin/xlnet_gemm\u001b[0m\n",
      "[ 52%] \u001b[32m\u001b[1mLinking CXX executable ../../../../bin/decoding_gemm\u001b[0m\n",
      "[ 53%] \u001b[32m\u001b[1mLinking CXX executable ../../../../bin/gpt_gemm\u001b[0m\n",
      "[ 53%] Built target gpt_gemm\n",
      "[ 53%] Built target swin_gemm\n",
      "[ 53%] Built target bert_gemm\n",
      "[ 53%] Built target xlnet_gemm\n",
      "[ 53%] Built target decoding_gemm\n",
      "[ 53%] \u001b[32m\u001b[1mLinking CUDA device code CMakeFiles/unfused_attention_int8_kernels.dir/cmake_device_link.o\u001b[0m\n",
      "[ 53%] \u001b[32m\u001b[1mLinking CUDA device code CMakeFiles/sampling_topk_kernels.dir/cmake_device_link.o\u001b[0m\n",
      "[ 53%] \u001b[32m\u001b[1mLinking CXX executable ../../../../bin/t5_gemm\u001b[0m\n",
      "[ 53%] \u001b[32m\u001b[1mLinking CUDA device code CMakeFiles/GptJDecoderLayerWeight.dir/cmake_device_link.o\u001b[0m\n",
      "[ 53%] \u001b[32m\u001b[1mLinking CUDA static library ../../../lib/libunfused_attention_int8_kernels.a\u001b[0m\n",
      "[ 53%] \u001b[32m\u001b[1mLinking CUDA static library ../../../lib/libsampling_topk_kernels.a\u001b[0m\n",
      "[ 54%] \u001b[32m\u001b[1mLinking CUDA device code CMakeFiles/FfnLayerINT8.dir/cmake_device_link.o\u001b[0m\n",
      "[ 54%] Built target unfused_attention_int8_kernels\n",
      "[ 54%] Built target sampling_topk_kernels\n",
      "\u001b[35m\u001b[1mScanning dependencies of target UnfusedAttentionLayerINT8\u001b[0m\n",
      "[ 54%] Built target t5_gemm\n",
      "\u001b[35m\u001b[1mScanning dependencies of target TopKSamplingLayer\u001b[0m\n",
      "[ 54%] \u001b[32m\u001b[1mLinking CXX static library ../../../../lib/libGptJDecoderLayerWeight.a\u001b[0m\n",
      "[ 54%] \u001b[32mBuilding CUDA object src/fastertransformer/layers/sampling_layers/CMakeFiles/TopKSamplingLayer.dir/TopKSamplingLayer.cu.o\u001b[0m\n",
      "[ 54%] \u001b[32mBuilding CXX object src/fastertransformer/layers/attention_layers_int8/CMakeFiles/UnfusedAttentionLayerINT8.dir/UnfusedAttentionLayerINT8.cc.o\u001b[0m\n",
      "[ 54%] Built target GptJDecoderLayerWeight\n",
      "[ 54%] \u001b[32m\u001b[1mLinking CXX static library ../../../lib/libFfnLayerINT8.a\u001b[0m\n",
      "\u001b[35m\u001b[1mScanning dependencies of target GptJWeight\u001b[0m\n",
      "[ 54%] \u001b[32mBuilding CXX object src/fastertransformer/models/gptj/CMakeFiles/GptJWeight.dir/GptJWeight.cc.o\u001b[0m\n",
      "[ 54%] Built target FfnLayerINT8\n",
      "[ 54%] \u001b[32m\u001b[1mLinking CUDA device code CMakeFiles/XlnetAttentionLayer.dir/cmake_device_link.o\u001b[0m\n",
      "[ 54%] \u001b[32m\u001b[1mLinking CXX static library ../../../../lib/libXlnetAttentionLayer.a\u001b[0m\n",
      "[ 54%] \u001b[32m\u001b[1mLinking CUDA device code CMakeFiles/custom_ar_comm.dir/cmake_device_link.o\u001b[0m\n",
      "[ 54%] Built target XlnetAttentionLayer\n",
      "[ 54%] \u001b[32m\u001b[1mLinking CUDA device code CMakeFiles/gemm.dir/cmake_device_link.o\u001b[0m\n",
      "[ 54%] \u001b[32m\u001b[1mLinking CXX static library ../../../lib/libcustom_ar_comm.a\u001b[0m\n",
      "[ 54%] \u001b[32m\u001b[1mLinking CXX static library ../../../lib/libgemm.a\u001b[0m\n",
      "[ 54%] Built target custom_ar_comm\n",
      "[ 54%] Built target gemm\n",
      "\u001b[35m\u001b[1mScanning dependencies of target test_gemm\u001b[0m\n",
      "[ 54%] \u001b[32mBuilding CUDA object tests/unittests/CMakeFiles/test_gemm.dir/test_gemm.cu.o\u001b[0m\n",
      "[ 55%] \u001b[32m\u001b[1mLinking CUDA device code CMakeFiles/ParallelGptDecoderLayerWeight.dir/cmake_device_link.o\u001b[0m\n",
      "[ 55%] \u001b[32m\u001b[1mLinking CXX static library ../../../../lib/libParallelGptDecoderLayerWeight.a\u001b[0m\n",
      "[ 55%] Built target ParallelGptDecoderLayerWeight\n",
      "\u001b[35m\u001b[1mScanning dependencies of target ParallelGptWeight\u001b[0m\n",
      "[ 55%] \u001b[32mBuilding CXX object src/fastertransformer/models/multi_gpu_gpt/CMakeFiles/ParallelGptWeight.dir/ParallelGptWeight.cc.o\u001b[0m\n",
      "[ 55%] \u001b[32m\u001b[1mLinking CUDA device code CMakeFiles/FfnLayer.dir/cmake_device_link.o\u001b[0m\n",
      "[ 55%] \u001b[32m\u001b[1mLinking CXX static library ../../../lib/libFfnLayer.a\u001b[0m\n",
      "[ 55%] Built target FfnLayer\n",
      "\u001b[35m\u001b[1mScanning dependencies of target TensorParallelGeluFfnLayer\u001b[0m\n",
      "\u001b[35m\u001b[1mScanning dependencies of target TensorParallelReluFfnLayer\u001b[0m\n",
      "[ 55%] \u001b[32mBuilding CXX object src/fastertransformer/layers/CMakeFiles/TensorParallelGeluFfnLayer.dir/TensorParallelGeluFfnLayer.cc.o\u001b[0m\n",
      "[ 55%] \u001b[32mBuilding CXX object src/fastertransformer/layers/CMakeFiles/TensorParallelReluFfnLayer.dir/TensorParallelReluFfnLayer.cc.o\u001b[0m\n",
      "[ 55%] \u001b[32m\u001b[1mLinking CUDA device code CMakeFiles/GptJWeight.dir/cmake_device_link.o\u001b[0m\n",
      "[ 55%] \u001b[32m\u001b[1mLinking CUDA device code CMakeFiles/UnfusedAttentionLayerINT8.dir/cmake_device_link.o\u001b[0m\n",
      "[ 55%] \u001b[32m\u001b[1mLinking CXX static library ../../../../lib/libGptJWeight.a\u001b[0m\n",
      "[ 55%] Built target GptJWeight\n",
      "[ 55%] \u001b[32m\u001b[1mLinking CXX static library ../../../../lib/libUnfusedAttentionLayerINT8.a\u001b[0m\n",
      "[ 55%] Built target UnfusedAttentionLayerINT8\n",
      "[ 55%] \u001b[32m\u001b[1mLinking CUDA device code CMakeFiles/longformer_kernels.dir/cmake_device_link.o\u001b[0m\n",
      "[ 55%] \u001b[32m\u001b[1mLinking CUDA static library ../../../lib/liblongformer_kernels.a\u001b[0m\n",
      "[ 55%] Built target longformer_kernels\n",
      "\u001b[35m\u001b[1mScanning dependencies of target LongformerAttentionLayer\u001b[0m\n",
      "[ 56%] \u001b[32mBuilding CXX object src/fastertransformer/layers/attention_layers/CMakeFiles/LongformerAttentionLayer.dir/LongformerAttentionLayer.cc.o\u001b[0m\n",
      "[ 56%] \u001b[32m\u001b[1mLinking CUDA device code CMakeFiles/TensorParallelReluFfnLayer.dir/cmake_device_link.o\u001b[0m\n",
      "[ 56%] \u001b[32m\u001b[1mLinking CUDA device code CMakeFiles/TensorParallelGeluFfnLayer.dir/cmake_device_link.o\u001b[0m\n",
      "[ 57%] \u001b[32m\u001b[1mLinking CXX static library ../../../lib/libTensorParallelReluFfnLayer.a\u001b[0m\n",
      "[ 57%] Built target TensorParallelReluFfnLayer\n",
      "[ 57%] \u001b[32m\u001b[1mLinking CXX static library ../../../lib/libTensorParallelGeluFfnLayer.a\u001b[0m\n",
      "[ 57%] Built target TensorParallelGeluFfnLayer\n",
      "[ 57%] \u001b[32m\u001b[1mLinking CUDA device code CMakeFiles/LongformerAttentionLayer.dir/cmake_device_link.o\u001b[0m\n",
      "[ 57%] \u001b[32m\u001b[1mLinking CUDA device code CMakeFiles/ParallelGptWeight.dir/cmake_device_link.o\u001b[0m\n",
      "[ 57%] \u001b[32m\u001b[1mLinking CXX static library ../../../../lib/libLongformerAttentionLayer.a\u001b[0m\n",
      "[ 57%] \u001b[32m\u001b[1mLinking CXX static library ../../../../lib/libParallelGptWeight.a\u001b[0m\n",
      "[ 57%] Built target LongformerAttentionLayer\n",
      "[ 57%] Built target ParallelGptWeight\n",
      "\u001b[01m\u001b[K/dli/task/FasterTransformer/3rdparty/trt_fused_multihead_attention/qkvToContext.cu:\u001b[m\u001b[K In function ‘\u001b[01m\u001b[Kvoid fastertransformer::set_alpha(uint32_t&, float, fastertransformer::Data_type)\u001b[m\u001b[K’:\n",
      "\u001b[01m\u001b[K/dli/task/FasterTransformer/3rdparty/trt_fused_multihead_attention/qkvToContext.cu:32:46:\u001b[m\u001b[K \u001b[01;35m\u001b[Kwarning: \u001b[m\u001b[Kdereferencing type-punned pointer will break strict-aliasing rules [\u001b[01;35m\u001b[K-Wstrict-aliasing\u001b[m\u001b[K]\n",
      "   32 |         alpha = reinterpret_cast<const uint32\u001b[01;35m\u001b[K_t\u001b[m\u001b[K&>(h2);\n",
      "      |                                              \u001b[01;35m\u001b[K^~\u001b[m\u001b[K\n",
      "\u001b[01m\u001b[K/dli/task/FasterTransformer/3rdparty/trt_fused_multihead_attention/qkvToContext.cu:36:46:\u001b[m\u001b[K \u001b[01;35m\u001b[Kwarning: \u001b[m\u001b[Kdereferencing type-punned pointer will break strict-aliasing rules [\u001b[01;35m\u001b[K-Wstrict-aliasing\u001b[m\u001b[K]\n",
      "   36 |         alpha = reinterpret_cast<const uint32\u001b[01;35m\u001b[K_t&>\u001b[m\u001b[K(norm);\n",
      "      |                                              \u001b[01;35m\u001b[K^~~~\u001b[m\u001b[K\n",
      "\u001b[01m\u001b[K/dli/task/FasterTransformer/3rdparty/trt_fused_multihead_attention/qkvToContext.cu:\u001b[m\u001b[K In member function ‘\u001b[01m\u001b[Kvoid fastertransformer::FusedMHARunnerInt8v2::mhaImpl::setup(int, int, int)\u001b[m\u001b[K’:\n",
      "\u001b[01m\u001b[K/dli/task/FasterTransformer/3rdparty/trt_fused_multihead_attention/qkvToContext.cu:416:62:\u001b[m\u001b[K \u001b[01;35m\u001b[Kwarning: \u001b[m\u001b[Kdereferencing type-punned pointer will break strict-aliasing rules [\u001b[01;35m\u001b[K-Wstrict-aliasing\u001b[m\u001b[K]\n",
      "  416 |         params.scale_bmm1 = reinterpret_cast<const uint32_t&>\u001b[01;35m\u001b[K(scaleBmm\u001b[m\u001b[K1);\n",
      "      |                                                              \u001b[01;35m\u001b[K^~~~~~~~~\u001b[m\u001b[K\n",
      "\u001b[01m\u001b[K/dli/task/FasterTransformer/3rdparty/trt_fused_multihead_attention/qkvToContext.cu:417:62:\u001b[m\u001b[K \u001b[01;35m\u001b[Kwarning: \u001b[m\u001b[Kdereferencing type-punned pointer will break strict-aliasing rules [\u001b[01;35m\u001b[K-Wstrict-aliasing\u001b[m\u001b[K]\n",
      "  417 |         params.scale_bmm2 = reinterpret_cast<const uint32_t&>\u001b[01;35m\u001b[K(scaleBmm\u001b[m\u001b[K2);\n",
      "      |                                                              \u001b[01;35m\u001b[K^~~~~~~~~\u001b[m\u001b[K\n",
      "\u001b[01m\u001b[K/dli/task/FasterTransformer/3rdparty/trt_fused_multihead_attention/qkvToContext.cu:418:65:\u001b[m\u001b[K \u001b[01;35m\u001b[Kwarning: \u001b[m\u001b[Kdereferencing type-punned pointer will break strict-aliasing rules [\u001b[01;35m\u001b[K-Wstrict-aliasing\u001b[m\u001b[K]\n",
      "  418 |         params.scale_softmax = reinterpret_cast<const uint32_t&>\u001b[01;35m\u001b[K(scaleSoftma\u001b[m\u001b[Kx);\n",
      "      |                                                                 \u001b[01;35m\u001b[K^~~~~~~~~~~~\u001b[m\u001b[K\n",
      "\u001b[01m\u001b[K/dli/task/FasterTransformer/3rdparty/trt_fused_multihead_attention/qkvToContext.cu:\u001b[m\u001b[K In member function ‘\u001b[01m\u001b[Kvoid fastertransformer::FusedMHARunnerInt8v2::mhaImpl::setup(int, int)\u001b[m\u001b[K’:\n",
      "\u001b[01m\u001b[K/dli/task/FasterTransformer/3rdparty/trt_fused_multihead_attention/qkvToContext.cu:498:62:\u001b[m\u001b[K \u001b[01;35m\u001b[Kwarning: \u001b[m\u001b[Kdereferencing type-punned pointer will break strict-aliasing rules [\u001b[01;35m\u001b[K-Wstrict-aliasing\u001b[m\u001b[K]\n",
      "  498 |         params.scale_bmm1 = reinterpret_cast<const uint32_t&>\u001b[01;35m\u001b[K(scaleBmm\u001b[m\u001b[K1);\n",
      "      |                                                              \u001b[01;35m\u001b[K^~~~~~~~~\u001b[m\u001b[K\n",
      "\u001b[01m\u001b[K/dli/task/FasterTransformer/3rdparty/trt_fused_multihead_attention/qkvToContext.cu:499:62:\u001b[m\u001b[K \u001b[01;35m\u001b[Kwarning: \u001b[m\u001b[Kdereferencing type-punned pointer will break strict-aliasing rules [\u001b[01;35m\u001b[K-Wstrict-aliasing\u001b[m\u001b[K]\n",
      "  499 |         params.scale_bmm2 = reinterpret_cast<const uint32_t&>\u001b[01;35m\u001b[K(scaleBmm\u001b[m\u001b[K2);\n",
      "      |                                                              \u001b[01;35m\u001b[K^~~~~~~~~\u001b[m\u001b[K\n",
      "\u001b[01m\u001b[K/dli/task/FasterTransformer/3rdparty/trt_fused_multihead_attention/qkvToContext.cu:500:65:\u001b[m\u001b[K \u001b[01;35m\u001b[Kwarning: \u001b[m\u001b[Kdereferencing type-punned pointer will break strict-aliasing rules [\u001b[01;35m\u001b[K-Wstrict-aliasing\u001b[m\u001b[K]\n",
      "  500 |         params.scale_softmax = reinterpret_cast<const uint32_t&>\u001b[01;35m\u001b[K(scaleSoftma\u001b[m\u001b[Kx);\n",
      "      |                                                                 \u001b[01;35m\u001b[K^~~~~~~~~~~~\u001b[m\u001b[K\n",
      "[ 57%] \u001b[32m\u001b[1mLinking CUDA device code CMakeFiles/BaseBeamSearchLayer.dir/cmake_device_link.o\u001b[0m\n",
      "[ 57%] \u001b[32m\u001b[1mLinking CUDA static library ../../../../lib/libBaseBeamSearchLayer.a\u001b[0m\n",
      "[ 57%] Built target BaseBeamSearchLayer\n",
      "\u001b[01m\u001b[K/dli/task/FasterTransformer/3rdparty/trt_fused_multihead_attention/qkvToContext.cu:\u001b[m\u001b[K In member function ‘\u001b[01m\u001b[Kvirtual void fastertransformer::FusedMHARunnerInt8v2::setup(int, int, int)\u001b[m\u001b[K’:\n",
      "\u001b[01m\u001b[K/dli/task/FasterTransformer/3rdparty/trt_fused_multihead_attention/qkvToContext.cu:368:8:\u001b[m\u001b[K \u001b[01;35m\u001b[Kwarning: \u001b[m\u001b[K‘\u001b[01m\u001b[Kwarps_m\u001b[m\u001b[K’ may be used uninitialized in this function [\u001b[01;35m\u001b[K-Wmaybe-uninitialized\u001b[m\u001b[K]\n",
      "  368 |        \u001b[01;35m\u001b[K size_t\u001b[m\u001b[K warps_m, warps_n, warps_k = 1;\n",
      "      |        \u001b[01;35m\u001b[K^~~~~~~\u001b[m\u001b[K\n",
      "\u001b[01m\u001b[K/dli/task/FasterTransformer/3rdparty/trt_fused_multihead_attention/qkvToContext.cu:368:17:\u001b[m\u001b[K \u001b[01;35m\u001b[Kwarning: \u001b[m\u001b[K‘\u001b[01m\u001b[Kwarps_n\u001b[m\u001b[K’ may be used uninitialized in this function [\u001b[01;35m\u001b[K-Wmaybe-uninitialized\u001b[m\u001b[K]\n",
      "  368 |         size_t w\u001b[01;35m\u001b[Karps_m,\u001b[m\u001b[K warps_n, warps_k = 1;\n",
      "      |                 \u001b[01;35m\u001b[K^~~~~~~\u001b[m\u001b[K\n",
      "\u001b[01m\u001b[K/dli/task/FasterTransformer/3rdparty/trt_fused_multihead_attention/qkvToContext.cu:\u001b[m\u001b[K In member function ‘\u001b[01m\u001b[Kvirtual void fastertransformer::FusedMHARunnerInt8v2::setup(int, int)\u001b[m\u001b[K’:\n",
      "\u001b[01m\u001b[K/dli/task/FasterTransformer/3rdparty/trt_fused_multihead_attention/qkvToContext.cu:449:8:\u001b[m\u001b[K \u001b[01;35m\u001b[Kwarning: \u001b[m\u001b[K‘\u001b[01m\u001b[Kwarps_m\u001b[m\u001b[K’ may be used uninitialized in this function [\u001b[01;35m\u001b[K-Wmaybe-uninitialized\u001b[m\u001b[K]\n",
      "  449 |        \u001b[01;35m\u001b[K size_t\u001b[m\u001b[K warps_m, warps_n, warps_k = 1;\n",
      "      |        \u001b[01;35m\u001b[K^~~~~~~\u001b[m\u001b[K\n",
      "\u001b[01m\u001b[K/dli/task/FasterTransformer/3rdparty/trt_fused_multihead_attention/qkvToContext.cu:449:17:\u001b[m\u001b[K \u001b[01;35m\u001b[Kwarning: \u001b[m\u001b[K‘\u001b[01m\u001b[Kwarps_n\u001b[m\u001b[K’ may be used uninitialized in this function [\u001b[01;35m\u001b[K-Wmaybe-uninitialized\u001b[m\u001b[K]\n",
      "  449 |         size_t w\u001b[01;35m\u001b[Karps_m,\u001b[m\u001b[K warps_n, warps_k = 1;\n",
      "      |                 \u001b[01;35m\u001b[K^~~~~~~\u001b[m\u001b[K\n",
      "[ 57%] \u001b[32m\u001b[1mLinking CUDA device code CMakeFiles/trt_fused_multi_head_attention.dir/cmake_device_link.o\u001b[0m\n",
      "[ 58%] \u001b[32m\u001b[1mLinking CXX static library ../../lib/libtrt_fused_multi_head_attention.a\u001b[0m\n",
      "[ 58%] Built target trt_fused_multi_head_attention\n",
      "\u001b[35m\u001b[1mScanning dependencies of target FusedAttentionLayer\u001b[0m\n",
      "\u001b[35m\u001b[1mScanning dependencies of target FusedAttentionLayerINT8\u001b[0m\n",
      "[ 58%] \u001b[32mBuilding CUDA object src/fastertransformer/layers/attention_layers/CMakeFiles/FusedAttentionLayer.dir/FusedAttentionLayer.cu.o\u001b[0m\n",
      "[ 59%] \u001b[32mBuilding CUDA object src/fastertransformer/layers/attention_layers_int8/CMakeFiles/FusedAttentionLayerINT8.dir/FusedAttentionLayerINT8.cu.o\u001b[0m\n",
      "[ 59%] \u001b[32m\u001b[1mLinking CUDA executable ../../bin/test_logprob_kernels\u001b[0m\n",
      "[ 59%] Built target test_logprob_kernels\n",
      "/dli/task/FasterTransformer/src/fastertransformer/kernels/unfused_attention_kernels.cu: In instantiation of ‘\u001b[01m\u001b[Kvoid fastertransformer::invokeTransposeQKV(T*, T*, int, int, int, int, cudaStream_t) [with T = __half; cudaStream_t = CUstream_st*]\u001b[m\u001b[K’:\n",
      "\u001b[01m\u001b[K/dli/task/FasterTransformer/src/fastertransformer/kernels/unfused_attention_kernels.cu:765:163:\u001b[m\u001b[K   required from here\n",
      "\u001b[01m\u001b[K/dli/task/FasterTransformer/src/fastertransformer/kernels/unfused_attention_kernels.cu:729:37:\u001b[m\u001b[K \u001b[01;35m\u001b[Kwarning: \u001b[m\u001b[Kcomparison of integer expressions of different signedness: ‘\u001b[01m\u001b[Kunsigned int\u001b[m\u001b[K’ and ‘\u001b[01m\u001b[Kint\u001b[m\u001b[K’ [\u001b[01;35m\u001b[K-Wsign-compare\u001b[m\u001b[K]\n",
      "  729 |         F\u001b[01;35m\u001b[KT_CHECK(grid.x * seq_per_block == batch_size * head_num * seq_len\u001b[m\u001b[K);\n",
      "      |          \u001b[01;35m\u001b[K~~~~~~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\u001b[m\u001b[K\n",
      "/dli/task/FasterTransformer/src/fastertransformer/kernels/unfused_attention_kernels.cu: In instantiation of ‘\u001b[01m\u001b[Kvoid fastertransformer::invokeTransposeQKV(T*, T*, int, int, int, int, cudaStream_t) [with T = __nv_bfloat16; cudaStream_t = CUstream_st*]\u001b[m\u001b[K’:\n",
      "\u001b[01m\u001b[K/dli/task/FasterTransformer/src/fastertransformer/kernels/unfused_attention_kernels.cu:774:181:\u001b[m\u001b[K   required from here\n",
      "\u001b[01m\u001b[K/dli/task/FasterTransformer/src/fastertransformer/kernels/unfused_attention_kernels.cu:729:37:\u001b[m\u001b[K \u001b[01;35m\u001b[Kwarning: \u001b[m\u001b[Kcomparison of integer expressions of different signedness: ‘\u001b[01m\u001b[Kunsigned int\u001b[m\u001b[K’ and ‘\u001b[01m\u001b[Kint\u001b[m\u001b[K’ [\u001b[01;35m\u001b[K-Wsign-compare\u001b[m\u001b[K]\n",
      "[ 60%] \u001b[32m\u001b[1mLinking CUDA device code CMakeFiles/unfused_attention_kernels.dir/cmake_device_link.o\u001b[0m\n",
      "[ 60%] \u001b[32m\u001b[1mLinking CUDA static library ../../../lib/libunfused_attention_kernels.a\u001b[0m\n",
      "[ 60%] Built target unfused_attention_kernels\n",
      "\u001b[35m\u001b[1mScanning dependencies of target UnfusedAttentionLayer\u001b[0m\n",
      "\u001b[35m\u001b[1mScanning dependencies of target WindowAttention\u001b[0m\n",
      "\u001b[35m\u001b[1mScanning dependencies of target GptContextAttentionLayer\u001b[0m\n",
      "[ 60%] \u001b[32mBuilding CXX object src/fastertransformer/layers/attention_layers/CMakeFiles/UnfusedAttentionLayer.dir/UnfusedAttentionLayer.cc.o\u001b[0m\n",
      "[ 60%] \u001b[32mBuilding CXX object src/fastertransformer/layers/attention_layers/CMakeFiles/WindowAttention.dir/WindowAttention.cc.o\u001b[0m\n",
      "[ 60%] \u001b[32mBuilding CXX object src/fastertransformer/layers/attention_layers/CMakeFiles/GptContextAttentionLayer.dir/GptContextAttentionLayer.cc.o\u001b[0m\n",
      "[ 60%] \u001b[32m\u001b[1mLinking CXX executable ../../bin/test_gemm\u001b[0m\n",
      "[ 60%] Built target test_gemm\n",
      "[ 61%] \u001b[32m\u001b[1mLinking CUDA device code CMakeFiles/TopKSamplingLayer.dir/cmake_device_link.o\u001b[0m\n",
      "[ 61%] \u001b[32m\u001b[1mLinking CXX static library ../../../../lib/libTopKSamplingLayer.a\u001b[0m\n",
      "[ 61%] Built target TopKSamplingLayer\n",
      "[ 61%] \u001b[32m\u001b[1mLinking CUDA device code CMakeFiles/WindowAttention.dir/cmake_device_link.o\u001b[0m\n",
      "[ 61%] \u001b[32m\u001b[1mLinking CXX static library ../../../../lib/libWindowAttention.a\u001b[0m\n",
      "[ 61%] Built target WindowAttention\n",
      "[ 61%] \u001b[32m\u001b[1mLinking CUDA device code CMakeFiles/GptContextAttentionLayer.dir/cmake_device_link.o\u001b[0m\n",
      "[ 61%] \u001b[32m\u001b[1mLinking CUDA device code CMakeFiles/UnfusedAttentionLayer.dir/cmake_device_link.o\u001b[0m\n",
      "[ 62%] \u001b[32m\u001b[1mLinking CXX static library ../../../../lib/libGptContextAttentionLayer.a\u001b[0m\n",
      "[ 62%] Built target GptContextAttentionLayer\n",
      "[ 63%] \u001b[32m\u001b[1mLinking CXX static library ../../../../lib/libUnfusedAttentionLayer.a\u001b[0m\n",
      "\u001b[35m\u001b[1mScanning dependencies of target TensorParallelGptContextAttentionLayer\u001b[0m\n",
      "[ 63%] \u001b[32mBuilding CXX object src/fastertransformer/layers/attention_layers/CMakeFiles/TensorParallelGptContextAttentionLayer.dir/TensorParallelGptContextAttentionLayer.cc.o\u001b[0m\n",
      "[ 63%] Built target UnfusedAttentionLayer\n",
      "\u001b[35m\u001b[1mScanning dependencies of target TensorParallelUnfusedAttentionLayer\u001b[0m\n",
      "[ 63%] \u001b[32mBuilding CXX object src/fastertransformer/layers/attention_layers/CMakeFiles/TensorParallelUnfusedAttentionLayer.dir/TensorParallelUnfusedAttentionLayer.cc.o\u001b[0m\n",
      "[ 64%] \u001b[32m\u001b[1mLinking CUDA device code CMakeFiles/layernorm_kernels.dir/cmake_device_link.o\u001b[0m\n",
      "[ 64%] \u001b[32m\u001b[1mLinking CUDA static library ../../../lib/liblayernorm_kernels.a\u001b[0m\n",
      "[ 64%] Built target layernorm_kernels\n",
      "\u001b[35m\u001b[1mScanning dependencies of target Xlnet\u001b[0m\n",
      "\u001b[35m\u001b[1mScanning dependencies of target WindowAttentionINT8\u001b[0m\n",
      "\u001b[35m\u001b[1mScanning dependencies of target layernorm_test\u001b[0m\n",
      "\u001b[35m\u001b[1mScanning dependencies of target SwinBlock\u001b[0m\n",
      "\u001b[35m\u001b[1mScanning dependencies of target LongformerEncoder\u001b[0m\n",
      "[ 64%] \u001b[32mBuilding CUDA object src/fastertransformer/layers/attention_layers_int8/CMakeFiles/WindowAttentionINT8.dir/WindowAttentionINT8.cu.o\u001b[0m\n",
      "[ 64%] \u001b[32mBuilding CXX object src/fastertransformer/models/xlnet/CMakeFiles/Xlnet.dir/Xlnet.cc.o\u001b[0m\n",
      "[ 64%] \u001b[32mBuilding CXX object examples/cpp/decoding/CMakeFiles/layernorm_test.dir/layernorm_test.cc.o\u001b[0m\n",
      "[ 64%] \u001b[32mBuilding CXX object src/fastertransformer/models/longformer/CMakeFiles/LongformerEncoder.dir/LongformerEncoder.cc.o\u001b[0m\n",
      "[ 64%] \u001b[32mBuilding CXX object src/fastertransformer/models/swin/CMakeFiles/SwinBlock.dir/SwinBlock.cc.o\u001b[0m\n",
      "/dli/task/FasterTransformer/examples/cpp/decoding/layernorm_test.cc: In instantiation of ‘\u001b[01m\u001b[Kvoid add_bias_residual_layernorm_test(int, int) [with T = __half]\u001b[m\u001b[K’:\n",
      "\u001b[01m\u001b[K/dli/task/FasterTransformer/examples/cpp/decoding/layernorm_test.cc:100:60:\u001b[m\u001b[K   required from here\n",
      "\u001b[01m\u001b[K/dli/task/FasterTransformer/examples/cpp/decoding/layernorm_test.cc:191:50:\u001b[m\u001b[K \u001b[01;35m\u001b[Kwarning: \u001b[m\u001b[KISO C++ forbids converting a string constant to ‘\u001b[01m\u001b[Kchar*\u001b[m\u001b[K’ [\u001b[01;35m\u001b[K-Wwrite-strings\u001b[m\u001b[K]\n",
      "  191 |     checkMat(output_baseline, output_opt, m * n, \u001b[01;35m\u001b[K\"output_baseline vs output_opt\"\u001b[m\u001b[K);\n",
      "      |                                                  \u001b[01;35m\u001b[K^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\u001b[m\u001b[K\n",
      "\u001b[01m\u001b[K/dli/task/FasterTransformer/examples/cpp/decoding/layernorm_test.cc:192:64:\u001b[m\u001b[K \u001b[01;35m\u001b[Kwarning: \u001b[m\u001b[KISO C++ forbids converting a string constant to ‘\u001b[01m\u001b[Kchar*\u001b[m\u001b[K’ [\u001b[01;35m\u001b[K-Wwrite-strings\u001b[m\u001b[K]\n",
      "  192 |     checkMat(normed_output_baseline, normed_output_opt, m * n, \u001b[01;35m\u001b[K\"normed_output_baseline vs normed_output_opt\"\u001b[m\u001b[K);\n",
      "      |                                                                \u001b[01;35m\u001b[K^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\u001b[m\u001b[K\n",
      "/dli/task/FasterTransformer/examples/cpp/decoding/layernorm_test.cc: In instantiation of ‘\u001b[01m\u001b[Kvoid add_bias_residual_layernorm_test(int, int) [with T = float]\u001b[m\u001b[K’:\n",
      "\u001b[01m\u001b[K/dli/task/FasterTransformer/examples/cpp/decoding/layernorm_test.cc:103:61:\u001b[m\u001b[K   required from here\n",
      "\u001b[01m\u001b[K/dli/task/FasterTransformer/examples/cpp/decoding/layernorm_test.cc:191:50:\u001b[m\u001b[K \u001b[01;35m\u001b[Kwarning: \u001b[m\u001b[KISO C++ forbids converting a string constant to ‘\u001b[01m\u001b[Kchar*\u001b[m\u001b[K’ [\u001b[01;35m\u001b[K-Wwrite-strings\u001b[m\u001b[K]\n",
      "  191 |     checkMat(output_baseline, output_opt, m * n, \u001b[01;35m\u001b[K\"output_baseline vs output_opt\"\u001b[m\u001b[K);\n",
      "      |                                                  \u001b[01;35m\u001b[K^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\u001b[m\u001b[K\n",
      "\u001b[01m\u001b[K/dli/task/FasterTransformer/examples/cpp/decoding/layernorm_test.cc:192:64:\u001b[m\u001b[K \u001b[01;35m\u001b[Kwarning: \u001b[m\u001b[KISO C++ forbids converting a string constant to ‘\u001b[01m\u001b[Kchar*\u001b[m\u001b[K’ [\u001b[01;35m\u001b[K-Wwrite-strings\u001b[m\u001b[K]\n",
      "  192 |     checkMat(normed_output_baseline, normed_output_opt, m * n, \u001b[01;35m\u001b[K\"normed_output_baseline vs normed_output_opt\"\u001b[m\u001b[K);\n",
      "      |                                                                \u001b[01;35m\u001b[K^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\u001b[m\u001b[K\n",
      "[ 64%] \u001b[32m\u001b[1mLinking CUDA device code CMakeFiles/TensorParallelUnfusedAttentionLayer.dir/cmake_device_link.o\u001b[0m\n",
      "[ 65%] \u001b[32m\u001b[1mLinking CUDA device code CMakeFiles/TensorParallelGptContextAttentionLayer.dir/cmake_device_link.o\u001b[0m\n",
      "[ 65%] \u001b[32m\u001b[1mLinking CXX static library ../../../../lib/libTensorParallelUnfusedAttentionLayer.a\u001b[0m\n",
      "[ 65%] Built target TensorParallelUnfusedAttentionLayer\n",
      "[ 65%] \u001b[32m\u001b[1mLinking CXX static library ../../../../lib/libTensorParallelGptContextAttentionLayer.a\u001b[0m\n",
      "[ 65%] Built target TensorParallelGptContextAttentionLayer\n",
      "\u001b[35m\u001b[1mScanning dependencies of target GptJContextDecoder\u001b[0m\n",
      "\u001b[35m\u001b[1mScanning dependencies of target ParallelGptContextDecoder\u001b[0m\n",
      "[ 65%] \u001b[32m\u001b[1mLinking CXX executable ../../../bin/layernorm_test\u001b[0m\n",
      "[ 65%] \u001b[32mBuilding CXX object src/fastertransformer/models/gptj/CMakeFiles/GptJContextDecoder.dir/GptJContextDecoder.cc.o\u001b[0m\n",
      "[ 65%] \u001b[32mBuilding CXX object src/fastertransformer/models/multi_gpu_gpt/CMakeFiles/ParallelGptContextDecoder.dir/ParallelGptContextDecoder.cc.o\u001b[0m\n",
      "[ 65%] Built target layernorm_test\n",
      "\u001b[01m\u001b[0m\u001b[01m/dli/task/FasterTransformer/src/fastertransformer/kernels/sampling_topp_kernels.cu(532)\u001b[0m: \u001b[01;35mwarning\u001b[0m #177-D: variable \u001b[01m\"KEYS_PER_LDG\"\u001b[0m was declared but never referenced\n",
      "\n",
      "[ 65%] \u001b[32m\u001b[1mLinking CUDA device code CMakeFiles/FusedAttentionLayerINT8.dir/cmake_device_link.o\u001b[0m\n",
      "[ 65%] \u001b[32m\u001b[1mLinking CXX static library ../../../../lib/libFusedAttentionLayerINT8.a\u001b[0m\n",
      "[ 66%] \u001b[32m\u001b[1mLinking CUDA device code CMakeFiles/LongformerEncoder.dir/cmake_device_link.o\u001b[0m\n",
      "[ 66%] Built target FusedAttentionLayerINT8\n",
      "\u001b[35m\u001b[1mScanning dependencies of target ViTINT8\u001b[0m\n",
      "\u001b[35m\u001b[1mScanning dependencies of target BertLayerINT8\u001b[0m\n",
      "[ 66%] \u001b[32mBuilding CXX object src/fastertransformer/models/vit_int8/CMakeFiles/ViTINT8.dir/ViTINT8.cc.o\u001b[0m\n",
      "[ 66%] \u001b[32mBuilding CXX object src/fastertransformer/models/bert_int8/CMakeFiles/BertLayerINT8.dir/BertLayerINT8.cc.o\u001b[0m\n",
      "[ 66%] \u001b[32m\u001b[1mLinking CUDA device code CMakeFiles/SwinBlock.dir/cmake_device_link.o\u001b[0m\n",
      "[ 66%] \u001b[32m\u001b[1mLinking CXX static library ../../../../lib/libLongformerEncoder.a\u001b[0m\n",
      "[ 66%] \u001b[32m\u001b[1mLinking CUDA device code CMakeFiles/Xlnet.dir/cmake_device_link.o\u001b[0m\n",
      "[ 66%] Built target LongformerEncoder\n",
      "[ 67%] \u001b[32m\u001b[1mLinking CXX static library ../../../../lib/libSwinBlock.a\u001b[0m\n",
      "[ 67%] \u001b[32m\u001b[1mLinking CXX static library ../../../../lib/libXlnet.a\u001b[0m\n",
      "[ 67%] Built target SwinBlock\n",
      "\u001b[35m\u001b[1mScanning dependencies of target SwinBasicLayer\u001b[0m\n",
      "[ 67%] Built target Xlnet\n",
      "\u001b[35m\u001b[1mScanning dependencies of target xlnet_correctness_example\u001b[0m\n",
      "\u001b[35m\u001b[1mScanning dependencies of target xlnet_example\u001b[0m\n",
      "[ 67%] \u001b[32mBuilding CXX object src/fastertransformer/models/swin/CMakeFiles/SwinBasicLayer.dir/SwinBasicLayer.cc.o\u001b[0m\n",
      "[ 67%] \u001b[32mBuilding CXX object examples/cpp/xlnet/CMakeFiles/xlnet_correctness_example.dir/xlnet_correctness_example.cc.o\u001b[0m\n",
      "[ 68%] \u001b[32mBuilding CXX object examples/cpp/xlnet/CMakeFiles/xlnet_example.dir/xlnet_example.cc.o\u001b[0m\n",
      "[ 68%] \u001b[32m\u001b[1mLinking CUDA device code CMakeFiles/FusedAttentionLayer.dir/cmake_device_link.o\u001b[0m\n",
      "[ 68%] \u001b[32m\u001b[1mLinking CXX static library ../../../../lib/libFusedAttentionLayer.a\u001b[0m\n",
      "[ 68%] Built target FusedAttentionLayer\n",
      "\u001b[35m\u001b[1mScanning dependencies of target Bert\u001b[0m\n",
      "\u001b[35m\u001b[1mScanning dependencies of target T5Encoder\u001b[0m\n",
      "\u001b[35m\u001b[1mScanning dependencies of target ViT\u001b[0m\n",
      "[ 69%] \u001b[32mBuilding CXX object src/fastertransformer/models/bert/CMakeFiles/Bert.dir/Bert.cc.o\u001b[0m\n",
      "[ 70%] \u001b[32mBuilding CXX object src/fastertransformer/models/t5/CMakeFiles/T5Encoder.dir/T5Encoder.cc.o\u001b[0m\n",
      "[ 70%] \u001b[32mBuilding CXX object src/fastertransformer/models/t5/CMakeFiles/T5Encoder.dir/T5EncoderWeight.cc.o\u001b[0m\n",
      "[ 70%] \u001b[32mBuilding CXX object src/fastertransformer/models/vit/CMakeFiles/ViT.dir/ViT.cc.o\u001b[0m\n",
      "[ 70%] \u001b[32mBuilding CXX object src/fastertransformer/models/t5/CMakeFiles/T5Encoder.dir/T5EncoderLayerWeight.cc.o\u001b[0m\n",
      "[ 71%] \u001b[32m\u001b[1mLinking CUDA device code CMakeFiles/GptJContextDecoder.dir/cmake_device_link.o\u001b[0m\n",
      "[ 71%] \u001b[32m\u001b[1mLinking CXX static library ../../../../lib/libGptJContextDecoder.a\u001b[0m\n",
      "[ 71%] Built target GptJContextDecoder\n",
      "[ 71%] \u001b[32m\u001b[1mLinking CUDA device code CMakeFiles/SwinBasicLayer.dir/cmake_device_link.o\u001b[0m\n",
      "[ 71%] \u001b[32m\u001b[1mLinking CXX static library ../../../../lib/libSwinBasicLayer.a\u001b[0m\n",
      "[ 71%] Built target SwinBasicLayer\n",
      "\u001b[35m\u001b[1mScanning dependencies of target Swin\u001b[0m\n",
      "[ 71%] \u001b[32mBuilding CXX object src/fastertransformer/models/swin/CMakeFiles/Swin.dir/Swin.cc.o\u001b[0m\n",
      "[ 71%] \u001b[32m\u001b[1mLinking CUDA device code CMakeFiles/BertLayerINT8.dir/cmake_device_link.o\u001b[0m\n",
      "[ 71%] \u001b[32m\u001b[1mLinking CUDA device code CMakeFiles/ParallelGptContextDecoder.dir/cmake_device_link.o\u001b[0m\n",
      "[ 71%] \u001b[32m\u001b[1mLinking CXX static library ../../../../lib/libBertLayerINT8.a\u001b[0m\n",
      "[ 72%] \u001b[32m\u001b[1mLinking CXX static library ../../../../lib/libParallelGptContextDecoder.a\u001b[0m\n",
      "[ 72%] Built target BertLayerINT8\n",
      "[ 72%] Built target ParallelGptContextDecoder\n",
      "\u001b[35m\u001b[1mScanning dependencies of target BertINT8\u001b[0m\n",
      "[ 72%] \u001b[32mBuilding CXX object src/fastertransformer/models/bert_int8/CMakeFiles/BertINT8.dir/BertINT8.cc.o\u001b[0m\n",
      "[ 72%] \u001b[32m\u001b[1mLinking CXX executable ../../../bin/xlnet_example\u001b[0m\n",
      "[ 72%] Built target xlnet_example\n",
      "[ 72%] \u001b[32m\u001b[1mLinking CUDA device code CMakeFiles/ViTINT8.dir/cmake_device_link.o\u001b[0m\n",
      "[ 73%] \u001b[32m\u001b[1mLinking CXX static library ../../../../lib/libViTINT8.a\u001b[0m\n",
      "[ 73%] Built target ViTINT8\n",
      "\u001b[35m\u001b[1mScanning dependencies of target vit_int8_example\u001b[0m\n",
      "[ 73%] \u001b[32mBuilding CXX object examples/cpp/vit_int8/CMakeFiles/vit_int8_example.dir/vit_int8_example.cc.o\u001b[0m\n",
      "[ 73%] \u001b[32m\u001b[1mLinking CUDA device code CMakeFiles/Bert.dir/cmake_device_link.o\u001b[0m\n",
      "[ 73%] \u001b[32m\u001b[1mLinking CXX static library ../../../../lib/libBert.a\u001b[0m\n",
      "[ 73%] Built target Bert\n",
      "\u001b[35m\u001b[1mScanning dependencies of target bert_example\u001b[0m\n",
      "[ 73%] \u001b[32mBuilding CXX object examples/cpp/bert/CMakeFiles/bert_example.dir/bert_example.cc.o\u001b[0m\n",
      "[ 74%] \u001b[32m\u001b[1mLinking CUDA device code CMakeFiles/T5Encoder.dir/cmake_device_link.o\u001b[0m\n",
      "[ 74%] \u001b[32m\u001b[1mLinking CXX executable ../../../bin/xlnet_correctness_example\u001b[0m\n",
      "[ 74%] \u001b[32m\u001b[1mLinking CXX static library ../../../../lib/libT5Encoder.a\u001b[0m\n",
      "[ 74%] \u001b[32m\u001b[1mLinking CUDA device code CMakeFiles/ViT.dir/cmake_device_link.o\u001b[0m\n",
      "[ 74%] Built target T5Encoder\n",
      "[ 74%] \u001b[32m\u001b[1mLinking CXX static library ../../../../lib/libViT.a\u001b[0m\n",
      "[ 74%] Built target xlnet_correctness_example\n",
      "[ 74%] Built target ViT\n",
      "\u001b[35m\u001b[1mScanning dependencies of target vit_example\u001b[0m\n",
      "[ 74%] \u001b[32mBuilding CXX object examples/cpp/vit/CMakeFiles/vit_example.dir/vit_example.cc.o\u001b[0m\n",
      "[ 75%] \u001b[32m\u001b[1mLinking CUDA device code CMakeFiles/Swin.dir/cmake_device_link.o\u001b[0m\n",
      "[ 75%] \u001b[32m\u001b[1mLinking CXX static library ../../../../lib/libSwin.a\u001b[0m\n",
      "[ 75%] Built target Swin\n",
      "\u001b[35m\u001b[1mScanning dependencies of target swin_example\u001b[0m\n",
      "[ 76%] \u001b[32mBuilding CXX object examples/cpp/swin/CMakeFiles/swin_example.dir/swin_example.cc.o\u001b[0m\n",
      "[ 76%] \u001b[32m\u001b[1mLinking CUDA device code CMakeFiles/BertINT8.dir/cmake_device_link.o\u001b[0m\n",
      "[ 77%] \u001b[32m\u001b[1mLinking CXX static library ../../../../lib/libBertINT8.a\u001b[0m\n",
      "[ 77%] Built target BertINT8\n",
      "\u001b[35m\u001b[1mScanning dependencies of target bert_int8_example\u001b[0m\n",
      "[ 78%] \u001b[32mBuilding CXX object examples/cpp/bert_int8/CMakeFiles/bert_int8_example.dir/bert_int8_example.cc.o\u001b[0m\n",
      "[ 78%] \u001b[32m\u001b[1mLinking CXX executable ../../../bin/vit_int8_example\u001b[0m\n",
      "[ 78%] Built target vit_int8_example\n",
      "[ 78%] \u001b[32m\u001b[1mLinking CXX executable ../../../bin/bert_example\u001b[0m\n",
      "[ 78%] Built target bert_example\n",
      "[ 78%] \u001b[32m\u001b[1mLinking CXX executable ../../../bin/vit_example\u001b[0m\n",
      "[ 78%] Built target vit_example\n",
      "[ 78%] \u001b[32m\u001b[1mLinking CXX executable ../../../bin/swin_example\u001b[0m\n",
      "[ 78%] \u001b[32m\u001b[1mLinking CXX executable ../../../bin/bert_int8_example\u001b[0m\n",
      "[ 78%] Built target bert_int8_example\n",
      "[ 78%] Built target swin_example\n",
      "[ 79%] \u001b[32m\u001b[1mLinking CUDA device code CMakeFiles/WindowAttentionINT8.dir/cmake_device_link.o\u001b[0m\n",
      "[ 79%] \u001b[32m\u001b[1mLinking CXX static library ../../../../lib/libWindowAttentionINT8.a\u001b[0m\n",
      "[ 79%] Built target WindowAttentionINT8\n",
      "\u001b[35m\u001b[1mScanning dependencies of target SwinBlockINT8\u001b[0m\n",
      "[ 79%] \u001b[32mBuilding CXX object src/fastertransformer/models/swin_int8/CMakeFiles/SwinBlockINT8.dir/SwinBlockINT8.cc.o\u001b[0m\n",
      "[ 79%] \u001b[32m\u001b[1mLinking CUDA device code CMakeFiles/SwinBlockINT8.dir/cmake_device_link.o\u001b[0m\n",
      "[ 79%] \u001b[32m\u001b[1mLinking CXX static library ../../../../lib/libSwinBlockINT8.a\u001b[0m\n",
      "[ 79%] Built target SwinBlockINT8\n",
      "\u001b[35m\u001b[1mScanning dependencies of target SwinBasicLayerINT8\u001b[0m\n",
      "[ 80%] \u001b[32mBuilding CXX object src/fastertransformer/models/swin_int8/CMakeFiles/SwinBasicLayerINT8.dir/SwinBasicLayerINT8.cc.o\u001b[0m\n",
      "[ 80%] \u001b[32m\u001b[1mLinking CUDA device code CMakeFiles/SwinBasicLayerINT8.dir/cmake_device_link.o\u001b[0m\n",
      "[ 80%] \u001b[32m\u001b[1mLinking CXX static library ../../../../lib/libSwinBasicLayerINT8.a\u001b[0m\n",
      "[ 80%] Built target SwinBasicLayerINT8\n",
      "\u001b[35m\u001b[1mScanning dependencies of target SwinINT8\u001b[0m\n",
      "[ 81%] \u001b[32mBuilding CXX object src/fastertransformer/models/swin_int8/CMakeFiles/SwinINT8.dir/SwinINT8.cc.o\u001b[0m\n",
      "[ 81%] \u001b[32m\u001b[1mLinking CUDA device code CMakeFiles/SwinINT8.dir/cmake_device_link.o\u001b[0m\n",
      "[ 81%] \u001b[32m\u001b[1mLinking CXX static library ../../../../lib/libSwinINT8.a\u001b[0m\n",
      "[ 81%] Built target SwinINT8\n",
      "\u001b[35m\u001b[1mScanning dependencies of target swin_int8_example\u001b[0m\n",
      "[ 82%] \u001b[32mBuilding CXX object examples/cpp/swin_int8/CMakeFiles/swin_int8_example.dir/swin_int8_example.cc.o\u001b[0m\n",
      "[ 82%] \u001b[32m\u001b[1mLinking CXX executable ../../../bin/swin_int8_example\u001b[0m\n",
      "[ 82%] Built target swin_int8_example\n",
      "\u001b[01m\u001b[0m\u001b[01m/dli/task/FasterTransformer/src/fastertransformer/kernels/sampling_topp_kernels.cu(532)\u001b[0m: \u001b[01;35mwarning\u001b[0m #177-D: variable \u001b[01m\"KEYS_PER_LDG\"\u001b[0m was declared but never referenced\n",
      "\n",
      "\u001b[01m\u001b[0m\u001b[01m/dli/task/FasterTransformer/src/fastertransformer/kernels/sampling_topp_kernels.cu(532)\u001b[0m: \u001b[01;35mwarning\u001b[0m #177-D: variable \u001b[01m\"KEYS_PER_LDG\"\u001b[0m was declared but never referenced\n",
      "\n",
      "[ 82%] \u001b[32m\u001b[1mLinking CUDA device code CMakeFiles/beam_search_topk_kernels.dir/cmake_device_link.o\u001b[0m\n",
      "[ 83%] \u001b[32m\u001b[1mLinking CUDA static library ../../../lib/libbeam_search_topk_kernels.a\u001b[0m\n",
      "[ 83%] Built target beam_search_topk_kernels\n",
      "\u001b[35m\u001b[1mScanning dependencies of target BeamSearchLayer\u001b[0m\n",
      "[ 83%] \u001b[32mBuilding CUDA object src/fastertransformer/layers/beam_search_layers/CMakeFiles/BeamSearchLayer.dir/BeamSearchLayer.cu.o\u001b[0m\n",
      "[ 83%] \u001b[32m\u001b[1mLinking CUDA device code CMakeFiles/BeamSearchLayer.dir/cmake_device_link.o\u001b[0m\n",
      "[ 83%] \u001b[32m\u001b[1mLinking CUDA static library ../../../../lib/libBeamSearchLayer.a\u001b[0m\n",
      "[ 83%] Built target BeamSearchLayer\n",
      "/dli/task/FasterTransformer/src/fastertransformer/kernels/sampling_topp_kernels.cu: In instantiation of ‘\u001b[01m\u001b[Kint fastertransformer::segmented_topp_impl::getSmemSizeAndCheck(const fastertransformer::segmented_topp_impl::TopKPerSegmentContext&, const fastertransformer::segmented_topp_impl::TopKPerSegmentParams&) [with Kernel_params = fastertransformer::segmented_topp_impl::Segmented_topk_kernel_params<float, int, 256, 2>]\u001b[m\u001b[K’:\n",
      "\u001b[01m\u001b[K/dli/task/FasterTransformer/src/fastertransformer/kernels/sampling_topp_kernels.cu:556:104:\u001b[m\u001b[K   required from here\n",
      "\u001b[01m\u001b[K/dli/task/FasterTransformer/src/fastertransformer/kernels/sampling_topp_kernels.cu:537:56:\u001b[m\u001b[K \u001b[01;35m\u001b[Kwarning: \u001b[m\u001b[Kcomparison of integer expressions of different signedness: ‘\u001b[01m\u001b[Klong unsigned int\u001b[m\u001b[K’ and ‘\u001b[01m\u001b[Kconst int\u001b[m\u001b[K’ [\u001b[01;35m\u001b[K-Wsign-compare\u001b[m\u001b[K]\n",
      "  537 |     if (\u001b[01;35m\u001b[Ksmem_size + BLOCK_THREADS * sizeof(float) > context.sm_shared_size ||  // \u001b[m\u001b[Kdynamic + static memory\n",
      "      |         \u001b[01;35m\u001b[K~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~\u001b[m\u001b[K\n",
      "/dli/task/FasterTransformer/src/fastertransformer/kernels/sampling_topp_kernels.cu: In instantiation of ‘\u001b[01m\u001b[Kint fastertransformer::segmented_topp_impl::getSmemSizeAndCheck(const fastertransformer::segmented_topp_impl::TopKPerSegmentContext&, const fastertransformer::segmented_topp_impl::TopKPerSegmentParams&) [with Kernel_params = fastertransformer::segmented_topp_impl::Segmented_topk_kernel_params<float, int, 256, 1>]\u001b[m\u001b[K’:\n",
      "\u001b[01m\u001b[K/dli/task/FasterTransformer/src/fastertransformer/kernels/sampling_topp_kernels.cu:559:104:\u001b[m\u001b[K   required from here\n",
      "\u001b[01m\u001b[K/dli/task/FasterTransformer/src/fastertransformer/kernels/sampling_topp_kernels.cu:537:56:\u001b[m\u001b[K \u001b[01;35m\u001b[Kwarning: \u001b[m\u001b[Kcomparison of integer expressions of different signedness: ‘\u001b[01m\u001b[Klong unsigned int\u001b[m\u001b[K’ and ‘\u001b[01m\u001b[Kconst int\u001b[m\u001b[K’ [\u001b[01;35m\u001b[K-Wsign-compare\u001b[m\u001b[K]\n",
      "/dli/task/FasterTransformer/src/fastertransformer/kernels/sampling_topp_kernels.cu: In instantiation of ‘\u001b[01m\u001b[Kint fastertransformer::segmented_topp_impl::getSmemSizeAndCheck(const fastertransformer::segmented_topp_impl::TopKPerSegmentContext&, const fastertransformer::segmented_topp_impl::TopKPerSegmentParams&) [with Kernel_params = fastertransformer::segmented_topp_impl::Segmented_topk_kernel_params<__half, int, 256, 4>]\u001b[m\u001b[K’:\n",
      "\u001b[01m\u001b[K/dli/task/FasterTransformer/src/fastertransformer/kernels/sampling_topp_kernels.cu:564:105:\u001b[m\u001b[K   required from here\n",
      "\u001b[01m\u001b[K/dli/task/FasterTransformer/src/fastertransformer/kernels/sampling_topp_kernels.cu:537:56:\u001b[m\u001b[K \u001b[01;35m\u001b[Kwarning: \u001b[m\u001b[Kcomparison of integer expressions of different signedness: ‘\u001b[01m\u001b[Klong unsigned int\u001b[m\u001b[K’ and ‘\u001b[01m\u001b[Kconst int\u001b[m\u001b[K’ [\u001b[01;35m\u001b[K-Wsign-compare\u001b[m\u001b[K]\n",
      "/dli/task/FasterTransformer/src/fastertransformer/kernels/sampling_topp_kernels.cu: In instantiation of ‘\u001b[01m\u001b[Kint fastertransformer::segmented_topp_impl::getSmemSizeAndCheck(const fastertransformer::segmented_topp_impl::TopKPerSegmentContext&, const fastertransformer::segmented_topp_impl::TopKPerSegmentParams&) [with Kernel_params = fastertransformer::segmented_topp_impl::Segmented_topk_kernel_params<__half, int, 256, 1>]\u001b[m\u001b[K’:\n",
      "\u001b[01m\u001b[K/dli/task/FasterTransformer/src/fastertransformer/kernels/sampling_topp_kernels.cu:567:105:\u001b[m\u001b[K   required from here\n",
      "\u001b[01m\u001b[K/dli/task/FasterTransformer/src/fastertransformer/kernels/sampling_topp_kernels.cu:537:56:\u001b[m\u001b[K \u001b[01;35m\u001b[Kwarning: \u001b[m\u001b[Kcomparison of integer expressions of different signedness: ‘\u001b[01m\u001b[Klong unsigned int\u001b[m\u001b[K’ and ‘\u001b[01m\u001b[Kconst int\u001b[m\u001b[K’ [\u001b[01;35m\u001b[K-Wsign-compare\u001b[m\u001b[K]\n",
      "/dli/task/FasterTransformer/src/fastertransformer/kernels/sampling_topp_kernels.cu: In instantiation of ‘\u001b[01m\u001b[Kvoid fastertransformer::invokeTopPSampling(void*, size_t&, size_t&, int*, int*, bool*, float*, float*, const T*, const int*, int*, int*, curandState_t*, int, size_t, const int*, float, cudaStream_t, cudaDeviceProp*) [with T = float; size_t = long unsigned int; curandState_t = curandStateXORWOW; cudaStream_t = CUstream_st*]\u001b[m\u001b[K’:\n",
      "\u001b[01m\u001b[K/dli/task/FasterTransformer/src/fastertransformer/kernels/sampling_topp_kernels.cu:1127:485:\u001b[m\u001b[K   required from here\n",
      "\u001b[01m\u001b[K/dli/task/FasterTransformer/src/fastertransformer/kernels/sampling_topp_kernels.cu:1032:7:\u001b[m\u001b[K \u001b[01;35m\u001b[Kwarning: \u001b[m\u001b[K‘\u001b[01m\u001b[Kvoid* memset(void*, int, size_t)\u001b[m\u001b[K’ clearing an object of non-trivial type ‘\u001b[01m\u001b[Kstruct fastertransformer::segmented_topp_impl::TopKPerSegmentContext\u001b[m\u001b[K’; use assignment or value-initialization instead [\u001b[01;35m\u001b[K-Wclass-memaccess\u001b[m\u001b[K]\n",
      " 1032 | \u001b[01;35m\u001b[K        memset(&context, 0, sizeof(\u001b[m\u001b[Kcontext));\n",
      "      |       \u001b[01;35m\u001b[K^\u001b[m\u001b[K \u001b[01;35m\u001b[K~~~~~~~~~~~~~~~~~~~~~~~~~~~\u001b[m\u001b[K\n",
      "\u001b[01m\u001b[K/dli/task/FasterTransformer/src/fastertransformer/kernels/sampling_topp_kernels.h:86:8:\u001b[m\u001b[K \u001b[01;36m\u001b[Knote: \u001b[m\u001b[K‘\u001b[01m\u001b[Kstruct fastertransformer::segmented_topp_impl::TopKPerSegmentContext\u001b[m\u001b[K’ declared here\n",
      "   86 | struct \u001b[01;36m\u001b[KTopKPerSegmentContext\u001b[m\u001b[K {\n",
      "      |        \u001b[01;36m\u001b[K^~~~~~~~~~~~~~~~~~~~~\u001b[m\u001b[K\n",
      "/dli/task/FasterTransformer/src/fastertransformer/kernels/sampling_topp_kernels.cu: In instantiation of ‘\u001b[01m\u001b[Kvoid fastertransformer::invokeTopPSampling(void*, size_t&, size_t&, int*, int*, bool*, float*, float*, const T*, const int*, int*, int*, curandState_t*, int, size_t, const int*, float, cudaStream_t, cudaDeviceProp*) [with T = __half; size_t = long unsigned int; curandState_t = curandStateXORWOW; cudaStream_t = CUstream_st*]\u001b[m\u001b[K’:\n",
      "\u001b[01m\u001b[K/dli/task/FasterTransformer/src/fastertransformer/kernels/sampling_topp_kernels.cu:1147:484:\u001b[m\u001b[K   required from here\n",
      "\u001b[01m\u001b[K/dli/task/FasterTransformer/src/fastertransformer/kernels/sampling_topp_kernels.cu:1032:7:\u001b[m\u001b[K \u001b[01;35m\u001b[Kwarning: \u001b[m\u001b[K‘\u001b[01m\u001b[Kvoid* memset(void*, int, size_t)\u001b[m\u001b[K’ clearing an object of non-trivial type ‘\u001b[01m\u001b[Kstruct fastertransformer::segmented_topp_impl::TopKPerSegmentContext\u001b[m\u001b[K’; use assignment or value-initialization instead [\u001b[01;35m\u001b[K-Wclass-memaccess\u001b[m\u001b[K]\n",
      " 1032 | \u001b[01;35m\u001b[K        memset(&context, 0, sizeof(\u001b[m\u001b[Kcontext));\n",
      "      |       \u001b[01;35m\u001b[K^\u001b[m\u001b[K \u001b[01;35m\u001b[K~~~~~~~~~~~~~~~~~~~~~~~~~~~\u001b[m\u001b[K\n",
      "\u001b[01m\u001b[K/dli/task/FasterTransformer/src/fastertransformer/kernels/sampling_topp_kernels.h:86:8:\u001b[m\u001b[K \u001b[01;36m\u001b[Knote: \u001b[m\u001b[K‘\u001b[01m\u001b[Kstruct fastertransformer::segmented_topp_impl::TopKPerSegmentContext\u001b[m\u001b[K’ declared here\n",
      "   86 | struct \u001b[01;36m\u001b[KTopKPerSegmentContext\u001b[m\u001b[K {\n",
      "      |        \u001b[01;36m\u001b[K^~~~~~~~~~~~~~~~~~~~~\u001b[m\u001b[K\n",
      "[ 83%] \u001b[32m\u001b[1mLinking CUDA device code CMakeFiles/sampling_topp_kernels.dir/cmake_device_link.o\u001b[0m\n",
      "[ 83%] \u001b[32m\u001b[1mLinking CUDA static library ../../../lib/libsampling_topp_kernels.a\u001b[0m\n",
      "[ 83%] Built target sampling_topp_kernels\n",
      "\u001b[35m\u001b[1mScanning dependencies of target TopPSamplingLayer\u001b[0m\n",
      "\u001b[35m\u001b[1mScanning dependencies of target TopKTopPSamplingLayer\u001b[0m\n",
      "[ 83%] \u001b[32mBuilding CUDA object src/fastertransformer/layers/sampling_layers/CMakeFiles/TopKTopPSamplingLayer.dir/TopKTopPSamplingLayer.cu.o\u001b[0m\n",
      "[ 84%] \u001b[32mBuilding CUDA object src/fastertransformer/layers/sampling_layers/CMakeFiles/TopPSamplingLayer.dir/TopPSamplingLayer.cu.o\u001b[0m\n",
      "[ 84%] \u001b[32m\u001b[1mLinking CUDA device code CMakeFiles/TopKTopPSamplingLayer.dir/cmake_device_link.o\u001b[0m\n",
      "[ 84%] \u001b[32m\u001b[1mLinking CUDA device code CMakeFiles/TopPSamplingLayer.dir/cmake_device_link.o\u001b[0m\n",
      "[ 84%] \u001b[32m\u001b[1mLinking CXX static library ../../../../lib/libTopKTopPSamplingLayer.a\u001b[0m\n",
      "[ 84%] Built target TopKTopPSamplingLayer\n",
      "[ 84%] \u001b[32m\u001b[1mLinking CXX static library ../../../../lib/libTopPSamplingLayer.a\u001b[0m\n",
      "[ 84%] Built target TopPSamplingLayer\n",
      "[ 84%] \u001b[32m\u001b[1mLinking CUDA device code CMakeFiles/online_softmax_beamsearch_kernels.dir/cmake_device_link.o\u001b[0m\n",
      "[ 84%] \u001b[32m\u001b[1mLinking CUDA static library ../../../lib/libonline_softmax_beamsearch_kernels.a\u001b[0m\n",
      "[ 84%] Built target online_softmax_beamsearch_kernels\n",
      "\u001b[35m\u001b[1mScanning dependencies of target OnlineBeamSearchLayer\u001b[0m\n",
      "[ 84%] \u001b[32mBuilding CUDA object src/fastertransformer/layers/beam_search_layers/CMakeFiles/OnlineBeamSearchLayer.dir/OnlineBeamSearchLayer.cu.o\u001b[0m\n",
      "[ 84%] \u001b[32m\u001b[1mLinking CUDA device code CMakeFiles/OnlineBeamSearchLayer.dir/cmake_device_link.o\u001b[0m\n",
      "[ 84%] \u001b[32m\u001b[1mLinking CUDA static library ../../../../lib/libOnlineBeamSearchLayer.a\u001b[0m\n",
      "[ 84%] Built target OnlineBeamSearchLayer\n",
      "\u001b[35m\u001b[1mScanning dependencies of target DynamicDecodeLayer\u001b[0m\n",
      "[ 84%] \u001b[32mBuilding CXX object src/fastertransformer/layers/CMakeFiles/DynamicDecodeLayer.dir/DynamicDecodeLayer.cc.o\u001b[0m\n",
      "In file included from \u001b[01m\u001b[K/dli/task/FasterTransformer/src/fastertransformer/utils/Tensor.h:20\u001b[m\u001b[K,\n",
      "                 from \u001b[01m\u001b[K/dli/task/FasterTransformer/src/fastertransformer/layers/BaseLayer.h:21\u001b[m\u001b[K,\n",
      "                 from \u001b[01m\u001b[K/dli/task/FasterTransformer/src/fastertransformer/layers/DynamicDecodeLayer.h:22\u001b[m\u001b[K,\n",
      "                 from \u001b[01m\u001b[K/dli/task/FasterTransformer/src/fastertransformer/layers/DynamicDecodeLayer.cc:17\u001b[m\u001b[K:\n",
      "\u001b[01m\u001b[K/dli/task/FasterTransformer/src/fastertransformer/utils/Tensor.h:\u001b[m\u001b[K In member function ‘\u001b[01m\u001b[Kint fastertransformer::Tensor::getDataTypeByteNum(fastertransformer::DataType) const\u001b[m\u001b[K’:\n",
      "\u001b[01m\u001b[K/dli/task/FasterTransformer/src/fastertransformer/utils/cuda_utils.h:337:31:\u001b[m\u001b[K \u001b[01;35m\u001b[Kwarning: \u001b[m\u001b[Kcontrol reaches end of non-void function [\u001b[01;35m\u001b[K-Wreturn-type\u001b[m\u001b[K]\n",
      "  337 | #define FT_CHECK(val) \u001b[01;35m\u001b[KmyAssert(val, __FILE__, __LINE__)\u001b[m\u001b[K\n",
      "      |                       \u001b[01;35m\u001b[K~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~\u001b[m\u001b[K\n",
      "\u001b[01m\u001b[K/dli/task/FasterTransformer/src/fastertransformer/utils/Tensor.h:254:17:\u001b[m\u001b[K \u001b[01;36m\u001b[Knote: \u001b[m\u001b[Kin expansion of macro ‘\u001b[01m\u001b[KFT_CHECK\u001b[m\u001b[K’\n",
      "  254 |                 \u001b[01;36m\u001b[KFT_CHECK\u001b[m\u001b[K(false);\n",
      "      |                 \u001b[01;36m\u001b[K^~~~~~~~\u001b[m\u001b[K\n",
      "[ 84%] \u001b[32m\u001b[1mLinking CUDA device code CMakeFiles/DynamicDecodeLayer.dir/cmake_device_link.o\u001b[0m\n",
      "[ 84%] \u001b[32m\u001b[1mLinking CXX static library ../../../lib/libDynamicDecodeLayer.a\u001b[0m\n",
      "[ 84%] Built target DynamicDecodeLayer\n",
      "\u001b[35m\u001b[1mScanning dependencies of target test_sampling\u001b[0m\n",
      "[ 84%] \u001b[32mBuilding CUDA object tests/unittests/CMakeFiles/test_sampling.dir/test_sampling.cu.o\u001b[0m\n",
      "\u001b[01m\u001b[K/dli/task/FasterTransformer/src/fastertransformer/utils/Tensor.h:\u001b[m\u001b[K In member function ‘\u001b[01m\u001b[Kint fastertransformer::Tensor::getDataTypeByteNum(fastertransformer::DataType) const\u001b[m\u001b[K’:\n",
      "\u001b[01m\u001b[K/dli/task/FasterTransformer/src/fastertransformer/utils/Tensor.h:254:9:\u001b[m\u001b[K \u001b[01;35m\u001b[Kwarning: \u001b[m\u001b[Kcontrol reaches end of non-void function [\u001b[01;35m\u001b[K-Wreturn-type\u001b[m\u001b[K]\n",
      "  254 | \u001b[01;35m\u001b[K                FT_CHECK(false);\u001b[m\u001b[K\n",
      "      |         \u001b[01;35m\u001b[K^\u001b[m\u001b[K       \u001b[01;35m\u001b[K~~~~~~~~~~~~~~~~\u001b[m\u001b[K                                                        \n",
      "[ 84%] \u001b[32m\u001b[1mLinking CXX executable ../../bin/test_sampling\u001b[0m\n",
      "[ 84%] Built target test_sampling\n",
      "[ 84%] \u001b[32m\u001b[1mLinking CUDA device code CMakeFiles/decoder_masked_multihead_attention.dir/cmake_device_link.o\u001b[0m\n",
      "[ 84%] \u001b[32m\u001b[1mLinking CUDA static library ../../../lib/libdecoder_masked_multihead_attention.a\u001b[0m\n",
      "[ 84%] Built target decoder_masked_multihead_attention\n",
      "\u001b[35m\u001b[1mScanning dependencies of target DecoderSelfAttentionLayer\u001b[0m\n",
      "\u001b[35m\u001b[1mScanning dependencies of target DecoderCrossAttentionLayer\u001b[0m\n",
      "[ 84%] \u001b[32mBuilding CXX object src/fastertransformer/layers/attention_layers/CMakeFiles/DecoderSelfAttentionLayer.dir/DecoderSelfAttentionLayer.cc.o\u001b[0m\n",
      "[ 84%] \u001b[32mBuilding CUDA object src/fastertransformer/layers/attention_layers/CMakeFiles/DecoderCrossAttentionLayer.dir/DecoderCrossAttentionLayer.cu.o\u001b[0m\n",
      "\u001b[01m\u001b[0m\u001b[01m/dli/task/FasterTransformer/src/fastertransformer/layers/attention_layers/DecoderCrossAttentionLayer.cu(76)\u001b[0m: \u001b[01;35mwarning\u001b[0m #1886-D: specified alignment (2) is different from alignment (4) specified on a previous declaration\n",
      "          detected during:\n",
      "            instantiation of \u001b[01m\"void fastertransformer::cross_attention_kernel(T *, const T *, T *, const T *, T *, const T *, const int *, T *, const __nv_bool *, int, int, int, int, int, T) [with T=half]\"\u001b[0m \u001b[32m\n",
      "(443): here\u001b[0m\n",
      "            instantiation of \u001b[01m\"void fastertransformer::cross_attention_dispatch(T *, const T *, T *, const T *, T *, const T *, const int *, T *, const __nv_bool *, int, int, int, int, int, int, __nv_bool, float, cudaStream_t) [with T=half]\"\u001b[0m \u001b[32m\n",
      "(504): here\u001b[0m\n",
      "\n",
      "[ 84%] \u001b[32m\u001b[1mLinking CUDA device code CMakeFiles/DecoderSelfAttentionLayer.dir/cmake_device_link.o\u001b[0m\n",
      "[ 84%] \u001b[32m\u001b[1mLinking CXX static library ../../../../lib/libDecoderSelfAttentionLayer.a\u001b[0m\n",
      "[ 84%] Built target DecoderSelfAttentionLayer\n",
      "\u001b[35m\u001b[1mScanning dependencies of target TensorParallelDecoderSelfAttentionLayer\u001b[0m\n",
      "[ 84%] \u001b[32mBuilding CXX object src/fastertransformer/layers/attention_layers/CMakeFiles/TensorParallelDecoderSelfAttentionLayer.dir/TensorParallelDecoderSelfAttentionLayer.cc.o\u001b[0m\n",
      "[ 84%] \u001b[32m\u001b[1mLinking CUDA device code CMakeFiles/TensorParallelDecoderSelfAttentionLayer.dir/cmake_device_link.o\u001b[0m\n",
      "[ 85%] \u001b[32m\u001b[1mLinking CXX static library ../../../../lib/libTensorParallelDecoderSelfAttentionLayer.a\u001b[0m\n",
      "[ 85%] Built target TensorParallelDecoderSelfAttentionLayer\n",
      "\u001b[35m\u001b[1mScanning dependencies of target ParallelGptDecoder\u001b[0m\n",
      "\u001b[35m\u001b[1mScanning dependencies of target GptJDecoder\u001b[0m\n",
      "[ 85%] \u001b[32mBuilding CXX object src/fastertransformer/models/multi_gpu_gpt/CMakeFiles/ParallelGptDecoder.dir/ParallelGptDecoder.cc.o\u001b[0m\n",
      "[ 85%] \u001b[32mBuilding CXX object src/fastertransformer/models/gptj/CMakeFiles/GptJDecoder.dir/GptJDecoder.cc.o\u001b[0m\n",
      "\u001b[01m\u001b[0m\u001b[01m/dli/task/FasterTransformer/src/fastertransformer/layers/attention_layers/DecoderCrossAttentionLayer.cu(76)\u001b[0m: \u001b[01;35mwarning\u001b[0m #1886-D: specified alignment (2) is different from alignment (4) specified on a previous declaration\n",
      "          detected during:\n",
      "            instantiation of \u001b[01m\"void fastertransformer::cross_attention_kernel(T *, const T *, T *, const T *, T *, const T *, const int *, T *, const __nv_bool *, int, int, int, int, int, T) [with T=half]\"\u001b[0m \u001b[32m\n",
      "(443): here\u001b[0m\n",
      "            instantiation of \u001b[01m\"void fastertransformer::cross_attention_dispatch(T *, const T *, T *, const T *, T *, const T *, const int *, T *, const __nv_bool *, int, int, int, int, int, int, __nv_bool, float, cudaStream_t) [with T=half]\"\u001b[0m \u001b[32m\n",
      "(504): here\u001b[0m\n",
      "\n",
      "[ 85%] \u001b[32m\u001b[1mLinking CUDA device code CMakeFiles/GptJDecoder.dir/cmake_device_link.o\u001b[0m\n",
      "[ 86%] \u001b[32m\u001b[1mLinking CXX static library ../../../../lib/libGptJDecoder.a\u001b[0m\n",
      "[ 86%] Built target GptJDecoder\n",
      "\u001b[35m\u001b[1mScanning dependencies of target GptJ\u001b[0m\n",
      "[ 86%] \u001b[32mBuilding CXX object src/fastertransformer/models/gptj/CMakeFiles/GptJ.dir/GptJ.cc.o\u001b[0m\n",
      "[ 86%] \u001b[32m\u001b[1mLinking CUDA device code CMakeFiles/ParallelGptDecoder.dir/cmake_device_link.o\u001b[0m\n",
      "[ 86%] \u001b[32m\u001b[1mLinking CXX static library ../../../../lib/libParallelGptDecoder.a\u001b[0m\n",
      "[ 86%] Built target ParallelGptDecoder\n",
      "\u001b[35m\u001b[1mScanning dependencies of target ParallelGpt\u001b[0m\n",
      "[ 87%] \u001b[32mBuilding CXX object src/fastertransformer/models/multi_gpu_gpt/CMakeFiles/ParallelGpt.dir/ParallelGpt.cc.o\u001b[0m\n",
      "\u001b[01m\u001b[0m\u001b[01m/dli/task/FasterTransformer/src/fastertransformer/layers/attention_layers/DecoderCrossAttentionLayer.cu(76)\u001b[0m: \u001b[01;35mwarning\u001b[0m #1886-D: specified alignment (2) is different from alignment (4) specified on a previous declaration\n",
      "          detected during:\n",
      "            instantiation of \u001b[01m\"void fastertransformer::cross_attention_kernel(T *, const T *, T *, const T *, T *, const T *, const int *, T *, const __nv_bool *, int, int, int, int, int, T) [with T=half]\"\u001b[0m \u001b[32m\n",
      "(443): here\u001b[0m\n",
      "            instantiation of \u001b[01m\"void fastertransformer::cross_attention_dispatch(T *, const T *, T *, const T *, T *, const T *, const int *, T *, const __nv_bool *, int, int, int, int, int, int, __nv_bool, float, cudaStream_t) [with T=half]\"\u001b[0m \u001b[32m\n",
      "(504): here\u001b[0m\n",
      "\n",
      "[ 87%] \u001b[32m\u001b[1mLinking CUDA device code CMakeFiles/GptJ.dir/cmake_device_link.o\u001b[0m\n",
      "[ 87%] \u001b[32m\u001b[1mLinking CXX static library ../../../../lib/libGptJ.a\u001b[0m\n",
      "[ 87%] Built target GptJ\n",
      "\u001b[35m\u001b[1mScanning dependencies of target gptj_example\u001b[0m\n",
      "\u001b[35m\u001b[1mScanning dependencies of target GptJTritonBackend\u001b[0m\n",
      "[ 87%] \u001b[32mBuilding CXX object examples/cpp/gptj/CMakeFiles/gptj_example.dir/gptj_example.cc.o\u001b[0m\n",
      "[ 88%] \u001b[32mBuilding CXX object src/fastertransformer/triton_backend/gptj/CMakeFiles/GptJTritonBackend.dir/GptJTritonModel.cc.o\u001b[0m\n",
      "[ 88%] \u001b[32mBuilding CXX object src/fastertransformer/triton_backend/gptj/CMakeFiles/GptJTritonBackend.dir/GptJTritonModelInstance.cc.o\u001b[0m\n",
      "\u001b[01m\u001b[0m\u001b[01m/dli/task/FasterTransformer/src/fastertransformer/layers/attention_layers/DecoderCrossAttentionLayer.cu(76)\u001b[0m: \u001b[01;35mwarning\u001b[0m #1886-D: specified alignment (2) is different from alignment (4) specified on a previous declaration\n",
      "          detected during:\n",
      "            instantiation of \u001b[01m\"void fastertransformer::cross_attention_kernel(T *, const T *, T *, const T *, T *, const T *, const int *, T *, const __nv_bool *, int, int, int, int, int, T) [with T=half]\"\u001b[0m \u001b[32m\n",
      "(443): here\u001b[0m\n",
      "            instantiation of \u001b[01m\"void fastertransformer::cross_attention_dispatch(T *, const T *, T *, const T *, T *, const T *, const int *, T *, const __nv_bool *, int, int, int, int, int, int, __nv_bool, float, cudaStream_t) [with T=half]\"\u001b[0m \u001b[32m\n",
      "(504): here\u001b[0m\n",
      "\n",
      "[ 88%] \u001b[32m\u001b[1mLinking CUDA device code CMakeFiles/ParallelGpt.dir/cmake_device_link.o\u001b[0m\n",
      "[ 88%] \u001b[32m\u001b[1mLinking CXX static library ../../../../lib/libParallelGpt.a\u001b[0m\n",
      "[ 88%] Built target ParallelGpt\n",
      "\u001b[35m\u001b[1mScanning dependencies of target gpt_example\u001b[0m\n",
      "\u001b[35m\u001b[1mScanning dependencies of target ParallelGptTritonBackend\u001b[0m\n",
      "\u001b[35m\u001b[1mScanning dependencies of target multi_gpu_gpt_async_example\u001b[0m\n",
      "\u001b[35m\u001b[1mScanning dependencies of target multi_gpu_gpt_example\u001b[0m\n",
      "[ 88%] \u001b[32mBuilding CXX object examples/cpp/gpt/CMakeFiles/gpt_example.dir/gpt_example.cc.o\u001b[0m\n",
      "[ 88%] \u001b[32mBuilding CXX object examples/cpp/multi_gpu_gpt/CMakeFiles/multi_gpu_gpt_async_example.dir/multi_gpu_gpt_async_example.cc.o\u001b[0m\n",
      "[ 88%] \u001b[32mBuilding CXX object src/fastertransformer/triton_backend/multi_gpu_gpt/CMakeFiles/ParallelGptTritonBackend.dir/ParallelGptTritonModel.cc.o\u001b[0m\n",
      "[ 88%] \u001b[32mBuilding CXX object src/fastertransformer/triton_backend/multi_gpu_gpt/CMakeFiles/ParallelGptTritonBackend.dir/ParallelGptTritonModelInstance.cc.o\u001b[0m\n",
      "[ 88%] \u001b[32mBuilding CXX object examples/cpp/multi_gpu_gpt/CMakeFiles/multi_gpu_gpt_example.dir/multi_gpu_gpt_example.cc.o\u001b[0m\n",
      "[ 88%] \u001b[32m\u001b[1mLinking CXX executable ../../../bin/gptj_example\u001b[0m\n",
      "[ 88%] \u001b[32m\u001b[1mLinking CXX shared library ../../../../lib/libGptJTritonBackend.so\u001b[0m\n",
      "[ 88%] Built target GptJTritonBackend\n",
      "[ 88%] Built target gptj_example\n",
      "\u001b[35m\u001b[1mScanning dependencies of target gptj_triton_example\u001b[0m\n",
      "[ 88%] \u001b[32mBuilding CXX object examples/cpp/gptj/CMakeFiles/gptj_triton_example.dir/gptj_triton_example.cc.o\u001b[0m\n",
      "[ 89%] \u001b[32m\u001b[1mLinking CXX shared library ../../../../lib/libParallelGptTritonBackend.so\u001b[0m\n",
      "[ 89%] \u001b[32m\u001b[1mLinking CXX executable ../../../bin/multi_gpu_gpt_example\u001b[0m\n",
      "[ 90%] \u001b[32m\u001b[1mLinking CXX executable ../../../bin/gpt_example\u001b[0m\n",
      "[ 90%] Built target ParallelGptTritonBackend\n",
      "\u001b[35m\u001b[1mScanning dependencies of target multi_gpu_gpt_triton_example\u001b[0m\n",
      "[ 90%] \u001b[32mBuilding CXX object examples/cpp/multi_gpu_gpt/CMakeFiles/multi_gpu_gpt_triton_example.dir/multi_gpu_gpt_triton_example.cc.o\u001b[0m\n",
      "[ 90%] Built target multi_gpu_gpt_example\n",
      "[ 90%] Built target gpt_example\n",
      "/dli/task/FasterTransformer/src/fastertransformer/layers/attention_layers/DecoderCrossAttentionLayer.cu: In instantiation of ‘\u001b[01m\u001b[Kvoid fastertransformer::cross_attention_dispatch(T*, const T*, T*, const T*, T*, const T*, const int*, T*, const bool*, int, int, int, int, int, int, bool, float, cudaStream_t) [with T = float; cudaStream_t = CUstream_st*]\u001b[m\u001b[K’:\n",
      "\u001b[01m\u001b[K/dli/task/FasterTransformer/src/fastertransformer/layers/attention_layers/DecoderCrossAttentionLayer.cu:485:448:\u001b[m\u001b[K   required from here\n",
      "\u001b[01m\u001b[K/dli/task/FasterTransformer/src/fastertransformer/layers/attention_layers/DecoderCrossAttentionLayer.cu:452:7:\u001b[m\u001b[K \u001b[01;35m\u001b[Kwarning: \u001b[m\u001b[K‘\u001b[01m\u001b[Kvoid* memset(void*, int, size_t)\u001b[m\u001b[K’ clearing an object of non-trivial type ‘\u001b[01m\u001b[KCross_multihead_attention_params<float>\u001b[m\u001b[K’ {aka ‘\u001b[01m\u001b[Kstruct Multihead_attention_params<float, true>\u001b[m\u001b[K’}; use assignment or value-initialization instead [\u001b[01;35m\u001b[K-Wclass-memaccess\u001b[m\u001b[K]\n",
      "  452 | \u001b[01;35m\u001b[K        memset(&params, 0, sizeof(\u001b[m\u001b[Kparams));\n",
      "      |       \u001b[01;35m\u001b[K^\u001b[m\u001b[K \u001b[01;35m\u001b[K~~~~~~~~~~~~~~~~~~~~~~~~~~\u001b[m\u001b[K\n",
      "\u001b[01m\u001b[K/dli/task/FasterTransformer/src/fastertransformer/kernels/decoder_masked_multihead_attention.h:115:8:\u001b[m\u001b[K \u001b[01;36m\u001b[Knote: \u001b[m\u001b[K‘\u001b[01m\u001b[KCross_multihead_attention_params<float>\u001b[m\u001b[K’ {aka ‘\u001b[01m\u001b[Kstruct Multihead_attention_params<float, true>\u001b[m\u001b[K’} declared here\n",
      "  115 | struct \u001b[01;36m\u001b[KMultihead_attention_params<T, true>:\u001b[m\u001b[K public Multihead_attention_params_base<T> {\n",
      "      |        \u001b[01;36m\u001b[K^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\u001b[m\u001b[K\n",
      "/dli/task/FasterTransformer/src/fastertransformer/layers/attention_layers/DecoderCrossAttentionLayer.cu: In instantiation of ‘\u001b[01m\u001b[Kvoid fastertransformer::cross_attention_dispatch(T*, const T*, T*, const T*, T*, const T*, const int*, T*, const bool*, int, int, int, int, int, int, bool, float, cudaStream_t) [with T = __half; cudaStream_t = CUstream_st*]\u001b[m\u001b[K’:\n",
      "\u001b[01m\u001b[K/dli/task/FasterTransformer/src/fastertransformer/layers/attention_layers/DecoderCrossAttentionLayer.cu:504:442:\u001b[m\u001b[K   required from here\n",
      "\u001b[01m\u001b[K/dli/task/FasterTransformer/src/fastertransformer/layers/attention_layers/DecoderCrossAttentionLayer.cu:452:7:\u001b[m\u001b[K \u001b[01;35m\u001b[Kwarning: \u001b[m\u001b[K‘\u001b[01m\u001b[Kvoid* memset(void*, int, size_t)\u001b[m\u001b[K’ clearing an object of non-trivial type ‘\u001b[01m\u001b[KCross_multihead_attention_params<short unsigned int>\u001b[m\u001b[K’ {aka ‘\u001b[01m\u001b[Kstruct Multihead_attention_params<short unsigned int, true>\u001b[m\u001b[K’}; use assignment or value-initialization instead [\u001b[01;35m\u001b[K-Wclass-memaccess\u001b[m\u001b[K]\n",
      "  452 | \u001b[01;35m\u001b[K        memset(&params, 0, sizeof(\u001b[m\u001b[Kparams));\n",
      "      |       \u001b[01;35m\u001b[K^\u001b[m\u001b[K \u001b[01;35m\u001b[K~~~~~~~~~~~~~~~~~~~~~~~~~~\u001b[m\u001b[K\n",
      "\u001b[01m\u001b[K/dli/task/FasterTransformer/src/fastertransformer/kernels/decoder_masked_multihead_attention.h:115:8:\u001b[m\u001b[K \u001b[01;36m\u001b[Knote: \u001b[m\u001b[K‘\u001b[01m\u001b[KCross_multihead_attention_params<short unsigned int>\u001b[m\u001b[K’ {aka ‘\u001b[01m\u001b[Kstruct Multihead_attention_params<short unsigned int, true>\u001b[m\u001b[K’} declared here\n",
      "  115 | struct \u001b[01;36m\u001b[KMultihead_attention_params<T, true>:\u001b[m\u001b[K public Multihead_attention_params_base<T> {\n",
      "      |        \u001b[01;36m\u001b[K^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\u001b[m\u001b[K\n",
      "[ 91%] \u001b[32m\u001b[1mLinking CXX executable ../../../bin/multi_gpu_gpt_async_example\u001b[0m\n",
      "[ 92%] \u001b[32m\u001b[1mLinking CXX executable ../../../bin/gptj_triton_example\u001b[0m\n",
      "[ 92%] Built target gptj_triton_example\n",
      "[ 92%] Built target multi_gpu_gpt_async_example\n",
      "[ 92%] \u001b[32m\u001b[1mLinking CUDA device code CMakeFiles/DecoderCrossAttentionLayer.dir/cmake_device_link.o\u001b[0m\n",
      "[ 93%] \u001b[32m\u001b[1mLinking CXX static library ../../../../lib/libDecoderCrossAttentionLayer.a\u001b[0m\n",
      "[ 93%] Built target DecoderCrossAttentionLayer\n",
      "\u001b[35m\u001b[1mScanning dependencies of target Decoder\u001b[0m\n",
      "\u001b[35m\u001b[1mScanning dependencies of target TensorParallelDecoderCrossAttentionLayer\u001b[0m\n",
      "[ 94%] \u001b[32mBuilding CXX object src/fastertransformer/models/decoder/CMakeFiles/Decoder.dir/Decoder.cc.o\u001b[0m\n",
      "[ 95%] \u001b[32mBuilding CXX object src/fastertransformer/layers/attention_layers/CMakeFiles/TensorParallelDecoderCrossAttentionLayer.dir/TensorParallelDecoderCrossAttentionLayer.cc.o\u001b[0m\n",
      "[ 95%] \u001b[32m\u001b[1mLinking CUDA device code CMakeFiles/TensorParallelDecoderCrossAttentionLayer.dir/cmake_device_link.o\u001b[0m\n",
      "[ 95%] \u001b[32m\u001b[1mLinking CXX static library ../../../../lib/libTensorParallelDecoderCrossAttentionLayer.a\u001b[0m\n",
      "[ 95%] Built target TensorParallelDecoderCrossAttentionLayer\n",
      "\u001b[35m\u001b[1mScanning dependencies of target T5Decoder\u001b[0m\n",
      "[ 95%] \u001b[32mBuilding CXX object src/fastertransformer/models/t5/CMakeFiles/T5Decoder.dir/T5Decoder.cc.o\u001b[0m\n",
      "[ 95%] \u001b[32mBuilding CXX object src/fastertransformer/models/t5/CMakeFiles/T5Decoder.dir/T5DecoderLayerWeight.cc.o\u001b[0m\n",
      "[ 95%] \u001b[32m\u001b[1mLinking CXX executable ../../../bin/multi_gpu_gpt_triton_example\u001b[0m\n",
      "[ 95%] Built target multi_gpu_gpt_triton_example\n",
      "[ 95%] \u001b[32m\u001b[1mLinking CUDA device code CMakeFiles/Decoder.dir/cmake_device_link.o\u001b[0m\n",
      "[ 95%] \u001b[32m\u001b[1mLinking CXX static library ../../../../lib/libDecoder.a\u001b[0m\n",
      "[ 95%] Built target Decoder\n",
      "\u001b[35m\u001b[1mScanning dependencies of target Decoding\u001b[0m\n",
      "[ 95%] \u001b[32mBuilding CXX object src/fastertransformer/models/decoding/CMakeFiles/Decoding.dir/Decoding.cc.o\u001b[0m\n",
      "[ 96%] \u001b[32m\u001b[1mLinking CUDA device code CMakeFiles/T5Decoder.dir/cmake_device_link.o\u001b[0m\n",
      "[ 96%] \u001b[32m\u001b[1mLinking CXX static library ../../../../lib/libT5Decoder.a\u001b[0m\n",
      "[ 96%] Built target T5Decoder\n",
      "\u001b[35m\u001b[1mScanning dependencies of target T5Decoding\u001b[0m\n",
      "[ 96%] \u001b[32mBuilding CXX object src/fastertransformer/models/t5/CMakeFiles/T5Decoding.dir/T5DecodingWeight.cc.o\u001b[0m\n",
      "[ 96%] \u001b[32mBuilding CXX object src/fastertransformer/models/t5/CMakeFiles/T5Decoding.dir/T5Decoding.cc.o\u001b[0m\n",
      "/dli/task/FasterTransformer/src/fastertransformer/models/t5/T5DecodingWeight.cc: In instantiation of ‘\u001b[01m\u001b[Kvoid fastertransformer::T5DecodingWeight<T>::loadModel(std::string) [with T = float; std::string = std::__cxx11::basic_string<char>]\u001b[m\u001b[K’:\n",
      "\u001b[01m\u001b[K/dli/task/FasterTransformer/src/fastertransformer/models/t5/T5DecodingWeight.cc:275:17:\u001b[m\u001b[K   required from here\n",
      "\u001b[01m\u001b[K/dli/task/FasterTransformer/src/fastertransformer/models/t5/T5DecodingWeight.cc:231:29:\u001b[m\u001b[K \u001b[01;35m\u001b[Kwarning: \u001b[m\u001b[Knarrowing conversion of ‘\u001b[01m\u001b[K((fastertransformer::T5DecodingWeight<float>*)this)->fastertransformer::T5DecodingWeight<float>::weights_size[4]\u001b[m\u001b[K’ from ‘\u001b[01m\u001b[Ksize_t\u001b[m\u001b[K’ {aka ‘\u001b[01m\u001b[Klong unsigned int\u001b[m\u001b[K’} to ‘\u001b[01m\u001b[Kint\u001b[m\u001b[K’ [\u001b[01;35m\u001b[K-Wnarrowing\u001b[m\u001b[K]\n",
      "  231 |         \u001b[01;35m\u001b[KloadWeightFromBin<T>(weights_ptr[4], {weights_size[4]}, dir_path + \"/decoder.final_layer_norm.bias.bin\")\u001b[m\u001b[K;\n",
      "      |         \u001b[01;35m\u001b[K~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\u001b[m\u001b[K\n",
      "\u001b[01m\u001b[K/dli/task/FasterTransformer/src/fastertransformer/models/t5/T5DecodingWeight.cc:231:29:\u001b[m\u001b[K \u001b[01;35m\u001b[Kwarning: \u001b[m\u001b[Knarrowing conversion of ‘\u001b[01m\u001b[K((fastertransformer::T5DecodingWeight<float>*)this)->fastertransformer::T5DecodingWeight<float>::weights_size[4]\u001b[m\u001b[K’ from ‘\u001b[01m\u001b[Ksize_t\u001b[m\u001b[K’ {aka ‘\u001b[01m\u001b[Klong unsigned int\u001b[m\u001b[K’} to ‘\u001b[01m\u001b[Kint\u001b[m\u001b[K’ [\u001b[01;35m\u001b[K-Wnarrowing\u001b[m\u001b[K]\n",
      "\u001b[01m\u001b[K/dli/task/FasterTransformer/src/fastertransformer/models/t5/T5DecodingWeight.cc:232:29:\u001b[m\u001b[K \u001b[01;35m\u001b[Kwarning: \u001b[m\u001b[Knarrowing conversion of ‘\u001b[01m\u001b[K((fastertransformer::T5DecodingWeight<float>*)this)->fastertransformer::T5DecodingWeight<float>::weights_size[5]\u001b[m\u001b[K’ from ‘\u001b[01m\u001b[Ksize_t\u001b[m\u001b[K’ {aka ‘\u001b[01m\u001b[Klong unsigned int\u001b[m\u001b[K’} to ‘\u001b[01m\u001b[Kint\u001b[m\u001b[K’ [\u001b[01;35m\u001b[K-Wnarrowing\u001b[m\u001b[K]\n",
      "  232 |         \u001b[01;35m\u001b[KloadWeightFromBin<T>(weights_ptr[5], {weights_size[5]}, dir_path + \"/shared.bias.bin\")\u001b[m\u001b[K;\n",
      "      |         \u001b[01;35m\u001b[K~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\u001b[m\u001b[K\n",
      "\u001b[01m\u001b[K/dli/task/FasterTransformer/src/fastertransformer/models/t5/T5DecodingWeight.cc:232:29:\u001b[m\u001b[K \u001b[01;35m\u001b[Kwarning: \u001b[m\u001b[Knarrowing conversion of ‘\u001b[01m\u001b[K((fastertransformer::T5DecodingWeight<float>*)this)->fastertransformer::T5DecodingWeight<float>::weights_size[5]\u001b[m\u001b[K’ from ‘\u001b[01m\u001b[Ksize_t\u001b[m\u001b[K’ {aka ‘\u001b[01m\u001b[Klong unsigned int\u001b[m\u001b[K’} to ‘\u001b[01m\u001b[Kint\u001b[m\u001b[K’ [\u001b[01;35m\u001b[K-Wnarrowing\u001b[m\u001b[K]\n",
      "/dli/task/FasterTransformer/src/fastertransformer/models/t5/T5DecodingWeight.cc: In instantiation of ‘\u001b[01m\u001b[Kvoid fastertransformer::T5DecodingWeight<T>::loadModel(std::string) [with T = __half; std::string = std::__cxx11::basic_string<char>]\u001b[m\u001b[K’:\n",
      "\u001b[01m\u001b[K/dli/task/FasterTransformer/src/fastertransformer/models/t5/T5DecodingWeight.cc:276:17:\u001b[m\u001b[K   required from here\n",
      "\u001b[01m\u001b[K/dli/task/FasterTransformer/src/fastertransformer/models/t5/T5DecodingWeight.cc:231:29:\u001b[m\u001b[K \u001b[01;35m\u001b[Kwarning: \u001b[m\u001b[Knarrowing conversion of ‘\u001b[01m\u001b[K((fastertransformer::T5DecodingWeight<__half>*)this)->fastertransformer::T5DecodingWeight<__half>::weights_size[4]\u001b[m\u001b[K’ from ‘\u001b[01m\u001b[Ksize_t\u001b[m\u001b[K’ {aka ‘\u001b[01m\u001b[Klong unsigned int\u001b[m\u001b[K’} to ‘\u001b[01m\u001b[Kint\u001b[m\u001b[K’ [\u001b[01;35m\u001b[K-Wnarrowing\u001b[m\u001b[K]\n",
      "  231 |         \u001b[01;35m\u001b[KloadWeightFromBin<T>(weights_ptr[4], {weights_size[4]}, dir_path + \"/decoder.final_layer_norm.bias.bin\")\u001b[m\u001b[K;\n",
      "      |         \u001b[01;35m\u001b[K~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\u001b[m\u001b[K\n",
      "\u001b[01m\u001b[K/dli/task/FasterTransformer/src/fastertransformer/models/t5/T5DecodingWeight.cc:231:29:\u001b[m\u001b[K \u001b[01;35m\u001b[Kwarning: \u001b[m\u001b[Knarrowing conversion of ‘\u001b[01m\u001b[K((fastertransformer::T5DecodingWeight<__half>*)this)->fastertransformer::T5DecodingWeight<__half>::weights_size[4]\u001b[m\u001b[K’ from ‘\u001b[01m\u001b[Ksize_t\u001b[m\u001b[K’ {aka ‘\u001b[01m\u001b[Klong unsigned int\u001b[m\u001b[K’} to ‘\u001b[01m\u001b[Kint\u001b[m\u001b[K’ [\u001b[01;35m\u001b[K-Wnarrowing\u001b[m\u001b[K]\n",
      "\u001b[01m\u001b[K/dli/task/FasterTransformer/src/fastertransformer/models/t5/T5DecodingWeight.cc:232:29:\u001b[m\u001b[K \u001b[01;35m\u001b[Kwarning: \u001b[m\u001b[Knarrowing conversion of ‘\u001b[01m\u001b[K((fastertransformer::T5DecodingWeight<__half>*)this)->fastertransformer::T5DecodingWeight<__half>::weights_size[5]\u001b[m\u001b[K’ from ‘\u001b[01m\u001b[Ksize_t\u001b[m\u001b[K’ {aka ‘\u001b[01m\u001b[Klong unsigned int\u001b[m\u001b[K’} to ‘\u001b[01m\u001b[Kint\u001b[m\u001b[K’ [\u001b[01;35m\u001b[K-Wnarrowing\u001b[m\u001b[K]\n",
      "  232 |         \u001b[01;35m\u001b[KloadWeightFromBin<T>(weights_ptr[5], {weights_size[5]}, dir_path + \"/shared.bias.bin\")\u001b[m\u001b[K;\n",
      "      |         \u001b[01;35m\u001b[K~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\u001b[m\u001b[K\n",
      "\u001b[01m\u001b[K/dli/task/FasterTransformer/src/fastertransformer/models/t5/T5DecodingWeight.cc:232:29:\u001b[m\u001b[K \u001b[01;35m\u001b[Kwarning: \u001b[m\u001b[Knarrowing conversion of ‘\u001b[01m\u001b[K((fastertransformer::T5DecodingWeight<__half>*)this)->fastertransformer::T5DecodingWeight<__half>::weights_size[5]\u001b[m\u001b[K’ from ‘\u001b[01m\u001b[Ksize_t\u001b[m\u001b[K’ {aka ‘\u001b[01m\u001b[Klong unsigned int\u001b[m\u001b[K’} to ‘\u001b[01m\u001b[Kint\u001b[m\u001b[K’ [\u001b[01;35m\u001b[K-Wnarrowing\u001b[m\u001b[K]\n",
      "[ 97%] \u001b[32m\u001b[1mLinking CUDA device code CMakeFiles/Decoding.dir/cmake_device_link.o\u001b[0m\n",
      "[ 97%] \u001b[32m\u001b[1mLinking CXX static library ../../../../lib/libDecoding.a\u001b[0m\n",
      "[ 97%] Built target Decoding\n",
      "\u001b[35m\u001b[1mScanning dependencies of target decoding_example\u001b[0m\n",
      "[ 98%] \u001b[32mBuilding CXX object examples/cpp/decoding/CMakeFiles/decoding_example.dir/decoding_example.cc.o\u001b[0m\n",
      "[ 98%] \u001b[32m\u001b[1mLinking CUDA device code CMakeFiles/T5Decoding.dir/cmake_device_link.o\u001b[0m\n",
      "[ 99%] \u001b[32m\u001b[1mLinking CXX static library ../../../../lib/libT5Decoding.a\u001b[0m\n",
      "[ 99%] Built target T5Decoding\n",
      "\u001b[35m\u001b[1mScanning dependencies of target T5TritonBackend\u001b[0m\n",
      "[ 99%] \u001b[32mBuilding CXX object src/fastertransformer/triton_backend/t5/CMakeFiles/T5TritonBackend.dir/T5TritonModelInstance.cc.o\u001b[0m\n",
      "[ 99%] \u001b[32mBuilding CXX object src/fastertransformer/triton_backend/t5/CMakeFiles/T5TritonBackend.dir/T5TritonModel.cc.o\u001b[0m\n",
      "[ 99%] \u001b[32m\u001b[1mLinking CXX executable ../../../bin/decoding_example\u001b[0m\n",
      "[ 99%] Built target decoding_example\n",
      "[ 99%] \u001b[32m\u001b[1mLinking CXX shared library ../../../../lib/libT5TritonBackend.so\u001b[0m\n",
      "[ 99%] Built target T5TritonBackend\n",
      "\u001b[35m\u001b[1mScanning dependencies of target transformer-static\u001b[0m\n",
      "\u001b[35m\u001b[1mScanning dependencies of target transformer-shared\u001b[0m\n",
      "[100%] \u001b[32m\u001b[1mLinking CUDA device code CMakeFiles/transformer-shared.dir/cmake_device_link.o\u001b[0m\n",
      "[100%] \u001b[32m\u001b[1mLinking CUDA device code CMakeFiles/transformer-static.dir/cmake_device_link.o\u001b[0m\n",
      "[100%] \u001b[32m\u001b[1mLinking CXX shared library lib/libtransformer-shared.so\u001b[0m\n",
      "[100%] \u001b[32m\u001b[1mLinking CXX static library lib/libtransformer-static.a\u001b[0m\n",
      "[100%] Built target transformer-static\n",
      "[100%] Built target transformer-shared\n"
     ]
    }
   ],
   "source": [
    "!git submodule init && git submodule update\n",
    "!cmake -DSM=xx -DCMAKE_BUILD_TYPE=Release -DBUILD_MULTI_GPU=ON ..\n",
    "!make -j32"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's have a look at all pre-built binary files with examples and useful utilities."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "List of FasterTransformer examples:\n",
      "bert_example\t     layernorm_test\t\t   test_logprob_kernels\n",
      "bert_gemm\t     multi_gpu_gpt_async_example   test_sampling\n",
      "bert_int8_example    multi_gpu_gpt_example\t   vit_example\n",
      "decoding_example     multi_gpu_gpt_triton_example  vit_gemm\n",
      "decoding_gemm\t     swin_example\t\t   vit_int8_example\n",
      "gpt_example\t     swin_gemm\t\t\t   xlnet_correctness_example\n",
      "gpt_gemm\t     swin_int8_example\t\t   xlnet_example\n",
      "gptj_example\t     t5_gemm\t\t\t   xlnet_gemm\n",
      "gptj_triton_example  test_gemm\n"
     ]
    }
   ],
   "source": [
    "print(\"List of FasterTransformer examples:\")\n",
    "!ls ./bin"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Looks like the FasterTransformer library was successfully built and ready for the inference of our models."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3.4 Step 3: Download and prepare GPT-J checkpoint"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's see how to execute inference of a GPT model using C programming language. Before we start, we need to convert the checkpoints from the TensorFlow or PyTorch format to a binary representation that can be consumed by FasterTransformer C API. Currently there are no published checkpoints released by OpenAI. Fortunately, FasterTransformer provides several converters which will allow to consume models in several formats. FasterTransformer can consume Megatron-LM (which is trained by PyTorch) checkpoints. Another option includes using the OpenAI GPT-2 model format or consume Transformers library based checkpoints. In this example we will continue to use GPT-J 6B checkpoint."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We already downloaded the weights for you, but you can always uncomment the lines below to download if desired:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/dli/task\n"
     ]
    }
   ],
   "source": [
    "# Go to the main course dir\n",
    "%cd /dli/task"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "# We already downloaded the weights for you.\n",
    "# Please uncomment the below to download from scratch on your own system.\n",
    "# !wget https://mystic.the-eye.eu/public/AI/GPT-J-6B/step_383500_slim.tar.zstd\n",
    "!tar -axf ./weights/gpt-j/ft/step_383500_slim.tar.zstd -C ./model_repository/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's have a look at downloaded weights."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "shard_0  shard_1  shard_2  shard_3  shard_4  shard_5  shard_6  shard_7\n"
     ]
    }
   ],
   "source": [
    "!ls ./model_repository/step_383500/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Weights are split into multiple shards. That is how they were pretrained with the Megatron framework. Shards are parts of the weights that were placed onto different devices in the process of training."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "On top of the GPT-J model we will use Transformers’ library implementation of tokenizers as well as encoding/decoding components. We will also need the Vocab and Merge table files that will be needed at the inference stage to initialize the tokenizer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--2023-03-23 21:56:12--  https://s3.amazonaws.com/models.huggingface.co/bert/gpt2-vocab.json\n",
      "Resolving s3.amazonaws.com (s3.amazonaws.com)... 52.217.88.86, 52.216.240.174, 52.217.36.198, ...\n",
      "Connecting to s3.amazonaws.com (s3.amazonaws.com)|52.217.88.86|:443... connected.\n",
      "HTTP request sent, awaiting response... 200 OK\n",
      "Length: 1042301 (1018K) [application/json]\n",
      "Saving to: ‘models/gpt2-vocab.json’\n",
      "\n",
      "gpt2-vocab.json     100%[===================>]   1018K  4.55MB/s    in 0.2s    \n",
      "\n",
      "2023-03-23 21:56:12 (4.55 MB/s) - ‘models/gpt2-vocab.json’ saved [1042301/1042301]\n",
      "\n",
      "--2023-03-23 21:56:12--  https://s3.amazonaws.com/models.huggingface.co/bert/gpt2-merges.txt\n",
      "Resolving s3.amazonaws.com (s3.amazonaws.com)... 54.231.163.96, 52.216.140.54, 52.217.160.48, ...\n",
      "Connecting to s3.amazonaws.com (s3.amazonaws.com)|54.231.163.96|:443... connected.\n",
      "HTTP request sent, awaiting response... 200 OK\n",
      "Length: 456318 (446K) [text/plain]\n",
      "Saving to: ‘models/gpt2-merges.txt’\n",
      "\n",
      "gpt2-merges.txt     100%[===================>] 445.62K  --.-KB/s    in 0.1s    \n",
      "\n",
      "2023-03-23 21:56:13 (3.01 MB/s) - ‘models/gpt2-merges.txt’ saved [456318/456318]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "!wget https://s3.amazonaws.com/models.huggingface.co/bert/gpt2-vocab.json -P models\n",
    "!wget https://s3.amazonaws.com/models.huggingface.co/bert/gpt2-merges.txt -P models"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3.5 Step 4: Convert weights into FT format"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "FasterTransformer provides a number of tools/scripts for format conversion. </br> \n",
    "For GPT-J weights, we can use the following script `FasterTransformer/examples/pytorch/gptj/utils/gptj_ckpt_convert.py`.</br> \n",
    "The converter requires the following arguments: \n",
    "\n",
    "1. `--ckpt-dir`: The path of megatron model\n",
    "2. `--output-dir`: The output path of converted model\n",
    "3. `--n-inference-gpus`: The tensor parallel size of the pipeline\n",
    "\n",
    "Please note that because of the file sizes involved, the conversion can take a couple minutes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/dli/task/FasterTransformer/build\n"
     ]
    }
   ],
   "source": [
    "# get to directoy with build FasterTransformer library \n",
    "%cd FasterTransformer/build"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "scrolled": true,
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "loading\n",
      "loading shards for part 0\n",
      "read from checkpoint\n",
      "< (8, 4096) to (4096,)\n",
      "> transformer.wte.bias torch.Size([4096])\n",
      "< (8, 6300, 4096) to (1, 50400, 4096)\n",
      "> transformer.wte.weight torch.Size([4096, 50400])\n",
      "< (8, 4096, 512) to (1, 4096, 4096)\n",
      "WARNING:absl:No GPU/TPU found, falling back to CPU. (Set TF_CPP_MIN_LOG_LEVEL=0 and rerun for more info.)\n",
      "> transformer.h.0.attn.attention.q_proj.weight torch.Size([4096, 4096])\n",
      "< (8, 4096, 512) to (1, 4096, 4096)\n",
      "> transformer.h.0.attn.attention.v_proj.weight torch.Size([4096, 4096])\n",
      "< (8, 4096, 512) to (1, 4096, 4096)\n",
      "> transformer.h.0.attn.attention.k_proj.weight torch.Size([4096, 4096])\n",
      "< (8, 512, 4096) to (1, 4096, 4096)\n",
      "> transformer.h.0.attn.attention.out_proj.weight torch.Size([4096, 4096])\n",
      "< (8, 2048) to (1, 16384)\n",
      "> transformer.h.0.mlp.c_fc.bias torch.Size([16384])\n",
      "< (8, 4096, 2048) to (1, 4096, 16384)\n",
      "> transformer.h.0.mlp.c_fc.weight torch.Size([16384, 4096])\n",
      "< (8, 4096) to (4096,)\n",
      "> transformer.h.0.mlp.c_proj.bias torch.Size([4096])\n",
      "< (8, 2048, 4096) to (1, 16384, 4096)\n",
      "> transformer.h.0.mlp.c_proj.weight torch.Size([4096, 16384])\n",
      "< (8, 4096) to (4096,)\n",
      "> transformer.h.0.ln_1.bias torch.Size([4096])\n",
      "< (8, 4096) to (4096,)\n",
      "> transformer.h.0.ln_1.weight torch.Size([4096])\n",
      "< (8, 4096, 512) to (1, 4096, 4096)\n",
      "> transformer.h.1.attn.attention.q_proj.weight torch.Size([4096, 4096])\n",
      "< (8, 4096, 512) to (1, 4096, 4096)\n",
      "> transformer.h.1.attn.attention.v_proj.weight torch.Size([4096, 4096])\n",
      "< (8, 4096, 512) to (1, 4096, 4096)\n",
      "> transformer.h.1.attn.attention.k_proj.weight torch.Size([4096, 4096])\n",
      "< (8, 512, 4096) to (1, 4096, 4096)\n",
      "> transformer.h.1.attn.attention.out_proj.weight torch.Size([4096, 4096])\n",
      "< (8, 2048) to (1, 16384)\n",
      "> transformer.h.1.mlp.c_fc.bias torch.Size([16384])\n",
      "< (8, 4096, 2048) to (1, 4096, 16384)\n",
      "> transformer.h.1.mlp.c_fc.weight torch.Size([16384, 4096])\n",
      "loading shards for part 1\n",
      "read from checkpoint\n",
      "< (8, 4096) to (4096,)\n",
      "> transformer.h.1.mlp.c_proj.bias torch.Size([4096])\n",
      "< (8, 2048, 4096) to (1, 16384, 4096)\n",
      "> transformer.h.1.mlp.c_proj.weight torch.Size([4096, 16384])\n",
      "< (8, 4096) to (4096,)\n",
      "> transformer.h.1.ln_1.bias torch.Size([4096])\n",
      "< (8, 4096) to (4096,)\n",
      "> transformer.h.1.ln_1.weight torch.Size([4096])\n",
      "< (8, 4096, 512) to (1, 4096, 4096)\n",
      "> transformer.h.10.attn.attention.q_proj.weight torch.Size([4096, 4096])\n",
      "< (8, 4096, 512) to (1, 4096, 4096)\n",
      "> transformer.h.10.attn.attention.v_proj.weight torch.Size([4096, 4096])\n",
      "< (8, 4096, 512) to (1, 4096, 4096)\n",
      "> transformer.h.10.attn.attention.k_proj.weight torch.Size([4096, 4096])\n",
      "< (8, 512, 4096) to (1, 4096, 4096)\n",
      "> transformer.h.10.attn.attention.out_proj.weight torch.Size([4096, 4096])\n",
      "< (8, 2048) to (1, 16384)\n",
      "> transformer.h.10.mlp.c_fc.bias torch.Size([16384])\n",
      "< (8, 4096, 2048) to (1, 4096, 16384)\n",
      "> transformer.h.10.mlp.c_fc.weight torch.Size([16384, 4096])\n",
      "< (8, 4096) to (4096,)\n",
      "> transformer.h.10.mlp.c_proj.bias torch.Size([4096])\n",
      "< (8, 2048, 4096) to (1, 16384, 4096)\n",
      "> transformer.h.10.mlp.c_proj.weight torch.Size([4096, 16384])\n",
      "< (8, 4096) to (4096,)\n",
      "> transformer.h.10.ln_1.bias torch.Size([4096])\n",
      "< (8, 4096) to (4096,)\n",
      "> transformer.h.10.ln_1.weight torch.Size([4096])\n",
      "< (8, 4096, 512) to (1, 4096, 4096)\n",
      "> transformer.h.11.attn.attention.q_proj.weight torch.Size([4096, 4096])\n",
      "< (8, 4096, 512) to (1, 4096, 4096)\n",
      "> transformer.h.11.attn.attention.v_proj.weight torch.Size([4096, 4096])\n",
      "< (8, 4096, 512) to (1, 4096, 4096)\n",
      "> transformer.h.11.attn.attention.k_proj.weight torch.Size([4096, 4096])\n",
      "< (8, 512, 4096) to (1, 4096, 4096)\n",
      "> transformer.h.11.attn.attention.out_proj.weight torch.Size([4096, 4096])\n",
      "loading shards for part 2\n",
      "read from checkpoint\n",
      "< (8, 2048) to (1, 16384)\n",
      "> transformer.h.11.mlp.c_fc.bias torch.Size([16384])\n",
      "< (8, 4096, 2048) to (1, 4096, 16384)\n",
      "> transformer.h.11.mlp.c_fc.weight torch.Size([16384, 4096])\n",
      "< (8, 4096) to (4096,)\n",
      "> transformer.h.11.mlp.c_proj.bias torch.Size([4096])\n",
      "< (8, 2048, 4096) to (1, 16384, 4096)\n",
      "> transformer.h.11.mlp.c_proj.weight torch.Size([4096, 16384])\n",
      "< (8, 4096) to (4096,)\n",
      "> transformer.h.11.ln_1.bias torch.Size([4096])\n",
      "< (8, 4096) to (4096,)\n",
      "> transformer.h.11.ln_1.weight torch.Size([4096])\n",
      "< (8, 4096, 512) to (1, 4096, 4096)\n",
      "> transformer.h.12.attn.attention.q_proj.weight torch.Size([4096, 4096])\n",
      "< (8, 4096, 512) to (1, 4096, 4096)\n",
      "> transformer.h.12.attn.attention.v_proj.weight torch.Size([4096, 4096])\n",
      "< (8, 4096, 512) to (1, 4096, 4096)\n",
      "> transformer.h.12.attn.attention.k_proj.weight torch.Size([4096, 4096])\n",
      "< (8, 512, 4096) to (1, 4096, 4096)\n",
      "> transformer.h.12.attn.attention.out_proj.weight torch.Size([4096, 4096])\n",
      "< (8, 2048) to (1, 16384)\n",
      "> transformer.h.12.mlp.c_fc.bias torch.Size([16384])\n",
      "< (8, 4096, 2048) to (1, 4096, 16384)\n",
      "> transformer.h.12.mlp.c_fc.weight torch.Size([16384, 4096])\n",
      "< (8, 4096) to (4096,)\n",
      "> transformer.h.12.mlp.c_proj.bias torch.Size([4096])\n",
      "< (8, 2048, 4096) to (1, 16384, 4096)\n",
      "> transformer.h.12.mlp.c_proj.weight torch.Size([4096, 16384])\n",
      "< (8, 4096) to (4096,)\n",
      "> transformer.h.12.ln_1.bias torch.Size([4096])\n",
      "< (8, 4096) to (4096,)\n",
      "> transformer.h.12.ln_1.weight torch.Size([4096])\n",
      "< (8, 4096, 512) to (1, 4096, 4096)\n",
      "> transformer.h.13.attn.attention.q_proj.weight torch.Size([4096, 4096])\n",
      "< (8, 4096, 512) to (1, 4096, 4096)\n",
      "> transformer.h.13.attn.attention.v_proj.weight torch.Size([4096, 4096])\n",
      "loading shards for part 3\n",
      "read from checkpoint\n",
      "< (8, 4096, 512) to (1, 4096, 4096)\n",
      "> transformer.h.13.attn.attention.k_proj.weight torch.Size([4096, 4096])\n",
      "< (8, 512, 4096) to (1, 4096, 4096)\n",
      "> transformer.h.13.attn.attention.out_proj.weight torch.Size([4096, 4096])\n",
      "< (8, 2048) to (1, 16384)\n",
      "> transformer.h.13.mlp.c_fc.bias torch.Size([16384])\n",
      "< (8, 4096, 2048) to (1, 4096, 16384)\n",
      "> transformer.h.13.mlp.c_fc.weight torch.Size([16384, 4096])\n",
      "< (8, 4096) to (4096,)\n",
      "> transformer.h.13.mlp.c_proj.bias torch.Size([4096])\n",
      "< (8, 2048, 4096) to (1, 16384, 4096)\n",
      "> transformer.h.13.mlp.c_proj.weight torch.Size([4096, 16384])\n",
      "< (8, 4096) to (4096,)\n",
      "> transformer.h.13.ln_1.bias torch.Size([4096])\n",
      "< (8, 4096) to (4096,)\n",
      "> transformer.h.13.ln_1.weight torch.Size([4096])\n",
      "< (8, 4096, 512) to (1, 4096, 4096)\n",
      "> transformer.h.14.attn.attention.q_proj.weight torch.Size([4096, 4096])\n",
      "< (8, 4096, 512) to (1, 4096, 4096)\n",
      "> transformer.h.14.attn.attention.v_proj.weight torch.Size([4096, 4096])\n",
      "< (8, 4096, 512) to (1, 4096, 4096)\n",
      "> transformer.h.14.attn.attention.k_proj.weight torch.Size([4096, 4096])\n",
      "< (8, 512, 4096) to (1, 4096, 4096)\n",
      "> transformer.h.14.attn.attention.out_proj.weight torch.Size([4096, 4096])\n",
      "< (8, 2048) to (1, 16384)\n",
      "> transformer.h.14.mlp.c_fc.bias torch.Size([16384])\n",
      "< (8, 4096, 2048) to (1, 4096, 16384)\n",
      "> transformer.h.14.mlp.c_fc.weight torch.Size([16384, 4096])\n",
      "< (8, 4096) to (4096,)\n",
      "> transformer.h.14.mlp.c_proj.bias torch.Size([4096])\n",
      "< (8, 2048, 4096) to (1, 16384, 4096)\n",
      "> transformer.h.14.mlp.c_proj.weight torch.Size([4096, 16384])\n",
      "< (8, 4096) to (4096,)\n",
      "> transformer.h.14.ln_1.bias torch.Size([4096])\n",
      "< (8, 4096) to (4096,)\n",
      "> transformer.h.14.ln_1.weight torch.Size([4096])\n",
      "loading shards for part 4\n",
      "read from checkpoint\n",
      "< (8, 4096, 512) to (1, 4096, 4096)\n",
      "> transformer.h.15.attn.attention.q_proj.weight torch.Size([4096, 4096])\n",
      "< (8, 4096, 512) to (1, 4096, 4096)\n",
      "> transformer.h.15.attn.attention.v_proj.weight torch.Size([4096, 4096])\n",
      "< (8, 4096, 512) to (1, 4096, 4096)\n",
      "> transformer.h.15.attn.attention.k_proj.weight torch.Size([4096, 4096])\n",
      "< (8, 512, 4096) to (1, 4096, 4096)\n",
      "> transformer.h.15.attn.attention.out_proj.weight torch.Size([4096, 4096])\n",
      "< (8, 2048) to (1, 16384)\n",
      "> transformer.h.15.mlp.c_fc.bias torch.Size([16384])\n",
      "< (8, 4096, 2048) to (1, 4096, 16384)\n",
      "> transformer.h.15.mlp.c_fc.weight torch.Size([16384, 4096])\n",
      "< (8, 4096) to (4096,)\n",
      "> transformer.h.15.mlp.c_proj.bias torch.Size([4096])\n",
      "< (8, 2048, 4096) to (1, 16384, 4096)\n",
      "> transformer.h.15.mlp.c_proj.weight torch.Size([4096, 16384])\n",
      "< (8, 4096) to (4096,)\n",
      "> transformer.h.15.ln_1.bias torch.Size([4096])\n",
      "< (8, 4096) to (4096,)\n",
      "> transformer.h.15.ln_1.weight torch.Size([4096])\n",
      "< (8, 4096, 512) to (1, 4096, 4096)\n",
      "> transformer.h.16.attn.attention.q_proj.weight torch.Size([4096, 4096])\n",
      "< (8, 4096, 512) to (1, 4096, 4096)\n",
      "> transformer.h.16.attn.attention.v_proj.weight torch.Size([4096, 4096])\n",
      "< (8, 4096, 512) to (1, 4096, 4096)\n",
      "> transformer.h.16.attn.attention.k_proj.weight torch.Size([4096, 4096])\n",
      "< (8, 512, 4096) to (1, 4096, 4096)\n",
      "> transformer.h.16.attn.attention.out_proj.weight torch.Size([4096, 4096])\n",
      "< (8, 2048) to (1, 16384)\n",
      "> transformer.h.16.mlp.c_fc.bias torch.Size([16384])\n",
      "< (8, 4096, 2048) to (1, 4096, 16384)\n",
      "> transformer.h.16.mlp.c_fc.weight torch.Size([16384, 4096])\n",
      "< (8, 4096) to (4096,)\n",
      "> transformer.h.16.mlp.c_proj.bias torch.Size([4096])\n",
      "< (8, 2048, 4096) to (1, 16384, 4096)\n",
      "> transformer.h.16.mlp.c_proj.weight torch.Size([4096, 16384])\n",
      "loading shards for part 5\n",
      "read from checkpoint\n",
      "< (8, 4096) to (4096,)\n",
      "> transformer.h.16.ln_1.bias torch.Size([4096])\n",
      "< (8, 4096) to (4096,)\n",
      "> transformer.h.16.ln_1.weight torch.Size([4096])\n",
      "< (8, 4096, 512) to (1, 4096, 4096)\n",
      "> transformer.h.17.attn.attention.q_proj.weight torch.Size([4096, 4096])\n",
      "< (8, 4096, 512) to (1, 4096, 4096)\n",
      "> transformer.h.17.attn.attention.v_proj.weight torch.Size([4096, 4096])\n",
      "< (8, 4096, 512) to (1, 4096, 4096)\n",
      "> transformer.h.17.attn.attention.k_proj.weight torch.Size([4096, 4096])\n",
      "< (8, 512, 4096) to (1, 4096, 4096)\n",
      "> transformer.h.17.attn.attention.out_proj.weight torch.Size([4096, 4096])\n",
      "< (8, 2048) to (1, 16384)\n",
      "> transformer.h.17.mlp.c_fc.bias torch.Size([16384])\n",
      "< (8, 4096, 2048) to (1, 4096, 16384)\n",
      "> transformer.h.17.mlp.c_fc.weight torch.Size([16384, 4096])\n",
      "< (8, 4096) to (4096,)\n",
      "> transformer.h.17.mlp.c_proj.bias torch.Size([4096])\n",
      "< (8, 2048, 4096) to (1, 16384, 4096)\n",
      "> transformer.h.17.mlp.c_proj.weight torch.Size([4096, 16384])\n",
      "< (8, 4096) to (4096,)\n",
      "> transformer.h.17.ln_1.bias torch.Size([4096])\n",
      "< (8, 4096) to (4096,)\n",
      "> transformer.h.17.ln_1.weight torch.Size([4096])\n",
      "< (8, 4096, 512) to (1, 4096, 4096)\n",
      "> transformer.h.18.attn.attention.q_proj.weight torch.Size([4096, 4096])\n",
      "< (8, 4096, 512) to (1, 4096, 4096)\n",
      "> transformer.h.18.attn.attention.v_proj.weight torch.Size([4096, 4096])\n",
      "< (8, 4096, 512) to (1, 4096, 4096)\n",
      "> transformer.h.18.attn.attention.k_proj.weight torch.Size([4096, 4096])\n",
      "< (8, 512, 4096) to (1, 4096, 4096)\n",
      "> transformer.h.18.attn.attention.out_proj.weight torch.Size([4096, 4096])\n",
      "< (8, 2048) to (1, 16384)\n",
      "> transformer.h.18.mlp.c_fc.bias torch.Size([16384])\n",
      "< (8, 4096, 2048) to (1, 4096, 16384)\n",
      "> transformer.h.18.mlp.c_fc.weight torch.Size([16384, 4096])\n",
      "loading shards for part 6\n",
      "read from checkpoint\n",
      "< (8, 4096) to (4096,)\n",
      "> transformer.h.18.mlp.c_proj.bias torch.Size([4096])\n",
      "< (8, 2048, 4096) to (1, 16384, 4096)\n",
      "> transformer.h.18.mlp.c_proj.weight torch.Size([4096, 16384])\n",
      "< (8, 4096) to (4096,)\n",
      "> transformer.h.18.ln_1.bias torch.Size([4096])\n",
      "< (8, 4096) to (4096,)\n",
      "> transformer.h.18.ln_1.weight torch.Size([4096])\n",
      "< (8, 4096, 512) to (1, 4096, 4096)\n",
      "> transformer.h.19.attn.attention.q_proj.weight torch.Size([4096, 4096])\n",
      "< (8, 4096, 512) to (1, 4096, 4096)\n",
      "> transformer.h.19.attn.attention.v_proj.weight torch.Size([4096, 4096])\n",
      "< (8, 4096, 512) to (1, 4096, 4096)\n",
      "> transformer.h.19.attn.attention.k_proj.weight torch.Size([4096, 4096])\n",
      "< (8, 512, 4096) to (1, 4096, 4096)\n",
      "> transformer.h.19.attn.attention.out_proj.weight torch.Size([4096, 4096])\n",
      "< (8, 2048) to (1, 16384)\n",
      "> transformer.h.19.mlp.c_fc.bias torch.Size([16384])\n",
      "< (8, 4096, 2048) to (1, 4096, 16384)\n",
      "> transformer.h.19.mlp.c_fc.weight torch.Size([16384, 4096])\n",
      "< (8, 4096) to (4096,)\n",
      "> transformer.h.19.mlp.c_proj.bias torch.Size([4096])\n",
      "< (8, 2048, 4096) to (1, 16384, 4096)\n",
      "> transformer.h.19.mlp.c_proj.weight torch.Size([4096, 16384])\n",
      "< (8, 4096) to (4096,)\n",
      "> transformer.h.19.ln_1.bias torch.Size([4096])\n",
      "< (8, 4096) to (4096,)\n",
      "> transformer.h.19.ln_1.weight torch.Size([4096])\n",
      "< (8, 4096, 512) to (1, 4096, 4096)\n",
      "> transformer.h.2.attn.attention.q_proj.weight torch.Size([4096, 4096])\n",
      "< (8, 4096, 512) to (1, 4096, 4096)\n",
      "> transformer.h.2.attn.attention.v_proj.weight torch.Size([4096, 4096])\n",
      "< (8, 4096, 512) to (1, 4096, 4096)\n",
      "> transformer.h.2.attn.attention.k_proj.weight torch.Size([4096, 4096])\n",
      "< (8, 512, 4096) to (1, 4096, 4096)\n",
      "> transformer.h.2.attn.attention.out_proj.weight torch.Size([4096, 4096])\n",
      "loading shards for part 7\n",
      "read from checkpoint\n",
      "< (8, 2048) to (1, 16384)\n",
      "> transformer.h.2.mlp.c_fc.bias torch.Size([16384])\n",
      "< (8, 4096, 2048) to (1, 4096, 16384)\n",
      "> transformer.h.2.mlp.c_fc.weight torch.Size([16384, 4096])\n",
      "< (8, 4096) to (4096,)\n",
      "> transformer.h.2.mlp.c_proj.bias torch.Size([4096])\n",
      "< (8, 2048, 4096) to (1, 16384, 4096)\n",
      "> transformer.h.2.mlp.c_proj.weight torch.Size([4096, 16384])\n",
      "< (8, 4096) to (4096,)\n",
      "> transformer.h.2.ln_1.bias torch.Size([4096])\n",
      "< (8, 4096) to (4096,)\n",
      "> transformer.h.2.ln_1.weight torch.Size([4096])\n",
      "< (8, 4096, 512) to (1, 4096, 4096)\n",
      "> transformer.h.20.attn.attention.q_proj.weight torch.Size([4096, 4096])\n",
      "< (8, 4096, 512) to (1, 4096, 4096)\n",
      "> transformer.h.20.attn.attention.v_proj.weight torch.Size([4096, 4096])\n",
      "< (8, 4096, 512) to (1, 4096, 4096)\n",
      "> transformer.h.20.attn.attention.k_proj.weight torch.Size([4096, 4096])\n",
      "< (8, 512, 4096) to (1, 4096, 4096)\n",
      "> transformer.h.20.attn.attention.out_proj.weight torch.Size([4096, 4096])\n",
      "< (8, 2048) to (1, 16384)\n",
      "> transformer.h.20.mlp.c_fc.bias torch.Size([16384])\n",
      "< (8, 4096, 2048) to (1, 4096, 16384)\n",
      "> transformer.h.20.mlp.c_fc.weight torch.Size([16384, 4096])\n",
      "< (8, 4096) to (4096,)\n",
      "> transformer.h.20.mlp.c_proj.bias torch.Size([4096])\n",
      "< (8, 2048, 4096) to (1, 16384, 4096)\n",
      "> transformer.h.20.mlp.c_proj.weight torch.Size([4096, 16384])\n",
      "< (8, 4096) to (4096,)\n",
      "> transformer.h.20.ln_1.bias torch.Size([4096])\n",
      "< (8, 4096) to (4096,)\n",
      "> transformer.h.20.ln_1.weight torch.Size([4096])\n",
      "< (8, 4096, 512) to (1, 4096, 4096)\n",
      "> transformer.h.21.attn.attention.q_proj.weight torch.Size([4096, 4096])\n",
      "< (8, 4096, 512) to (1, 4096, 4096)\n",
      "> transformer.h.21.attn.attention.v_proj.weight torch.Size([4096, 4096])\n",
      "loading shards for part 8\n",
      "read from checkpoint\n",
      "< (8, 4096, 512) to (1, 4096, 4096)\n",
      "> transformer.h.21.attn.attention.k_proj.weight torch.Size([4096, 4096])\n",
      "< (8, 512, 4096) to (1, 4096, 4096)\n",
      "> transformer.h.21.attn.attention.out_proj.weight torch.Size([4096, 4096])\n",
      "< (8, 2048) to (1, 16384)\n",
      "> transformer.h.21.mlp.c_fc.bias torch.Size([16384])\n",
      "< (8, 4096, 2048) to (1, 4096, 16384)\n",
      "> transformer.h.21.mlp.c_fc.weight torch.Size([16384, 4096])\n",
      "< (8, 4096) to (4096,)\n",
      "> transformer.h.21.mlp.c_proj.bias torch.Size([4096])\n",
      "< (8, 2048, 4096) to (1, 16384, 4096)\n",
      "> transformer.h.21.mlp.c_proj.weight torch.Size([4096, 16384])\n",
      "< (8, 4096) to (4096,)\n",
      "> transformer.h.21.ln_1.bias torch.Size([4096])\n",
      "< (8, 4096) to (4096,)\n",
      "> transformer.h.21.ln_1.weight torch.Size([4096])\n",
      "< (8, 4096, 512) to (1, 4096, 4096)\n",
      "> transformer.h.22.attn.attention.q_proj.weight torch.Size([4096, 4096])\n",
      "< (8, 4096, 512) to (1, 4096, 4096)\n",
      "> transformer.h.22.attn.attention.v_proj.weight torch.Size([4096, 4096])\n",
      "< (8, 4096, 512) to (1, 4096, 4096)\n",
      "> transformer.h.22.attn.attention.k_proj.weight torch.Size([4096, 4096])\n",
      "< (8, 512, 4096) to (1, 4096, 4096)\n",
      "> transformer.h.22.attn.attention.out_proj.weight torch.Size([4096, 4096])\n",
      "< (8, 2048) to (1, 16384)\n",
      "> transformer.h.22.mlp.c_fc.bias torch.Size([16384])\n",
      "< (8, 4096, 2048) to (1, 4096, 16384)\n",
      "> transformer.h.22.mlp.c_fc.weight torch.Size([16384, 4096])\n",
      "< (8, 4096) to (4096,)\n",
      "> transformer.h.22.mlp.c_proj.bias torch.Size([4096])\n",
      "< (8, 2048, 4096) to (1, 16384, 4096)\n",
      "> transformer.h.22.mlp.c_proj.weight torch.Size([4096, 16384])\n",
      "< (8, 4096) to (4096,)\n",
      "> transformer.h.22.ln_1.bias torch.Size([4096])\n",
      "< (8, 4096) to (4096,)\n",
      "> transformer.h.22.ln_1.weight torch.Size([4096])\n",
      "loading shards for part 9\n",
      "read from checkpoint\n",
      "< (8, 4096, 512) to (1, 4096, 4096)\n",
      "> transformer.h.23.attn.attention.q_proj.weight torch.Size([4096, 4096])\n",
      "< (8, 4096, 512) to (1, 4096, 4096)\n",
      "> transformer.h.23.attn.attention.v_proj.weight torch.Size([4096, 4096])\n",
      "< (8, 4096, 512) to (1, 4096, 4096)\n",
      "> transformer.h.23.attn.attention.k_proj.weight torch.Size([4096, 4096])\n",
      "< (8, 512, 4096) to (1, 4096, 4096)\n",
      "> transformer.h.23.attn.attention.out_proj.weight torch.Size([4096, 4096])\n",
      "< (8, 2048) to (1, 16384)\n",
      "> transformer.h.23.mlp.c_fc.bias torch.Size([16384])\n",
      "< (8, 4096, 2048) to (1, 4096, 16384)\n",
      "> transformer.h.23.mlp.c_fc.weight torch.Size([16384, 4096])\n",
      "< (8, 4096) to (4096,)\n",
      "> transformer.h.23.mlp.c_proj.bias torch.Size([4096])\n",
      "< (8, 2048, 4096) to (1, 16384, 4096)\n",
      "> transformer.h.23.mlp.c_proj.weight torch.Size([4096, 16384])\n",
      "< (8, 4096) to (4096,)\n",
      "> transformer.h.23.ln_1.bias torch.Size([4096])\n",
      "< (8, 4096) to (4096,)\n",
      "> transformer.h.23.ln_1.weight torch.Size([4096])\n",
      "< (8, 4096, 512) to (1, 4096, 4096)\n",
      "> transformer.h.24.attn.attention.q_proj.weight torch.Size([4096, 4096])\n",
      "< (8, 4096, 512) to (1, 4096, 4096)\n",
      "> transformer.h.24.attn.attention.v_proj.weight torch.Size([4096, 4096])\n",
      "< (8, 4096, 512) to (1, 4096, 4096)\n",
      "> transformer.h.24.attn.attention.k_proj.weight torch.Size([4096, 4096])\n",
      "< (8, 512, 4096) to (1, 4096, 4096)\n",
      "> transformer.h.24.attn.attention.out_proj.weight torch.Size([4096, 4096])\n",
      "< (8, 2048) to (1, 16384)\n",
      "> transformer.h.24.mlp.c_fc.bias torch.Size([16384])\n",
      "< (8, 4096, 2048) to (1, 4096, 16384)\n",
      "> transformer.h.24.mlp.c_fc.weight torch.Size([16384, 4096])\n",
      "< (8, 4096) to (4096,)\n",
      "> transformer.h.24.mlp.c_proj.bias torch.Size([4096])\n",
      "< (8, 2048, 4096) to (1, 16384, 4096)\n",
      "> transformer.h.24.mlp.c_proj.weight torch.Size([4096, 16384])\n",
      "loading shards for part 10\n",
      "read from checkpoint\n",
      "< (8, 4096) to (4096,)\n",
      "> transformer.h.24.ln_1.bias torch.Size([4096])\n",
      "< (8, 4096) to (4096,)\n",
      "> transformer.h.24.ln_1.weight torch.Size([4096])\n",
      "< (8, 4096, 512) to (1, 4096, 4096)\n",
      "> transformer.h.25.attn.attention.q_proj.weight torch.Size([4096, 4096])\n",
      "< (8, 4096, 512) to (1, 4096, 4096)\n",
      "> transformer.h.25.attn.attention.v_proj.weight torch.Size([4096, 4096])\n",
      "< (8, 4096, 512) to (1, 4096, 4096)\n",
      "> transformer.h.25.attn.attention.k_proj.weight torch.Size([4096, 4096])\n",
      "< (8, 512, 4096) to (1, 4096, 4096)\n",
      "> transformer.h.25.attn.attention.out_proj.weight torch.Size([4096, 4096])\n",
      "< (8, 2048) to (1, 16384)\n",
      "> transformer.h.25.mlp.c_fc.bias torch.Size([16384])\n",
      "< (8, 4096, 2048) to (1, 4096, 16384)\n",
      "> transformer.h.25.mlp.c_fc.weight torch.Size([16384, 4096])\n",
      "< (8, 4096) to (4096,)\n",
      "> transformer.h.25.mlp.c_proj.bias torch.Size([4096])\n",
      "< (8, 2048, 4096) to (1, 16384, 4096)\n",
      "> transformer.h.25.mlp.c_proj.weight torch.Size([4096, 16384])\n",
      "< (8, 4096) to (4096,)\n",
      "> transformer.h.25.ln_1.bias torch.Size([4096])\n",
      "< (8, 4096) to (4096,)\n",
      "> transformer.h.25.ln_1.weight torch.Size([4096])\n",
      "< (8, 4096, 512) to (1, 4096, 4096)\n",
      "> transformer.h.26.attn.attention.q_proj.weight torch.Size([4096, 4096])\n",
      "< (8, 4096, 512) to (1, 4096, 4096)\n",
      "> transformer.h.26.attn.attention.v_proj.weight torch.Size([4096, 4096])\n",
      "< (8, 4096, 512) to (1, 4096, 4096)\n",
      "> transformer.h.26.attn.attention.k_proj.weight torch.Size([4096, 4096])\n",
      "< (8, 512, 4096) to (1, 4096, 4096)\n",
      "> transformer.h.26.attn.attention.out_proj.weight torch.Size([4096, 4096])\n",
      "< (8, 2048) to (1, 16384)\n",
      "> transformer.h.26.mlp.c_fc.bias torch.Size([16384])\n",
      "< (8, 4096, 2048) to (1, 4096, 16384)\n",
      "> transformer.h.26.mlp.c_fc.weight torch.Size([16384, 4096])\n",
      "loading shards for part 11\n",
      "read from checkpoint\n",
      "< (8, 4096) to (4096,)\n",
      "> transformer.h.26.mlp.c_proj.bias torch.Size([4096])\n",
      "< (8, 2048, 4096) to (1, 16384, 4096)\n",
      "> transformer.h.26.mlp.c_proj.weight torch.Size([4096, 16384])\n",
      "< (8, 4096) to (4096,)\n",
      "> transformer.h.26.ln_1.bias torch.Size([4096])\n",
      "< (8, 4096) to (4096,)\n",
      "> transformer.h.26.ln_1.weight torch.Size([4096])\n",
      "< (8, 4096, 512) to (1, 4096, 4096)\n",
      "> transformer.h.27.attn.attention.q_proj.weight torch.Size([4096, 4096])\n",
      "< (8, 4096, 512) to (1, 4096, 4096)\n",
      "> transformer.h.27.attn.attention.v_proj.weight torch.Size([4096, 4096])\n",
      "< (8, 4096, 512) to (1, 4096, 4096)\n",
      "> transformer.h.27.attn.attention.k_proj.weight torch.Size([4096, 4096])\n",
      "< (8, 512, 4096) to (1, 4096, 4096)\n",
      "> transformer.h.27.attn.attention.out_proj.weight torch.Size([4096, 4096])\n",
      "< (8, 2048) to (1, 16384)\n",
      "> transformer.h.27.mlp.c_fc.bias torch.Size([16384])\n",
      "< (8, 4096, 2048) to (1, 4096, 16384)\n",
      "> transformer.h.27.mlp.c_fc.weight torch.Size([16384, 4096])\n",
      "< (8, 4096) to (4096,)\n",
      "> transformer.h.27.mlp.c_proj.bias torch.Size([4096])\n",
      "< (8, 2048, 4096) to (1, 16384, 4096)\n",
      "> transformer.h.27.mlp.c_proj.weight torch.Size([4096, 16384])\n",
      "< (8, 4096) to (4096,)\n",
      "> transformer.h.27.ln_1.bias torch.Size([4096])\n",
      "< (8, 4096) to (4096,)\n",
      "> transformer.h.27.ln_1.weight torch.Size([4096])\n",
      "< (8, 4096, 512) to (1, 4096, 4096)\n",
      "> transformer.h.3.attn.attention.q_proj.weight torch.Size([4096, 4096])\n",
      "< (8, 4096, 512) to (1, 4096, 4096)\n",
      "> transformer.h.3.attn.attention.v_proj.weight torch.Size([4096, 4096])\n",
      "< (8, 4096, 512) to (1, 4096, 4096)\n",
      "> transformer.h.3.attn.attention.k_proj.weight torch.Size([4096, 4096])\n",
      "< (8, 512, 4096) to (1, 4096, 4096)\n",
      "> transformer.h.3.attn.attention.out_proj.weight torch.Size([4096, 4096])\n",
      "loading shards for part 12\n",
      "read from checkpoint\n",
      "< (8, 2048) to (1, 16384)\n",
      "> transformer.h.3.mlp.c_fc.bias torch.Size([16384])\n",
      "< (8, 4096, 2048) to (1, 4096, 16384)\n",
      "> transformer.h.3.mlp.c_fc.weight torch.Size([16384, 4096])\n",
      "< (8, 4096) to (4096,)\n",
      "> transformer.h.3.mlp.c_proj.bias torch.Size([4096])\n",
      "< (8, 2048, 4096) to (1, 16384, 4096)\n",
      "> transformer.h.3.mlp.c_proj.weight torch.Size([4096, 16384])\n",
      "< (8, 4096) to (4096,)\n",
      "> transformer.h.3.ln_1.bias torch.Size([4096])\n",
      "< (8, 4096) to (4096,)\n",
      "> transformer.h.3.ln_1.weight torch.Size([4096])\n",
      "< (8, 4096, 512) to (1, 4096, 4096)\n",
      "> transformer.h.4.attn.attention.q_proj.weight torch.Size([4096, 4096])\n",
      "< (8, 4096, 512) to (1, 4096, 4096)\n",
      "> transformer.h.4.attn.attention.v_proj.weight torch.Size([4096, 4096])\n",
      "< (8, 4096, 512) to (1, 4096, 4096)\n",
      "> transformer.h.4.attn.attention.k_proj.weight torch.Size([4096, 4096])\n",
      "< (8, 512, 4096) to (1, 4096, 4096)\n",
      "> transformer.h.4.attn.attention.out_proj.weight torch.Size([4096, 4096])\n",
      "< (8, 2048) to (1, 16384)\n",
      "> transformer.h.4.mlp.c_fc.bias torch.Size([16384])\n",
      "< (8, 4096, 2048) to (1, 4096, 16384)\n",
      "> transformer.h.4.mlp.c_fc.weight torch.Size([16384, 4096])\n",
      "< (8, 4096) to (4096,)\n",
      "> transformer.h.4.mlp.c_proj.bias torch.Size([4096])\n",
      "< (8, 2048, 4096) to (1, 16384, 4096)\n",
      "> transformer.h.4.mlp.c_proj.weight torch.Size([4096, 16384])\n",
      "< (8, 4096) to (4096,)\n",
      "> transformer.h.4.ln_1.bias torch.Size([4096])\n",
      "< (8, 4096) to (4096,)\n",
      "> transformer.h.4.ln_1.weight torch.Size([4096])\n",
      "< (8, 4096, 512) to (1, 4096, 4096)\n",
      "> transformer.h.5.attn.attention.q_proj.weight torch.Size([4096, 4096])\n",
      "< (8, 4096, 512) to (1, 4096, 4096)\n",
      "> transformer.h.5.attn.attention.v_proj.weight torch.Size([4096, 4096])\n",
      "loading shards for part 13\n",
      "read from checkpoint\n",
      "< (8, 4096, 512) to (1, 4096, 4096)\n",
      "> transformer.h.5.attn.attention.k_proj.weight torch.Size([4096, 4096])\n",
      "< (8, 512, 4096) to (1, 4096, 4096)\n",
      "> transformer.h.5.attn.attention.out_proj.weight torch.Size([4096, 4096])\n",
      "< (8, 2048) to (1, 16384)\n",
      "> transformer.h.5.mlp.c_fc.bias torch.Size([16384])\n",
      "< (8, 4096, 2048) to (1, 4096, 16384)\n",
      "> transformer.h.5.mlp.c_fc.weight torch.Size([16384, 4096])\n",
      "< (8, 4096) to (4096,)\n",
      "> transformer.h.5.mlp.c_proj.bias torch.Size([4096])\n",
      "< (8, 2048, 4096) to (1, 16384, 4096)\n",
      "> transformer.h.5.mlp.c_proj.weight torch.Size([4096, 16384])\n",
      "< (8, 4096) to (4096,)\n",
      "> transformer.h.5.ln_1.bias torch.Size([4096])\n",
      "< (8, 4096) to (4096,)\n",
      "> transformer.h.5.ln_1.weight torch.Size([4096])\n",
      "< (8, 4096, 512) to (1, 4096, 4096)\n",
      "> transformer.h.6.attn.attention.q_proj.weight torch.Size([4096, 4096])\n",
      "< (8, 4096, 512) to (1, 4096, 4096)\n",
      "> transformer.h.6.attn.attention.v_proj.weight torch.Size([4096, 4096])\n",
      "< (8, 4096, 512) to (1, 4096, 4096)\n",
      "> transformer.h.6.attn.attention.k_proj.weight torch.Size([4096, 4096])\n",
      "< (8, 512, 4096) to (1, 4096, 4096)\n",
      "> transformer.h.6.attn.attention.out_proj.weight torch.Size([4096, 4096])\n",
      "< (8, 2048) to (1, 16384)\n",
      "> transformer.h.6.mlp.c_fc.bias torch.Size([16384])\n",
      "< (8, 4096, 2048) to (1, 4096, 16384)\n",
      "> transformer.h.6.mlp.c_fc.weight torch.Size([16384, 4096])\n",
      "< (8, 4096) to (4096,)\n",
      "> transformer.h.6.mlp.c_proj.bias torch.Size([4096])\n",
      "< (8, 2048, 4096) to (1, 16384, 4096)\n",
      "> transformer.h.6.mlp.c_proj.weight torch.Size([4096, 16384])\n",
      "< (8, 4096) to (4096,)\n",
      "> transformer.h.6.ln_1.bias torch.Size([4096])\n",
      "< (8, 4096) to (4096,)\n",
      "> transformer.h.6.ln_1.weight torch.Size([4096])\n",
      "loading shards for part 14\n",
      "read from checkpoint\n",
      "< (8, 4096, 512) to (1, 4096, 4096)\n",
      "> transformer.h.7.attn.attention.q_proj.weight torch.Size([4096, 4096])\n",
      "< (8, 4096, 512) to (1, 4096, 4096)\n",
      "> transformer.h.7.attn.attention.v_proj.weight torch.Size([4096, 4096])\n",
      "< (8, 4096, 512) to (1, 4096, 4096)\n",
      "> transformer.h.7.attn.attention.k_proj.weight torch.Size([4096, 4096])\n",
      "< (8, 512, 4096) to (1, 4096, 4096)\n",
      "> transformer.h.7.attn.attention.out_proj.weight torch.Size([4096, 4096])\n",
      "< (8, 2048) to (1, 16384)\n",
      "> transformer.h.7.mlp.c_fc.bias torch.Size([16384])\n",
      "< (8, 4096, 2048) to (1, 4096, 16384)\n",
      "> transformer.h.7.mlp.c_fc.weight torch.Size([16384, 4096])\n",
      "< (8, 4096) to (4096,)\n",
      "> transformer.h.7.mlp.c_proj.bias torch.Size([4096])\n",
      "< (8, 2048, 4096) to (1, 16384, 4096)\n",
      "> transformer.h.7.mlp.c_proj.weight torch.Size([4096, 16384])\n",
      "< (8, 4096) to (4096,)\n",
      "> transformer.h.7.ln_1.bias torch.Size([4096])\n",
      "< (8, 4096) to (4096,)\n",
      "> transformer.h.7.ln_1.weight torch.Size([4096])\n",
      "< (8, 4096, 512) to (1, 4096, 4096)\n",
      "> transformer.h.8.attn.attention.q_proj.weight torch.Size([4096, 4096])\n",
      "< (8, 4096, 512) to (1, 4096, 4096)\n",
      "> transformer.h.8.attn.attention.v_proj.weight torch.Size([4096, 4096])\n",
      "< (8, 4096, 512) to (1, 4096, 4096)\n",
      "> transformer.h.8.attn.attention.k_proj.weight torch.Size([4096, 4096])\n",
      "< (8, 512, 4096) to (1, 4096, 4096)\n",
      "> transformer.h.8.attn.attention.out_proj.weight torch.Size([4096, 4096])\n",
      "< (8, 2048) to (1, 16384)\n",
      "> transformer.h.8.mlp.c_fc.bias torch.Size([16384])\n",
      "< (8, 4096, 2048) to (1, 4096, 16384)\n",
      "> transformer.h.8.mlp.c_fc.weight torch.Size([16384, 4096])\n",
      "< (8, 4096) to (4096,)\n",
      "> transformer.h.8.mlp.c_proj.bias torch.Size([4096])\n",
      "< (8, 2048, 4096) to (1, 16384, 4096)\n",
      "> transformer.h.8.mlp.c_proj.weight torch.Size([4096, 16384])\n",
      "loading shards for part 15\n",
      "read from checkpoint\n",
      "< (8, 4096) to (4096,)\n",
      "> transformer.h.8.ln_1.bias torch.Size([4096])\n",
      "< (8, 4096) to (4096,)\n",
      "> transformer.h.8.ln_1.weight torch.Size([4096])\n",
      "< (8, 4096, 512) to (1, 4096, 4096)\n",
      "> transformer.h.9.attn.attention.q_proj.weight torch.Size([4096, 4096])\n",
      "< (8, 4096, 512) to (1, 4096, 4096)\n",
      "> transformer.h.9.attn.attention.v_proj.weight torch.Size([4096, 4096])\n",
      "< (8, 4096, 512) to (1, 4096, 4096)\n",
      "> transformer.h.9.attn.attention.k_proj.weight torch.Size([4096, 4096])\n",
      "< (8, 512, 4096) to (1, 4096, 4096)\n",
      "> transformer.h.9.attn.attention.out_proj.weight torch.Size([4096, 4096])\n",
      "< (8, 2048) to (1, 16384)\n",
      "> transformer.h.9.mlp.c_fc.bias torch.Size([16384])\n",
      "< (8, 4096, 2048) to (1, 4096, 16384)\n",
      "> transformer.h.9.mlp.c_fc.weight torch.Size([16384, 4096])\n",
      "< (8, 4096) to (4096,)\n",
      "> transformer.h.9.mlp.c_proj.bias torch.Size([4096])\n",
      "< (8, 2048, 4096) to (1, 16384, 4096)\n",
      "> transformer.h.9.mlp.c_proj.weight torch.Size([4096, 16384])\n",
      "< (8, 4096) to (4096,)\n",
      "> transformer.h.9.ln_1.bias torch.Size([4096])\n",
      "< (8, 4096) to (4096,)\n",
      "> transformer.h.9.ln_1.weight torch.Size([4096])\n",
      "< (8, 6300) to (1, 50400)\n",
      "> lm_head.bias torch.Size([50400])\n",
      "< (8, 4096, 6300) to (1, 4096, 50400)\n",
      "> lm_head.weight torch.Size([50400, 4096])\n",
      "< (8, 4096) to (4096,)\n",
      "> transformer.ln_f.bias torch.Size([4096])\n",
      "< (8, 4096) to (4096,)\n",
      "> transformer.ln_f.weight torch.Size([4096])\n",
      "left over: [array([383502., 383502., 383502., 383502., 383502., 383502., 383502., 383502.], dtype=float32)]\n",
      "saving\n",
      "Saving layer 0 / 28\n",
      "Saving layer 1 / 28\n",
      "Saving layer 2 / 28\n",
      "Saving layer 3 / 28\n",
      "Saving layer 4 / 28\n",
      "Saving layer 5 / 28\n",
      "Saving layer 6 / 28\n",
      "Saving layer 7 / 28\n",
      "Saving layer 8 / 28\n",
      "Saving layer 9 / 28\n",
      "Saving layer 10 / 28\n",
      "Saving layer 11 / 28\n",
      "Saving layer 12 / 28\n",
      "Saving layer 13 / 28\n",
      "Saving layer 14 / 28\n",
      "Saving layer 15 / 28\n",
      "Saving layer 16 / 28\n",
      "Saving layer 17 / 28\n",
      "Saving layer 18 / 28\n",
      "Saving layer 19 / 28\n",
      "Saving layer 20 / 28\n",
      "Saving layer 21 / 28\n",
      "Saving layer 22 / 28\n",
      "Saving layer 23 / 28\n",
      "Saving layer 24 / 28\n",
      "Saving layer 25 / 28\n",
      "Saving layer 26 / 28\n",
      "Saving layer 27 / 28\n",
      "done\n"
     ]
    }
   ],
   "source": [
    "# Converting weights for 1 GPU without parallelism\n",
    "!python3 ../examples/pytorch/gptj/utils/gptj_ckpt_convert.py \\\n",
    "                                --output-dir ../../models/j6b_ckpt \\\n",
    "                                --ckpt-dir ../../model_repository/step_383500/ \\\n",
    "                                --n-inference-gpus 1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let us have a look at the generated files. The conversion script has re-combined sharded weights into groups of weights that will be placed onto GPUs that we want to use for inference. For now, we have configured the pipeline for a single GPU execution."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1-gpu\n"
     ]
    }
   ],
   "source": [
    "!ls ../../models/j6b_ckpt/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3.6 Step 5: Kernel autotuning. Seeking the fastest CUDA kernels for our model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let us run `./FasterTransformer/build/bin/gpt_gemm` binary file that we have built earlier. This file takes as an input the parameters of our model as well as additional configuration parameters, namely:  \n",
    "* `batch_size`  \n",
    "* `beam_width`  \n",
    "* `max_input_len`  \n",
    "* `head_number`  \n",
    "* `size_per_head` \n",
    "* `inter_size` \n",
    "* `vocab_size`  \n",
    "* `data_type`  \n",
    "* `tensor_para_size` \n",
    "\n",
    "All these parameters are needed to simulate the execution of the neural network and to identify the fastest algorithms to use during the inference process. We will need to set up these parameters manually. For the GPT-J model, the FasterTransformer team has prepared a `../examples/cpp/gptj/gptj_config.ini` config that contains the information about the GPT-J model as well as parameters required for inference. </br> \n",
    "\n",
    "<b>All these tests need to be run on the GPU that will be used for the inference later as those optimizations are GPU specific.</b> "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "scrolled": true,
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[INFO] arguments: \n",
      "  batch_size: 1 \n",
      "  beam_width: 1 \n",
      "  max_input_len: 128 \n",
      "  head_num: 16 \n",
      "  size_per_head: 256 \n",
      "  inter_size: 16384 \n",
      "  vocab_size: 50256 \n",
      "  data_type: 1 \n",
      "  tensor_para_size: 1 \n",
      "\n",
      "Device Graphics Device\n",
      "***Encoder Gemm Testing Begin***\n",
      "***Cublas Gemm Testing Begin***\n",
      "\n",
      "-----------------------------\n",
      "GEMM test 0: [M: 128, K: 4096, N: 12288] context from_tensor * weightQKV\n",
      "algo_99 costs 0.086ms \n",
      "algo_100 costs 0.078ms \n",
      "algo_101 costs 0.079ms \n",
      "algo_102 costs 0.079ms \n",
      "algo_103 costs 0.078ms \n",
      "algo_104 costs 0.078ms \n",
      "algo_105 costs 0.079ms \n",
      "algo_106 costs 0.079ms \n",
      "algo_107 costs 0.079ms \n",
      "algo_108 costs 0.079ms \n",
      "algo_109 costs 0.078ms \n",
      "algo_110 costs 0.078ms \n",
      "algo_111 costs 0.079ms \n",
      "algo_112 costs 0.078ms \n",
      "algo_113 costs 0.078ms \n",
      "algo_114 costs 0.078ms \n",
      "algo_115 costs 0.078ms \n",
      "fast_algo 115 costs 0.078 ms\n",
      "***cublasLt Gemm Testing Beign***\n",
      "AlgoCount: 3181\n",
      "return 1 and run heuristic algo\n",
      "workspacesize==0, run 170 algos\n",
      "result 000 : algo={ Id=6, tileIdx=20 (128x128) splitK=1 reduc=0 swizzle=0 custom=0 stages=11} status 0 time 0.077885ms workspace=0 mathMode=0 waves=0.444444\n",
      "result 001 : algo={ Id=6, tileIdx=20 (128x128) splitK=0 reduc=0 swizzle=1 custom=0 stages=11} status 0 time 0.078193ms workspace=0 mathMode=0 waves=0.444444\n",
      "result 002 : algo={ Id=6, tileIdx=20 (128x128) splitK=0 reduc=0 swizzle=0 custom=0 stages=11} status 0 time 0.078316ms workspace=0 mathMode=0 waves=0.444444\n",
      "result 003 : algo={ Id=6, tileIdx=18 (128x64) splitK=0 reduc=0 swizzle=1 custom=0 stages=12} status 0 time 0.081449ms workspace=0 mathMode=0 waves=0.888889\n",
      "result 004 : algo={ Id=6, tileIdx=18 (128x64) splitK=0 reduc=0 swizzle=1 custom=0 stages=15} status 0 time 0.081541ms workspace=0 mathMode=0 waves=0.888889\n",
      "result 005 : algo={ Id=6, tileIdx=18 (128x64) splitK=0 reduc=0 swizzle=0 custom=0 stages=15} status 0 time 0.082954ms workspace=0 mathMode=0 waves=0.888889\n",
      "result 006 : algo={ Id=6, tileIdx=18 (128x64) splitK=0 reduc=0 swizzle=1 custom=0 stages=16} status 0 time 0.083722ms workspace=0 mathMode=0 waves=1.777778\n",
      "result 007 : algo={ Id=6, tileIdx=20 (128x128) splitK=0 reduc=0 swizzle=0 custom=0 stages=15} status 0 time 0.086641ms workspace=0 mathMode=0 waves=0.888889\n",
      "result 008 : algo={ Id=6, tileIdx=20 (128x128) splitK=0 reduc=0 swizzle=1 custom=0 stages=15} status 0 time 0.087142ms workspace=0 mathMode=0 waves=0.888889\n",
      "result 009 : algo={ Id=6, tileIdx=24 (256x128) splitK=0 reduc=0 swizzle=0 custom=0 stages=9} status 0 time 0.103188ms workspace=0 mathMode=0 waves=0.444444\n",
      "result 010 : algo={ Id=6, tileIdx=24 (256x128) splitK=0 reduc=0 swizzle=1 custom=0 stages=15} status 0 time 0.103639ms workspace=0 mathMode=0 waves=0.444444\n",
      "result 011 : algo={ Id=6, tileIdx=24 (256x128) splitK=0 reduc=0 swizzle=0 custom=0 stages=15} status 0 time 0.104376ms workspace=0 mathMode=0 waves=0.444444\n",
      "result 012 : algo={ Id=6, tileIdx=23 (128x256) splitK=0 reduc=0 swizzle=0 custom=0 stages=15} status 0 time 0.104561ms workspace=0 mathMode=0 waves=0.888889\n",
      "result 013 : algo={ Id=6, tileIdx=23 (128x256) splitK=0 reduc=0 swizzle=1 custom=0 stages=15} status 0 time 0.104571ms workspace=0 mathMode=0 waves=0.888889\n",
      "result 014 : algo={ Id=5, tileIdx=20 (128x128) splitK=0 reduc=0 swizzle=0 custom=0 stages=0} status 0 time 0.115999ms workspace=0 mathMode=0 waves=0.444444\n",
      "result 015 : algo={ Id=5, tileIdx=20 (128x128) splitK=0 reduc=0 swizzle=1 custom=0 stages=0} status 0 time 0.116285ms workspace=0 mathMode=0 waves=0.444444\n",
      "result 016 : algo={ Id=6, tileIdx=18 (128x64) splitK=0 reduc=0 swizzle=0 custom=0 stages=12} status 0 time 0.116818ms workspace=0 mathMode=0 waves=0.888889\n",
      "result 017 : algo={ Id=5, tileIdx=21 (256x64) splitK=0 reduc=0 swizzle=0 custom=0 stages=7} status 0 time 0.121344ms workspace=0 mathMode=0 waves=0.444444\n",
      "result 018 : algo={ Id=5, tileIdx=21 (256x64) splitK=0 reduc=0 swizzle=1 custom=0 stages=7} status 0 time 0.122706ms workspace=0 mathMode=0 waves=0.444444\n",
      "result 019 : algo={ Id=5, tileIdx=20 (128x128) splitK=0 reduc=0 swizzle=1 custom=0 stages=7} status 0 time 0.125716ms workspace=0 mathMode=0 waves=0.444444\n",
      "result 020 : algo={ Id=5, tileIdx=20 (128x128) splitK=0 reduc=0 swizzle=0 custom=0 stages=7} status 0 time 0.125747ms workspace=0 mathMode=0 waves=0.444444\n",
      "result 021 : algo={ Id=5, tileIdx=23 (128x256) splitK=0 reduc=0 swizzle=0 custom=0 stages=0} status 0 time 0.129096ms workspace=0 mathMode=0 waves=0.888889\n",
      "result 022 : algo={ Id=5, tileIdx=23 (128x256) splitK=0 reduc=0 swizzle=1 custom=0 stages=0} status 0 time 0.129219ms workspace=0 mathMode=0 waves=0.888889\n",
      "result 023 : algo={ Id=5, tileIdx=24 (256x128) splitK=0 reduc=0 swizzle=0 custom=0 stages=0} status 0 time 0.133274ms workspace=0 mathMode=0 waves=0.444444\n",
      "result 024 : algo={ Id=6, tileIdx=18 (128x64) splitK=0 reduc=0 swizzle=0 custom=0 stages=16} status 0 time 0.133315ms workspace=0 mathMode=0 waves=1.777778\n",
      "result 025 : algo={ Id=5, tileIdx=24 (256x128) splitK=0 reduc=0 swizzle=1 custom=0 stages=0} status 0 time 0.134830ms workspace=0 mathMode=0 waves=0.444444\n",
      "result 026 : algo={ Id=5, tileIdx=23 (128x256) splitK=0 reduc=0 swizzle=1 custom=0 stages=7} status 0 time 0.137923ms workspace=0 mathMode=0 waves=0.888889\n",
      "result 027 : algo={ Id=5, tileIdx=23 (128x256) splitK=0 reduc=0 swizzle=0 custom=0 stages=7} status 0 time 0.137943ms workspace=0 mathMode=0 waves=0.888889\n",
      "result 028 : algo={ Id=6, tileIdx=23 (128x256) splitK=0 reduc=0 swizzle=1 custom=0 stages=0} status 0 time 0.143770ms workspace=0 mathMode=0 waves=0.888889\n",
      "result 029 : algo={ Id=6, tileIdx=23 (128x256) splitK=0 reduc=0 swizzle=0 custom=0 stages=0} status 0 time 0.143800ms workspace=0 mathMode=0 waves=0.888889\n",
      "result 030 : algo={ Id=6, tileIdx=18 (128x64) splitK=0 reduc=0 swizzle=0 custom=0 stages=0} status 0 time 0.145510ms workspace=0 mathMode=0 waves=0.444444\n",
      "result 031 : algo={ Id=6, tileIdx=18 (128x64) splitK=0 reduc=0 swizzle=1 custom=0 stages=0} status 0 time 0.149617ms workspace=0 mathMode=0 waves=0.444444\n",
      "result 032 : algo={ Id=5, tileIdx=21 (256x64) splitK=0 reduc=0 swizzle=0 custom=0 stages=0} status 0 time 0.156969ms workspace=0 mathMode=0 waves=0.444444\n",
      "result 033 : algo={ Id=5, tileIdx=21 (256x64) splitK=0 reduc=0 swizzle=1 custom=0 stages=0} status 0 time 0.157696ms workspace=0 mathMode=0 waves=0.444444\n",
      "result 034 : algo={ Id=0, tileIdx=20 (128x128) splitK=0 reduc=0 swizzle=1 custom=0 stages=0} status 0 time 0.764508ms workspace=0 mathMode=0 waves=0.444444\n",
      "result 035 : algo={ Id=0, tileIdx=20 (128x128) splitK=0 reduc=0 swizzle=0 custom=0 stages=0} status 0 time 0.786913ms workspace=0 mathMode=0 waves=0.444444\n",
      "result 036 : algo={ Id=0, tileIdx=15 (64x64) splitK=0 reduc=0 swizzle=0 custom=0 stages=0} status 0 time 0.800563ms workspace=0 mathMode=0 waves=0.444444\n",
      "result 037 : algo={ Id=0, tileIdx=15 (64x64) splitK=0 reduc=0 swizzle=1 custom=0 stages=0} status 0 time 0.800891ms workspace=0 mathMode=0 waves=0.444444\n",
      "result 038 : algo={ Id=0, tileIdx=18 (128x64) splitK=0 reduc=0 swizzle=1 custom=0 stages=0} status 0 time 0.804649ms workspace=0 mathMode=0 waves=0.444444\n",
      "result 039 : algo={ Id=0, tileIdx=18 (128x64) splitK=0 reduc=0 swizzle=0 custom=0 stages=0} status 0 time 0.804803ms workspace=0 mathMode=0 waves=0.444444\n",
      "result 040 : algo={ Id=1, tileIdx=16 (128x32) splitK=0 reduc=0 swizzle=1 custom=0 stages=0} status 0 time 0.807229ms workspace=0 mathMode=0 waves=3.555556\n",
      "result 041 : algo={ Id=1, tileIdx=16 (128x32) splitK=0 reduc=0 swizzle=0 custom=0 stages=0} status 0 time 0.807997ms workspace=0 mathMode=0 waves=3.555556\n",
      "result 042 : algo={ Id=1, tileIdx=13 (64x32) splitK=0 reduc=0 swizzle=1 custom=0 stages=0} status 0 time 0.817285ms workspace=0 mathMode=0 waves=3.555556\n",
      "result 043 : algo={ Id=1, tileIdx=13 (64x32) splitK=0 reduc=0 swizzle=0 custom=0 stages=0} status 0 time 0.823685ms workspace=0 mathMode=0 waves=3.555556\n",
      "result 044 : algo={ Id=1, tileIdx=11 (32x32) splitK=0 reduc=0 swizzle=1 custom=0 stages=0} status 0 time 0.825057ms workspace=0 mathMode=0 waves=3.555556\n",
      "result 045 : algo={ Id=1, tileIdx=11 (32x32) splitK=0 reduc=0 swizzle=0 custom=0 stages=0} status 0 time 0.850524ms workspace=0 mathMode=0 waves=3.555556\n",
      "result 046 : algo={ Id=0, tileIdx=14 (32x128) splitK=0 reduc=0 swizzle=1 custom=0 stages=0} status 0 time 0.879053ms workspace=0 mathMode=0 waves=0.888889\n",
      "result 047 : algo={ Id=0, tileIdx=14 (32x128) splitK=0 reduc=0 swizzle=0 custom=0 stages=0} status 0 time 0.879268ms workspace=0 mathMode=0 waves=0.888889\n",
      "result 048 : algo={ Id=0, tileIdx=16 (128x32) splitK=0 reduc=0 swizzle=0 custom=0 stages=0} status 0 time 0.880845ms workspace=0 mathMode=0 waves=0.888889\n",
      "result 049 : algo={ Id=0, tileIdx=16 (128x32) splitK=0 reduc=0 swizzle=1 custom=0 stages=0} status 0 time 0.882401ms workspace=0 mathMode=0 waves=0.888889\n",
      "algo={ Id=6, tileIdx=20 (128x128) splitK=1 reduc=0 swizzle=0 custom=0 stages=11} status 0 time 0.077885ms workspace=0 mathMode=0 waves=0.444444\n",
      "***cublasLt Gemm Testing End***\n",
      "\n",
      "-----------------------------\n",
      "GEMM test 1: [M: 128, K: 256, N: 128] context batch gemm Q*K^T\n",
      "algo_99 costs 0.009ms \n",
      "algo_100 costs 0.006ms \n",
      "algo_101 costs 0.006ms \n",
      "algo_102 costs 0.006ms \n",
      "algo_103 costs 0.006ms \n",
      "algo_104 costs 0.006ms \n",
      "algo_105 costs 0.006ms \n",
      "algo_106 costs 0.006ms \n",
      "algo_107 costs 0.006ms \n",
      "algo_108 costs 0.006ms \n",
      "algo_109 costs 0.006ms \n",
      "algo_110 costs 0.006ms \n",
      "algo_111 costs 0.007ms \n",
      "algo_112 costs 0.007ms \n",
      "algo_113 costs 0.007ms \n",
      "algo_114 costs 0.007ms \n",
      "algo_115 costs 0.006ms \n",
      "fast_algo 106 costs 0.006 ms\n",
      "\n",
      "-----------------------------\n",
      "GEMM test 2: [M: 128, K: 128, N: 256] context batch gemm QK*V^T\n",
      "algo_99 costs 0.006ms \n",
      "algo_100 costs 0.006ms \n",
      "algo_101 costs 0.006ms \n",
      "algo_102 costs 0.006ms \n",
      "algo_103 costs 0.006ms \n",
      "algo_104 costs 0.006ms \n",
      "algo_105 costs 0.006ms \n",
      "algo_106 costs 0.006ms \n",
      "algo_107 costs 0.006ms \n",
      "algo_108 costs 0.006ms \n",
      "algo_109 costs 0.006ms \n",
      "algo_110 costs 0.007ms \n",
      "algo_111 costs 0.007ms \n",
      "algo_112 costs 0.007ms \n",
      "algo_113 costs 0.007ms \n",
      "algo_114 costs 0.006ms \n",
      "algo_115 costs 0.006ms \n",
      "fast_algo 100 costs 0.006 ms\n",
      "\n",
      "-----------------------------\n",
      "GEMM test 3: [M: 128, K: 4096, N: 4096] context attr * output_kernel\n",
      "algo_99 costs 0.033ms \n",
      "algo_100 costs 0.033ms \n",
      "algo_101 costs 0.033ms \n",
      "algo_102 costs 0.033ms \n",
      "algo_103 costs 0.033ms \n",
      "algo_104 costs 0.033ms \n",
      "algo_105 costs 0.033ms \n",
      "algo_106 costs 0.033ms \n",
      "algo_107 costs 0.033ms \n",
      "algo_108 costs 0.033ms \n",
      "algo_109 costs 0.033ms \n",
      "algo_110 costs 0.033ms \n",
      "algo_111 costs 0.033ms \n",
      "algo_112 costs 0.033ms \n",
      "algo_113 costs 0.033ms \n",
      "algo_114 costs 0.033ms \n",
      "algo_115 costs 0.033ms \n",
      "fast_algo 104 costs 0.033 ms\n",
      "***cublasLt Gemm Testing Beign***\n",
      "AlgoCount: 4287\n",
      "return 1 and run heuristic algo\n",
      "workspacesize==0, run 170 algos\n",
      "result 000 : algo={ Id=6, tileIdx=18 (128x64) splitK=0 reduc=0 swizzle=0 custom=0 stages=16} status 0 time 0.032164ms workspace=0 mathMode=0 waves=0.592593\n",
      "result 001 : algo={ Id=6, tileIdx=18 (128x64) splitK=0 reduc=0 swizzle=1 custom=0 stages=16} status 0 time 0.032205ms workspace=0 mathMode=0 waves=0.592593\n",
      "result 002 : algo={ Id=6, tileIdx=18 (128x64) splitK=0 reduc=0 swizzle=1 custom=0 stages=12} status 0 time 0.032317ms workspace=0 mathMode=0 waves=0.296296\n",
      "result 003 : algo={ Id=6, tileIdx=18 (128x64) splitK=0 reduc=0 swizzle=0 custom=0 stages=12} status 0 time 0.032348ms workspace=0 mathMode=0 waves=0.296296\n",
      "result 004 : algo={ Id=21, tileIdx=18 (128x64) splitK=1 reduc=0 swizzle=0 custom=0 stages=18} status 0 time 0.032717ms workspace=0 mathMode=0 waves=0.592593\n",
      "result 005 : algo={ Id=6, tileIdx=18 (128x64) splitK=0 reduc=0 swizzle=1 custom=0 stages=15} status 0 time 0.039516ms workspace=0 mathMode=0 waves=0.296296\n",
      "result 006 : algo={ Id=6, tileIdx=18 (128x64) splitK=0 reduc=0 swizzle=0 custom=0 stages=15} status 0 time 0.039619ms workspace=0 mathMode=0 waves=0.296296\n",
      "result 007 : algo={ Id=6, tileIdx=20 (128x128) splitK=0 reduc=0 swizzle=0 custom=0 stages=15} status 0 time 0.055368ms workspace=0 mathMode=0 waves=0.296296\n",
      "result 008 : algo={ Id=6, tileIdx=20 (128x128) splitK=0 reduc=0 swizzle=1 custom=0 stages=15} status 0 time 0.055409ms workspace=0 mathMode=0 waves=0.296296\n",
      "result 009 : algo={ Id=6, tileIdx=20 (128x128) splitK=0 reduc=0 swizzle=1 custom=0 stages=11} status 0 time 0.055532ms workspace=0 mathMode=0 waves=0.148148\n",
      "result 010 : algo={ Id=6, tileIdx=20 (128x128) splitK=0 reduc=0 swizzle=0 custom=0 stages=11} status 0 time 0.055542ms workspace=0 mathMode=0 waves=0.148148\n",
      "result 011 : algo={ Id=6, tileIdx=24 (256x128) splitK=0 reduc=0 swizzle=0 custom=0 stages=9} status 0 time 0.102093ms workspace=0 mathMode=0 waves=0.148148\n",
      "result 012 : algo={ Id=5, tileIdx=20 (128x128) splitK=0 reduc=0 swizzle=1 custom=0 stages=0} status 0 time 0.102492ms workspace=0 mathMode=0 waves=0.148148\n",
      "result 013 : algo={ Id=5, tileIdx=20 (128x128) splitK=0 reduc=0 swizzle=0 custom=0 stages=0} status 0 time 0.102523ms workspace=0 mathMode=0 waves=0.148148\n",
      "result 014 : algo={ Id=6, tileIdx=24 (256x128) splitK=0 reduc=0 swizzle=0 custom=0 stages=15} status 0 time 0.102574ms workspace=0 mathMode=0 waves=0.148148\n",
      "result 015 : algo={ Id=6, tileIdx=24 (256x128) splitK=0 reduc=0 swizzle=1 custom=0 stages=15} status 0 time 0.102574ms workspace=0 mathMode=0 waves=0.148148\n",
      "result 016 : algo={ Id=6, tileIdx=23 (128x256) splitK=0 reduc=0 swizzle=1 custom=0 stages=15} status 0 time 0.103137ms workspace=0 mathMode=0 waves=0.296296\n",
      "result 017 : algo={ Id=6, tileIdx=23 (128x256) splitK=0 reduc=0 swizzle=0 custom=0 stages=15} status 0 time 0.103148ms workspace=0 mathMode=0 waves=0.296296\n",
      "result 018 : algo={ Id=5, tileIdx=21 (256x64) splitK=0 reduc=0 swizzle=0 custom=0 stages=7} status 0 time 0.112302ms workspace=0 mathMode=0 waves=0.148148\n",
      "result 019 : algo={ Id=5, tileIdx=21 (256x64) splitK=0 reduc=0 swizzle=1 custom=0 stages=7} status 0 time 0.112691ms workspace=0 mathMode=0 waves=0.148148\n",
      "result 020 : algo={ Id=6, tileIdx=23 (128x256) splitK=0 reduc=0 swizzle=0 custom=0 stages=0} status 0 time 0.114330ms workspace=0 mathMode=0 waves=0.296296\n",
      "result 021 : algo={ Id=6, tileIdx=23 (128x256) splitK=0 reduc=0 swizzle=1 custom=0 stages=0} status 0 time 0.114432ms workspace=0 mathMode=0 waves=0.296296\n",
      "result 022 : algo={ Id=5, tileIdx=20 (128x128) splitK=0 reduc=0 swizzle=1 custom=0 stages=7} status 0 time 0.115415ms workspace=0 mathMode=0 waves=0.148148\n",
      "result 023 : algo={ Id=5, tileIdx=20 (128x128) splitK=0 reduc=0 swizzle=0 custom=0 stages=7} status 0 time 0.115476ms workspace=0 mathMode=0 waves=0.148148\n",
      "result 024 : algo={ Id=6, tileIdx=18 (128x64) splitK=0 reduc=0 swizzle=0 custom=0 stages=0} status 0 time 0.120136ms workspace=0 mathMode=0 waves=0.148148\n",
      "result 025 : algo={ Id=6, tileIdx=18 (128x64) splitK=0 reduc=0 swizzle=1 custom=0 stages=0} status 0 time 0.120320ms workspace=0 mathMode=0 waves=0.148148\n",
      "result 026 : algo={ Id=5, tileIdx=23 (128x256) splitK=0 reduc=0 swizzle=0 custom=0 stages=0} status 0 time 0.126751ms workspace=0 mathMode=0 waves=0.296296\n",
      "result 027 : algo={ Id=5, tileIdx=23 (128x256) splitK=0 reduc=0 swizzle=1 custom=0 stages=0} status 0 time 0.126802ms workspace=0 mathMode=0 waves=0.296296\n",
      "result 028 : algo={ Id=5, tileIdx=24 (256x128) splitK=0 reduc=0 swizzle=1 custom=0 stages=0} status 0 time 0.132588ms workspace=0 mathMode=0 waves=0.148148\n",
      "result 029 : algo={ Id=5, tileIdx=24 (256x128) splitK=0 reduc=0 swizzle=0 custom=0 stages=0} status 0 time 0.132598ms workspace=0 mathMode=0 waves=0.148148\n",
      "result 030 : algo={ Id=5, tileIdx=23 (128x256) splitK=0 reduc=0 swizzle=1 custom=0 stages=7} status 0 time 0.134011ms workspace=0 mathMode=0 waves=0.296296\n",
      "result 031 : algo={ Id=5, tileIdx=23 (128x256) splitK=0 reduc=0 swizzle=0 custom=0 stages=7} status 0 time 0.134052ms workspace=0 mathMode=0 waves=0.296296\n",
      "result 032 : algo={ Id=5, tileIdx=21 (256x64) splitK=0 reduc=0 swizzle=0 custom=0 stages=0} status 0 time 0.143841ms workspace=0 mathMode=0 waves=0.148148\n",
      "result 033 : algo={ Id=5, tileIdx=21 (256x64) splitK=0 reduc=0 swizzle=1 custom=0 stages=0} status 0 time 0.144763ms workspace=0 mathMode=0 waves=0.148148\n",
      "result 034 : algo={ Id=1, tileIdx=11 (32x32) splitK=0 reduc=0 swizzle=0 custom=0 stages=0} status 0 time 0.321300ms workspace=0 mathMode=0 waves=1.185185\n",
      "result 035 : algo={ Id=1, tileIdx=13 (64x32) splitK=0 reduc=0 swizzle=1 custom=0 stages=0} status 0 time 0.329994ms workspace=0 mathMode=0 waves=1.185185\n",
      "result 036 : algo={ Id=1, tileIdx=11 (32x32) splitK=0 reduc=0 swizzle=1 custom=0 stages=0} status 0 time 0.346501ms workspace=0 mathMode=0 waves=1.185185\n",
      "result 037 : algo={ Id=1, tileIdx=13 (64x32) splitK=0 reduc=0 swizzle=0 custom=0 stages=0} status 0 time 0.374231ms workspace=0 mathMode=0 waves=1.185185\n",
      "result 038 : algo={ Id=1, tileIdx=16 (128x32) splitK=0 reduc=0 swizzle=0 custom=0 stages=0} status 0 time 0.404920ms workspace=0 mathMode=0 waves=1.185185\n",
      "result 039 : algo={ Id=1, tileIdx=16 (128x32) splitK=0 reduc=0 swizzle=1 custom=0 stages=0} status 0 time 0.404920ms workspace=0 mathMode=0 waves=1.185185\n",
      "result 040 : algo={ Id=0, tileIdx=16 (128x32) splitK=0 reduc=0 swizzle=0 custom=0 stages=0} status 0 time 0.420946ms workspace=0 mathMode=0 waves=0.296296\n",
      "result 041 : algo={ Id=0, tileIdx=16 (128x32) splitK=0 reduc=0 swizzle=1 custom=0 stages=0} status 0 time 0.421949ms workspace=0 mathMode=0 waves=0.296296\n",
      "result 042 : algo={ Id=0, tileIdx=14 (32x128) splitK=0 reduc=0 swizzle=1 custom=0 stages=0} status 0 time 0.450396ms workspace=0 mathMode=0 waves=0.296296\n",
      "result 043 : algo={ Id=0, tileIdx=14 (32x128) splitK=0 reduc=0 swizzle=0 custom=0 stages=0} status 0 time 0.450611ms workspace=0 mathMode=0 waves=0.296296\n",
      "result 044 : algo={ Id=0, tileIdx=18 (128x64) splitK=0 reduc=0 swizzle=0 custom=0 stages=0} status 0 time 0.471931ms workspace=0 mathMode=0 waves=0.148148\n",
      "result 045 : algo={ Id=0, tileIdx=18 (128x64) splitK=0 reduc=0 swizzle=1 custom=0 stages=0} status 0 time 0.472504ms workspace=0 mathMode=0 waves=0.148148\n",
      "result 046 : algo={ Id=0, tileIdx=20 (128x128) splitK=0 reduc=0 swizzle=1 custom=0 stages=0} status 0 time 0.763443ms workspace=0 mathMode=0 waves=0.148148\n",
      "result 047 : algo={ Id=0, tileIdx=20 (128x128) splitK=0 reduc=0 swizzle=0 custom=0 stages=0} status 0 time 0.770140ms workspace=0 mathMode=0 waves=0.148148\n",
      "result 048 : algo={ Id=0, tileIdx=15 (64x64) splitK=0 reduc=0 swizzle=0 custom=0 stages=0} status 0 time 0.778854ms workspace=0 mathMode=0 waves=0.148148\n",
      "result 049 : algo={ Id=0, tileIdx=15 (64x64) splitK=0 reduc=0 swizzle=1 custom=0 stages=0} status 0 time 0.779069ms workspace=0 mathMode=0 waves=0.148148\n",
      "algo={ Id=6, tileIdx=18 (128x64) splitK=0 reduc=0 swizzle=0 custom=0 stages=16} status 0 time 0.032164ms workspace=0 mathMode=0 waves=0.592593\n",
      "***cublasLt Gemm Testing End***\n",
      "\n",
      "-----------------------------\n",
      "GEMM test 4: [M: 128, K: 4096, N: 16384] context ffn gemm 1\n",
      "algo_99 costs 0.116ms \n",
      "algo_100 costs 0.116ms \n",
      "algo_101 costs 0.116ms \n",
      "algo_102 costs 0.116ms \n",
      "algo_103 costs 0.116ms \n",
      "algo_104 costs 0.116ms \n",
      "algo_105 costs 0.116ms \n",
      "algo_106 costs 0.116ms \n",
      "algo_107 costs 0.116ms \n",
      "algo_108 costs 0.116ms \n",
      "algo_109 costs 0.116ms \n",
      "algo_110 costs 0.117ms \n",
      "algo_111 costs 0.118ms \n",
      "algo_112 costs 0.118ms \n",
      "algo_113 costs 0.118ms \n",
      "algo_114 costs 0.118ms \n",
      "algo_115 costs 0.119ms \n",
      "fast_algo 101 costs 0.116 ms\n",
      "***cublasLt Gemm Testing Beign***\n",
      "AlgoCount: 3023\n",
      "return 1 and run heuristic algo\n",
      "workspacesize==0, run 170 algos\n",
      "result 000 : algo={ Id=6, tileIdx=24 (256x128) splitK=0 reduc=0 swizzle=0 custom=0 stages=9} status 0 time 0.108677ms workspace=0 mathMode=0 waves=0.592593\n",
      "result 001 : algo={ Id=6, tileIdx=24 (256x128) splitK=0 reduc=0 swizzle=0 custom=0 stages=15} status 0 time 0.109711ms workspace=0 mathMode=0 waves=0.592593\n",
      "result 002 : algo={ Id=6, tileIdx=24 (256x128) splitK=0 reduc=0 swizzle=1 custom=0 stages=15} status 0 time 0.110285ms workspace=0 mathMode=0 waves=0.592593\n",
      "result 003 : algo={ Id=6, tileIdx=18 (128x64) splitK=0 reduc=0 swizzle=1 custom=0 stages=15} status 0 time 0.116244ms workspace=0 mathMode=0 waves=1.185185\n",
      "result 004 : algo={ Id=6, tileIdx=18 (128x64) splitK=0 reduc=0 swizzle=1 custom=0 stages=12} status 0 time 0.116890ms workspace=0 mathMode=0 waves=1.185185\n",
      "result 005 : algo={ Id=6, tileIdx=18 (128x64) splitK=0 reduc=0 swizzle=1 custom=0 stages=16} status 0 time 0.118006ms workspace=0 mathMode=0 waves=2.370370\n",
      "result 006 : algo={ Id=6, tileIdx=20 (128x128) splitK=4 reduc=1 swizzle=1 custom=0 stages=11} status 0 time 0.118088ms workspace=512 mathMode=0 waves=2.370370\n",
      "result 007 : algo={ Id=6, tileIdx=20 (128x128) splitK=0 reduc=0 swizzle=0 custom=0 stages=11} status 0 time 0.128389ms workspace=0 mathMode=0 waves=0.592593\n",
      "result 008 : algo={ Id=6, tileIdx=20 (128x128) splitK=0 reduc=0 swizzle=1 custom=0 stages=11} status 0 time 0.128563ms workspace=0 mathMode=0 waves=0.592593\n",
      "result 009 : algo={ Id=6, tileIdx=18 (128x64) splitK=0 reduc=0 swizzle=0 custom=0 stages=15} status 0 time 0.132383ms workspace=0 mathMode=0 waves=1.185185\n",
      "result 010 : algo={ Id=5, tileIdx=24 (256x128) splitK=0 reduc=0 swizzle=0 custom=0 stages=0} status 0 time 0.135823ms workspace=0 mathMode=0 waves=0.592593\n",
      "result 011 : algo={ Id=5, tileIdx=24 (256x128) splitK=0 reduc=0 swizzle=1 custom=0 stages=0} status 0 time 0.135864ms workspace=0 mathMode=0 waves=0.592593\n",
      "result 012 : algo={ Id=6, tileIdx=20 (128x128) splitK=0 reduc=0 swizzle=1 custom=0 stages=15} status 0 time 0.141906ms workspace=0 mathMode=0 waves=1.185185\n",
      "result 013 : algo={ Id=6, tileIdx=20 (128x128) splitK=0 reduc=0 swizzle=0 custom=0 stages=15} status 0 time 0.142213ms workspace=0 mathMode=0 waves=1.185185\n",
      "result 014 : algo={ Id=5, tileIdx=20 (128x128) splitK=0 reduc=0 swizzle=1 custom=0 stages=0} status 0 time 0.151235ms workspace=0 mathMode=0 waves=0.592593\n",
      "result 015 : algo={ Id=5, tileIdx=20 (128x128) splitK=0 reduc=0 swizzle=0 custom=0 stages=0} status 0 time 0.151347ms workspace=0 mathMode=0 waves=0.592593\n",
      "result 016 : algo={ Id=6, tileIdx=18 (128x64) splitK=0 reduc=0 swizzle=0 custom=0 stages=0} status 0 time 0.153641ms workspace=0 mathMode=0 waves=0.592593\n",
      "result 017 : algo={ Id=6, tileIdx=18 (128x64) splitK=0 reduc=0 swizzle=1 custom=0 stages=0} status 0 time 0.154194ms workspace=0 mathMode=0 waves=0.592593\n",
      "result 018 : algo={ Id=6, tileIdx=18 (128x64) splitK=0 reduc=0 swizzle=0 custom=0 stages=12} status 0 time 0.154624ms workspace=0 mathMode=0 waves=1.185185\n",
      "result 019 : algo={ Id=5, tileIdx=21 (256x64) splitK=0 reduc=0 swizzle=1 custom=0 stages=7} status 0 time 0.157553ms workspace=0 mathMode=0 waves=0.592593\n",
      "result 020 : algo={ Id=5, tileIdx=21 (256x64) splitK=0 reduc=0 swizzle=0 custom=0 stages=7} status 0 time 0.157655ms workspace=0 mathMode=0 waves=0.592593\n",
      "result 021 : algo={ Id=5, tileIdx=21 (256x64) splitK=0 reduc=0 swizzle=0 custom=0 stages=0} status 0 time 0.158628ms workspace=0 mathMode=0 waves=0.592593\n",
      "result 022 : algo={ Id=5, tileIdx=20 (128x128) splitK=0 reduc=0 swizzle=1 custom=0 stages=7} status 0 time 0.162089ms workspace=0 mathMode=0 waves=0.592593\n",
      "result 023 : algo={ Id=5, tileIdx=20 (128x128) splitK=0 reduc=0 swizzle=0 custom=0 stages=7} status 0 time 0.162150ms workspace=0 mathMode=0 waves=0.592593\n",
      "result 024 : algo={ Id=5, tileIdx=21 (256x64) splitK=0 reduc=0 swizzle=1 custom=0 stages=0} status 0 time 0.168735ms workspace=0 mathMode=0 waves=0.592593\n",
      "result 025 : algo={ Id=6, tileIdx=18 (128x64) splitK=0 reduc=0 swizzle=0 custom=0 stages=16} status 0 time 0.183214ms workspace=0 mathMode=0 waves=2.370370\n",
      "result 026 : algo={ Id=6, tileIdx=23 (128x256) splitK=0 reduc=0 swizzle=1 custom=0 stages=15} status 0 time 0.204339ms workspace=0 mathMode=0 waves=1.185185\n",
      "result 027 : algo={ Id=6, tileIdx=23 (128x256) splitK=0 reduc=0 swizzle=0 custom=0 stages=15} status 0 time 0.204360ms workspace=0 mathMode=0 waves=1.185185\n",
      "result 028 : algo={ Id=5, tileIdx=23 (128x256) splitK=0 reduc=0 swizzle=1 custom=0 stages=0} status 0 time 0.251187ms workspace=0 mathMode=0 waves=1.185185\n",
      "result 029 : algo={ Id=5, tileIdx=23 (128x256) splitK=0 reduc=0 swizzle=0 custom=0 stages=0} status 0 time 0.251238ms workspace=0 mathMode=0 waves=1.185185\n",
      "result 030 : algo={ Id=6, tileIdx=23 (128x256) splitK=0 reduc=0 swizzle=0 custom=0 stages=0} status 0 time 0.257618ms workspace=0 mathMode=0 waves=1.185185\n",
      "result 031 : algo={ Id=6, tileIdx=23 (128x256) splitK=0 reduc=0 swizzle=1 custom=0 stages=0} status 0 time 0.258263ms workspace=0 mathMode=0 waves=1.185185\n",
      "result 032 : algo={ Id=5, tileIdx=23 (128x256) splitK=0 reduc=0 swizzle=0 custom=0 stages=7} status 0 time 0.265728ms workspace=0 mathMode=0 waves=1.185185\n",
      "result 033 : algo={ Id=5, tileIdx=23 (128x256) splitK=0 reduc=0 swizzle=1 custom=0 stages=7} status 0 time 0.265748ms workspace=0 mathMode=0 waves=1.185185\n",
      "result 034 : algo={ Id=1, tileIdx=16 (128x32) splitK=0 reduc=0 swizzle=1 custom=0 stages=0} status 0 time 1.008599ms workspace=0 mathMode=0 waves=4.740741\n",
      "result 035 : algo={ Id=1, tileIdx=16 (128x32) splitK=0 reduc=0 swizzle=0 custom=0 stages=0} status 0 time 1.009285ms workspace=0 mathMode=0 waves=4.740741\n",
      "result 036 : algo={ Id=1, tileIdx=13 (64x32) splitK=0 reduc=0 swizzle=1 custom=0 stages=0} status 0 time 1.014088ms workspace=0 mathMode=0 waves=4.740741\n",
      "result 037 : algo={ Id=1, tileIdx=13 (64x32) splitK=0 reduc=0 swizzle=0 custom=0 stages=0} status 0 time 1.022218ms workspace=0 mathMode=0 waves=4.740741\n",
      "result 038 : algo={ Id=1, tileIdx=11 (32x32) splitK=0 reduc=0 swizzle=1 custom=0 stages=0} status 0 time 1.041418ms workspace=0 mathMode=0 waves=4.740741\n",
      "result 039 : algo={ Id=0, tileIdx=16 (128x32) splitK=0 reduc=0 swizzle=1 custom=0 stages=0} status 0 time 1.060905ms workspace=0 mathMode=0 waves=1.185185\n",
      "result 040 : algo={ Id=1, tileIdx=11 (32x32) splitK=0 reduc=0 swizzle=0 custom=0 stages=0} status 0 time 1.072210ms workspace=0 mathMode=0 waves=4.740741\n",
      "result 041 : algo={ Id=0, tileIdx=16 (128x32) splitK=0 reduc=0 swizzle=0 custom=0 stages=0} status 0 time 1.088502ms workspace=0 mathMode=0 waves=1.185185\n",
      "result 042 : algo={ Id=0, tileIdx=14 (32x128) splitK=0 reduc=0 swizzle=1 custom=0 stages=0} status 0 time 1.090734ms workspace=0 mathMode=0 waves=1.185185\n",
      "result 043 : algo={ Id=0, tileIdx=14 (32x128) splitK=0 reduc=0 swizzle=0 custom=0 stages=0} status 0 time 1.105756ms workspace=0 mathMode=0 waves=1.185185\n",
      "result 044 : algo={ Id=0, tileIdx=18 (128x64) splitK=0 reduc=0 swizzle=1 custom=0 stages=0} status 0 time 1.200046ms workspace=0 mathMode=0 waves=0.592593\n",
      "result 045 : algo={ Id=0, tileIdx=18 (128x64) splitK=0 reduc=0 swizzle=0 custom=0 stages=0} status 0 time 1.200333ms workspace=0 mathMode=0 waves=0.592593\n",
      "result 046 : algo={ Id=0, tileIdx=15 (64x64) splitK=0 reduc=0 swizzle=0 custom=0 stages=0} status 0 time 1.215283ms workspace=0 mathMode=0 waves=0.592593\n",
      "result 047 : algo={ Id=0, tileIdx=15 (64x64) splitK=0 reduc=0 swizzle=1 custom=0 stages=0} status 0 time 1.217997ms workspace=0 mathMode=0 waves=0.592593\n",
      "result 048 : algo={ Id=0, tileIdx=20 (128x128) splitK=0 reduc=0 swizzle=1 custom=0 stages=0} status 0 time 1.521326ms workspace=0 mathMode=0 waves=0.592593\n",
      "result 049 : algo={ Id=0, tileIdx=20 (128x128) splitK=0 reduc=0 swizzle=0 custom=0 stages=0} status 0 time 1.525248ms workspace=0 mathMode=0 waves=0.592593\n",
      "algo={ Id=6, tileIdx=24 (256x128) splitK=0 reduc=0 swizzle=0 custom=0 stages=9} status 0 time 0.108677ms workspace=0 mathMode=0 waves=0.592593\n",
      "***cublasLt Gemm Testing End***\n",
      "\n",
      "-----------------------------\n",
      "GEMM test 5: [M: 128, K: 16384, N: 4096] context ffn gemm 2\n",
      "algo_99 costs 0.115ms \n",
      "algo_100 costs 0.115ms \n",
      "algo_101 costs 0.116ms \n",
      "algo_102 costs 0.117ms \n",
      "algo_103 costs 0.119ms \n",
      "algo_104 costs 0.120ms \n",
      "algo_105 costs 0.123ms \n",
      "algo_106 costs 0.125ms \n",
      "algo_107 costs 0.128ms \n",
      "algo_108 costs 0.130ms \n",
      "algo_109 costs 0.131ms \n",
      "algo_110 costs 0.132ms \n",
      "algo_111 costs 0.131ms \n",
      "algo_112 costs 0.129ms \n",
      "algo_113 costs 0.125ms \n",
      "algo_114 costs 0.122ms \n",
      "algo_115 costs 0.120ms \n",
      "fast_algo 99 costs 0.115 ms\n",
      "***cublasLt Gemm Testing Beign***\n",
      "AlgoCount: 4287\n",
      "return 7 and run heuristic algo\n",
      "workspacesize==0, run 170 algos\n",
      "result 000 : algo={ Id=6, tileIdx=18 (128x64) splitK=0 reduc=0 swizzle=0 custom=0 stages=12} status 0 time 0.109271ms workspace=0 mathMode=0 waves=0.296296\n",
      "result 001 : algo={ Id=6, tileIdx=17 (64x128) splitK=1 reduc=0 swizzle=0 custom=0 stages=16} status 0 time 0.115702ms workspace=0 mathMode=0 waves=0.592593\n",
      "result 002 : algo={ Id=6, tileIdx=24 (256x128) splitK=5 reduc=1 swizzle=1 custom=0 stages=15} status 0 time 0.118067ms workspace=64 mathMode=0 waves=0.740741\n",
      "result 003 : algo={ Id=6, tileIdx=24 (256x128) splitK=4 reduc=1 swizzle=1 custom=0 stages=15} status 0 time 0.120535ms workspace=64 mathMode=0 waves=0.592593\n",
      "result 004 : algo={ Id=6, tileIdx=17 (64x128) splitK=3 reduc=4 swizzle=0 custom=0 stages=16} status 0 time 0.121457ms workspace=3145728 mathMode=0 waves=1.777778\n",
      "result 005 : algo={ Id=6, tileIdx=18 (128x64) splitK=0 reduc=0 swizzle=1 custom=0 stages=15} status 0 time 0.139387ms workspace=0 mathMode=0 waves=0.296296\n",
      "result 006 : algo={ Id=6, tileIdx=18 (128x64) splitK=0 reduc=0 swizzle=0 custom=0 stages=15} status 0 time 0.139960ms workspace=0 mathMode=0 waves=0.296296\n",
      "result 007 : algo={ Id=23, tileIdx=20 (128x128) splitK=2 reduc=4 swizzle=0 custom=0 stages=15} status 0 time 0.196198ms workspace=2097152 mathMode=0 waves=0.592593\n",
      "result 008 : algo={ Id=6, tileIdx=20 (128x128) splitK=1 reduc=0 swizzle=1 custom=0 stages=15} status 0 time 0.199148ms workspace=0 mathMode=0 waves=0.296296\n",
      "result 009 : algo={ Id=6, tileIdx=20 (128x128) splitK=0 reduc=0 swizzle=0 custom=0 stages=15} status 0 time 0.199332ms workspace=0 mathMode=0 waves=0.296296\n",
      "result 010 : algo={ Id=6, tileIdx=20 (128x128) splitK=0 reduc=0 swizzle=1 custom=0 stages=15} status 0 time 0.199537ms workspace=0 mathMode=0 waves=0.296296\n",
      "result 011 : algo={ Id=6, tileIdx=20 (128x128) splitK=0 reduc=0 swizzle=1 custom=0 stages=11} status 0 time 0.200786ms workspace=0 mathMode=0 waves=0.148148\n",
      "result 012 : algo={ Id=6, tileIdx=20 (128x128) splitK=0 reduc=0 swizzle=0 custom=0 stages=11} status 0 time 0.200847ms workspace=0 mathMode=0 waves=0.148148\n",
      "result 013 : algo={ Id=6, tileIdx=23 (128x256) splitK=2 reduc=1 swizzle=1 custom=0 stages=15} status 0 time 0.206141ms workspace=128 mathMode=0 waves=0.592593\n",
      "result 014 : algo={ Id=6, tileIdx=23 (128x256) splitK=0 reduc=0 swizzle=0 custom=0 stages=15} status 0 time 0.388956ms workspace=0 mathMode=0 waves=0.296296\n",
      "result 015 : algo={ Id=6, tileIdx=23 (128x256) splitK=0 reduc=0 swizzle=1 custom=0 stages=15} status 0 time 0.388966ms workspace=0 mathMode=0 waves=0.296296\n",
      "result 016 : algo={ Id=5, tileIdx=20 (128x128) splitK=0 reduc=0 swizzle=1 custom=0 stages=0} status 0 time 0.389386ms workspace=0 mathMode=0 waves=0.148148\n",
      "result 017 : algo={ Id=5, tileIdx=20 (128x128) splitK=0 reduc=0 swizzle=0 custom=0 stages=0} status 0 time 0.389396ms workspace=0 mathMode=0 waves=0.148148\n",
      "result 018 : algo={ Id=5, tileIdx=21 (256x64) splitK=0 reduc=0 swizzle=0 custom=0 stages=7} status 0 time 0.423567ms workspace=0 mathMode=0 waves=0.148148\n",
      "result 019 : algo={ Id=5, tileIdx=21 (256x64) splitK=0 reduc=0 swizzle=1 custom=0 stages=7} status 0 time 0.424724ms workspace=0 mathMode=0 waves=0.148148\n",
      "result 020 : algo={ Id=6, tileIdx=23 (128x256) splitK=0 reduc=0 swizzle=1 custom=0 stages=0} status 0 time 0.433705ms workspace=0 mathMode=0 waves=0.296296\n",
      "result 021 : algo={ Id=6, tileIdx=23 (128x256) splitK=0 reduc=0 swizzle=0 custom=0 stages=0} status 0 time 0.433818ms workspace=0 mathMode=0 waves=0.296296\n",
      "result 022 : algo={ Id=5, tileIdx=20 (128x128) splitK=0 reduc=0 swizzle=0 custom=0 stages=7} status 0 time 0.437135ms workspace=0 mathMode=0 waves=0.148148\n",
      "result 023 : algo={ Id=5, tileIdx=20 (128x128) splitK=0 reduc=0 swizzle=1 custom=0 stages=7} status 0 time 0.437176ms workspace=0 mathMode=0 waves=0.148148\n",
      "result 024 : algo={ Id=6, tileIdx=18 (128x64) splitK=0 reduc=0 swizzle=1 custom=0 stages=0} status 0 time 0.459776ms workspace=0 mathMode=0 waves=0.148148\n",
      "result 025 : algo={ Id=6, tileIdx=18 (128x64) splitK=0 reduc=0 swizzle=0 custom=0 stages=0} status 0 time 0.460677ms workspace=0 mathMode=0 waves=0.148148\n",
      "result 026 : algo={ Id=5, tileIdx=23 (128x256) splitK=0 reduc=0 swizzle=1 custom=0 stages=0} status 0 time 0.483953ms workspace=0 mathMode=0 waves=0.296296\n",
      "result 027 : algo={ Id=5, tileIdx=23 (128x256) splitK=0 reduc=0 swizzle=0 custom=0 stages=0} status 0 time 0.483983ms workspace=0 mathMode=0 waves=0.296296\n",
      "result 028 : algo={ Id=5, tileIdx=24 (256x128) splitK=0 reduc=0 swizzle=1 custom=0 stages=0} status 0 time 0.506460ms workspace=0 mathMode=0 waves=0.148148\n",
      "result 029 : algo={ Id=5, tileIdx=24 (256x128) splitK=0 reduc=0 swizzle=0 custom=0 stages=0} status 0 time 0.506511ms workspace=0 mathMode=0 waves=0.148148\n",
      "result 030 : algo={ Id=5, tileIdx=23 (128x256) splitK=0 reduc=0 swizzle=0 custom=0 stages=7} status 0 time 0.510515ms workspace=0 mathMode=0 waves=0.296296\n",
      "result 031 : algo={ Id=5, tileIdx=23 (128x256) splitK=0 reduc=0 swizzle=1 custom=0 stages=7} status 0 time 0.510546ms workspace=0 mathMode=0 waves=0.296296\n",
      "result 032 : algo={ Id=5, tileIdx=21 (256x64) splitK=0 reduc=0 swizzle=0 custom=0 stages=0} status 0 time 0.550011ms workspace=0 mathMode=0 waves=0.148148\n",
      "result 033 : algo={ Id=5, tileIdx=21 (256x64) splitK=0 reduc=0 swizzle=1 custom=0 stages=0} status 0 time 0.554803ms workspace=0 mathMode=0 waves=0.148148\n",
      "result 034 : algo={ Id=1, tileIdx=11 (32x32) splitK=0 reduc=0 swizzle=1 custom=0 stages=0} status 0 time 1.238549ms workspace=0 mathMode=0 waves=1.185185\n",
      "result 035 : algo={ Id=1, tileIdx=13 (64x32) splitK=0 reduc=0 swizzle=1 custom=0 stages=0} status 0 time 1.254052ms workspace=0 mathMode=0 waves=1.185185\n",
      "result 036 : algo={ Id=1, tileIdx=13 (64x32) splitK=0 reduc=0 swizzle=0 custom=0 stages=0} status 0 time 1.279314ms workspace=0 mathMode=0 waves=1.185185\n",
      "result 037 : algo={ Id=1, tileIdx=11 (32x32) splitK=0 reduc=0 swizzle=0 custom=0 stages=0} status 0 time 1.280881ms workspace=0 mathMode=0 waves=1.185185\n",
      "result 038 : algo={ Id=1, tileIdx=16 (128x32) splitK=0 reduc=0 swizzle=0 custom=0 stages=0} status 0 time 1.583145ms workspace=0 mathMode=0 waves=1.185185\n",
      "result 039 : algo={ Id=1, tileIdx=16 (128x32) splitK=0 reduc=0 swizzle=1 custom=0 stages=0} status 0 time 1.584220ms workspace=0 mathMode=0 waves=1.185185\n",
      "result 040 : algo={ Id=0, tileIdx=14 (32x128) splitK=0 reduc=0 swizzle=0 custom=0 stages=0} status 0 time 1.793874ms workspace=0 mathMode=0 waves=0.296296\n",
      "result 041 : algo={ Id=0, tileIdx=14 (32x128) splitK=0 reduc=0 swizzle=1 custom=0 stages=0} status 0 time 1.814292ms workspace=0 mathMode=0 waves=0.296296\n",
      "result 042 : algo={ Id=0, tileIdx=16 (128x32) splitK=0 reduc=0 swizzle=1 custom=0 stages=0} status 0 time 1.818849ms workspace=0 mathMode=0 waves=0.296296\n",
      "result 043 : algo={ Id=0, tileIdx=16 (128x32) splitK=0 reduc=0 swizzle=0 custom=0 stages=0} status 0 time 1.832151ms workspace=0 mathMode=0 waves=0.296296\n",
      "result 044 : algo={ Id=0, tileIdx=18 (128x64) splitK=0 reduc=0 swizzle=1 custom=0 stages=0} status 0 time 1.863629ms workspace=0 mathMode=0 waves=0.148148\n",
      "result 045 : algo={ Id=0, tileIdx=18 (128x64) splitK=0 reduc=0 swizzle=0 custom=0 stages=0} status 0 time 1.863629ms workspace=0 mathMode=0 waves=0.148148\n",
      "result 046 : algo={ Id=0, tileIdx=20 (128x128) splitK=0 reduc=0 swizzle=0 custom=0 stages=0} status 0 time 3.028900ms workspace=0 mathMode=0 waves=0.148148\n",
      "result 047 : algo={ Id=0, tileIdx=20 (128x128) splitK=0 reduc=0 swizzle=1 custom=0 stages=0} status 0 time 3.030334ms workspace=0 mathMode=0 waves=0.148148\n",
      "result 048 : algo={ Id=0, tileIdx=15 (64x64) splitK=0 reduc=0 swizzle=1 custom=0 stages=0} status 0 time 3.113103ms workspace=0 mathMode=0 waves=0.148148\n",
      "result 049 : algo={ Id=0, tileIdx=15 (64x64) splitK=0 reduc=0 swizzle=0 custom=0 stages=0} status 0 time 3.139942ms workspace=0 mathMode=0 waves=0.148148\n",
      "algo={ Id=6, tileIdx=18 (128x64) splitK=0 reduc=0 swizzle=0 custom=0 stages=12} status 0 time 0.109271ms workspace=0 mathMode=0 waves=0.296296\n",
      "***cublasLt Gemm Testing End***\n",
      "\n",
      "-----------------------------\n",
      "GEMM test 6: [M: 1, K: 4096, N: 12288] from_tensor * weightQKV\n",
      "algo_99 costs 0.071ms \n",
      "algo_100 costs 0.070ms \n",
      "algo_101 costs 0.070ms \n",
      "algo_102 costs 0.070ms \n",
      "algo_103 costs 0.070ms \n",
      "algo_104 costs 0.070ms \n",
      "algo_105 costs 0.070ms \n",
      "algo_106 costs 0.070ms \n",
      "algo_107 costs 0.070ms \n",
      "algo_108 costs 0.070ms \n",
      "algo_109 costs 0.070ms \n",
      "algo_110 costs 0.070ms \n",
      "algo_111 costs 0.070ms \n",
      "algo_112 costs 0.070ms \n",
      "algo_113 costs 0.070ms \n",
      "algo_114 costs 0.070ms \n",
      "algo_115 costs 0.070ms \n",
      "fast_algo 101 costs 0.070 ms\n",
      "***cublasLt Gemm Testing Beign***\n",
      "AlgoCount: 5000\n",
      "return 8 and run heuristic algo\n",
      "workspacesize==0, run 245 algos\n",
      "result 000 : algo={ Id=6, tileIdx=18 (128x64) splitK=1 reduc=0 swizzle=1 custom=0 stages=16} status 0 time 0.069171ms workspace=0 mathMode=0 waves=0.888889\n",
      "result 001 : algo={ Id=6, tileIdx=18 (128x64) splitK=1 reduc=0 swizzle=0 custom=0 stages=12} status 0 time 0.069990ms workspace=0 mathMode=0 waves=0.444444\n",
      "result 002 : algo={ Id=6, tileIdx=18 (128x64) splitK=2 reduc=1 swizzle=1 custom=0 stages=12} status 0 time 0.070103ms workspace=384 mathMode=0 waves=0.888889\n",
      "result 003 : algo={ Id=6, tileIdx=18 (128x64) splitK=0 reduc=0 swizzle=0 custom=0 stages=15} status 0 time 0.072233ms workspace=0 mathMode=0 waves=0.444444\n",
      "result 004 : algo={ Id=21, tileIdx=21 (256x64) splitK=4 reduc=4 swizzle=0 custom=0 stages=10} status 0 time 0.072253ms workspace=98304 mathMode=0 waves=0.888889\n",
      "result 005 : algo={ Id=6, tileIdx=18 (128x64) splitK=0 reduc=0 swizzle=1 custom=0 stages=15} status 0 time 0.072489ms workspace=0 mathMode=0 waves=0.444444\n",
      "result 006 : algo={ Id=6, tileIdx=21 (256x64) splitK=2 reduc=1 swizzle=1 custom=0 stages=15} status 0 time 0.073943ms workspace=192 mathMode=0 waves=0.888889\n",
      "result 007 : algo={ Id=6, tileIdx=20 (128x128) splitK=0 reduc=0 swizzle=0 custom=0 stages=11} status 0 time 0.074281ms workspace=0 mathMode=0 waves=0.444444\n",
      "result 008 : algo={ Id=6, tileIdx=20 (128x128) splitK=0 reduc=0 swizzle=1 custom=0 stages=11} status 0 time 0.074291ms workspace=0 mathMode=0 waves=0.444444\n",
      "result 009 : algo={ Id=6, tileIdx=20 (128x128) splitK=0 reduc=0 swizzle=0 custom=0 stages=15} status 0 time 0.074363ms workspace=0 mathMode=0 waves=0.888889\n",
      "result 010 : algo={ Id=6, tileIdx=20 (128x128) splitK=0 reduc=0 swizzle=1 custom=0 stages=15} status 0 time 0.074711ms workspace=0 mathMode=0 waves=0.888889\n",
      "result 011 : algo={ Id=6, tileIdx=21 (256x64) splitK=3 reduc=4 swizzle=0 custom=0 stages=15} status 0 time 0.075530ms workspace=73728 mathMode=0 waves=1.333333\n",
      "result 012 : algo={ Id=13, tileIdx=0 (UNDEF) splitK=1 reduc=0 swizzle=0 custom=103 stages=0} status 0 time 0.078418ms workspace=0 mathMode=0 waves=0.888889\n",
      "result 013 : algo={ Id=13, tileIdx=0 (UNDEF) splitK=32 reduc=2 swizzle=0 custom=7 stages=0} status 0 time 0.101601ms workspace=1572880 mathMode=0 waves=2.844445\n",
      "result 014 : algo={ Id=5, tileIdx=20 (128x128) splitK=0 reduc=0 swizzle=1 custom=0 stages=0} status 0 time 0.103537ms workspace=0 mathMode=0 waves=0.444444\n",
      "result 015 : algo={ Id=5, tileIdx=20 (128x128) splitK=0 reduc=0 swizzle=0 custom=0 stages=0} status 0 time 0.103608ms workspace=0 mathMode=0 waves=0.444444\n",
      "result 016 : algo={ Id=6, tileIdx=23 (128x256) splitK=0 reduc=0 swizzle=0 custom=0 stages=15} status 0 time 0.104253ms workspace=0 mathMode=0 waves=0.888889\n",
      "result 017 : algo={ Id=6, tileIdx=23 (128x256) splitK=0 reduc=0 swizzle=1 custom=0 stages=15} status 0 time 0.104305ms workspace=0 mathMode=0 waves=0.888889\n",
      "result 018 : algo={ Id=5, tileIdx=20 (128x128) splitK=0 reduc=0 swizzle=1 custom=0 stages=7} status 0 time 0.112865ms workspace=0 mathMode=0 waves=0.444444\n",
      "result 019 : algo={ Id=5, tileIdx=20 (128x128) splitK=0 reduc=0 swizzle=0 custom=0 stages=7} status 0 time 0.112865ms workspace=0 mathMode=0 waves=0.444444\n",
      "result 020 : algo={ Id=5, tileIdx=21 (256x64) splitK=0 reduc=0 swizzle=1 custom=0 stages=7} status 0 time 0.118180ms workspace=0 mathMode=0 waves=0.222222\n",
      "result 021 : algo={ Id=5, tileIdx=21 (256x64) splitK=0 reduc=0 swizzle=0 custom=0 stages=7} status 0 time 0.118682ms workspace=0 mathMode=0 waves=0.222222\n",
      "result 022 : algo={ Id=6, tileIdx=23 (128x256) splitK=0 reduc=0 swizzle=0 custom=0 stages=0} status 0 time 0.121979ms workspace=0 mathMode=0 waves=0.888889\n",
      "result 023 : algo={ Id=6, tileIdx=23 (128x256) splitK=0 reduc=0 swizzle=1 custom=0 stages=0} status 0 time 0.122061ms workspace=0 mathMode=0 waves=0.888889\n",
      "result 024 : algo={ Id=5, tileIdx=23 (128x256) splitK=0 reduc=0 swizzle=1 custom=0 stages=0} status 0 time 0.126628ms workspace=0 mathMode=0 waves=0.888889\n",
      "result 025 : algo={ Id=5, tileIdx=23 (128x256) splitK=0 reduc=0 swizzle=0 custom=0 stages=0} status 0 time 0.126669ms workspace=0 mathMode=0 waves=0.888889\n",
      "result 026 : algo={ Id=5, tileIdx=23 (128x256) splitK=0 reduc=0 swizzle=1 custom=0 stages=7} status 0 time 0.128246ms workspace=0 mathMode=0 waves=0.888889\n",
      "result 027 : algo={ Id=5, tileIdx=23 (128x256) splitK=0 reduc=0 swizzle=0 custom=0 stages=7} status 0 time 0.128287ms workspace=0 mathMode=0 waves=0.888889\n",
      "result 028 : algo={ Id=5, tileIdx=24 (256x128) splitK=0 reduc=0 swizzle=0 custom=0 stages=0} status 0 time 0.128891ms workspace=0 mathMode=0 waves=0.444444\n",
      "result 029 : algo={ Id=5, tileIdx=24 (256x128) splitK=0 reduc=0 swizzle=1 custom=0 stages=0} status 0 time 0.129485ms workspace=0 mathMode=0 waves=0.444444\n",
      "result 030 : algo={ Id=6, tileIdx=18 (128x64) splitK=0 reduc=0 swizzle=0 custom=0 stages=0} status 0 time 0.129536ms workspace=0 mathMode=0 waves=0.222222\n",
      "result 031 : algo={ Id=6, tileIdx=18 (128x64) splitK=0 reduc=0 swizzle=1 custom=0 stages=0} status 0 time 0.129669ms workspace=0 mathMode=0 waves=0.222222\n",
      "result 032 : algo={ Id=5, tileIdx=21 (256x64) splitK=0 reduc=0 swizzle=1 custom=0 stages=0} status 0 time 0.151316ms workspace=0 mathMode=0 waves=0.222222\n",
      "result 033 : algo={ Id=5, tileIdx=21 (256x64) splitK=0 reduc=0 swizzle=0 custom=0 stages=0} status 0 time 0.152197ms workspace=0 mathMode=0 waves=0.222222\n",
      "result 034 : algo={ Id=1, tileIdx=16 (128x32) splitK=0 reduc=0 swizzle=1 custom=0 stages=0} status 0 time 0.205220ms workspace=0 mathMode=0 waves=0.888889\n",
      "result 035 : algo={ Id=1, tileIdx=16 (128x32) splitK=0 reduc=0 swizzle=0 custom=0 stages=0} status 0 time 0.205322ms workspace=0 mathMode=0 waves=0.888889\n",
      "result 036 : algo={ Id=1, tileIdx=13 (64x32) splitK=0 reduc=0 swizzle=1 custom=0 stages=0} status 0 time 0.219648ms workspace=0 mathMode=0 waves=0.888889\n",
      "result 037 : algo={ Id=1, tileIdx=13 (64x32) splitK=0 reduc=0 swizzle=0 custom=0 stages=0} status 0 time 0.219658ms workspace=0 mathMode=0 waves=0.888889\n",
      "result 038 : algo={ Id=1, tileIdx=11 (32x32) splitK=0 reduc=0 swizzle=1 custom=0 stages=0} status 0 time 0.235776ms workspace=0 mathMode=0 waves=0.888889\n",
      "result 039 : algo={ Id=1, tileIdx=11 (32x32) splitK=0 reduc=0 swizzle=0 custom=0 stages=0} status 0 time 0.235837ms workspace=0 mathMode=0 waves=0.888889\n",
      "result 040 : algo={ Id=0, tileIdx=16 (128x32) splitK=0 reduc=0 swizzle=1 custom=0 stages=0} status 0 time 0.381409ms workspace=0 mathMode=0 waves=0.222222\n",
      "result 041 : algo={ Id=0, tileIdx=16 (128x32) splitK=0 reduc=0 swizzle=0 custom=0 stages=0} status 0 time 0.381583ms workspace=0 mathMode=0 waves=0.222222\n",
      "result 042 : algo={ Id=0, tileIdx=18 (128x64) splitK=0 reduc=0 swizzle=1 custom=0 stages=0} status 0 time 0.468255ms workspace=0 mathMode=0 waves=0.222222\n",
      "result 043 : algo={ Id=0, tileIdx=18 (128x64) splitK=0 reduc=0 swizzle=0 custom=0 stages=0} status 0 time 0.468347ms workspace=0 mathMode=0 waves=0.222222\n",
      "result 044 : algo={ Id=0, tileIdx=20 (128x128) splitK=0 reduc=0 swizzle=1 custom=0 stages=0} status 0 time 0.762419ms workspace=0 mathMode=0 waves=0.444444\n",
      "result 045 : algo={ Id=0, tileIdx=20 (128x128) splitK=0 reduc=0 swizzle=0 custom=0 stages=0} status 0 time 0.762429ms workspace=0 mathMode=0 waves=0.444444\n",
      "result 046 : algo={ Id=0, tileIdx=15 (64x64) splitK=0 reduc=0 swizzle=1 custom=0 stages=0} status 0 time 0.788931ms workspace=0 mathMode=0 waves=0.222222\n",
      "result 047 : algo={ Id=0, tileIdx=15 (64x64) splitK=0 reduc=0 swizzle=0 custom=0 stages=0} status 0 time 0.790528ms workspace=0 mathMode=0 waves=0.222222\n",
      "result 048 : algo={ Id=0, tileIdx=14 (32x128) splitK=0 reduc=0 swizzle=1 custom=0 stages=0} status 0 time 0.847575ms workspace=0 mathMode=0 waves=0.888889\n",
      "result 049 : algo={ Id=0, tileIdx=14 (32x128) splitK=0 reduc=0 swizzle=0 custom=0 stages=0} status 0 time 0.847770ms workspace=0 mathMode=0 waves=0.888889\n",
      "algo={ Id=6, tileIdx=18 (128x64) splitK=1 reduc=0 swizzle=1 custom=0 stages=16} status 0 time 0.069171ms workspace=0 mathMode=0 waves=0.888889\n",
      "***cublasLt Gemm Testing End***\n",
      "\n",
      "-----------------------------\n",
      "GEMM test 7: [M: 1, K: 4096, N: 4096] attr * output_kernel\n",
      "algo_99 costs 0.023ms \n",
      "algo_100 costs 0.022ms \n",
      "algo_101 costs 0.022ms \n",
      "algo_102 costs 0.022ms \n",
      "algo_103 costs 0.022ms \n",
      "algo_104 costs 0.022ms \n",
      "algo_105 costs 0.022ms \n",
      "algo_106 costs 0.022ms \n",
      "algo_107 costs 0.022ms \n",
      "algo_108 costs 0.022ms \n",
      "algo_109 costs 0.022ms \n",
      "algo_110 costs 0.022ms \n",
      "algo_111 costs 0.022ms \n",
      "algo_112 costs 0.022ms \n",
      "algo_113 costs 0.022ms \n",
      "algo_114 costs 0.022ms \n",
      "algo_115 costs 0.022ms \n",
      "fast_algo 114 costs 0.022 ms\n",
      "***cublasLt Gemm Testing Beign***\n",
      "AlgoCount: 5000\n",
      "return 8 and run heuristic algo\n",
      "workspacesize==0, run 245 algos\n",
      "result 000 : algo={ Id=31, tileIdx=15 (64x64) splitK=1 reduc=0 swizzle=0 custom=0 stages=17} status 0 time 0.021914ms workspace=0 mathMode=0 waves=0.296296\n",
      "result 001 : algo={ Id=6, tileIdx=18 (128x64) splitK=3 reduc=4 swizzle=0 custom=0 stages=16} status 0 time 0.022815ms workspace=24576 mathMode=0 waves=0.888889\n",
      "result 002 : algo={ Id=6, tileIdx=15 (64x64) splitK=1 reduc=0 swizzle=1 custom=0 stages=18} status 0 time 0.022938ms workspace=0 mathMode=0 waves=0.592593\n",
      "result 003 : algo={ Id=6, tileIdx=18 (128x64) splitK=2 reduc=1 swizzle=1 custom=0 stages=12} status 0 time 0.024412ms workspace=128 mathMode=0 waves=0.296296\n",
      "result 004 : algo={ Id=21, tileIdx=21 (256x64) splitK=4 reduc=4 swizzle=0 custom=0 stages=10} status 0 time 0.025293ms workspace=32768 mathMode=0 waves=0.296296\n",
      "result 005 : algo={ Id=6, tileIdx=18 (128x64) splitK=4 reduc=4 swizzle=1 custom=0 stages=12} status 0 time 0.026429ms workspace=32768 mathMode=0 waves=0.592593\n",
      "result 006 : algo={ Id=6, tileIdx=18 (128x64) splitK=7 reduc=1 swizzle=0 custom=0 stages=16} status 0 time 0.031222ms workspace=128 mathMode=0 waves=2.074074\n",
      "result 007 : algo={ Id=6, tileIdx=18 (128x64) splitK=0 reduc=0 swizzle=0 custom=0 stages=15} status 0 time 0.034714ms workspace=0 mathMode=0 waves=0.148148\n",
      "result 008 : algo={ Id=6, tileIdx=18 (128x64) splitK=0 reduc=0 swizzle=1 custom=0 stages=15} status 0 time 0.034714ms workspace=0 mathMode=0 waves=0.148148\n",
      "result 009 : algo={ Id=13, tileIdx=0 (UNDEF) splitK=64 reduc=2 swizzle=0 custom=7 stages=0} status 0 time 0.040806ms workspace=1048592 mathMode=0 waves=1.896296\n",
      "result 010 : algo={ Id=6, tileIdx=20 (128x128) splitK=0 reduc=0 swizzle=1 custom=0 stages=15} status 0 time 0.055009ms workspace=0 mathMode=0 waves=0.296296\n",
      "result 011 : algo={ Id=6, tileIdx=20 (128x128) splitK=0 reduc=0 swizzle=0 custom=0 stages=15} status 0 time 0.055050ms workspace=0 mathMode=0 waves=0.296296\n",
      "result 012 : algo={ Id=6, tileIdx=20 (128x128) splitK=0 reduc=0 swizzle=0 custom=0 stages=11} status 0 time 0.055153ms workspace=0 mathMode=0 waves=0.148148\n",
      "result 013 : algo={ Id=6, tileIdx=20 (128x128) splitK=0 reduc=0 swizzle=1 custom=0 stages=11} status 0 time 0.055194ms workspace=0 mathMode=0 waves=0.148148\n",
      "result 014 : algo={ Id=5, tileIdx=20 (128x128) splitK=0 reduc=0 swizzle=0 custom=0 stages=0} status 0 time 0.102185ms workspace=0 mathMode=0 waves=0.148148\n",
      "result 015 : algo={ Id=5, tileIdx=20 (128x128) splitK=0 reduc=0 swizzle=1 custom=0 stages=0} status 0 time 0.102185ms workspace=0 mathMode=0 waves=0.148148\n",
      "result 016 : algo={ Id=6, tileIdx=23 (128x256) splitK=0 reduc=0 swizzle=1 custom=0 stages=15} status 0 time 0.102881ms workspace=0 mathMode=0 waves=0.296296\n",
      "result 017 : algo={ Id=6, tileIdx=23 (128x256) splitK=0 reduc=0 swizzle=0 custom=0 stages=15} status 0 time 0.102912ms workspace=0 mathMode=0 waves=0.296296\n",
      "result 018 : algo={ Id=6, tileIdx=23 (128x256) splitK=0 reduc=0 swizzle=0 custom=0 stages=0} status 0 time 0.107397ms workspace=0 mathMode=0 waves=0.296296\n",
      "result 019 : algo={ Id=6, tileIdx=23 (128x256) splitK=0 reduc=0 swizzle=1 custom=0 stages=0} status 0 time 0.107407ms workspace=0 mathMode=0 waves=0.296296\n",
      "result 020 : algo={ Id=5, tileIdx=20 (128x128) splitK=0 reduc=0 swizzle=0 custom=0 stages=7} status 0 time 0.108247ms workspace=0 mathMode=0 waves=0.148148\n",
      "result 021 : algo={ Id=5, tileIdx=20 (128x128) splitK=0 reduc=0 swizzle=1 custom=0 stages=7} status 0 time 0.108339ms workspace=0 mathMode=0 waves=0.148148\n",
      "result 022 : algo={ Id=5, tileIdx=21 (256x64) splitK=0 reduc=0 swizzle=1 custom=0 stages=7} status 0 time 0.110172ms workspace=0 mathMode=0 waves=0.074074\n",
      "result 023 : algo={ Id=5, tileIdx=21 (256x64) splitK=0 reduc=0 swizzle=0 custom=0 stages=7} status 0 time 0.110244ms workspace=0 mathMode=0 waves=0.074074\n",
      "result 024 : algo={ Id=6, tileIdx=18 (128x64) splitK=0 reduc=0 swizzle=0 custom=0 stages=0} status 0 time 0.114678ms workspace=0 mathMode=0 waves=0.074074\n",
      "result 025 : algo={ Id=6, tileIdx=18 (128x64) splitK=0 reduc=0 swizzle=1 custom=0 stages=0} status 0 time 0.114780ms workspace=0 mathMode=0 waves=0.074074\n",
      "result 026 : algo={ Id=5, tileIdx=23 (128x256) splitK=0 reduc=0 swizzle=0 custom=0 stages=0} status 0 time 0.125788ms workspace=0 mathMode=0 waves=0.296296\n",
      "result 027 : algo={ Id=5, tileIdx=23 (128x256) splitK=0 reduc=0 swizzle=1 custom=0 stages=0} status 0 time 0.125788ms workspace=0 mathMode=0 waves=0.296296\n",
      "result 028 : algo={ Id=5, tileIdx=23 (128x256) splitK=0 reduc=0 swizzle=1 custom=0 stages=7} status 0 time 0.127119ms workspace=0 mathMode=0 waves=0.296296\n",
      "result 029 : algo={ Id=5, tileIdx=23 (128x256) splitK=0 reduc=0 swizzle=0 custom=0 stages=7} status 0 time 0.127160ms workspace=0 mathMode=0 waves=0.296296\n",
      "result 030 : algo={ Id=5, tileIdx=24 (256x128) splitK=0 reduc=0 swizzle=1 custom=0 stages=0} status 0 time 0.127212ms workspace=0 mathMode=0 waves=0.148148\n",
      "result 031 : algo={ Id=5, tileIdx=24 (256x128) splitK=0 reduc=0 swizzle=0 custom=0 stages=0} status 0 time 0.127212ms workspace=0 mathMode=0 waves=0.148148\n",
      "result 032 : algo={ Id=1, tileIdx=13 (64x32) splitK=0 reduc=0 swizzle=1 custom=0 stages=0} status 0 time 0.132721ms workspace=0 mathMode=0 waves=0.296296\n",
      "result 033 : algo={ Id=1, tileIdx=13 (64x32) splitK=0 reduc=0 swizzle=0 custom=0 stages=0} status 0 time 0.132936ms workspace=0 mathMode=0 waves=0.296296\n",
      "result 034 : algo={ Id=1, tileIdx=11 (32x32) splitK=0 reduc=0 swizzle=0 custom=0 stages=0} status 0 time 0.138199ms workspace=0 mathMode=0 waves=0.296296\n",
      "result 035 : algo={ Id=1, tileIdx=11 (32x32) splitK=0 reduc=0 swizzle=1 custom=0 stages=0} status 0 time 0.138291ms workspace=0 mathMode=0 waves=0.296296\n",
      "result 036 : algo={ Id=5, tileIdx=21 (256x64) splitK=0 reduc=0 swizzle=1 custom=0 stages=0} status 0 time 0.140913ms workspace=0 mathMode=0 waves=0.074074\n",
      "result 037 : algo={ Id=5, tileIdx=21 (256x64) splitK=0 reduc=0 swizzle=0 custom=0 stages=0} status 0 time 0.140995ms workspace=0 mathMode=0 waves=0.074074\n",
      "result 038 : algo={ Id=1, tileIdx=16 (128x32) splitK=0 reduc=0 swizzle=0 custom=0 stages=0} status 0 time 0.204247ms workspace=0 mathMode=0 waves=0.296296\n",
      "result 039 : algo={ Id=1, tileIdx=16 (128x32) splitK=0 reduc=0 swizzle=1 custom=0 stages=0} status 0 time 0.204257ms workspace=0 mathMode=0 waves=0.296296\n",
      "result 040 : algo={ Id=0, tileIdx=16 (128x32) splitK=0 reduc=0 swizzle=1 custom=0 stages=0} status 0 time 0.362414ms workspace=0 mathMode=0 waves=0.074074\n",
      "result 041 : algo={ Id=0, tileIdx=16 (128x32) splitK=0 reduc=0 swizzle=0 custom=0 stages=0} status 0 time 0.362865ms workspace=0 mathMode=0 waves=0.074074\n",
      "result 042 : algo={ Id=0, tileIdx=14 (32x128) splitK=0 reduc=0 swizzle=1 custom=0 stages=0} status 0 time 0.435579ms workspace=0 mathMode=0 waves=0.296296\n",
      "result 043 : algo={ Id=0, tileIdx=14 (32x128) splitK=0 reduc=0 swizzle=0 custom=0 stages=0} status 0 time 0.435722ms workspace=0 mathMode=0 waves=0.296296\n",
      "result 044 : algo={ Id=0, tileIdx=18 (128x64) splitK=0 reduc=0 swizzle=1 custom=0 stages=0} status 0 time 0.455148ms workspace=0 mathMode=0 waves=0.074074\n",
      "result 045 : algo={ Id=0, tileIdx=18 (128x64) splitK=0 reduc=0 swizzle=0 custom=0 stages=0} status 0 time 0.455342ms workspace=0 mathMode=0 waves=0.074074\n",
      "result 046 : algo={ Id=0, tileIdx=15 (64x64) splitK=0 reduc=0 swizzle=1 custom=0 stages=0} status 0 time 0.466698ms workspace=0 mathMode=0 waves=0.074074\n",
      "result 047 : algo={ Id=0, tileIdx=15 (64x64) splitK=0 reduc=0 swizzle=0 custom=0 stages=0} status 0 time 0.467200ms workspace=0 mathMode=0 waves=0.074074\n",
      "result 048 : algo={ Id=0, tileIdx=20 (128x128) splitK=0 reduc=0 swizzle=1 custom=0 stages=0} status 0 time 0.761580ms workspace=0 mathMode=0 waves=0.148148\n",
      "result 049 : algo={ Id=0, tileIdx=20 (128x128) splitK=0 reduc=0 swizzle=0 custom=0 stages=0} status 0 time 0.761590ms workspace=0 mathMode=0 waves=0.148148\n",
      "algo={ Id=31, tileIdx=15 (64x64) splitK=1 reduc=0 swizzle=0 custom=0 stages=17} status 0 time 0.021914ms workspace=0 mathMode=0 waves=0.296296\n",
      "***cublasLt Gemm Testing End***\n",
      "\n",
      "-----------------------------\n",
      "GEMM test 8: [M: 1, K: 4096, N: 16384] ffn gemm 1\n",
      "algo_99 costs 0.094ms \n",
      "algo_100 costs 0.094ms \n",
      "algo_101 costs 0.094ms \n",
      "algo_102 costs 0.094ms \n",
      "algo_103 costs 0.094ms \n",
      "algo_104 costs 0.094ms \n",
      "algo_105 costs 0.094ms \n",
      "algo_106 costs 0.094ms \n",
      "algo_107 costs 0.094ms \n",
      "algo_108 costs 0.094ms \n",
      "algo_109 costs 0.094ms \n",
      "algo_110 costs 0.094ms \n",
      "algo_111 costs 0.094ms \n",
      "algo_112 costs 0.094ms \n",
      "algo_113 costs 0.094ms \n",
      "algo_114 costs 0.094ms \n",
      "algo_115 costs 0.094ms \n",
      "fast_algo 102 costs 0.094 ms\n",
      "***cublasLt Gemm Testing Beign***\n",
      "AlgoCount: 5000\n",
      "return 6 and run heuristic algo\n",
      "workspacesize==0, run 245 algos\n",
      "result 000 : algo={ Id=6, tileIdx=18 (128x64) splitK=0 reduc=0 swizzle=0 custom=0 stages=12} status 0 time 0.092488ms workspace=0 mathMode=0 waves=0.592593\n",
      "result 001 : algo={ Id=6, tileIdx=18 (128x64) splitK=0 reduc=0 swizzle=1 custom=0 stages=12} status 0 time 0.092641ms workspace=0 mathMode=0 waves=0.592593\n",
      "result 002 : algo={ Id=21, tileIdx=21 (256x64) splitK=1 reduc=0 swizzle=0 custom=0 stages=10} status 0 time 0.092979ms workspace=0 mathMode=0 waves=0.296296\n",
      "result 003 : algo={ Id=6, tileIdx=21 (256x64) splitK=1 reduc=0 swizzle=1 custom=0 stages=15} status 0 time 0.093972ms workspace=0 mathMode=0 waves=0.592593\n",
      "result 004 : algo={ Id=6, tileIdx=18 (128x64) splitK=0 reduc=0 swizzle=1 custom=0 stages=15} status 0 time 0.096492ms workspace=0 mathMode=0 waves=0.592593\n",
      "result 005 : algo={ Id=6, tileIdx=18 (128x64) splitK=0 reduc=0 swizzle=0 custom=0 stages=15} status 0 time 0.096512ms workspace=0 mathMode=0 waves=0.592593\n",
      "result 006 : algo={ Id=5, tileIdx=21 (256x64) splitK=5 reduc=2 swizzle=1 custom=0 stages=7} status 0 time 0.099707ms workspace=327680 mathMode=0 waves=1.481481\n",
      "result 007 : algo={ Id=5, tileIdx=21 (256x64) splitK=5 reduc=2 swizzle=0 custom=0 stages=7} status 0 time 0.099860ms workspace=327680 mathMode=0 waves=1.481481\n",
      "result 008 : algo={ Id=5, tileIdx=21 (256x64) splitK=14 reduc=1 swizzle=1 custom=0 stages=0} status 0 time 0.103772ms workspace=256 mathMode=0 waves=3.851852\n",
      "result 009 : algo={ Id=21, tileIdx=18 (128x64) splitK=7 reduc=4 swizzle=0 custom=0 stages=16} status 0 time 0.105851ms workspace=229376 mathMode=0 waves=8.296296\n",
      "result 010 : algo={ Id=6, tileIdx=20 (128x128) splitK=0 reduc=0 swizzle=0 custom=0 stages=11} status 0 time 0.113439ms workspace=0 mathMode=0 waves=0.592593\n",
      "result 011 : algo={ Id=6, tileIdx=20 (128x128) splitK=0 reduc=0 swizzle=1 custom=0 stages=11} status 0 time 0.113623ms workspace=0 mathMode=0 waves=0.592593\n",
      "result 012 : algo={ Id=5, tileIdx=21 (256x64) splitK=0 reduc=0 swizzle=0 custom=0 stages=7} status 0 time 0.119337ms workspace=0 mathMode=0 waves=0.296296\n",
      "result 013 : algo={ Id=5, tileIdx=21 (256x64) splitK=0 reduc=0 swizzle=1 custom=0 stages=7} status 0 time 0.119521ms workspace=0 mathMode=0 waves=0.296296\n",
      "result 014 : algo={ Id=6, tileIdx=20 (128x128) splitK=0 reduc=0 swizzle=1 custom=0 stages=15} status 0 time 0.126116ms workspace=0 mathMode=0 waves=1.185185\n",
      "result 015 : algo={ Id=6, tileIdx=20 (128x128) splitK=0 reduc=0 swizzle=0 custom=0 stages=15} status 0 time 0.126177ms workspace=0 mathMode=0 waves=1.185185\n",
      "result 016 : algo={ Id=5, tileIdx=24 (256x128) splitK=0 reduc=0 swizzle=0 custom=0 stages=0} status 0 time 0.128942ms workspace=0 mathMode=0 waves=0.592593\n",
      "result 017 : algo={ Id=5, tileIdx=24 (256x128) splitK=0 reduc=0 swizzle=1 custom=0 stages=0} status 0 time 0.128993ms workspace=0 mathMode=0 waves=0.592593\n",
      "result 018 : algo={ Id=5, tileIdx=20 (128x128) splitK=0 reduc=0 swizzle=0 custom=0 stages=0} status 0 time 0.132710ms workspace=0 mathMode=0 waves=0.592593\n",
      "result 019 : algo={ Id=5, tileIdx=20 (128x128) splitK=0 reduc=0 swizzle=1 custom=0 stages=0} status 0 time 0.132741ms workspace=0 mathMode=0 waves=0.592593\n",
      "result 020 : algo={ Id=5, tileIdx=20 (128x128) splitK=0 reduc=0 swizzle=1 custom=0 stages=7} status 0 time 0.138301ms workspace=0 mathMode=0 waves=0.592593\n",
      "result 021 : algo={ Id=5, tileIdx=20 (128x128) splitK=0 reduc=0 swizzle=0 custom=0 stages=7} status 0 time 0.138394ms workspace=0 mathMode=0 waves=0.592593\n",
      "result 022 : algo={ Id=6, tileIdx=18 (128x64) splitK=0 reduc=0 swizzle=1 custom=0 stages=0} status 0 time 0.146350ms workspace=0 mathMode=0 waves=0.296296\n",
      "result 023 : algo={ Id=6, tileIdx=18 (128x64) splitK=0 reduc=0 swizzle=0 custom=0 stages=0} status 0 time 0.146371ms workspace=0 mathMode=0 waves=0.296296\n",
      "result 024 : algo={ Id=5, tileIdx=21 (256x64) splitK=0 reduc=0 swizzle=0 custom=0 stages=0} status 0 time 0.151050ms workspace=0 mathMode=0 waves=0.296296\n",
      "result 025 : algo={ Id=5, tileIdx=21 (256x64) splitK=0 reduc=0 swizzle=1 custom=0 stages=0} status 0 time 0.151132ms workspace=0 mathMode=0 waves=0.296296\n",
      "result 026 : algo={ Id=6, tileIdx=23 (128x256) splitK=0 reduc=0 swizzle=0 custom=0 stages=15} status 0 time 0.203776ms workspace=0 mathMode=0 waves=1.185185\n",
      "result 027 : algo={ Id=6, tileIdx=23 (128x256) splitK=0 reduc=0 swizzle=1 custom=0 stages=15} status 0 time 0.203817ms workspace=0 mathMode=0 waves=1.185185\n",
      "result 028 : algo={ Id=6, tileIdx=23 (128x256) splitK=0 reduc=0 swizzle=0 custom=0 stages=0} status 0 time 0.219597ms workspace=0 mathMode=0 waves=1.185185\n",
      "result 029 : algo={ Id=6, tileIdx=23 (128x256) splitK=0 reduc=0 swizzle=1 custom=0 stages=0} status 0 time 0.219668ms workspace=0 mathMode=0 waves=1.185185\n",
      "result 030 : algo={ Id=5, tileIdx=23 (128x256) splitK=0 reduc=0 swizzle=1 custom=0 stages=0} status 0 time 0.248678ms workspace=0 mathMode=0 waves=1.185185\n",
      "result 031 : algo={ Id=5, tileIdx=23 (128x256) splitK=0 reduc=0 swizzle=0 custom=0 stages=0} status 0 time 0.248678ms workspace=0 mathMode=0 waves=1.185185\n",
      "result 032 : algo={ Id=5, tileIdx=23 (128x256) splitK=0 reduc=0 swizzle=1 custom=0 stages=7} status 0 time 0.250900ms workspace=0 mathMode=0 waves=1.185185\n",
      "result 033 : algo={ Id=5, tileIdx=23 (128x256) splitK=0 reduc=0 swizzle=0 custom=0 stages=7} status 0 time 0.250921ms workspace=0 mathMode=0 waves=1.185185\n",
      "result 034 : algo={ Id=1, tileIdx=11 (32x32) splitK=0 reduc=0 swizzle=1 custom=0 stages=0} status 0 time 0.312361ms workspace=0 mathMode=0 waves=1.185185\n",
      "result 035 : algo={ Id=1, tileIdx=11 (32x32) splitK=0 reduc=0 swizzle=0 custom=0 stages=0} status 0 time 0.312586ms workspace=0 mathMode=0 waves=1.185185\n",
      "result 036 : algo={ Id=1, tileIdx=13 (64x32) splitK=0 reduc=0 swizzle=1 custom=0 stages=0} status 0 time 0.327076ms workspace=0 mathMode=0 waves=1.185185\n",
      "result 037 : algo={ Id=1, tileIdx=13 (64x32) splitK=0 reduc=0 swizzle=0 custom=0 stages=0} status 0 time 0.327096ms workspace=0 mathMode=0 waves=1.185185\n",
      "result 038 : algo={ Id=1, tileIdx=16 (128x32) splitK=0 reduc=0 swizzle=1 custom=0 stages=0} status 0 time 0.405002ms workspace=0 mathMode=0 waves=1.185185\n",
      "result 039 : algo={ Id=1, tileIdx=16 (128x32) splitK=0 reduc=0 swizzle=0 custom=0 stages=0} status 0 time 0.405023ms workspace=0 mathMode=0 waves=1.185185\n",
      "result 040 : algo={ Id=0, tileIdx=16 (128x32) splitK=0 reduc=0 swizzle=0 custom=0 stages=0} status 0 time 0.439839ms workspace=0 mathMode=0 waves=0.296296\n",
      "result 041 : algo={ Id=0, tileIdx=16 (128x32) splitK=0 reduc=0 swizzle=1 custom=0 stages=0} status 0 time 0.439849ms workspace=0 mathMode=0 waves=0.296296\n",
      "result 042 : algo={ Id=0, tileIdx=18 (128x64) splitK=0 reduc=0 swizzle=1 custom=0 stages=0} status 0 time 0.792402ms workspace=0 mathMode=0 waves=0.296296\n",
      "result 043 : algo={ Id=0, tileIdx=18 (128x64) splitK=0 reduc=0 swizzle=0 custom=0 stages=0} status 0 time 0.792474ms workspace=0 mathMode=0 waves=0.296296\n",
      "result 044 : algo={ Id=0, tileIdx=15 (64x64) splitK=0 reduc=0 swizzle=0 custom=0 stages=0} status 0 time 0.803482ms workspace=0 mathMode=0 waves=0.296296\n",
      "result 045 : algo={ Id=0, tileIdx=15 (64x64) splitK=0 reduc=0 swizzle=1 custom=0 stages=0} status 0 time 0.803656ms workspace=0 mathMode=0 waves=0.296296\n",
      "result 046 : algo={ Id=0, tileIdx=14 (32x128) splitK=0 reduc=0 swizzle=1 custom=0 stages=0} status 0 time 1.054894ms workspace=0 mathMode=0 waves=1.185185\n",
      "result 047 : algo={ Id=0, tileIdx=14 (32x128) splitK=0 reduc=0 swizzle=0 custom=0 stages=0} status 0 time 1.055529ms workspace=0 mathMode=0 waves=1.185185\n",
      "result 048 : algo={ Id=0, tileIdx=20 (128x128) splitK=0 reduc=0 swizzle=0 custom=0 stages=0} status 0 time 1.517722ms workspace=0 mathMode=0 waves=0.592593\n",
      "result 049 : algo={ Id=0, tileIdx=20 (128x128) splitK=0 reduc=0 swizzle=1 custom=0 stages=0} status 0 time 1.518162ms workspace=0 mathMode=0 waves=0.592593\n",
      "algo={ Id=6, tileIdx=18 (128x64) splitK=0 reduc=0 swizzle=0 custom=0 stages=12} status 0 time 0.092488ms workspace=0 mathMode=0 waves=0.592593\n",
      "***cublasLt Gemm Testing End***\n",
      "\n",
      "-----------------------------\n",
      "GEMM test 9: [M: 1, K: 16384, N: 4096] ffn gemm 2\n",
      "algo_99 costs 0.097ms \n",
      "algo_100 costs 0.097ms \n",
      "algo_101 costs 0.097ms \n",
      "algo_102 costs 0.097ms \n",
      "algo_103 costs 0.097ms \n",
      "algo_104 costs 0.097ms \n",
      "algo_105 costs 0.097ms \n",
      "algo_106 costs 0.097ms \n",
      "algo_107 costs 0.097ms \n",
      "algo_108 costs 0.097ms \n",
      "algo_109 costs 0.097ms \n",
      "algo_110 costs 0.097ms \n",
      "algo_111 costs 0.097ms \n",
      "algo_112 costs 0.097ms \n",
      "algo_113 costs 0.096ms \n",
      "algo_114 costs 0.097ms \n",
      "algo_115 costs 0.097ms \n",
      "fast_algo 113 costs 0.096 ms\n",
      "***cublasLt Gemm Testing Beign***\n",
      "AlgoCount: 5000\n",
      "return 8 and run heuristic algo\n",
      "workspacesize==0, run 245 algos\n",
      "result 000 : algo={ Id=31, tileIdx=15 (64x64) splitK=1 reduc=0 swizzle=0 custom=0 stages=17} status 0 time 0.089784ms workspace=0 mathMode=0 waves=0.296296\n",
      "result 001 : algo={ Id=6, tileIdx=18 (128x64) splitK=3 reduc=4 swizzle=0 custom=0 stages=16} status 0 time 0.091894ms workspace=24576 mathMode=0 waves=0.888889\n",
      "result 002 : algo={ Id=6, tileIdx=15 (64x64) splitK=1 reduc=0 swizzle=1 custom=0 stages=18} status 0 time 0.093112ms workspace=0 mathMode=0 waves=0.592593\n",
      "result 003 : algo={ Id=6, tileIdx=18 (128x64) splitK=4 reduc=4 swizzle=1 custom=0 stages=12} status 0 time 0.094587ms workspace=32768 mathMode=0 waves=0.592593\n",
      "result 004 : algo={ Id=21, tileIdx=21 (256x64) splitK=4 reduc=4 swizzle=0 custom=0 stages=10} status 0 time 0.095048ms workspace=32768 mathMode=0 waves=0.296296\n",
      "result 005 : algo={ Id=6, tileIdx=18 (128x64) splitK=2 reduc=1 swizzle=1 custom=0 stages=12} status 0 time 0.096410ms workspace=128 mathMode=0 waves=0.296296\n",
      "result 006 : algo={ Id=6, tileIdx=18 (128x64) splitK=7 reduc=1 swizzle=0 custom=0 stages=16} status 0 time 0.098693ms workspace=128 mathMode=0 waves=2.074074\n",
      "result 007 : algo={ Id=6, tileIdx=18 (128x64) splitK=0 reduc=0 swizzle=1 custom=0 stages=15} status 0 time 0.126147ms workspace=0 mathMode=0 waves=0.148148\n",
      "result 008 : algo={ Id=6, tileIdx=18 (128x64) splitK=0 reduc=0 swizzle=0 custom=0 stages=15} status 0 time 0.126249ms workspace=0 mathMode=0 waves=0.148148\n",
      "result 009 : algo={ Id=13, tileIdx=0 (UNDEF) splitK=108 reduc=2 swizzle=0 custom=7 stages=0} status 0 time 0.146289ms workspace=1769488 mathMode=0 waves=3.200000\n",
      "result 010 : algo={ Id=6, tileIdx=20 (128x128) splitK=0 reduc=0 swizzle=0 custom=0 stages=15} status 0 time 0.198963ms workspace=0 mathMode=0 waves=0.296296\n",
      "result 011 : algo={ Id=6, tileIdx=20 (128x128) splitK=0 reduc=0 swizzle=1 custom=0 stages=15} status 0 time 0.199004ms workspace=0 mathMode=0 waves=0.296296\n",
      "result 012 : algo={ Id=6, tileIdx=20 (128x128) splitK=0 reduc=0 swizzle=1 custom=0 stages=11} status 0 time 0.199967ms workspace=0 mathMode=0 waves=0.148148\n",
      "result 013 : algo={ Id=6, tileIdx=20 (128x128) splitK=0 reduc=0 swizzle=0 custom=0 stages=11} status 0 time 0.200038ms workspace=0 mathMode=0 waves=0.148148\n",
      "result 014 : algo={ Id=6, tileIdx=23 (128x256) splitK=0 reduc=0 swizzle=0 custom=0 stages=15} status 0 time 0.388301ms workspace=0 mathMode=0 waves=0.296296\n",
      "result 015 : algo={ Id=6, tileIdx=23 (128x256) splitK=0 reduc=0 swizzle=1 custom=0 stages=15} status 0 time 0.388424ms workspace=0 mathMode=0 waves=0.296296\n",
      "result 016 : algo={ Id=5, tileIdx=20 (128x128) splitK=0 reduc=0 swizzle=1 custom=0 stages=0} status 0 time 0.388905ms workspace=0 mathMode=0 waves=0.148148\n",
      "result 017 : algo={ Id=5, tileIdx=20 (128x128) splitK=0 reduc=0 swizzle=0 custom=0 stages=0} status 0 time 0.389018ms workspace=0 mathMode=0 waves=0.148148\n",
      "result 018 : algo={ Id=6, tileIdx=23 (128x256) splitK=0 reduc=0 swizzle=1 custom=0 stages=0} status 0 time 0.409887ms workspace=0 mathMode=0 waves=0.296296\n",
      "result 019 : algo={ Id=6, tileIdx=23 (128x256) splitK=0 reduc=0 swizzle=0 custom=0 stages=0} status 0 time 0.409938ms workspace=0 mathMode=0 waves=0.296296\n",
      "result 020 : algo={ Id=5, tileIdx=20 (128x128) splitK=0 reduc=0 swizzle=0 custom=0 stages=7} status 0 time 0.414413ms workspace=0 mathMode=0 waves=0.148148\n",
      "result 021 : algo={ Id=5, tileIdx=20 (128x128) splitK=0 reduc=0 swizzle=1 custom=0 stages=7} status 0 time 0.414423ms workspace=0 mathMode=0 waves=0.148148\n",
      "result 022 : algo={ Id=5, tileIdx=21 (256x64) splitK=0 reduc=0 swizzle=1 custom=0 stages=7} status 0 time 0.425564ms workspace=0 mathMode=0 waves=0.074074\n",
      "result 023 : algo={ Id=5, tileIdx=21 (256x64) splitK=0 reduc=0 swizzle=0 custom=0 stages=7} status 0 time 0.425636ms workspace=0 mathMode=0 waves=0.074074\n",
      "result 024 : algo={ Id=6, tileIdx=18 (128x64) splitK=0 reduc=0 swizzle=0 custom=0 stages=0} status 0 time 0.456448ms workspace=0 mathMode=0 waves=0.074074\n",
      "result 025 : algo={ Id=6, tileIdx=18 (128x64) splitK=0 reduc=0 swizzle=1 custom=0 stages=0} status 0 time 0.456571ms workspace=0 mathMode=0 waves=0.074074\n",
      "result 026 : algo={ Id=5, tileIdx=23 (128x256) splitK=0 reduc=0 swizzle=1 custom=0 stages=0} status 0 time 0.479406ms workspace=0 mathMode=0 waves=0.296296\n",
      "result 027 : algo={ Id=5, tileIdx=23 (128x256) splitK=0 reduc=0 swizzle=0 custom=0 stages=0} status 0 time 0.479519ms workspace=0 mathMode=0 waves=0.296296\n",
      "result 028 : algo={ Id=5, tileIdx=23 (128x256) splitK=0 reduc=0 swizzle=1 custom=0 stages=7} status 0 time 0.483164ms workspace=0 mathMode=0 waves=0.296296\n",
      "result 029 : algo={ Id=5, tileIdx=23 (128x256) splitK=0 reduc=0 swizzle=0 custom=0 stages=7} status 0 time 0.483318ms workspace=0 mathMode=0 waves=0.296296\n",
      "result 030 : algo={ Id=5, tileIdx=24 (256x128) splitK=0 reduc=0 swizzle=0 custom=0 stages=0} status 0 time 0.484628ms workspace=0 mathMode=0 waves=0.148148\n",
      "result 031 : algo={ Id=5, tileIdx=24 (256x128) splitK=0 reduc=0 swizzle=1 custom=0 stages=0} status 0 time 0.484690ms workspace=0 mathMode=0 waves=0.148148\n",
      "result 032 : algo={ Id=1, tileIdx=13 (64x32) splitK=0 reduc=0 swizzle=1 custom=0 stages=0} status 0 time 0.524421ms workspace=0 mathMode=0 waves=0.296296\n",
      "result 033 : algo={ Id=1, tileIdx=13 (64x32) splitK=0 reduc=0 swizzle=0 custom=0 stages=0} status 0 time 0.524677ms workspace=0 mathMode=0 waves=0.296296\n",
      "result 034 : algo={ Id=1, tileIdx=11 (32x32) splitK=0 reduc=0 swizzle=1 custom=0 stages=0} status 0 time 0.533064ms workspace=0 mathMode=0 waves=0.296296\n",
      "result 035 : algo={ Id=1, tileIdx=11 (32x32) splitK=0 reduc=0 swizzle=0 custom=0 stages=0} status 0 time 0.534682ms workspace=0 mathMode=0 waves=0.296296\n",
      "result 036 : algo={ Id=5, tileIdx=21 (256x64) splitK=0 reduc=0 swizzle=0 custom=0 stages=0} status 0 time 0.558715ms workspace=0 mathMode=0 waves=0.074074\n",
      "result 037 : algo={ Id=5, tileIdx=21 (256x64) splitK=0 reduc=0 swizzle=1 custom=0 stages=0} status 0 time 0.558899ms workspace=0 mathMode=0 waves=0.074074\n",
      "result 038 : algo={ Id=1, tileIdx=16 (128x32) splitK=0 reduc=0 swizzle=1 custom=0 stages=0} status 0 time 0.793651ms workspace=0 mathMode=0 waves=0.296296\n",
      "result 039 : algo={ Id=1, tileIdx=16 (128x32) splitK=0 reduc=0 swizzle=0 custom=0 stages=0} status 0 time 0.793825ms workspace=0 mathMode=0 waves=0.296296\n",
      "result 040 : algo={ Id=0, tileIdx=16 (128x32) splitK=0 reduc=0 swizzle=0 custom=0 stages=0} status 0 time 1.493371ms workspace=0 mathMode=0 waves=0.074074\n",
      "result 041 : algo={ Id=0, tileIdx=16 (128x32) splitK=0 reduc=0 swizzle=1 custom=0 stages=0} status 0 time 1.493381ms workspace=0 mathMode=0 waves=0.074074\n",
      "result 042 : algo={ Id=0, tileIdx=14 (32x128) splitK=0 reduc=0 swizzle=1 custom=0 stages=0} status 0 time 1.721323ms workspace=0 mathMode=0 waves=0.296296\n",
      "result 043 : algo={ Id=0, tileIdx=14 (32x128) splitK=0 reduc=0 swizzle=0 custom=0 stages=0} status 0 time 1.721395ms workspace=0 mathMode=0 waves=0.296296\n",
      "result 044 : algo={ Id=0, tileIdx=18 (128x64) splitK=0 reduc=0 swizzle=0 custom=0 stages=0} status 0 time 1.840281ms workspace=0 mathMode=0 waves=0.074074\n",
      "result 045 : algo={ Id=0, tileIdx=18 (128x64) splitK=0 reduc=0 swizzle=1 custom=0 stages=0} status 0 time 1.840364ms workspace=0 mathMode=0 waves=0.074074\n",
      "result 046 : algo={ Id=0, tileIdx=15 (64x64) splitK=0 reduc=0 swizzle=0 custom=0 stages=0} status 0 time 1.873858ms workspace=0 mathMode=0 waves=0.074074\n",
      "result 047 : algo={ Id=0, tileIdx=15 (64x64) splitK=0 reduc=0 swizzle=1 custom=0 stages=0} status 0 time 1.883320ms workspace=0 mathMode=0 waves=0.074074\n",
      "result 048 : algo={ Id=0, tileIdx=20 (128x128) splitK=0 reduc=0 swizzle=0 custom=0 stages=0} status 0 time 3.022633ms workspace=0 mathMode=0 waves=0.148148\n",
      "result 049 : algo={ Id=0, tileIdx=20 (128x128) splitK=0 reduc=0 swizzle=1 custom=0 stages=0} status 0 time 3.023718ms workspace=0 mathMode=0 waves=0.148148\n",
      "algo={ Id=31, tileIdx=15 (64x64) splitK=1 reduc=0 swizzle=0 custom=0 stages=17} status 0 time 0.089784ms workspace=0 mathMode=0 waves=0.296296\n",
      "***cublasLt Gemm Testing End***\n",
      "\n",
      "-----------------------------\n",
      "GEMM test 10: [M: 1, K: 4096, N: 50256] logits gemm\n",
      "algo_99 costs 0.257ms \n",
      "algo_100 costs 0.257ms \n",
      "algo_101 costs 0.257ms \n",
      "algo_102 costs 0.257ms \n",
      "algo_103 costs 0.257ms \n",
      "algo_104 costs 0.258ms \n",
      "algo_105 costs 0.258ms \n",
      "algo_106 costs 0.257ms \n",
      "algo_107 costs 0.258ms \n",
      "algo_108 costs 0.258ms \n",
      "algo_109 costs 0.258ms \n",
      "algo_110 costs 0.257ms \n",
      "algo_111 costs 0.257ms \n",
      "algo_112 costs 0.258ms \n",
      "algo_113 costs 0.258ms \n",
      "algo_114 costs 0.258ms \n",
      "algo_115 costs 0.257ms \n",
      "fast_algo 100 costs 0.257 ms\n",
      "***cublas Gemm Testing End***\n",
      "\n",
      "***GPT Gemm Testing End***\n"
     ]
    }
   ],
   "source": [
    "!CUDA_VISIBLE_DEVICES=1  ./bin/gpt_gemm 1 1 128 16 256 16384 50256 1 1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As a result of kernel autotuning, we received a `gemm_config.in` file in our directory that contains information about algorithms which were tested and the execution time for each algorithm.  This configuration file will be used later by the FT library in any of our inference examples whether through bindings or C++ code."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 3.6 Step 6: GPT-J inference using C++ bindings "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3.6.1 Inference on 1 GPU "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Before we start, we have to look one more time at the configuration produced earlier: `./FasterTransformer/examples/cpp/gptj/gptj_config.ini`. \n",
    "\n",
    "It contains information about our GPT-J model like the number of GPT decoder layers, number of heads, number of hidden neurons in MLP layers, etc. At the same time, it contains hyperparameters for our inference like the batch size, search temperature, top_k, top_p, beam_search parameters, as well as the precision mode (e.g. FP16 or FP32)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Please modify the [gptj_config.ini](FasterTransformer/examples/cpp/gptj/gptj_config.ini) to make inference execution consistent to the inference pipeline we have used in the previous notebook:</br>\n",
    "* max_batch_size=<b>1</b> \n",
    "* top_k=<b>50</b> \n",
    "* top_p=<b>1</b> \n",
    "* is_half=<b>1</b> #set to 1 to run model in FP16 mode.\n",
    "* tensor_para_size=<b>1</b> # we will run on a single GPU without Tensor Parallelism\n",
    "* model_dir=<b>../../models/j6b_ckpt/</b> # Path to model weights\n",
    "* request_batch_size=<b>1</b>\n",
    "* request_output_len=<b>120</b> # Output length\n",
    "\n",
    "All other parameters should be left unmodified for a fair comparison."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's run the pre-built script `./FasterTransformer/build/bin/gptj_example`. It was built by us from C++ file `./FasterTransformer/examples/cpp/gptj/gptj_example.cpp`. </br>\n",
    "This file takes weights that we prepared and the config from `./FasterTransformer/examples/cpp/gptj/gptj_config.ini`. This binary file takes as input tokens from `./FasterTransformer/examples/cpp/gptj/start_ids.csv` as a prompt and generates an `out` text file that contains tokens generated in the process of work of GPT-J."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`CUDA_VISIBLE_DEVICES` environment variable allows you to choose what GPU will be used for current task/process/script. We have 4 GPUs on our system (0-3 numerations starts with 0) and we will use this variable to run current example onto GPU 1 (2nd GPU). We will use GPU 1 because previous task (Notebook 02) was deployed to GPU 0 and model is already there if you have not stopped the Jupyter kernel."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Total ranks: 1.\n",
      "Device Graphics Device\n",
      "P0 is runing with 0 GPU.\n",
      "after allocation, free 67.23 GB total 79.35 GB\n",
      "Writing 128 elements\n",
      "  818   262   938  3155   286  1528    11   257 10089   517 \n",
      "zeroCount = 0\n",
      "[INFO] request_batch_size 1 beam_width 1 head_num 16 size_per_head 256 total_output_len 128 decoder_layers 28 vocab_size 50400 FT-CPP-decoding-beamsearch-time 1170.47 ms\n"
     ]
    }
   ],
   "source": [
    "!CUDA_VISIBLE_DEVICES=1 ./bin/gptj_example"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We see that the inference time of the sentence with the same length is <b>1.8 seconds</b>, which is <b>x3.5</b> times faster than the default PyTorch + Transformers pipeline! Please note that since we are using different GPUs between classes, your performance can be different than the above.</br>\n",
    "Let's decode the `out` file and see what was generated by our neural network."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[INFO] batch 0: In the last couple of days, a handful more cases have been added to this list. Each time (or sometimes several times in one year) is usually seen as an \"uptick.\" In other words: while most people don't go nuts with buying things out right away when there's such high demand and many who would be affected by that price hike want no part at all - we can easily say over 10% or 20%. This has certainly never happened before for so long on any single item without also having prices falling instead because they're just too expensive anyways...\n",
      "When i first posted about it nearly 3 months ago.. I actually\n",
      "[[  818   262   938  3155   286  1528    11   257 10089   517  2663   423\n",
      "    587  2087   284   428  1351    13  5501   640   357   273  3360  1811\n",
      "   1661   287   530   614     8   318  3221  1775   355   281   366 37623\n",
      "    624   526   554   584  2456    25   981   749   661   836   470   467\n",
      "  14380   351  7067  1243   503   826  1497   618   612   338   884  1029\n",
      "   3512   290   867   508   561   307  5676   416   326  2756 18390   765\n",
      "    645   636   379   477   532   356   460  3538   910   625   838     4\n",
      "    393  1160  7225   770   468  3729  1239  3022   878   329   523   890\n",
      "    319   597  2060  2378  1231   635  1719  4536  7463  2427   780   484\n",
      "    821   655  1165  5789 32845   986   198  2215  1312   717  4481   546\n",
      "    340  3016   513  1933  2084   492   314  1682]]\n"
     ]
    }
   ],
   "source": [
    "!python3 ../examples/pytorch/gpt/utils/gpt_token_converter.py \\\n",
    "                       --vocab_file=../../models/gpt2-vocab.json  \\\n",
    "                       --bpe_file=../../models/gpt2-merges.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Users can see the details of arguments in `examples/cpp/multi_gpu_gpt/gpt_config.ini`. It controls the model path, model size, tensor parallelism size, and some hyper-parameters."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3.6.2 Inference on 2 GPUs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can run the same inference step but almost two times faster using Tensor-Parallelism."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let us run the inference of the GPT-J in Tensor-Parallel=2 mode. This means that we must split our weights into 2 parts and place them onto multiple GPUs connected with NVLink. FasterTransformer hides the complexity of model distribution, and we can enable this functionality by setting the appropriate export arguments. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For the weights preparation, we will use the same script, but set up `--n-inference-gpus` to `2` to prepare weights for TP=2 mode. Because of the size of the files involved, this step will take a couple of minutes. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "scrolled": true,
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "loading\n",
      "loading shards for part 0\n",
      "read from checkpoint\n",
      "< (8, 4096) to (4096,)\n",
      "> transformer.wte.bias torch.Size([4096])\n",
      "< (8, 6300, 4096) to (1, 50400, 4096)\n",
      "> transformer.wte.weight torch.Size([4096, 50400])\n",
      "< (8, 4096, 512) to (1, 4096, 4096)\n",
      "WARNING:absl:No GPU/TPU found, falling back to CPU. (Set TF_CPP_MIN_LOG_LEVEL=0 and rerun for more info.)\n",
      "> transformer.h.0.attn.attention.q_proj.weight torch.Size([4096, 4096])\n",
      "< (8, 4096, 512) to (1, 4096, 4096)\n",
      "> transformer.h.0.attn.attention.v_proj.weight torch.Size([4096, 4096])\n",
      "< (8, 4096, 512) to (1, 4096, 4096)\n",
      "> transformer.h.0.attn.attention.k_proj.weight torch.Size([4096, 4096])\n",
      "< (8, 512, 4096) to (1, 4096, 4096)\n",
      "> transformer.h.0.attn.attention.out_proj.weight torch.Size([4096, 4096])\n",
      "< (8, 2048) to (1, 16384)\n",
      "> transformer.h.0.mlp.c_fc.bias torch.Size([16384])\n",
      "< (8, 4096, 2048) to (1, 4096, 16384)\n",
      "> transformer.h.0.mlp.c_fc.weight torch.Size([16384, 4096])\n",
      "< (8, 4096) to (4096,)\n",
      "> transformer.h.0.mlp.c_proj.bias torch.Size([4096])\n",
      "< (8, 2048, 4096) to (1, 16384, 4096)\n",
      "> transformer.h.0.mlp.c_proj.weight torch.Size([4096, 16384])\n",
      "< (8, 4096) to (4096,)\n",
      "> transformer.h.0.ln_1.bias torch.Size([4096])\n",
      "< (8, 4096) to (4096,)\n",
      "> transformer.h.0.ln_1.weight torch.Size([4096])\n",
      "< (8, 4096, 512) to (1, 4096, 4096)\n",
      "> transformer.h.1.attn.attention.q_proj.weight torch.Size([4096, 4096])\n",
      "< (8, 4096, 512) to (1, 4096, 4096)\n",
      "> transformer.h.1.attn.attention.v_proj.weight torch.Size([4096, 4096])\n",
      "< (8, 4096, 512) to (1, 4096, 4096)\n",
      "> transformer.h.1.attn.attention.k_proj.weight torch.Size([4096, 4096])\n",
      "< (8, 512, 4096) to (1, 4096, 4096)\n",
      "> transformer.h.1.attn.attention.out_proj.weight torch.Size([4096, 4096])\n",
      "< (8, 2048) to (1, 16384)\n",
      "> transformer.h.1.mlp.c_fc.bias torch.Size([16384])\n",
      "< (8, 4096, 2048) to (1, 4096, 16384)\n",
      "> transformer.h.1.mlp.c_fc.weight torch.Size([16384, 4096])\n",
      "loading shards for part 1\n",
      "read from checkpoint\n",
      "< (8, 4096) to (4096,)\n",
      "> transformer.h.1.mlp.c_proj.bias torch.Size([4096])\n",
      "< (8, 2048, 4096) to (1, 16384, 4096)\n",
      "> transformer.h.1.mlp.c_proj.weight torch.Size([4096, 16384])\n",
      "< (8, 4096) to (4096,)\n",
      "> transformer.h.1.ln_1.bias torch.Size([4096])\n",
      "< (8, 4096) to (4096,)\n",
      "> transformer.h.1.ln_1.weight torch.Size([4096])\n",
      "< (8, 4096, 512) to (1, 4096, 4096)\n",
      "> transformer.h.10.attn.attention.q_proj.weight torch.Size([4096, 4096])\n",
      "< (8, 4096, 512) to (1, 4096, 4096)\n",
      "> transformer.h.10.attn.attention.v_proj.weight torch.Size([4096, 4096])\n",
      "< (8, 4096, 512) to (1, 4096, 4096)\n",
      "> transformer.h.10.attn.attention.k_proj.weight torch.Size([4096, 4096])\n",
      "< (8, 512, 4096) to (1, 4096, 4096)\n",
      "> transformer.h.10.attn.attention.out_proj.weight torch.Size([4096, 4096])\n",
      "< (8, 2048) to (1, 16384)\n",
      "> transformer.h.10.mlp.c_fc.bias torch.Size([16384])\n",
      "< (8, 4096, 2048) to (1, 4096, 16384)\n",
      "> transformer.h.10.mlp.c_fc.weight torch.Size([16384, 4096])\n",
      "< (8, 4096) to (4096,)\n",
      "> transformer.h.10.mlp.c_proj.bias torch.Size([4096])\n",
      "< (8, 2048, 4096) to (1, 16384, 4096)\n",
      "> transformer.h.10.mlp.c_proj.weight torch.Size([4096, 16384])\n",
      "< (8, 4096) to (4096,)\n",
      "> transformer.h.10.ln_1.bias torch.Size([4096])\n",
      "< (8, 4096) to (4096,)\n",
      "> transformer.h.10.ln_1.weight torch.Size([4096])\n",
      "< (8, 4096, 512) to (1, 4096, 4096)\n",
      "> transformer.h.11.attn.attention.q_proj.weight torch.Size([4096, 4096])\n",
      "< (8, 4096, 512) to (1, 4096, 4096)\n",
      "> transformer.h.11.attn.attention.v_proj.weight torch.Size([4096, 4096])\n",
      "< (8, 4096, 512) to (1, 4096, 4096)\n",
      "> transformer.h.11.attn.attention.k_proj.weight torch.Size([4096, 4096])\n",
      "< (8, 512, 4096) to (1, 4096, 4096)\n",
      "> transformer.h.11.attn.attention.out_proj.weight torch.Size([4096, 4096])\n",
      "loading shards for part 2\n",
      "read from checkpoint\n",
      "< (8, 2048) to (1, 16384)\n",
      "> transformer.h.11.mlp.c_fc.bias torch.Size([16384])\n",
      "< (8, 4096, 2048) to (1, 4096, 16384)\n",
      "> transformer.h.11.mlp.c_fc.weight torch.Size([16384, 4096])\n",
      "< (8, 4096) to (4096,)\n",
      "> transformer.h.11.mlp.c_proj.bias torch.Size([4096])\n",
      "< (8, 2048, 4096) to (1, 16384, 4096)\n",
      "> transformer.h.11.mlp.c_proj.weight torch.Size([4096, 16384])\n",
      "< (8, 4096) to (4096,)\n",
      "> transformer.h.11.ln_1.bias torch.Size([4096])\n",
      "< (8, 4096) to (4096,)\n",
      "> transformer.h.11.ln_1.weight torch.Size([4096])\n",
      "< (8, 4096, 512) to (1, 4096, 4096)\n",
      "> transformer.h.12.attn.attention.q_proj.weight torch.Size([4096, 4096])\n",
      "< (8, 4096, 512) to (1, 4096, 4096)\n",
      "> transformer.h.12.attn.attention.v_proj.weight torch.Size([4096, 4096])\n",
      "< (8, 4096, 512) to (1, 4096, 4096)\n",
      "> transformer.h.12.attn.attention.k_proj.weight torch.Size([4096, 4096])\n",
      "< (8, 512, 4096) to (1, 4096, 4096)\n",
      "> transformer.h.12.attn.attention.out_proj.weight torch.Size([4096, 4096])\n",
      "< (8, 2048) to (1, 16384)\n",
      "> transformer.h.12.mlp.c_fc.bias torch.Size([16384])\n",
      "< (8, 4096, 2048) to (1, 4096, 16384)\n",
      "> transformer.h.12.mlp.c_fc.weight torch.Size([16384, 4096])\n",
      "< (8, 4096) to (4096,)\n",
      "> transformer.h.12.mlp.c_proj.bias torch.Size([4096])\n",
      "< (8, 2048, 4096) to (1, 16384, 4096)\n",
      "> transformer.h.12.mlp.c_proj.weight torch.Size([4096, 16384])\n",
      "< (8, 4096) to (4096,)\n",
      "> transformer.h.12.ln_1.bias torch.Size([4096])\n",
      "< (8, 4096) to (4096,)\n",
      "> transformer.h.12.ln_1.weight torch.Size([4096])\n",
      "< (8, 4096, 512) to (1, 4096, 4096)\n",
      "> transformer.h.13.attn.attention.q_proj.weight torch.Size([4096, 4096])\n",
      "< (8, 4096, 512) to (1, 4096, 4096)\n",
      "> transformer.h.13.attn.attention.v_proj.weight torch.Size([4096, 4096])\n",
      "loading shards for part 3\n",
      "read from checkpoint\n",
      "< (8, 4096, 512) to (1, 4096, 4096)\n",
      "> transformer.h.13.attn.attention.k_proj.weight torch.Size([4096, 4096])\n",
      "< (8, 512, 4096) to (1, 4096, 4096)\n",
      "> transformer.h.13.attn.attention.out_proj.weight torch.Size([4096, 4096])\n",
      "< (8, 2048) to (1, 16384)\n",
      "> transformer.h.13.mlp.c_fc.bias torch.Size([16384])\n",
      "< (8, 4096, 2048) to (1, 4096, 16384)\n",
      "> transformer.h.13.mlp.c_fc.weight torch.Size([16384, 4096])\n",
      "< (8, 4096) to (4096,)\n",
      "> transformer.h.13.mlp.c_proj.bias torch.Size([4096])\n",
      "< (8, 2048, 4096) to (1, 16384, 4096)\n",
      "> transformer.h.13.mlp.c_proj.weight torch.Size([4096, 16384])\n",
      "< (8, 4096) to (4096,)\n",
      "> transformer.h.13.ln_1.bias torch.Size([4096])\n",
      "< (8, 4096) to (4096,)\n",
      "> transformer.h.13.ln_1.weight torch.Size([4096])\n",
      "< (8, 4096, 512) to (1, 4096, 4096)\n",
      "> transformer.h.14.attn.attention.q_proj.weight torch.Size([4096, 4096])\n",
      "< (8, 4096, 512) to (1, 4096, 4096)\n",
      "> transformer.h.14.attn.attention.v_proj.weight torch.Size([4096, 4096])\n",
      "< (8, 4096, 512) to (1, 4096, 4096)\n",
      "> transformer.h.14.attn.attention.k_proj.weight torch.Size([4096, 4096])\n",
      "< (8, 512, 4096) to (1, 4096, 4096)\n",
      "> transformer.h.14.attn.attention.out_proj.weight torch.Size([4096, 4096])\n",
      "< (8, 2048) to (1, 16384)\n",
      "> transformer.h.14.mlp.c_fc.bias torch.Size([16384])\n",
      "< (8, 4096, 2048) to (1, 4096, 16384)\n",
      "> transformer.h.14.mlp.c_fc.weight torch.Size([16384, 4096])\n",
      "< (8, 4096) to (4096,)\n",
      "> transformer.h.14.mlp.c_proj.bias torch.Size([4096])\n",
      "< (8, 2048, 4096) to (1, 16384, 4096)\n",
      "> transformer.h.14.mlp.c_proj.weight torch.Size([4096, 16384])\n",
      "< (8, 4096) to (4096,)\n",
      "> transformer.h.14.ln_1.bias torch.Size([4096])\n",
      "< (8, 4096) to (4096,)\n",
      "> transformer.h.14.ln_1.weight torch.Size([4096])\n",
      "loading shards for part 4\n",
      "read from checkpoint\n",
      "< (8, 4096, 512) to (1, 4096, 4096)\n",
      "> transformer.h.15.attn.attention.q_proj.weight torch.Size([4096, 4096])\n",
      "< (8, 4096, 512) to (1, 4096, 4096)\n",
      "> transformer.h.15.attn.attention.v_proj.weight torch.Size([4096, 4096])\n",
      "< (8, 4096, 512) to (1, 4096, 4096)\n",
      "> transformer.h.15.attn.attention.k_proj.weight torch.Size([4096, 4096])\n",
      "< (8, 512, 4096) to (1, 4096, 4096)\n",
      "> transformer.h.15.attn.attention.out_proj.weight torch.Size([4096, 4096])\n",
      "< (8, 2048) to (1, 16384)\n",
      "> transformer.h.15.mlp.c_fc.bias torch.Size([16384])\n",
      "< (8, 4096, 2048) to (1, 4096, 16384)\n",
      "> transformer.h.15.mlp.c_fc.weight torch.Size([16384, 4096])\n",
      "< (8, 4096) to (4096,)\n",
      "> transformer.h.15.mlp.c_proj.bias torch.Size([4096])\n",
      "< (8, 2048, 4096) to (1, 16384, 4096)\n",
      "> transformer.h.15.mlp.c_proj.weight torch.Size([4096, 16384])\n",
      "< (8, 4096) to (4096,)\n",
      "> transformer.h.15.ln_1.bias torch.Size([4096])\n",
      "< (8, 4096) to (4096,)\n",
      "> transformer.h.15.ln_1.weight torch.Size([4096])\n",
      "< (8, 4096, 512) to (1, 4096, 4096)\n",
      "> transformer.h.16.attn.attention.q_proj.weight torch.Size([4096, 4096])\n",
      "< (8, 4096, 512) to (1, 4096, 4096)\n",
      "> transformer.h.16.attn.attention.v_proj.weight torch.Size([4096, 4096])\n",
      "< (8, 4096, 512) to (1, 4096, 4096)\n",
      "> transformer.h.16.attn.attention.k_proj.weight torch.Size([4096, 4096])\n",
      "< (8, 512, 4096) to (1, 4096, 4096)\n",
      "> transformer.h.16.attn.attention.out_proj.weight torch.Size([4096, 4096])\n",
      "< (8, 2048) to (1, 16384)\n",
      "> transformer.h.16.mlp.c_fc.bias torch.Size([16384])\n",
      "< (8, 4096, 2048) to (1, 4096, 16384)\n",
      "> transformer.h.16.mlp.c_fc.weight torch.Size([16384, 4096])\n",
      "< (8, 4096) to (4096,)\n",
      "> transformer.h.16.mlp.c_proj.bias torch.Size([4096])\n",
      "< (8, 2048, 4096) to (1, 16384, 4096)\n",
      "> transformer.h.16.mlp.c_proj.weight torch.Size([4096, 16384])\n",
      "loading shards for part 5\n",
      "read from checkpoint\n",
      "< (8, 4096) to (4096,)\n",
      "> transformer.h.16.ln_1.bias torch.Size([4096])\n",
      "< (8, 4096) to (4096,)\n",
      "> transformer.h.16.ln_1.weight torch.Size([4096])\n",
      "< (8, 4096, 512) to (1, 4096, 4096)\n",
      "> transformer.h.17.attn.attention.q_proj.weight torch.Size([4096, 4096])\n",
      "< (8, 4096, 512) to (1, 4096, 4096)\n",
      "> transformer.h.17.attn.attention.v_proj.weight torch.Size([4096, 4096])\n",
      "< (8, 4096, 512) to (1, 4096, 4096)\n",
      "> transformer.h.17.attn.attention.k_proj.weight torch.Size([4096, 4096])\n",
      "< (8, 512, 4096) to (1, 4096, 4096)\n",
      "> transformer.h.17.attn.attention.out_proj.weight torch.Size([4096, 4096])\n",
      "< (8, 2048) to (1, 16384)\n",
      "> transformer.h.17.mlp.c_fc.bias torch.Size([16384])\n",
      "< (8, 4096, 2048) to (1, 4096, 16384)\n",
      "> transformer.h.17.mlp.c_fc.weight torch.Size([16384, 4096])\n",
      "< (8, 4096) to (4096,)\n",
      "> transformer.h.17.mlp.c_proj.bias torch.Size([4096])\n",
      "< (8, 2048, 4096) to (1, 16384, 4096)\n",
      "> transformer.h.17.mlp.c_proj.weight torch.Size([4096, 16384])\n",
      "< (8, 4096) to (4096,)\n",
      "> transformer.h.17.ln_1.bias torch.Size([4096])\n",
      "< (8, 4096) to (4096,)\n",
      "> transformer.h.17.ln_1.weight torch.Size([4096])\n",
      "< (8, 4096, 512) to (1, 4096, 4096)\n",
      "> transformer.h.18.attn.attention.q_proj.weight torch.Size([4096, 4096])\n",
      "< (8, 4096, 512) to (1, 4096, 4096)\n",
      "> transformer.h.18.attn.attention.v_proj.weight torch.Size([4096, 4096])\n",
      "< (8, 4096, 512) to (1, 4096, 4096)\n",
      "> transformer.h.18.attn.attention.k_proj.weight torch.Size([4096, 4096])\n",
      "< (8, 512, 4096) to (1, 4096, 4096)\n",
      "> transformer.h.18.attn.attention.out_proj.weight torch.Size([4096, 4096])\n",
      "< (8, 2048) to (1, 16384)\n",
      "> transformer.h.18.mlp.c_fc.bias torch.Size([16384])\n",
      "< (8, 4096, 2048) to (1, 4096, 16384)\n",
      "> transformer.h.18.mlp.c_fc.weight torch.Size([16384, 4096])\n",
      "loading shards for part 6\n",
      "read from checkpoint\n",
      "< (8, 4096) to (4096,)\n",
      "> transformer.h.18.mlp.c_proj.bias torch.Size([4096])\n",
      "< (8, 2048, 4096) to (1, 16384, 4096)\n",
      "> transformer.h.18.mlp.c_proj.weight torch.Size([4096, 16384])\n",
      "< (8, 4096) to (4096,)\n",
      "> transformer.h.18.ln_1.bias torch.Size([4096])\n",
      "< (8, 4096) to (4096,)\n",
      "> transformer.h.18.ln_1.weight torch.Size([4096])\n",
      "< (8, 4096, 512) to (1, 4096, 4096)\n",
      "> transformer.h.19.attn.attention.q_proj.weight torch.Size([4096, 4096])\n",
      "< (8, 4096, 512) to (1, 4096, 4096)\n",
      "> transformer.h.19.attn.attention.v_proj.weight torch.Size([4096, 4096])\n",
      "< (8, 4096, 512) to (1, 4096, 4096)\n",
      "> transformer.h.19.attn.attention.k_proj.weight torch.Size([4096, 4096])\n",
      "< (8, 512, 4096) to (1, 4096, 4096)\n",
      "> transformer.h.19.attn.attention.out_proj.weight torch.Size([4096, 4096])\n",
      "< (8, 2048) to (1, 16384)\n",
      "> transformer.h.19.mlp.c_fc.bias torch.Size([16384])\n",
      "< (8, 4096, 2048) to (1, 4096, 16384)\n",
      "> transformer.h.19.mlp.c_fc.weight torch.Size([16384, 4096])\n",
      "< (8, 4096) to (4096,)\n",
      "> transformer.h.19.mlp.c_proj.bias torch.Size([4096])\n",
      "< (8, 2048, 4096) to (1, 16384, 4096)\n",
      "> transformer.h.19.mlp.c_proj.weight torch.Size([4096, 16384])\n",
      "< (8, 4096) to (4096,)\n",
      "> transformer.h.19.ln_1.bias torch.Size([4096])\n",
      "< (8, 4096) to (4096,)\n",
      "> transformer.h.19.ln_1.weight torch.Size([4096])\n",
      "< (8, 4096, 512) to (1, 4096, 4096)\n",
      "> transformer.h.2.attn.attention.q_proj.weight torch.Size([4096, 4096])\n",
      "< (8, 4096, 512) to (1, 4096, 4096)\n",
      "> transformer.h.2.attn.attention.v_proj.weight torch.Size([4096, 4096])\n",
      "< (8, 4096, 512) to (1, 4096, 4096)\n",
      "> transformer.h.2.attn.attention.k_proj.weight torch.Size([4096, 4096])\n",
      "< (8, 512, 4096) to (1, 4096, 4096)\n",
      "> transformer.h.2.attn.attention.out_proj.weight torch.Size([4096, 4096])\n",
      "loading shards for part 7\n",
      "read from checkpoint\n",
      "< (8, 2048) to (1, 16384)\n",
      "> transformer.h.2.mlp.c_fc.bias torch.Size([16384])\n",
      "< (8, 4096, 2048) to (1, 4096, 16384)\n",
      "> transformer.h.2.mlp.c_fc.weight torch.Size([16384, 4096])\n",
      "< (8, 4096) to (4096,)\n",
      "> transformer.h.2.mlp.c_proj.bias torch.Size([4096])\n",
      "< (8, 2048, 4096) to (1, 16384, 4096)\n",
      "> transformer.h.2.mlp.c_proj.weight torch.Size([4096, 16384])\n",
      "< (8, 4096) to (4096,)\n",
      "> transformer.h.2.ln_1.bias torch.Size([4096])\n",
      "< (8, 4096) to (4096,)\n",
      "> transformer.h.2.ln_1.weight torch.Size([4096])\n",
      "< (8, 4096, 512) to (1, 4096, 4096)\n",
      "> transformer.h.20.attn.attention.q_proj.weight torch.Size([4096, 4096])\n",
      "< (8, 4096, 512) to (1, 4096, 4096)\n",
      "> transformer.h.20.attn.attention.v_proj.weight torch.Size([4096, 4096])\n",
      "< (8, 4096, 512) to (1, 4096, 4096)\n",
      "> transformer.h.20.attn.attention.k_proj.weight torch.Size([4096, 4096])\n",
      "< (8, 512, 4096) to (1, 4096, 4096)\n",
      "> transformer.h.20.attn.attention.out_proj.weight torch.Size([4096, 4096])\n",
      "< (8, 2048) to (1, 16384)\n",
      "> transformer.h.20.mlp.c_fc.bias torch.Size([16384])\n",
      "< (8, 4096, 2048) to (1, 4096, 16384)\n",
      "> transformer.h.20.mlp.c_fc.weight torch.Size([16384, 4096])\n",
      "< (8, 4096) to (4096,)\n",
      "> transformer.h.20.mlp.c_proj.bias torch.Size([4096])\n",
      "< (8, 2048, 4096) to (1, 16384, 4096)\n",
      "> transformer.h.20.mlp.c_proj.weight torch.Size([4096, 16384])\n",
      "< (8, 4096) to (4096,)\n",
      "> transformer.h.20.ln_1.bias torch.Size([4096])\n",
      "< (8, 4096) to (4096,)\n",
      "> transformer.h.20.ln_1.weight torch.Size([4096])\n",
      "< (8, 4096, 512) to (1, 4096, 4096)\n",
      "> transformer.h.21.attn.attention.q_proj.weight torch.Size([4096, 4096])\n",
      "< (8, 4096, 512) to (1, 4096, 4096)\n",
      "> transformer.h.21.attn.attention.v_proj.weight torch.Size([4096, 4096])\n",
      "loading shards for part 8\n",
      "read from checkpoint\n",
      "< (8, 4096, 512) to (1, 4096, 4096)\n",
      "> transformer.h.21.attn.attention.k_proj.weight torch.Size([4096, 4096])\n",
      "< (8, 512, 4096) to (1, 4096, 4096)\n",
      "> transformer.h.21.attn.attention.out_proj.weight torch.Size([4096, 4096])\n",
      "< (8, 2048) to (1, 16384)\n",
      "> transformer.h.21.mlp.c_fc.bias torch.Size([16384])\n",
      "< (8, 4096, 2048) to (1, 4096, 16384)\n",
      "> transformer.h.21.mlp.c_fc.weight torch.Size([16384, 4096])\n",
      "< (8, 4096) to (4096,)\n",
      "> transformer.h.21.mlp.c_proj.bias torch.Size([4096])\n",
      "< (8, 2048, 4096) to (1, 16384, 4096)\n",
      "> transformer.h.21.mlp.c_proj.weight torch.Size([4096, 16384])\n",
      "< (8, 4096) to (4096,)\n",
      "> transformer.h.21.ln_1.bias torch.Size([4096])\n",
      "< (8, 4096) to (4096,)\n",
      "> transformer.h.21.ln_1.weight torch.Size([4096])\n",
      "< (8, 4096, 512) to (1, 4096, 4096)\n",
      "> transformer.h.22.attn.attention.q_proj.weight torch.Size([4096, 4096])\n",
      "< (8, 4096, 512) to (1, 4096, 4096)\n",
      "> transformer.h.22.attn.attention.v_proj.weight torch.Size([4096, 4096])\n",
      "< (8, 4096, 512) to (1, 4096, 4096)\n",
      "> transformer.h.22.attn.attention.k_proj.weight torch.Size([4096, 4096])\n",
      "< (8, 512, 4096) to (1, 4096, 4096)\n",
      "> transformer.h.22.attn.attention.out_proj.weight torch.Size([4096, 4096])\n",
      "< (8, 2048) to (1, 16384)\n",
      "> transformer.h.22.mlp.c_fc.bias torch.Size([16384])\n",
      "< (8, 4096, 2048) to (1, 4096, 16384)\n",
      "> transformer.h.22.mlp.c_fc.weight torch.Size([16384, 4096])\n",
      "< (8, 4096) to (4096,)\n",
      "> transformer.h.22.mlp.c_proj.bias torch.Size([4096])\n",
      "< (8, 2048, 4096) to (1, 16384, 4096)\n",
      "> transformer.h.22.mlp.c_proj.weight torch.Size([4096, 16384])\n",
      "< (8, 4096) to (4096,)\n",
      "> transformer.h.22.ln_1.bias torch.Size([4096])\n",
      "< (8, 4096) to (4096,)\n",
      "> transformer.h.22.ln_1.weight torch.Size([4096])\n",
      "loading shards for part 9\n",
      "read from checkpoint\n",
      "< (8, 4096, 512) to (1, 4096, 4096)\n",
      "> transformer.h.23.attn.attention.q_proj.weight torch.Size([4096, 4096])\n",
      "< (8, 4096, 512) to (1, 4096, 4096)\n",
      "> transformer.h.23.attn.attention.v_proj.weight torch.Size([4096, 4096])\n",
      "< (8, 4096, 512) to (1, 4096, 4096)\n",
      "> transformer.h.23.attn.attention.k_proj.weight torch.Size([4096, 4096])\n",
      "< (8, 512, 4096) to (1, 4096, 4096)\n",
      "> transformer.h.23.attn.attention.out_proj.weight torch.Size([4096, 4096])\n",
      "< (8, 2048) to (1, 16384)\n",
      "> transformer.h.23.mlp.c_fc.bias torch.Size([16384])\n",
      "< (8, 4096, 2048) to (1, 4096, 16384)\n",
      "> transformer.h.23.mlp.c_fc.weight torch.Size([16384, 4096])\n",
      "< (8, 4096) to (4096,)\n",
      "> transformer.h.23.mlp.c_proj.bias torch.Size([4096])\n",
      "< (8, 2048, 4096) to (1, 16384, 4096)\n",
      "> transformer.h.23.mlp.c_proj.weight torch.Size([4096, 16384])\n",
      "< (8, 4096) to (4096,)\n",
      "> transformer.h.23.ln_1.bias torch.Size([4096])\n",
      "< (8, 4096) to (4096,)\n",
      "> transformer.h.23.ln_1.weight torch.Size([4096])\n",
      "< (8, 4096, 512) to (1, 4096, 4096)\n",
      "> transformer.h.24.attn.attention.q_proj.weight torch.Size([4096, 4096])\n",
      "< (8, 4096, 512) to (1, 4096, 4096)\n",
      "> transformer.h.24.attn.attention.v_proj.weight torch.Size([4096, 4096])\n",
      "< (8, 4096, 512) to (1, 4096, 4096)\n",
      "> transformer.h.24.attn.attention.k_proj.weight torch.Size([4096, 4096])\n",
      "< (8, 512, 4096) to (1, 4096, 4096)\n",
      "> transformer.h.24.attn.attention.out_proj.weight torch.Size([4096, 4096])\n",
      "< (8, 2048) to (1, 16384)\n",
      "> transformer.h.24.mlp.c_fc.bias torch.Size([16384])\n",
      "< (8, 4096, 2048) to (1, 4096, 16384)\n",
      "> transformer.h.24.mlp.c_fc.weight torch.Size([16384, 4096])\n",
      "< (8, 4096) to (4096,)\n",
      "> transformer.h.24.mlp.c_proj.bias torch.Size([4096])\n",
      "< (8, 2048, 4096) to (1, 16384, 4096)\n",
      "> transformer.h.24.mlp.c_proj.weight torch.Size([4096, 16384])\n",
      "loading shards for part 10\n",
      "read from checkpoint\n",
      "< (8, 4096) to (4096,)\n",
      "> transformer.h.24.ln_1.bias torch.Size([4096])\n",
      "< (8, 4096) to (4096,)\n",
      "> transformer.h.24.ln_1.weight torch.Size([4096])\n",
      "< (8, 4096, 512) to (1, 4096, 4096)\n",
      "> transformer.h.25.attn.attention.q_proj.weight torch.Size([4096, 4096])\n",
      "< (8, 4096, 512) to (1, 4096, 4096)\n",
      "> transformer.h.25.attn.attention.v_proj.weight torch.Size([4096, 4096])\n",
      "< (8, 4096, 512) to (1, 4096, 4096)\n",
      "> transformer.h.25.attn.attention.k_proj.weight torch.Size([4096, 4096])\n",
      "< (8, 512, 4096) to (1, 4096, 4096)\n",
      "> transformer.h.25.attn.attention.out_proj.weight torch.Size([4096, 4096])\n",
      "< (8, 2048) to (1, 16384)\n",
      "> transformer.h.25.mlp.c_fc.bias torch.Size([16384])\n",
      "< (8, 4096, 2048) to (1, 4096, 16384)\n",
      "> transformer.h.25.mlp.c_fc.weight torch.Size([16384, 4096])\n",
      "< (8, 4096) to (4096,)\n",
      "> transformer.h.25.mlp.c_proj.bias torch.Size([4096])\n",
      "< (8, 2048, 4096) to (1, 16384, 4096)\n",
      "> transformer.h.25.mlp.c_proj.weight torch.Size([4096, 16384])\n",
      "< (8, 4096) to (4096,)\n",
      "> transformer.h.25.ln_1.bias torch.Size([4096])\n",
      "< (8, 4096) to (4096,)\n",
      "> transformer.h.25.ln_1.weight torch.Size([4096])\n",
      "< (8, 4096, 512) to (1, 4096, 4096)\n",
      "> transformer.h.26.attn.attention.q_proj.weight torch.Size([4096, 4096])\n",
      "< (8, 4096, 512) to (1, 4096, 4096)\n",
      "> transformer.h.26.attn.attention.v_proj.weight torch.Size([4096, 4096])\n",
      "< (8, 4096, 512) to (1, 4096, 4096)\n",
      "> transformer.h.26.attn.attention.k_proj.weight torch.Size([4096, 4096])\n",
      "< (8, 512, 4096) to (1, 4096, 4096)\n",
      "> transformer.h.26.attn.attention.out_proj.weight torch.Size([4096, 4096])\n",
      "< (8, 2048) to (1, 16384)\n",
      "> transformer.h.26.mlp.c_fc.bias torch.Size([16384])\n",
      "< (8, 4096, 2048) to (1, 4096, 16384)\n",
      "> transformer.h.26.mlp.c_fc.weight torch.Size([16384, 4096])\n",
      "loading shards for part 11\n",
      "read from checkpoint\n",
      "< (8, 4096) to (4096,)\n",
      "> transformer.h.26.mlp.c_proj.bias torch.Size([4096])\n",
      "< (8, 2048, 4096) to (1, 16384, 4096)\n",
      "> transformer.h.26.mlp.c_proj.weight torch.Size([4096, 16384])\n",
      "< (8, 4096) to (4096,)\n",
      "> transformer.h.26.ln_1.bias torch.Size([4096])\n",
      "< (8, 4096) to (4096,)\n",
      "> transformer.h.26.ln_1.weight torch.Size([4096])\n",
      "< (8, 4096, 512) to (1, 4096, 4096)\n",
      "> transformer.h.27.attn.attention.q_proj.weight torch.Size([4096, 4096])\n",
      "< (8, 4096, 512) to (1, 4096, 4096)\n",
      "> transformer.h.27.attn.attention.v_proj.weight torch.Size([4096, 4096])\n",
      "< (8, 4096, 512) to (1, 4096, 4096)\n",
      "> transformer.h.27.attn.attention.k_proj.weight torch.Size([4096, 4096])\n",
      "< (8, 512, 4096) to (1, 4096, 4096)\n",
      "> transformer.h.27.attn.attention.out_proj.weight torch.Size([4096, 4096])\n",
      "< (8, 2048) to (1, 16384)\n",
      "> transformer.h.27.mlp.c_fc.bias torch.Size([16384])\n",
      "< (8, 4096, 2048) to (1, 4096, 16384)\n",
      "> transformer.h.27.mlp.c_fc.weight torch.Size([16384, 4096])\n",
      "< (8, 4096) to (4096,)\n",
      "> transformer.h.27.mlp.c_proj.bias torch.Size([4096])\n",
      "< (8, 2048, 4096) to (1, 16384, 4096)\n",
      "> transformer.h.27.mlp.c_proj.weight torch.Size([4096, 16384])\n",
      "< (8, 4096) to (4096,)\n",
      "> transformer.h.27.ln_1.bias torch.Size([4096])\n",
      "< (8, 4096) to (4096,)\n",
      "> transformer.h.27.ln_1.weight torch.Size([4096])\n",
      "< (8, 4096, 512) to (1, 4096, 4096)\n",
      "> transformer.h.3.attn.attention.q_proj.weight torch.Size([4096, 4096])\n",
      "< (8, 4096, 512) to (1, 4096, 4096)\n",
      "> transformer.h.3.attn.attention.v_proj.weight torch.Size([4096, 4096])\n",
      "< (8, 4096, 512) to (1, 4096, 4096)\n",
      "> transformer.h.3.attn.attention.k_proj.weight torch.Size([4096, 4096])\n",
      "< (8, 512, 4096) to (1, 4096, 4096)\n",
      "> transformer.h.3.attn.attention.out_proj.weight torch.Size([4096, 4096])\n",
      "loading shards for part 12\n",
      "read from checkpoint\n",
      "< (8, 2048) to (1, 16384)\n",
      "> transformer.h.3.mlp.c_fc.bias torch.Size([16384])\n",
      "< (8, 4096, 2048) to (1, 4096, 16384)\n",
      "> transformer.h.3.mlp.c_fc.weight torch.Size([16384, 4096])\n",
      "< (8, 4096) to (4096,)\n",
      "> transformer.h.3.mlp.c_proj.bias torch.Size([4096])\n",
      "< (8, 2048, 4096) to (1, 16384, 4096)\n",
      "> transformer.h.3.mlp.c_proj.weight torch.Size([4096, 16384])\n",
      "< (8, 4096) to (4096,)\n",
      "> transformer.h.3.ln_1.bias torch.Size([4096])\n",
      "< (8, 4096) to (4096,)\n",
      "> transformer.h.3.ln_1.weight torch.Size([4096])\n",
      "< (8, 4096, 512) to (1, 4096, 4096)\n",
      "> transformer.h.4.attn.attention.q_proj.weight torch.Size([4096, 4096])\n",
      "< (8, 4096, 512) to (1, 4096, 4096)\n",
      "> transformer.h.4.attn.attention.v_proj.weight torch.Size([4096, 4096])\n",
      "< (8, 4096, 512) to (1, 4096, 4096)\n",
      "> transformer.h.4.attn.attention.k_proj.weight torch.Size([4096, 4096])\n",
      "< (8, 512, 4096) to (1, 4096, 4096)\n",
      "> transformer.h.4.attn.attention.out_proj.weight torch.Size([4096, 4096])\n",
      "< (8, 2048) to (1, 16384)\n",
      "> transformer.h.4.mlp.c_fc.bias torch.Size([16384])\n",
      "< (8, 4096, 2048) to (1, 4096, 16384)\n",
      "> transformer.h.4.mlp.c_fc.weight torch.Size([16384, 4096])\n",
      "< (8, 4096) to (4096,)\n",
      "> transformer.h.4.mlp.c_proj.bias torch.Size([4096])\n",
      "< (8, 2048, 4096) to (1, 16384, 4096)\n",
      "> transformer.h.4.mlp.c_proj.weight torch.Size([4096, 16384])\n",
      "< (8, 4096) to (4096,)\n",
      "> transformer.h.4.ln_1.bias torch.Size([4096])\n",
      "< (8, 4096) to (4096,)\n",
      "> transformer.h.4.ln_1.weight torch.Size([4096])\n",
      "< (8, 4096, 512) to (1, 4096, 4096)\n",
      "> transformer.h.5.attn.attention.q_proj.weight torch.Size([4096, 4096])\n",
      "< (8, 4096, 512) to (1, 4096, 4096)\n",
      "> transformer.h.5.attn.attention.v_proj.weight torch.Size([4096, 4096])\n",
      "loading shards for part 13\n",
      "read from checkpoint\n",
      "< (8, 4096, 512) to (1, 4096, 4096)\n",
      "> transformer.h.5.attn.attention.k_proj.weight torch.Size([4096, 4096])\n",
      "< (8, 512, 4096) to (1, 4096, 4096)\n",
      "> transformer.h.5.attn.attention.out_proj.weight torch.Size([4096, 4096])\n",
      "< (8, 2048) to (1, 16384)\n",
      "> transformer.h.5.mlp.c_fc.bias torch.Size([16384])\n",
      "< (8, 4096, 2048) to (1, 4096, 16384)\n",
      "> transformer.h.5.mlp.c_fc.weight torch.Size([16384, 4096])\n",
      "< (8, 4096) to (4096,)\n",
      "> transformer.h.5.mlp.c_proj.bias torch.Size([4096])\n",
      "< (8, 2048, 4096) to (1, 16384, 4096)\n",
      "> transformer.h.5.mlp.c_proj.weight torch.Size([4096, 16384])\n",
      "< (8, 4096) to (4096,)\n",
      "> transformer.h.5.ln_1.bias torch.Size([4096])\n",
      "< (8, 4096) to (4096,)\n",
      "> transformer.h.5.ln_1.weight torch.Size([4096])\n",
      "< (8, 4096, 512) to (1, 4096, 4096)\n",
      "> transformer.h.6.attn.attention.q_proj.weight torch.Size([4096, 4096])\n",
      "< (8, 4096, 512) to (1, 4096, 4096)\n",
      "> transformer.h.6.attn.attention.v_proj.weight torch.Size([4096, 4096])\n",
      "< (8, 4096, 512) to (1, 4096, 4096)\n",
      "> transformer.h.6.attn.attention.k_proj.weight torch.Size([4096, 4096])\n",
      "< (8, 512, 4096) to (1, 4096, 4096)\n",
      "> transformer.h.6.attn.attention.out_proj.weight torch.Size([4096, 4096])\n",
      "< (8, 2048) to (1, 16384)\n",
      "> transformer.h.6.mlp.c_fc.bias torch.Size([16384])\n",
      "< (8, 4096, 2048) to (1, 4096, 16384)\n",
      "> transformer.h.6.mlp.c_fc.weight torch.Size([16384, 4096])\n",
      "< (8, 4096) to (4096,)\n",
      "> transformer.h.6.mlp.c_proj.bias torch.Size([4096])\n",
      "< (8, 2048, 4096) to (1, 16384, 4096)\n",
      "> transformer.h.6.mlp.c_proj.weight torch.Size([4096, 16384])\n",
      "< (8, 4096) to (4096,)\n",
      "> transformer.h.6.ln_1.bias torch.Size([4096])\n",
      "< (8, 4096) to (4096,)\n",
      "> transformer.h.6.ln_1.weight torch.Size([4096])\n",
      "loading shards for part 14\n",
      "read from checkpoint\n",
      "< (8, 4096, 512) to (1, 4096, 4096)\n",
      "> transformer.h.7.attn.attention.q_proj.weight torch.Size([4096, 4096])\n",
      "< (8, 4096, 512) to (1, 4096, 4096)\n",
      "> transformer.h.7.attn.attention.v_proj.weight torch.Size([4096, 4096])\n",
      "< (8, 4096, 512) to (1, 4096, 4096)\n",
      "> transformer.h.7.attn.attention.k_proj.weight torch.Size([4096, 4096])\n",
      "< (8, 512, 4096) to (1, 4096, 4096)\n",
      "> transformer.h.7.attn.attention.out_proj.weight torch.Size([4096, 4096])\n",
      "< (8, 2048) to (1, 16384)\n",
      "> transformer.h.7.mlp.c_fc.bias torch.Size([16384])\n",
      "< (8, 4096, 2048) to (1, 4096, 16384)\n",
      "> transformer.h.7.mlp.c_fc.weight torch.Size([16384, 4096])\n",
      "< (8, 4096) to (4096,)\n",
      "> transformer.h.7.mlp.c_proj.bias torch.Size([4096])\n",
      "< (8, 2048, 4096) to (1, 16384, 4096)\n",
      "> transformer.h.7.mlp.c_proj.weight torch.Size([4096, 16384])\n",
      "< (8, 4096) to (4096,)\n",
      "> transformer.h.7.ln_1.bias torch.Size([4096])\n",
      "< (8, 4096) to (4096,)\n",
      "> transformer.h.7.ln_1.weight torch.Size([4096])\n",
      "< (8, 4096, 512) to (1, 4096, 4096)\n",
      "> transformer.h.8.attn.attention.q_proj.weight torch.Size([4096, 4096])\n",
      "< (8, 4096, 512) to (1, 4096, 4096)\n",
      "> transformer.h.8.attn.attention.v_proj.weight torch.Size([4096, 4096])\n",
      "< (8, 4096, 512) to (1, 4096, 4096)\n",
      "> transformer.h.8.attn.attention.k_proj.weight torch.Size([4096, 4096])\n",
      "< (8, 512, 4096) to (1, 4096, 4096)\n",
      "> transformer.h.8.attn.attention.out_proj.weight torch.Size([4096, 4096])\n",
      "< (8, 2048) to (1, 16384)\n",
      "> transformer.h.8.mlp.c_fc.bias torch.Size([16384])\n",
      "< (8, 4096, 2048) to (1, 4096, 16384)\n",
      "> transformer.h.8.mlp.c_fc.weight torch.Size([16384, 4096])\n",
      "< (8, 4096) to (4096,)\n",
      "> transformer.h.8.mlp.c_proj.bias torch.Size([4096])\n",
      "< (8, 2048, 4096) to (1, 16384, 4096)\n",
      "> transformer.h.8.mlp.c_proj.weight torch.Size([4096, 16384])\n",
      "loading shards for part 15\n",
      "read from checkpoint\n",
      "< (8, 4096) to (4096,)\n",
      "> transformer.h.8.ln_1.bias torch.Size([4096])\n",
      "< (8, 4096) to (4096,)\n",
      "> transformer.h.8.ln_1.weight torch.Size([4096])\n",
      "< (8, 4096, 512) to (1, 4096, 4096)\n",
      "> transformer.h.9.attn.attention.q_proj.weight torch.Size([4096, 4096])\n",
      "< (8, 4096, 512) to (1, 4096, 4096)\n",
      "> transformer.h.9.attn.attention.v_proj.weight torch.Size([4096, 4096])\n",
      "< (8, 4096, 512) to (1, 4096, 4096)\n",
      "> transformer.h.9.attn.attention.k_proj.weight torch.Size([4096, 4096])\n",
      "< (8, 512, 4096) to (1, 4096, 4096)\n",
      "> transformer.h.9.attn.attention.out_proj.weight torch.Size([4096, 4096])\n",
      "< (8, 2048) to (1, 16384)\n",
      "> transformer.h.9.mlp.c_fc.bias torch.Size([16384])\n",
      "< (8, 4096, 2048) to (1, 4096, 16384)\n",
      "> transformer.h.9.mlp.c_fc.weight torch.Size([16384, 4096])\n",
      "< (8, 4096) to (4096,)\n",
      "> transformer.h.9.mlp.c_proj.bias torch.Size([4096])\n",
      "< (8, 2048, 4096) to (1, 16384, 4096)\n",
      "> transformer.h.9.mlp.c_proj.weight torch.Size([4096, 16384])\n",
      "< (8, 4096) to (4096,)\n",
      "> transformer.h.9.ln_1.bias torch.Size([4096])\n",
      "< (8, 4096) to (4096,)\n",
      "> transformer.h.9.ln_1.weight torch.Size([4096])\n",
      "< (8, 6300) to (1, 50400)\n",
      "> lm_head.bias torch.Size([50400])\n",
      "< (8, 4096, 6300) to (1, 4096, 50400)\n",
      "> lm_head.weight torch.Size([50400, 4096])\n",
      "< (8, 4096) to (4096,)\n",
      "> transformer.ln_f.bias torch.Size([4096])\n",
      "< (8, 4096) to (4096,)\n",
      "> transformer.ln_f.weight torch.Size([4096])\n",
      "left over: [array([383502., 383502., 383502., 383502., 383502., 383502., 383502., 383502.], dtype=float32)]\n",
      "saving\n",
      "Saving layer 0 / 28\n",
      "Saving layer 1 / 28\n",
      "Saving layer 2 / 28\n",
      "Saving layer 3 / 28\n",
      "Saving layer 4 / 28\n",
      "Saving layer 5 / 28\n",
      "Saving layer 6 / 28\n",
      "Saving layer 7 / 28\n",
      "Saving layer 8 / 28\n",
      "Saving layer 9 / 28\n",
      "Saving layer 10 / 28\n",
      "Saving layer 11 / 28\n",
      "Saving layer 12 / 28\n",
      "Saving layer 13 / 28\n",
      "Saving layer 14 / 28\n",
      "Saving layer 15 / 28\n",
      "Saving layer 16 / 28\n",
      "Saving layer 17 / 28\n",
      "Saving layer 18 / 28\n",
      "Saving layer 19 / 28\n",
      "Saving layer 20 / 28\n",
      "Saving layer 21 / 28\n",
      "Saving layer 22 / 28\n",
      "Saving layer 23 / 28\n",
      "Saving layer 24 / 28\n",
      "Saving layer 25 / 28\n",
      "Saving layer 26 / 28\n",
      "Saving layer 27 / 28\n",
      "done\n"
     ]
    }
   ],
   "source": [
    "# Converting weights for 2 GPU without parallelism\n",
    "!python3 ../examples/pytorch/gptj/utils/gptj_ckpt_convert.py \\\n",
    "                                --output-dir ../../models/j6b_ckpt \\\n",
    "                                --ckpt-dir ../../model_repository/step_383500/ \\\n",
    "                                --n-inference-gpus 2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let us have a look at the generated files. We have re-combined sharded weights into two groups of weights that will be placed onto two GPUs in our tensor parallel deployment. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1-gpu  2-gpu\n"
     ]
    }
   ],
   "source": [
    "!ls ../../models/j6b_ckpt/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we have to run kernel-autotuning again to find the fastest kernels for the 2xGPU mode. First, let's run inference with the kernels that we already found for 1xGPU mode on the previous step. We do this non-optimal inference just to compare results later with kernel-autotuning to see the acceleration this operation gives us. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Total ranks: 2.\n",
      "Device Graphics Device\n",
      "P0 is runing with 0 GPU.\n",
      "[INFO] Setting tensor_para_size to 2 \n",
      "Device Graphics Device\n",
      "P1 is runing with 1 GPU.\n",
      "[INFO] Setting tensor_para_size to 2 \n",
      "[FT][WARNING] Device 0 peer access Device 1 is not avaiable. This may lead to peer access errors when doing tensor/pipeline parallel!\n",
      "[FT][WARNING] Device 1 peer access Device 0 is not avaiable. This may lead to peer access errors when doing tensor/pipeline parallel!\n",
      "after allocation, free 72.48 GB total 79.35 GB\n",
      "after allocation, free 64.46 GB total 79.35 GB\n",
      "Writing 128 elements\n",
      "  818   262   938  3155   286  1528    11   257 10089   517 \n",
      "zeroCount = 0\n",
      "[INFO] request_batch_size 1 beam_width 1 head_num 16 size_per_head 256 total_output_len 128 decoder_layers 28 vocab_size 50400 FT-CPP-decoding-beamsearch-time 785.61 ms\n",
      "[INFO] request_batch_size 1 beam_width 1 head_num 16 size_per_head 256 total_output_len 128 decoder_layers 28 vocab_size 50400 FT-CPP-decoding-beamsearch-time 785.61 ms\n"
     ]
    }
   ],
   "source": [
    "!CUDA_VISIBLE_DEVICES=1,2 mpirun -n 2 --allow-run-as-root ./bin/gptj_example"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The inference with the non-optimal kernels is <b>~1.4s</b>. We achieved acceleration in comparison with 1xGPU mode, but it can be better. Now we have to do kernel-autotuning for Tp 2 regime because there is some probability that other algorithms may work better. We use the same parameters for our NN, but at this time we should set `tensor_para_size = 2` "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " We will use GPU 1 because previous task (Notebook 02) was atarted onto GPU 0 and model is already there if you haven't stopped the Jupyter kernel. So we just avoiding OOM error. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "scrolled": true,
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[INFO] arguments: \n",
      "  batch_size: 1 \n",
      "  beam_width: 1 \n",
      "  max_input_len: 128 \n",
      "  head_num: 16 \n",
      "  size_per_head: 256 \n",
      "  inter_size: 16384 \n",
      "  vocab_size: 50256 \n",
      "  data_type: 1 \n",
      "  tensor_para_size: 2 \n",
      "\n",
      "Device Graphics Device\n",
      "***Encoder Gemm Testing Begin***\n",
      "***Cublas Gemm Testing Begin***\n",
      "\n",
      "-----------------------------\n",
      "GEMM test 0: [M: 128, K: 4096, N: 6144] context from_tensor * weightQKV\n",
      "algo_99 costs 0.052ms \n",
      "algo_100 costs 0.044ms \n",
      "algo_101 costs 0.044ms \n",
      "algo_102 costs 0.044ms \n",
      "algo_103 costs 0.044ms \n",
      "algo_104 costs 0.044ms \n",
      "algo_105 costs 0.044ms \n",
      "algo_106 costs 0.044ms \n",
      "algo_107 costs 0.044ms \n",
      "algo_108 costs 0.044ms \n",
      "algo_109 costs 0.044ms \n",
      "algo_110 costs 0.044ms \n",
      "algo_111 costs 0.044ms \n",
      "algo_112 costs 0.044ms \n",
      "algo_113 costs 0.044ms \n",
      "algo_114 costs 0.044ms \n",
      "algo_115 costs 0.044ms \n",
      "fast_algo 105 costs 0.044 ms\n",
      "***cublasLt Gemm Testing Beign***\n",
      "AlgoCount: 3813\n",
      "return 1 and run heuristic algo\n",
      "workspacesize==0, run 170 algos\n",
      "result 000 : algo={ Id=6, tileIdx=18 (128x64) splitK=0 reduc=0 swizzle=0 custom=0 stages=16} status 0 time 0.043284ms workspace=0 mathMode=0 waves=0.888889\n",
      "result 001 : algo={ Id=6, tileIdx=18 (128x64) splitK=0 reduc=0 swizzle=1 custom=0 stages=16} status 0 time 0.043459ms workspace=0 mathMode=0 waves=0.888889\n",
      "result 002 : algo={ Id=21, tileIdx=18 (128x64) splitK=1 reduc=0 swizzle=0 custom=0 stages=18} status 0 time 0.043735ms workspace=0 mathMode=0 waves=0.888889\n",
      "result 003 : algo={ Id=6, tileIdx=18 (128x64) splitK=0 reduc=0 swizzle=0 custom=0 stages=12} status 0 time 0.043889ms workspace=0 mathMode=0 waves=0.444444\n",
      "result 004 : algo={ Id=6, tileIdx=18 (128x64) splitK=0 reduc=0 swizzle=1 custom=0 stages=12} status 0 time 0.044196ms workspace=0 mathMode=0 waves=0.444444\n",
      "result 005 : algo={ Id=6, tileIdx=18 (128x64) splitK=0 reduc=0 swizzle=0 custom=0 stages=15} status 0 time 0.045875ms workspace=0 mathMode=0 waves=0.444444\n",
      "result 006 : algo={ Id=6, tileIdx=18 (128x64) splitK=0 reduc=0 swizzle=1 custom=0 stages=15} status 0 time 0.046725ms workspace=0 mathMode=0 waves=0.444444\n",
      "result 007 : algo={ Id=6, tileIdx=20 (128x128) splitK=0 reduc=0 swizzle=0 custom=0 stages=15} status 0 time 0.055921ms workspace=0 mathMode=0 waves=0.444444\n",
      "result 008 : algo={ Id=6, tileIdx=20 (128x128) splitK=0 reduc=0 swizzle=1 custom=0 stages=11} status 0 time 0.055951ms workspace=0 mathMode=0 waves=0.222222\n",
      "result 009 : algo={ Id=6, tileIdx=20 (128x128) splitK=0 reduc=0 swizzle=1 custom=0 stages=15} status 0 time 0.055951ms workspace=0 mathMode=0 waves=0.444444\n",
      "result 010 : algo={ Id=6, tileIdx=20 (128x128) splitK=0 reduc=0 swizzle=0 custom=0 stages=11} status 0 time 0.056033ms workspace=0 mathMode=0 waves=0.222222\n",
      "result 011 : algo={ Id=6, tileIdx=24 (256x128) splitK=0 reduc=0 swizzle=0 custom=0 stages=9} status 0 time 0.102625ms workspace=0 mathMode=0 waves=0.222222\n",
      "result 012 : algo={ Id=6, tileIdx=24 (256x128) splitK=0 reduc=0 swizzle=0 custom=0 stages=15} status 0 time 0.103107ms workspace=0 mathMode=0 waves=0.222222\n",
      "result 013 : algo={ Id=6, tileIdx=24 (256x128) splitK=0 reduc=0 swizzle=1 custom=0 stages=15} status 0 time 0.103127ms workspace=0 mathMode=0 waves=0.222222\n",
      "result 014 : algo={ Id=6, tileIdx=23 (128x256) splitK=0 reduc=0 swizzle=1 custom=0 stages=15} status 0 time 0.103404ms workspace=0 mathMode=0 waves=0.444444\n",
      "result 015 : algo={ Id=6, tileIdx=23 (128x256) splitK=0 reduc=0 swizzle=0 custom=0 stages=15} status 0 time 0.103404ms workspace=0 mathMode=0 waves=0.444444\n",
      "result 016 : algo={ Id=5, tileIdx=20 (128x128) splitK=0 reduc=0 swizzle=1 custom=0 stages=0} status 0 time 0.104786ms workspace=0 mathMode=0 waves=0.222222\n",
      "result 017 : algo={ Id=5, tileIdx=20 (128x128) splitK=0 reduc=0 swizzle=0 custom=0 stages=0} status 0 time 0.104806ms workspace=0 mathMode=0 waves=0.222222\n",
      "result 018 : algo={ Id=5, tileIdx=21 (256x64) splitK=0 reduc=0 swizzle=0 custom=0 stages=7} status 0 time 0.113654ms workspace=0 mathMode=0 waves=0.222222\n",
      "result 019 : algo={ Id=5, tileIdx=21 (256x64) splitK=0 reduc=0 swizzle=1 custom=0 stages=7} status 0 time 0.113766ms workspace=0 mathMode=0 waves=0.222222\n",
      "result 020 : algo={ Id=5, tileIdx=20 (128x128) splitK=0 reduc=0 swizzle=0 custom=0 stages=7} status 0 time 0.120289ms workspace=0 mathMode=0 waves=0.222222\n",
      "result 021 : algo={ Id=5, tileIdx=20 (128x128) splitK=0 reduc=0 swizzle=1 custom=0 stages=7} status 0 time 0.120422ms workspace=0 mathMode=0 waves=0.222222\n",
      "result 022 : algo={ Id=6, tileIdx=23 (128x256) splitK=0 reduc=0 swizzle=1 custom=0 stages=0} status 0 time 0.120730ms workspace=0 mathMode=0 waves=0.444444\n",
      "result 023 : algo={ Id=6, tileIdx=23 (128x256) splitK=0 reduc=0 swizzle=0 custom=0 stages=0} status 0 time 0.120791ms workspace=0 mathMode=0 waves=0.444444\n",
      "result 024 : algo={ Id=5, tileIdx=23 (128x256) splitK=0 reduc=0 swizzle=1 custom=0 stages=0} status 0 time 0.126433ms workspace=0 mathMode=0 waves=0.444444\n",
      "result 025 : algo={ Id=5, tileIdx=23 (128x256) splitK=0 reduc=0 swizzle=0 custom=0 stages=0} status 0 time 0.126444ms workspace=0 mathMode=0 waves=0.444444\n",
      "result 026 : algo={ Id=6, tileIdx=18 (128x64) splitK=0 reduc=0 swizzle=1 custom=0 stages=0} status 0 time 0.127508ms workspace=0 mathMode=0 waves=0.222222\n",
      "result 027 : algo={ Id=6, tileIdx=18 (128x64) splitK=0 reduc=0 swizzle=0 custom=0 stages=0} status 0 time 0.127672ms workspace=0 mathMode=0 waves=0.222222\n",
      "result 028 : algo={ Id=5, tileIdx=24 (256x128) splitK=0 reduc=0 swizzle=0 custom=0 stages=0} status 0 time 0.132219ms workspace=0 mathMode=0 waves=0.222222\n",
      "result 029 : algo={ Id=5, tileIdx=24 (256x128) splitK=0 reduc=0 swizzle=1 custom=0 stages=0} status 0 time 0.132239ms workspace=0 mathMode=0 waves=0.222222\n",
      "result 030 : algo={ Id=5, tileIdx=23 (128x256) splitK=0 reduc=0 swizzle=1 custom=0 stages=7} status 0 time 0.136038ms workspace=0 mathMode=0 waves=0.444444\n",
      "result 031 : algo={ Id=5, tileIdx=23 (128x256) splitK=0 reduc=0 swizzle=0 custom=0 stages=7} status 0 time 0.136038ms workspace=0 mathMode=0 waves=0.444444\n",
      "result 032 : algo={ Id=5, tileIdx=21 (256x64) splitK=0 reduc=0 swizzle=0 custom=0 stages=0} status 0 time 0.146606ms workspace=0 mathMode=0 waves=0.222222\n",
      "result 033 : algo={ Id=5, tileIdx=21 (256x64) splitK=0 reduc=0 swizzle=1 custom=0 stages=0} status 0 time 0.146729ms workspace=0 mathMode=0 waves=0.222222\n",
      "result 034 : algo={ Id=1, tileIdx=16 (128x32) splitK=0 reduc=0 swizzle=0 custom=0 stages=0} status 0 time 0.406252ms workspace=0 mathMode=0 waves=1.777778\n",
      "result 035 : algo={ Id=1, tileIdx=16 (128x32) splitK=0 reduc=0 swizzle=1 custom=0 stages=0} status 0 time 0.406487ms workspace=0 mathMode=0 waves=1.777778\n",
      "result 036 : algo={ Id=0, tileIdx=16 (128x32) splitK=0 reduc=0 swizzle=1 custom=0 stages=0} status 0 time 0.418877ms workspace=0 mathMode=0 waves=0.444444\n",
      "result 037 : algo={ Id=1, tileIdx=13 (64x32) splitK=0 reduc=0 swizzle=1 custom=0 stages=0} status 0 time 0.420516ms workspace=0 mathMode=0 waves=1.777778\n",
      "result 038 : algo={ Id=0, tileIdx=16 (128x32) splitK=0 reduc=0 swizzle=0 custom=0 stages=0} status 0 time 0.420946ms workspace=0 mathMode=0 waves=0.444444\n",
      "result 039 : algo={ Id=1, tileIdx=13 (64x32) splitK=0 reduc=0 swizzle=0 custom=0 stages=0} status 0 time 0.424243ms workspace=0 mathMode=0 waves=1.777778\n",
      "result 040 : algo={ Id=1, tileIdx=11 (32x32) splitK=0 reduc=0 swizzle=1 custom=0 stages=0} status 0 time 0.451717ms workspace=0 mathMode=0 waves=1.777778\n",
      "result 041 : algo={ Id=0, tileIdx=14 (32x128) splitK=0 reduc=0 swizzle=0 custom=0 stages=0} status 0 time 0.452444ms workspace=0 mathMode=0 waves=0.444444\n",
      "result 042 : algo={ Id=0, tileIdx=14 (32x128) splitK=0 reduc=0 swizzle=1 custom=0 stages=0} status 0 time 0.452567ms workspace=0 mathMode=0 waves=0.444444\n",
      "result 043 : algo={ Id=1, tileIdx=11 (32x32) splitK=0 reduc=0 swizzle=0 custom=0 stages=0} status 0 time 0.453704ms workspace=0 mathMode=0 waves=1.777778\n",
      "result 044 : algo={ Id=0, tileIdx=18 (128x64) splitK=0 reduc=0 swizzle=1 custom=0 stages=0} status 0 time 0.477102ms workspace=0 mathMode=0 waves=0.222222\n",
      "result 045 : algo={ Id=0, tileIdx=18 (128x64) splitK=0 reduc=0 swizzle=0 custom=0 stages=0} status 0 time 0.480266ms workspace=0 mathMode=0 waves=0.222222\n",
      "result 046 : algo={ Id=0, tileIdx=15 (64x64) splitK=0 reduc=0 swizzle=0 custom=0 stages=0} status 0 time 0.502477ms workspace=0 mathMode=0 waves=0.222222\n",
      "result 047 : algo={ Id=0, tileIdx=15 (64x64) splitK=0 reduc=0 swizzle=1 custom=0 stages=0} status 0 time 0.516844ms workspace=0 mathMode=0 waves=0.222222\n",
      "result 048 : algo={ Id=0, tileIdx=20 (128x128) splitK=0 reduc=0 swizzle=0 custom=0 stages=0} status 0 time 0.765348ms workspace=0 mathMode=0 waves=0.222222\n",
      "result 049 : algo={ Id=0, tileIdx=20 (128x128) splitK=0 reduc=0 swizzle=1 custom=0 stages=0} status 0 time 0.766085ms workspace=0 mathMode=0 waves=0.222222\n",
      "algo={ Id=6, tileIdx=18 (128x64) splitK=0 reduc=0 swizzle=0 custom=0 stages=16} status 0 time 0.043284ms workspace=0 mathMode=0 waves=0.888889\n",
      "***cublasLt Gemm Testing End***\n",
      "\n",
      "-----------------------------\n",
      "GEMM test 1: [M: 128, K: 256, N: 128] context batch gemm Q*K^T\n",
      "algo_99 costs 0.013ms \n",
      "algo_100 costs 0.008ms \n",
      "algo_101 costs 0.007ms \n",
      "algo_102 costs 0.006ms \n",
      "algo_103 costs 0.006ms \n",
      "algo_104 costs 0.007ms \n",
      "algo_105 costs 0.007ms \n",
      "algo_106 costs 0.007ms \n",
      "algo_107 costs 0.006ms \n",
      "algo_108 costs 0.006ms \n",
      "algo_109 costs 0.006ms \n",
      "algo_110 costs 0.006ms \n",
      "algo_111 costs 0.006ms \n",
      "algo_112 costs 0.006ms \n",
      "algo_113 costs 0.006ms \n",
      "algo_114 costs 0.006ms \n",
      "algo_115 costs 0.006ms \n",
      "fast_algo 110 costs 0.006 ms\n",
      "\n",
      "-----------------------------\n",
      "GEMM test 2: [M: 128, K: 128, N: 256] context batch gemm QK*V^T\n",
      "algo_99 costs 0.006ms \n",
      "algo_100 costs 0.006ms \n",
      "algo_101 costs 0.006ms \n",
      "algo_102 costs 0.007ms \n",
      "algo_103 costs 0.007ms \n",
      "algo_104 costs 0.007ms \n",
      "algo_105 costs 0.007ms \n",
      "algo_106 costs 0.007ms \n",
      "algo_107 costs 0.007ms \n",
      "algo_108 costs 0.006ms \n",
      "algo_109 costs 0.006ms \n",
      "algo_110 costs 0.006ms \n",
      "algo_111 costs 0.006ms \n",
      "algo_112 costs 0.006ms \n",
      "algo_113 costs 0.006ms \n",
      "algo_114 costs 0.006ms \n",
      "algo_115 costs 0.006ms \n",
      "fast_algo 112 costs 0.006 ms\n",
      "\n",
      "-----------------------------\n",
      "GEMM test 3: [M: 128, K: 2048, N: 4096] context attr * output_kernel\n",
      "algo_99 costs 0.019ms \n",
      "algo_100 costs 0.019ms \n",
      "algo_101 costs 0.019ms \n",
      "algo_102 costs 0.019ms \n",
      "algo_103 costs 0.019ms \n",
      "algo_104 costs 0.019ms \n",
      "algo_105 costs 0.019ms \n",
      "algo_106 costs 0.019ms \n",
      "algo_107 costs 0.019ms \n",
      "algo_108 costs 0.019ms \n",
      "algo_109 costs 0.019ms \n",
      "algo_110 costs 0.019ms \n",
      "algo_111 costs 0.019ms \n",
      "algo_112 costs 0.019ms \n",
      "algo_113 costs 0.019ms \n",
      "algo_114 costs 0.019ms \n",
      "algo_115 costs 0.019ms \n",
      "fast_algo 115 costs 0.019 ms\n",
      "***cublasLt Gemm Testing Beign***\n",
      "AlgoCount: 4287\n",
      "return 1 and run heuristic algo\n",
      "workspacesize==0, run 170 algos\n",
      "result 000 : algo={ Id=6, tileIdx=18 (128x64) splitK=0 reduc=0 swizzle=1 custom=0 stages=15} status 0 time 0.017961ms workspace=0 mathMode=0 waves=0.296296\n",
      "result 001 : algo={ Id=6, tileIdx=18 (128x64) splitK=0 reduc=0 swizzle=1 custom=0 stages=12} status 0 time 0.017971ms workspace=0 mathMode=0 waves=0.296296\n",
      "result 002 : algo={ Id=6, tileIdx=18 (128x64) splitK=0 reduc=0 swizzle=0 custom=0 stages=15} status 0 time 0.017981ms workspace=0 mathMode=0 waves=0.296296\n",
      "result 003 : algo={ Id=6, tileIdx=18 (128x64) splitK=0 reduc=0 swizzle=0 custom=0 stages=12} status 0 time 0.017992ms workspace=0 mathMode=0 waves=0.296296\n",
      "result 004 : algo={ Id=6, tileIdx=18 (128x64) splitK=0 reduc=0 swizzle=0 custom=0 stages=16} status 0 time 0.018063ms workspace=0 mathMode=0 waves=0.592593\n",
      "result 005 : algo={ Id=6, tileIdx=18 (128x64) splitK=0 reduc=0 swizzle=1 custom=0 stages=16} status 0 time 0.018074ms workspace=0 mathMode=0 waves=0.592593\n",
      "result 006 : algo={ Id=21, tileIdx=18 (128x64) splitK=1 reduc=0 swizzle=0 custom=0 stages=18} status 0 time 0.018708ms workspace=0 mathMode=0 waves=0.592593\n",
      "result 007 : algo={ Id=6, tileIdx=20 (128x128) splitK=0 reduc=0 swizzle=1 custom=0 stages=11} status 0 time 0.030874ms workspace=0 mathMode=0 waves=0.148148\n",
      "result 008 : algo={ Id=6, tileIdx=20 (128x128) splitK=0 reduc=0 swizzle=0 custom=0 stages=11} status 0 time 0.030894ms workspace=0 mathMode=0 waves=0.148148\n",
      "result 009 : algo={ Id=6, tileIdx=20 (128x128) splitK=0 reduc=0 swizzle=0 custom=0 stages=15} status 0 time 0.031119ms workspace=0 mathMode=0 waves=0.296296\n",
      "result 010 : algo={ Id=6, tileIdx=20 (128x128) splitK=0 reduc=0 swizzle=1 custom=0 stages=15} status 0 time 0.031130ms workspace=0 mathMode=0 waves=0.296296\n",
      "result 011 : algo={ Id=6, tileIdx=18 (128x64) splitK=0 reduc=0 swizzle=0 custom=0 stages=0} status 0 time 0.044227ms workspace=0 mathMode=0 waves=0.148148\n",
      "result 012 : algo={ Id=6, tileIdx=18 (128x64) splitK=0 reduc=0 swizzle=1 custom=0 stages=0} status 0 time 0.044288ms workspace=0 mathMode=0 waves=0.148148\n",
      "result 013 : algo={ Id=5, tileIdx=20 (128x128) splitK=0 reduc=0 swizzle=1 custom=0 stages=0} status 0 time 0.053944ms workspace=0 mathMode=0 waves=0.148148\n",
      "result 014 : algo={ Id=5, tileIdx=20 (128x128) splitK=0 reduc=0 swizzle=0 custom=0 stages=0} status 0 time 0.053955ms workspace=0 mathMode=0 waves=0.148148\n",
      "result 015 : algo={ Id=6, tileIdx=24 (256x128) splitK=0 reduc=0 swizzle=0 custom=0 stages=9} status 0 time 0.054518ms workspace=0 mathMode=0 waves=0.148148\n",
      "result 016 : algo={ Id=6, tileIdx=24 (256x128) splitK=0 reduc=0 swizzle=0 custom=0 stages=15} status 0 time 0.054866ms workspace=0 mathMode=0 waves=0.148148\n",
      "result 017 : algo={ Id=6, tileIdx=24 (256x128) splitK=0 reduc=0 swizzle=1 custom=0 stages=15} status 0 time 0.054876ms workspace=0 mathMode=0 waves=0.148148\n",
      "result 018 : algo={ Id=6, tileIdx=23 (128x256) splitK=0 reduc=0 swizzle=1 custom=0 stages=15} status 0 time 0.055214ms workspace=0 mathMode=0 waves=0.296296\n",
      "result 019 : algo={ Id=6, tileIdx=23 (128x256) splitK=0 reduc=0 swizzle=0 custom=0 stages=15} status 0 time 0.055245ms workspace=0 mathMode=0 waves=0.296296\n",
      "result 020 : algo={ Id=6, tileIdx=23 (128x256) splitK=0 reduc=0 swizzle=0 custom=0 stages=0} status 0 time 0.056003ms workspace=0 mathMode=0 waves=0.296296\n",
      "result 021 : algo={ Id=6, tileIdx=23 (128x256) splitK=0 reduc=0 swizzle=1 custom=0 stages=0} status 0 time 0.056125ms workspace=0 mathMode=0 waves=0.296296\n",
      "result 022 : algo={ Id=5, tileIdx=20 (128x128) splitK=0 reduc=0 swizzle=0 custom=0 stages=7} status 0 time 0.057364ms workspace=0 mathMode=0 waves=0.148148\n",
      "result 023 : algo={ Id=5, tileIdx=20 (128x128) splitK=0 reduc=0 swizzle=1 custom=0 stages=7} status 0 time 0.057426ms workspace=0 mathMode=0 waves=0.148148\n",
      "result 024 : algo={ Id=5, tileIdx=21 (256x64) splitK=0 reduc=0 swizzle=0 custom=0 stages=7} status 0 time 0.057487ms workspace=0 mathMode=0 waves=0.148148\n",
      "result 025 : algo={ Id=5, tileIdx=21 (256x64) splitK=0 reduc=0 swizzle=1 custom=0 stages=7} status 0 time 0.057559ms workspace=0 mathMode=0 waves=0.148148\n",
      "result 026 : algo={ Id=5, tileIdx=21 (256x64) splitK=0 reduc=0 swizzle=0 custom=0 stages=0} status 0 time 0.057856ms workspace=0 mathMode=0 waves=0.148148\n",
      "result 027 : algo={ Id=5, tileIdx=21 (256x64) splitK=0 reduc=0 swizzle=1 custom=0 stages=0} status 0 time 0.057866ms workspace=0 mathMode=0 waves=0.148148\n",
      "result 028 : algo={ Id=5, tileIdx=23 (128x256) splitK=0 reduc=0 swizzle=0 custom=0 stages=7} status 0 time 0.067318ms workspace=0 mathMode=0 waves=0.296296\n",
      "result 029 : algo={ Id=5, tileIdx=23 (128x256) splitK=0 reduc=0 swizzle=1 custom=0 stages=7} status 0 time 0.067318ms workspace=0 mathMode=0 waves=0.296296\n",
      "result 030 : algo={ Id=5, tileIdx=24 (256x128) splitK=0 reduc=0 swizzle=0 custom=0 stages=0} status 0 time 0.067809ms workspace=0 mathMode=0 waves=0.148148\n",
      "result 031 : algo={ Id=5, tileIdx=24 (256x128) splitK=0 reduc=0 swizzle=1 custom=0 stages=0} status 0 time 0.067840ms workspace=0 mathMode=0 waves=0.148148\n",
      "result 032 : algo={ Id=5, tileIdx=23 (128x256) splitK=0 reduc=0 swizzle=1 custom=0 stages=0} status 0 time 0.068188ms workspace=0 mathMode=0 waves=0.296296\n",
      "result 033 : algo={ Id=5, tileIdx=23 (128x256) splitK=0 reduc=0 swizzle=0 custom=0 stages=0} status 0 time 0.068229ms workspace=0 mathMode=0 waves=0.296296\n",
      "result 034 : algo={ Id=1, tileIdx=11 (32x32) splitK=0 reduc=0 swizzle=1 custom=0 stages=0} status 0 time 0.146053ms workspace=0 mathMode=0 waves=1.185185\n",
      "result 035 : algo={ Id=1, tileIdx=11 (32x32) splitK=0 reduc=0 swizzle=0 custom=0 stages=0} status 0 time 0.146074ms workspace=0 mathMode=0 waves=1.185185\n",
      "result 036 : algo={ Id=1, tileIdx=13 (64x32) splitK=0 reduc=0 swizzle=1 custom=0 stages=0} status 0 time 0.155914ms workspace=0 mathMode=0 waves=1.185185\n",
      "result 037 : algo={ Id=1, tileIdx=13 (64x32) splitK=0 reduc=0 swizzle=0 custom=0 stages=0} status 0 time 0.155945ms workspace=0 mathMode=0 waves=1.185185\n",
      "result 038 : algo={ Id=0, tileIdx=16 (128x32) splitK=0 reduc=0 swizzle=1 custom=0 stages=0} status 0 time 0.204411ms workspace=0 mathMode=0 waves=0.296296\n",
      "result 039 : algo={ Id=0, tileIdx=16 (128x32) splitK=0 reduc=0 swizzle=0 custom=0 stages=0} status 0 time 0.204667ms workspace=0 mathMode=0 waves=0.296296\n",
      "result 040 : algo={ Id=1, tileIdx=16 (128x32) splitK=0 reduc=0 swizzle=1 custom=0 stages=0} status 0 time 0.207135ms workspace=0 mathMode=0 waves=1.185185\n",
      "result 041 : algo={ Id=1, tileIdx=16 (128x32) splitK=0 reduc=0 swizzle=0 custom=0 stages=0} status 0 time 0.207135ms workspace=0 mathMode=0 waves=1.185185\n",
      "result 042 : algo={ Id=0, tileIdx=14 (32x128) splitK=0 reduc=0 swizzle=0 custom=0 stages=0} status 0 time 0.212316ms workspace=0 mathMode=0 waves=0.296296\n",
      "result 043 : algo={ Id=0, tileIdx=14 (32x128) splitK=0 reduc=0 swizzle=1 custom=0 stages=0} status 0 time 0.212378ms workspace=0 mathMode=0 waves=0.296296\n",
      "result 044 : algo={ Id=0, tileIdx=18 (128x64) splitK=0 reduc=0 swizzle=0 custom=0 stages=0} status 0 time 0.217201ms workspace=0 mathMode=0 waves=0.148148\n",
      "result 045 : algo={ Id=0, tileIdx=18 (128x64) splitK=0 reduc=0 swizzle=1 custom=0 stages=0} status 0 time 0.217221ms workspace=0 mathMode=0 waves=0.148148\n",
      "result 046 : algo={ Id=0, tileIdx=20 (128x128) splitK=0 reduc=0 swizzle=0 custom=0 stages=0} status 0 time 0.386396ms workspace=0 mathMode=0 waves=0.148148\n",
      "result 047 : algo={ Id=0, tileIdx=20 (128x128) splitK=0 reduc=0 swizzle=1 custom=0 stages=0} status 0 time 0.386632ms workspace=0 mathMode=0 waves=0.148148\n",
      "result 048 : algo={ Id=0, tileIdx=15 (64x64) splitK=0 reduc=0 swizzle=0 custom=0 stages=0} status 0 time 0.393759ms workspace=0 mathMode=0 waves=0.148148\n",
      "result 049 : algo={ Id=0, tileIdx=15 (64x64) splitK=0 reduc=0 swizzle=1 custom=0 stages=0} status 0 time 0.393953ms workspace=0 mathMode=0 waves=0.148148\n",
      "algo={ Id=6, tileIdx=18 (128x64) splitK=0 reduc=0 swizzle=1 custom=0 stages=15} status 0 time 0.017961ms workspace=0 mathMode=0 waves=0.296296\n",
      "***cublasLt Gemm Testing End***\n",
      "\n",
      "-----------------------------\n",
      "GEMM test 4: [M: 128, K: 4096, N: 8192] context ffn gemm 1\n",
      "algo_99 costs 0.058ms \n",
      "algo_100 costs 0.057ms \n",
      "algo_101 costs 0.057ms \n",
      "algo_102 costs 0.057ms \n",
      "algo_103 costs 0.057ms \n",
      "algo_104 costs 0.057ms \n",
      "algo_105 costs 0.057ms \n",
      "algo_106 costs 0.057ms \n",
      "algo_107 costs 0.057ms \n",
      "algo_108 costs 0.057ms \n",
      "algo_109 costs 0.057ms \n",
      "algo_110 costs 0.057ms \n",
      "algo_111 costs 0.057ms \n",
      "algo_112 costs 0.057ms \n",
      "algo_113 costs 0.057ms \n",
      "algo_114 costs 0.057ms \n",
      "algo_115 costs 0.057ms \n",
      "fast_algo 109 costs 0.057 ms\n",
      "***cublasLt Gemm Testing Beign***\n",
      "AlgoCount: 3813\n",
      "return 2 and run heuristic algo\n",
      "workspacesize==0, run 170 algos\n",
      "result 000 : algo={ Id=6, tileIdx=21 (256x64) splitK=1 reduc=0 swizzle=1 custom=0 stages=15} status 0 time 0.056945ms workspace=0 mathMode=0 waves=0.592593\n",
      "result 001 : algo={ Id=6, tileIdx=20 (128x128) splitK=0 reduc=0 swizzle=1 custom=0 stages=11} status 0 time 0.057477ms workspace=0 mathMode=0 waves=0.296296\n",
      "result 002 : algo={ Id=6, tileIdx=20 (128x128) splitK=0 reduc=0 swizzle=0 custom=0 stages=11} status 0 time 0.057508ms workspace=0 mathMode=0 waves=0.296296\n",
      "result 003 : algo={ Id=6, tileIdx=20 (128x128) splitK=1 reduc=0 swizzle=0 custom=0 stages=15} status 0 time 0.060314ms workspace=0 mathMode=0 waves=0.592593\n",
      "result 004 : algo={ Id=6, tileIdx=20 (128x128) splitK=0 reduc=0 swizzle=0 custom=0 stages=15} status 0 time 0.060559ms workspace=0 mathMode=0 waves=0.592593\n",
      "result 005 : algo={ Id=6, tileIdx=20 (128x128) splitK=0 reduc=0 swizzle=1 custom=0 stages=15} status 0 time 0.060641ms workspace=0 mathMode=0 waves=0.592593\n",
      "result 006 : algo={ Id=6, tileIdx=18 (128x64) splitK=0 reduc=0 swizzle=0 custom=0 stages=15} status 0 time 0.062638ms workspace=0 mathMode=0 waves=0.592593\n",
      "result 007 : algo={ Id=6, tileIdx=18 (128x64) splitK=0 reduc=0 swizzle=1 custom=0 stages=15} status 0 time 0.063130ms workspace=0 mathMode=0 waves=0.592593\n",
      "result 008 : algo={ Id=6, tileIdx=18 (128x64) splitK=0 reduc=0 swizzle=1 custom=0 stages=12} status 0 time 0.066007ms workspace=0 mathMode=0 waves=0.592593\n",
      "result 009 : algo={ Id=6, tileIdx=18 (128x64) splitK=0 reduc=0 swizzle=0 custom=0 stages=12} status 0 time 0.067645ms workspace=0 mathMode=0 waves=0.592593\n",
      "result 010 : algo={ Id=6, tileIdx=18 (128x64) splitK=0 reduc=0 swizzle=1 custom=0 stages=16} status 0 time 0.074496ms workspace=0 mathMode=0 waves=1.185185\n",
      "result 011 : algo={ Id=6, tileIdx=18 (128x64) splitK=0 reduc=0 swizzle=0 custom=0 stages=16} status 0 time 0.074762ms workspace=0 mathMode=0 waves=1.185185\n",
      "result 012 : algo={ Id=6, tileIdx=24 (256x128) splitK=0 reduc=0 swizzle=0 custom=0 stages=15} status 0 time 0.103107ms workspace=0 mathMode=0 waves=0.296296\n",
      "result 013 : algo={ Id=6, tileIdx=24 (256x128) splitK=0 reduc=0 swizzle=1 custom=0 stages=15} status 0 time 0.103158ms workspace=0 mathMode=0 waves=0.296296\n",
      "result 014 : algo={ Id=6, tileIdx=23 (128x256) splitK=0 reduc=0 swizzle=0 custom=0 stages=15} status 0 time 0.103690ms workspace=0 mathMode=0 waves=0.592593\n",
      "result 015 : algo={ Id=6, tileIdx=23 (128x256) splitK=0 reduc=0 swizzle=1 custom=0 stages=15} status 0 time 0.103711ms workspace=0 mathMode=0 waves=0.592593\n",
      "result 016 : algo={ Id=5, tileIdx=20 (128x128) splitK=0 reduc=0 swizzle=1 custom=0 stages=0} status 0 time 0.108431ms workspace=0 mathMode=0 waves=0.296296\n",
      "result 017 : algo={ Id=5, tileIdx=20 (128x128) splitK=0 reduc=0 swizzle=0 custom=0 stages=0} status 0 time 0.108483ms workspace=0 mathMode=0 waves=0.296296\n",
      "result 018 : algo={ Id=5, tileIdx=21 (256x64) splitK=0 reduc=0 swizzle=0 custom=0 stages=7} status 0 time 0.114739ms workspace=0 mathMode=0 waves=0.296296\n",
      "result 019 : algo={ Id=5, tileIdx=21 (256x64) splitK=0 reduc=0 swizzle=1 custom=0 stages=7} status 0 time 0.114811ms workspace=0 mathMode=0 waves=0.296296\n",
      "result 020 : algo={ Id=5, tileIdx=20 (128x128) splitK=0 reduc=0 swizzle=1 custom=0 stages=7} status 0 time 0.118948ms workspace=0 mathMode=0 waves=0.296296\n",
      "result 021 : algo={ Id=5, tileIdx=20 (128x128) splitK=0 reduc=0 swizzle=0 custom=0 stages=7} status 0 time 0.119091ms workspace=0 mathMode=0 waves=0.296296\n",
      "result 022 : algo={ Id=6, tileIdx=23 (128x256) splitK=0 reduc=0 swizzle=1 custom=0 stages=0} status 0 time 0.125276ms workspace=0 mathMode=0 waves=0.592593\n",
      "result 023 : algo={ Id=6, tileIdx=23 (128x256) splitK=0 reduc=0 swizzle=0 custom=0 stages=0} status 0 time 0.125286ms workspace=0 mathMode=0 waves=0.592593\n",
      "result 024 : algo={ Id=5, tileIdx=23 (128x256) splitK=0 reduc=0 swizzle=0 custom=0 stages=0} status 0 time 0.127427ms workspace=0 mathMode=0 waves=0.592593\n",
      "result 025 : algo={ Id=5, tileIdx=23 (128x256) splitK=0 reduc=0 swizzle=1 custom=0 stages=0} status 0 time 0.127427ms workspace=0 mathMode=0 waves=0.592593\n",
      "result 026 : algo={ Id=6, tileIdx=18 (128x64) splitK=0 reduc=0 swizzle=0 custom=0 stages=0} status 0 time 0.132096ms workspace=0 mathMode=0 waves=0.296296\n",
      "result 027 : algo={ Id=5, tileIdx=24 (256x128) splitK=0 reduc=0 swizzle=0 custom=0 stages=0} status 0 time 0.133806ms workspace=0 mathMode=0 waves=0.296296\n",
      "result 028 : algo={ Id=5, tileIdx=24 (256x128) splitK=0 reduc=0 swizzle=1 custom=0 stages=0} status 0 time 0.133827ms workspace=0 mathMode=0 waves=0.296296\n",
      "result 029 : algo={ Id=5, tileIdx=23 (128x256) splitK=0 reduc=0 swizzle=0 custom=0 stages=7} status 0 time 0.135885ms workspace=0 mathMode=0 waves=0.592593\n",
      "result 030 : algo={ Id=5, tileIdx=23 (128x256) splitK=0 reduc=0 swizzle=1 custom=0 stages=7} status 0 time 0.135905ms workspace=0 mathMode=0 waves=0.592593\n",
      "result 031 : algo={ Id=6, tileIdx=18 (128x64) splitK=0 reduc=0 swizzle=1 custom=0 stages=0} status 0 time 0.136796ms workspace=0 mathMode=0 waves=0.296296\n",
      "result 032 : algo={ Id=5, tileIdx=21 (256x64) splitK=0 reduc=0 swizzle=0 custom=0 stages=0} status 0 time 0.146391ms workspace=0 mathMode=0 waves=0.296296\n",
      "result 033 : algo={ Id=5, tileIdx=21 (256x64) splitK=0 reduc=0 swizzle=1 custom=0 stages=0} status 0 time 0.146545ms workspace=0 mathMode=0 waves=0.296296\n",
      "result 034 : algo={ Id=1, tileIdx=13 (64x32) splitK=0 reduc=0 swizzle=1 custom=0 stages=0} status 0 time 0.519649ms workspace=0 mathMode=0 waves=2.370370\n",
      "result 035 : algo={ Id=1, tileIdx=13 (64x32) splitK=0 reduc=0 swizzle=0 custom=0 stages=0} status 0 time 0.527421ms workspace=0 mathMode=0 waves=2.370370\n",
      "result 036 : algo={ Id=1, tileIdx=11 (32x32) splitK=0 reduc=0 swizzle=1 custom=0 stages=0} status 0 time 0.560497ms workspace=0 mathMode=0 waves=2.370370\n",
      "result 037 : algo={ Id=1, tileIdx=11 (32x32) splitK=0 reduc=0 swizzle=0 custom=0 stages=0} status 0 time 0.578693ms workspace=0 mathMode=0 waves=2.370370\n",
      "result 038 : algo={ Id=1, tileIdx=16 (128x32) splitK=0 reduc=0 swizzle=1 custom=0 stages=0} status 0 time 0.606689ms workspace=0 mathMode=0 waves=2.370370\n",
      "result 039 : algo={ Id=1, tileIdx=16 (128x32) splitK=0 reduc=0 swizzle=0 custom=0 stages=0} status 0 time 0.606740ms workspace=0 mathMode=0 waves=2.370370\n",
      "result 040 : algo={ Id=0, tileIdx=14 (32x128) splitK=0 reduc=0 swizzle=1 custom=0 stages=0} status 0 time 0.682230ms workspace=0 mathMode=0 waves=0.592593\n",
      "result 041 : algo={ Id=0, tileIdx=14 (32x128) splitK=0 reduc=0 swizzle=0 custom=0 stages=0} status 0 time 0.682629ms workspace=0 mathMode=0 waves=0.592593\n",
      "result 042 : algo={ Id=0, tileIdx=16 (128x32) splitK=0 reduc=0 swizzle=1 custom=0 stages=0} status 0 time 0.691231ms workspace=0 mathMode=0 waves=0.592593\n",
      "result 043 : algo={ Id=0, tileIdx=16 (128x32) splitK=0 reduc=0 swizzle=0 custom=0 stages=0} status 0 time 0.695378ms workspace=0 mathMode=0 waves=0.592593\n",
      "result 044 : algo={ Id=0, tileIdx=20 (128x128) splitK=0 reduc=0 swizzle=0 custom=0 stages=0} status 0 time 0.766577ms workspace=0 mathMode=0 waves=0.296296\n",
      "result 045 : algo={ Id=0, tileIdx=20 (128x128) splitK=0 reduc=0 swizzle=1 custom=0 stages=0} status 0 time 0.766587ms workspace=0 mathMode=0 waves=0.296296\n",
      "result 046 : algo={ Id=0, tileIdx=18 (128x64) splitK=0 reduc=0 swizzle=1 custom=0 stages=0} status 0 time 0.784794ms workspace=0 mathMode=0 waves=0.296296\n",
      "result 047 : algo={ Id=0, tileIdx=18 (128x64) splitK=0 reduc=0 swizzle=0 custom=0 stages=0} status 0 time 0.792494ms workspace=0 mathMode=0 waves=0.296296\n",
      "result 048 : algo={ Id=0, tileIdx=15 (64x64) splitK=0 reduc=0 swizzle=1 custom=0 stages=0} status 0 time 0.803676ms workspace=0 mathMode=0 waves=0.296296\n",
      "result 049 : algo={ Id=0, tileIdx=15 (64x64) splitK=0 reduc=0 swizzle=0 custom=0 stages=0} status 0 time 0.805888ms workspace=0 mathMode=0 waves=0.296296\n",
      "algo={ Id=6, tileIdx=21 (256x64) splitK=1 reduc=0 swizzle=1 custom=0 stages=15} status 0 time 0.056945ms workspace=0 mathMode=0 waves=0.592593\n",
      "***cublasLt Gemm Testing End***\n",
      "\n",
      "-----------------------------\n",
      "GEMM test 5: [M: 128, K: 8192, N: 4096] context ffn gemm 2\n",
      "algo_99 costs 0.062ms \n",
      "algo_100 costs 0.061ms \n",
      "algo_101 costs 0.061ms \n",
      "algo_102 costs 0.061ms \n",
      "algo_103 costs 0.061ms \n",
      "algo_104 costs 0.061ms \n",
      "algo_105 costs 0.061ms \n",
      "algo_106 costs 0.061ms \n",
      "algo_107 costs 0.061ms \n",
      "algo_108 costs 0.061ms \n",
      "algo_109 costs 0.061ms \n",
      "algo_110 costs 0.061ms \n",
      "algo_111 costs 0.061ms \n",
      "algo_112 costs 0.061ms \n",
      "algo_113 costs 0.061ms \n",
      "algo_114 costs 0.061ms \n",
      "algo_115 costs 0.061ms \n",
      "fast_algo 114 costs 0.061 ms\n",
      "***cublasLt Gemm Testing Beign***\n",
      "AlgoCount: 4287\n",
      "return 7 and run heuristic algo\n",
      "workspacesize==0, run 170 algos\n",
      "result 000 : algo={ Id=6, tileIdx=18 (128x64) splitK=0 reduc=0 swizzle=0 custom=0 stages=12} status 0 time 0.058122ms workspace=0 mathMode=0 waves=0.296296\n",
      "result 001 : algo={ Id=6, tileIdx=17 (64x128) splitK=1 reduc=0 swizzle=0 custom=0 stages=16} status 0 time 0.062484ms workspace=0 mathMode=0 waves=0.592593\n",
      "result 002 : algo={ Id=6, tileIdx=17 (64x128) splitK=3 reduc=4 swizzle=0 custom=0 stages=16} status 0 time 0.072663ms workspace=3145728 mathMode=0 waves=1.777778\n",
      "result 003 : algo={ Id=6, tileIdx=18 (128x64) splitK=0 reduc=0 swizzle=1 custom=0 stages=15} status 0 time 0.072888ms workspace=0 mathMode=0 waves=0.296296\n",
      "result 004 : algo={ Id=6, tileIdx=18 (128x64) splitK=0 reduc=0 swizzle=0 custom=0 stages=15} status 0 time 0.073093ms workspace=0 mathMode=0 waves=0.296296\n",
      "result 005 : algo={ Id=6, tileIdx=24 (256x128) splitK=5 reduc=1 swizzle=1 custom=0 stages=15} status 0 time 0.075039ms workspace=64 mathMode=0 waves=0.740741\n",
      "result 006 : algo={ Id=6, tileIdx=24 (256x128) splitK=4 reduc=1 swizzle=1 custom=0 stages=15} status 0 time 0.075469ms workspace=64 mathMode=0 waves=0.592593\n",
      "result 007 : algo={ Id=6, tileIdx=20 (128x128) splitK=0 reduc=0 swizzle=1 custom=0 stages=15} status 0 time 0.103608ms workspace=0 mathMode=0 waves=0.296296\n",
      "result 008 : algo={ Id=6, tileIdx=20 (128x128) splitK=0 reduc=0 swizzle=0 custom=0 stages=15} status 0 time 0.103608ms workspace=0 mathMode=0 waves=0.296296\n",
      "result 009 : algo={ Id=6, tileIdx=20 (128x128) splitK=0 reduc=0 swizzle=1 custom=0 stages=11} status 0 time 0.104120ms workspace=0 mathMode=0 waves=0.148148\n",
      "result 010 : algo={ Id=6, tileIdx=20 (128x128) splitK=0 reduc=0 swizzle=0 custom=0 stages=11} status 0 time 0.104161ms workspace=0 mathMode=0 waves=0.148148\n",
      "result 011 : algo={ Id=6, tileIdx=20 (128x128) splitK=1 reduc=0 swizzle=1 custom=0 stages=15} status 0 time 0.105994ms workspace=0 mathMode=0 waves=0.296296\n",
      "result 012 : algo={ Id=23, tileIdx=20 (128x128) splitK=2 reduc=4 swizzle=0 custom=0 stages=15} status 0 time 0.107264ms workspace=2097152 mathMode=0 waves=0.592593\n",
      "result 013 : algo={ Id=6, tileIdx=23 (128x256) splitK=2 reduc=1 swizzle=1 custom=0 stages=15} status 0 time 0.111544ms workspace=128 mathMode=0 waves=0.592593\n",
      "result 014 : algo={ Id=5, tileIdx=20 (128x128) splitK=0 reduc=0 swizzle=0 custom=0 stages=0} status 0 time 0.198236ms workspace=0 mathMode=0 waves=0.148148\n",
      "result 015 : algo={ Id=5, tileIdx=20 (128x128) splitK=0 reduc=0 swizzle=1 custom=0 stages=0} status 0 time 0.198267ms workspace=0 mathMode=0 waves=0.148148\n",
      "result 016 : algo={ Id=6, tileIdx=23 (128x256) splitK=0 reduc=0 swizzle=1 custom=0 stages=15} status 0 time 0.198482ms workspace=0 mathMode=0 waves=0.296296\n",
      "result 017 : algo={ Id=6, tileIdx=23 (128x256) splitK=0 reduc=0 swizzle=0 custom=0 stages=15} status 0 time 0.198513ms workspace=0 mathMode=0 waves=0.296296\n",
      "result 018 : algo={ Id=5, tileIdx=21 (256x64) splitK=0 reduc=0 swizzle=0 custom=0 stages=7} status 0 time 0.216269ms workspace=0 mathMode=0 waves=0.148148\n",
      "result 019 : algo={ Id=5, tileIdx=21 (256x64) splitK=0 reduc=0 swizzle=1 custom=0 stages=7} status 0 time 0.217057ms workspace=0 mathMode=0 waves=0.148148\n",
      "result 020 : algo={ Id=6, tileIdx=23 (128x256) splitK=0 reduc=0 swizzle=1 custom=0 stages=0} status 0 time 0.221256ms workspace=0 mathMode=0 waves=0.296296\n",
      "result 021 : algo={ Id=6, tileIdx=23 (128x256) splitK=0 reduc=0 swizzle=0 custom=0 stages=0} status 0 time 0.221297ms workspace=0 mathMode=0 waves=0.296296\n",
      "result 022 : algo={ Id=5, tileIdx=20 (128x128) splitK=0 reduc=0 swizzle=1 custom=0 stages=7} status 0 time 0.222566ms workspace=0 mathMode=0 waves=0.148148\n",
      "result 023 : algo={ Id=5, tileIdx=20 (128x128) splitK=0 reduc=0 swizzle=0 custom=0 stages=7} status 0 time 0.222597ms workspace=0 mathMode=0 waves=0.148148\n",
      "result 024 : algo={ Id=6, tileIdx=18 (128x64) splitK=0 reduc=0 swizzle=1 custom=0 stages=0} status 0 time 0.233339ms workspace=0 mathMode=0 waves=0.148148\n",
      "result 025 : algo={ Id=6, tileIdx=18 (128x64) splitK=0 reduc=0 swizzle=0 custom=0 stages=0} status 0 time 0.234004ms workspace=0 mathMode=0 waves=0.148148\n",
      "result 026 : algo={ Id=5, tileIdx=23 (128x256) splitK=0 reduc=0 swizzle=0 custom=0 stages=0} status 0 time 0.246118ms workspace=0 mathMode=0 waves=0.296296\n",
      "result 027 : algo={ Id=5, tileIdx=23 (128x256) splitK=0 reduc=0 swizzle=1 custom=0 stages=0} status 0 time 0.246139ms workspace=0 mathMode=0 waves=0.296296\n",
      "result 028 : algo={ Id=5, tileIdx=24 (256x128) splitK=0 reduc=0 swizzle=1 custom=0 stages=0} status 0 time 0.257700ms workspace=0 mathMode=0 waves=0.148148\n",
      "result 029 : algo={ Id=5, tileIdx=24 (256x128) splitK=0 reduc=0 swizzle=0 custom=0 stages=0} status 0 time 0.257751ms workspace=0 mathMode=0 waves=0.148148\n",
      "result 030 : algo={ Id=5, tileIdx=23 (128x256) splitK=0 reduc=0 swizzle=1 custom=0 stages=7} status 0 time 0.259809ms workspace=0 mathMode=0 waves=0.296296\n",
      "result 031 : algo={ Id=5, tileIdx=23 (128x256) splitK=0 reduc=0 swizzle=0 custom=0 stages=7} status 0 time 0.259840ms workspace=0 mathMode=0 waves=0.296296\n",
      "result 032 : algo={ Id=5, tileIdx=21 (256x64) splitK=0 reduc=0 swizzle=0 custom=0 stages=0} status 0 time 0.278927ms workspace=0 mathMode=0 waves=0.148148\n",
      "result 033 : algo={ Id=5, tileIdx=21 (256x64) splitK=0 reduc=0 swizzle=1 custom=0 stages=0} status 0 time 0.280760ms workspace=0 mathMode=0 waves=0.148148\n",
      "result 034 : algo={ Id=1, tileIdx=11 (32x32) splitK=0 reduc=0 swizzle=1 custom=0 stages=0} status 0 time 0.616550ms workspace=0 mathMode=0 waves=1.185185\n",
      "result 035 : algo={ Id=1, tileIdx=13 (64x32) splitK=0 reduc=0 swizzle=1 custom=0 stages=0} status 0 time 0.632013ms workspace=0 mathMode=0 waves=1.185185\n",
      "result 036 : algo={ Id=1, tileIdx=11 (32x32) splitK=0 reduc=0 swizzle=0 custom=0 stages=0} status 0 time 0.636887ms workspace=0 mathMode=0 waves=1.185185\n",
      "result 037 : algo={ Id=1, tileIdx=13 (64x32) splitK=0 reduc=0 swizzle=0 custom=0 stages=0} status 0 time 0.644526ms workspace=0 mathMode=0 waves=1.185185\n",
      "result 038 : algo={ Id=1, tileIdx=16 (128x32) splitK=0 reduc=0 swizzle=1 custom=0 stages=0} status 0 time 0.797460ms workspace=0 mathMode=0 waves=1.185185\n",
      "result 039 : algo={ Id=1, tileIdx=16 (128x32) splitK=0 reduc=0 swizzle=0 custom=0 stages=0} status 0 time 0.797645ms workspace=0 mathMode=0 waves=1.185185\n",
      "result 040 : algo={ Id=0, tileIdx=16 (128x32) splitK=0 reduc=0 swizzle=1 custom=0 stages=0} status 0 time 0.852582ms workspace=0 mathMode=0 waves=0.296296\n",
      "result 041 : algo={ Id=0, tileIdx=16 (128x32) splitK=0 reduc=0 swizzle=0 custom=0 stages=0} status 0 time 0.855357ms workspace=0 mathMode=0 waves=0.296296\n",
      "result 042 : algo={ Id=0, tileIdx=14 (32x128) splitK=0 reduc=0 swizzle=1 custom=0 stages=0} status 0 time 0.897935ms workspace=0 mathMode=0 waves=0.296296\n",
      "result 043 : algo={ Id=0, tileIdx=14 (32x128) splitK=0 reduc=0 swizzle=0 custom=0 stages=0} status 0 time 0.898304ms workspace=0 mathMode=0 waves=0.296296\n",
      "result 044 : algo={ Id=0, tileIdx=18 (128x64) splitK=0 reduc=0 swizzle=1 custom=0 stages=0} status 0 time 0.935414ms workspace=0 mathMode=0 waves=0.148148\n",
      "result 045 : algo={ Id=0, tileIdx=18 (128x64) splitK=0 reduc=0 swizzle=0 custom=0 stages=0} status 0 time 0.935485ms workspace=0 mathMode=0 waves=0.148148\n",
      "result 046 : algo={ Id=0, tileIdx=20 (128x128) splitK=0 reduc=0 swizzle=0 custom=0 stages=0} status 0 time 1.519503ms workspace=0 mathMode=0 waves=0.148148\n",
      "result 047 : algo={ Id=0, tileIdx=20 (128x128) splitK=0 reduc=0 swizzle=1 custom=0 stages=0} status 0 time 1.521254ms workspace=0 mathMode=0 waves=0.148148\n",
      "result 048 : algo={ Id=0, tileIdx=15 (64x64) splitK=0 reduc=0 swizzle=1 custom=0 stages=0} status 0 time 1.554432ms workspace=0 mathMode=0 waves=0.148148\n",
      "result 049 : algo={ Id=0, tileIdx=15 (64x64) splitK=0 reduc=0 swizzle=0 custom=0 stages=0} status 0 time 1.555958ms workspace=0 mathMode=0 waves=0.148148\n",
      "algo={ Id=6, tileIdx=18 (128x64) splitK=0 reduc=0 swizzle=0 custom=0 stages=12} status 0 time 0.058122ms workspace=0 mathMode=0 waves=0.296296\n",
      "***cublasLt Gemm Testing End***\n",
      "\n",
      "-----------------------------\n",
      "GEMM test 6: [M: 1, K: 4096, N: 6144] from_tensor * weightQKV\n",
      "algo_99 costs 0.042ms \n",
      "algo_100 costs 0.042ms \n",
      "algo_101 costs 0.042ms \n",
      "algo_102 costs 0.042ms \n",
      "algo_103 costs 0.042ms \n",
      "algo_104 costs 0.042ms \n",
      "algo_105 costs 0.042ms \n",
      "algo_106 costs 0.042ms \n",
      "algo_107 costs 0.042ms \n",
      "algo_108 costs 0.042ms \n",
      "algo_109 costs 0.042ms \n",
      "algo_110 costs 0.042ms \n",
      "algo_111 costs 0.042ms \n",
      "algo_112 costs 0.042ms \n",
      "algo_113 costs 0.042ms \n",
      "algo_114 costs 0.042ms \n",
      "algo_115 costs 0.042ms \n",
      "fast_algo 110 costs 0.042 ms\n",
      "***cublasLt Gemm Testing Beign***\n",
      "AlgoCount: 5000\n",
      "return 8 and run heuristic algo\n",
      "workspacesize==0, run 245 algos\n",
      "result 000 : algo={ Id=6, tileIdx=15 (64x64) splitK=1 reduc=0 swizzle=1 custom=0 stages=18} status 0 time 0.038615ms workspace=0 mathMode=0 waves=0.888889\n",
      "result 001 : algo={ Id=31, tileIdx=15 (64x64) splitK=1 reduc=0 swizzle=0 custom=0 stages=17} status 0 time 0.039475ms workspace=0 mathMode=0 waves=0.444444\n",
      "result 002 : algo={ Id=6, tileIdx=18 (128x64) splitK=0 reduc=0 swizzle=0 custom=0 stages=15} status 0 time 0.039680ms workspace=0 mathMode=0 waves=0.222222\n",
      "result 003 : algo={ Id=6, tileIdx=18 (128x64) splitK=0 reduc=0 swizzle=1 custom=0 stages=15} status 0 time 0.039690ms workspace=0 mathMode=0 waves=0.222222\n",
      "result 004 : algo={ Id=6, tileIdx=18 (128x64) splitK=4 reduc=4 swizzle=1 custom=0 stages=12} status 0 time 0.041308ms workspace=49152 mathMode=0 waves=0.888889\n",
      "result 005 : algo={ Id=6, tileIdx=18 (128x64) splitK=2 reduc=1 swizzle=1 custom=0 stages=12} status 0 time 0.041390ms workspace=192 mathMode=0 waves=0.444444\n",
      "result 006 : algo={ Id=21, tileIdx=21 (256x64) splitK=4 reduc=4 swizzle=0 custom=0 stages=10} status 0 time 0.042158ms workspace=49152 mathMode=0 waves=0.444444\n",
      "result 007 : algo={ Id=6, tileIdx=18 (128x64) splitK=3 reduc=4 swizzle=0 custom=0 stages=16} status 0 time 0.043264ms workspace=36864 mathMode=0 waves=1.333333\n",
      "result 008 : algo={ Id=6, tileIdx=18 (128x64) splitK=7 reduc=1 swizzle=0 custom=0 stages=16} status 0 time 0.045752ms workspace=192 mathMode=0 waves=3.111111\n",
      "result 009 : algo={ Id=6, tileIdx=20 (128x128) splitK=0 reduc=0 swizzle=1 custom=0 stages=15} status 0 time 0.055470ms workspace=0 mathMode=0 waves=0.444444\n",
      "result 010 : algo={ Id=6, tileIdx=20 (128x128) splitK=0 reduc=0 swizzle=0 custom=0 stages=15} status 0 time 0.055511ms workspace=0 mathMode=0 waves=0.444444\n",
      "result 011 : algo={ Id=6, tileIdx=20 (128x128) splitK=0 reduc=0 swizzle=1 custom=0 stages=11} status 0 time 0.055624ms workspace=0 mathMode=0 waves=0.222222\n",
      "result 012 : algo={ Id=6, tileIdx=20 (128x128) splitK=0 reduc=0 swizzle=0 custom=0 stages=11} status 0 time 0.055634ms workspace=0 mathMode=0 waves=0.222222\n",
      "result 013 : algo={ Id=13, tileIdx=0 (UNDEF) splitK=64 reduc=2 swizzle=0 custom=7 stages=0} status 0 time 0.061645ms workspace=1572880 mathMode=0 waves=2.844445\n",
      "result 014 : algo={ Id=5, tileIdx=20 (128x128) splitK=0 reduc=0 swizzle=1 custom=0 stages=0} status 0 time 0.102431ms workspace=0 mathMode=0 waves=0.222222\n",
      "result 015 : algo={ Id=5, tileIdx=20 (128x128) splitK=0 reduc=0 swizzle=0 custom=0 stages=0} status 0 time 0.102441ms workspace=0 mathMode=0 waves=0.222222\n",
      "result 016 : algo={ Id=6, tileIdx=23 (128x256) splitK=0 reduc=0 swizzle=1 custom=0 stages=15} status 0 time 0.103137ms workspace=0 mathMode=0 waves=0.444444\n",
      "result 017 : algo={ Id=6, tileIdx=23 (128x256) splitK=0 reduc=0 swizzle=0 custom=0 stages=15} status 0 time 0.103148ms workspace=0 mathMode=0 waves=0.444444\n",
      "result 018 : algo={ Id=5, tileIdx=20 (128x128) splitK=0 reduc=0 swizzle=0 custom=0 stages=7} status 0 time 0.109425ms workspace=0 mathMode=0 waves=0.222222\n",
      "result 019 : algo={ Id=5, tileIdx=20 (128x128) splitK=0 reduc=0 swizzle=1 custom=0 stages=7} status 0 time 0.109486ms workspace=0 mathMode=0 waves=0.222222\n",
      "result 020 : algo={ Id=6, tileIdx=23 (128x256) splitK=0 reduc=0 swizzle=1 custom=0 stages=0} status 0 time 0.109916ms workspace=0 mathMode=0 waves=0.444444\n",
      "result 021 : algo={ Id=6, tileIdx=23 (128x256) splitK=0 reduc=0 swizzle=0 custom=0 stages=0} status 0 time 0.109947ms workspace=0 mathMode=0 waves=0.444444\n",
      "result 022 : algo={ Id=5, tileIdx=21 (256x64) splitK=0 reduc=0 swizzle=1 custom=0 stages=7} status 0 time 0.112937ms workspace=0 mathMode=0 waves=0.111111\n",
      "result 023 : algo={ Id=5, tileIdx=21 (256x64) splitK=0 reduc=0 swizzle=0 custom=0 stages=7} status 0 time 0.113060ms workspace=0 mathMode=0 waves=0.111111\n",
      "result 024 : algo={ Id=6, tileIdx=18 (128x64) splitK=0 reduc=0 swizzle=1 custom=0 stages=0} status 0 time 0.123689ms workspace=0 mathMode=0 waves=0.111111\n",
      "result 025 : algo={ Id=6, tileIdx=18 (128x64) splitK=0 reduc=0 swizzle=0 custom=0 stages=0} status 0 time 0.123720ms workspace=0 mathMode=0 waves=0.111111\n",
      "result 026 : algo={ Id=5, tileIdx=23 (128x256) splitK=0 reduc=0 swizzle=0 custom=0 stages=0} status 0 time 0.125768ms workspace=0 mathMode=0 waves=0.444444\n",
      "result 027 : algo={ Id=5, tileIdx=23 (128x256) splitK=0 reduc=0 swizzle=1 custom=0 stages=0} status 0 time 0.125778ms workspace=0 mathMode=0 waves=0.444444\n",
      "result 028 : algo={ Id=5, tileIdx=24 (256x128) splitK=0 reduc=0 swizzle=0 custom=0 stages=0} status 0 time 0.126904ms workspace=0 mathMode=0 waves=0.222222\n",
      "result 029 : algo={ Id=5, tileIdx=24 (256x128) splitK=0 reduc=0 swizzle=1 custom=0 stages=0} status 0 time 0.126925ms workspace=0 mathMode=0 waves=0.222222\n",
      "result 030 : algo={ Id=5, tileIdx=23 (128x256) splitK=0 reduc=0 swizzle=1 custom=0 stages=7} status 0 time 0.127396ms workspace=0 mathMode=0 waves=0.444444\n",
      "result 031 : algo={ Id=5, tileIdx=23 (128x256) splitK=0 reduc=0 swizzle=0 custom=0 stages=7} status 0 time 0.127406ms workspace=0 mathMode=0 waves=0.444444\n",
      "result 032 : algo={ Id=1, tileIdx=13 (64x32) splitK=0 reduc=0 swizzle=1 custom=0 stages=0} status 0 time 0.141199ms workspace=0 mathMode=0 waves=0.444444\n",
      "result 033 : algo={ Id=1, tileIdx=13 (64x32) splitK=0 reduc=0 swizzle=0 custom=0 stages=0} status 0 time 0.141271ms workspace=0 mathMode=0 waves=0.444444\n",
      "result 034 : algo={ Id=1, tileIdx=11 (32x32) splitK=0 reduc=0 swizzle=1 custom=0 stages=0} status 0 time 0.141681ms workspace=0 mathMode=0 waves=0.444444\n",
      "result 035 : algo={ Id=1, tileIdx=11 (32x32) splitK=0 reduc=0 swizzle=0 custom=0 stages=0} status 0 time 0.141752ms workspace=0 mathMode=0 waves=0.444444\n",
      "result 036 : algo={ Id=5, tileIdx=21 (256x64) splitK=0 reduc=0 swizzle=1 custom=0 stages=0} status 0 time 0.147651ms workspace=0 mathMode=0 waves=0.111111\n",
      "result 037 : algo={ Id=5, tileIdx=21 (256x64) splitK=0 reduc=0 swizzle=0 custom=0 stages=0} status 0 time 0.147804ms workspace=0 mathMode=0 waves=0.111111\n",
      "result 038 : algo={ Id=1, tileIdx=16 (128x32) splitK=0 reduc=0 swizzle=0 custom=0 stages=0} status 0 time 0.204421ms workspace=0 mathMode=0 waves=0.444444\n",
      "result 039 : algo={ Id=1, tileIdx=16 (128x32) splitK=0 reduc=0 swizzle=1 custom=0 stages=0} status 0 time 0.204462ms workspace=0 mathMode=0 waves=0.444444\n",
      "result 040 : algo={ Id=0, tileIdx=16 (128x32) splitK=0 reduc=0 swizzle=1 custom=0 stages=0} status 0 time 0.380498ms workspace=0 mathMode=0 waves=0.111111\n",
      "result 041 : algo={ Id=0, tileIdx=16 (128x32) splitK=0 reduc=0 swizzle=0 custom=0 stages=0} status 0 time 0.380764ms workspace=0 mathMode=0 waves=0.111111\n",
      "result 042 : algo={ Id=0, tileIdx=14 (32x128) splitK=0 reduc=0 swizzle=1 custom=0 stages=0} status 0 time 0.437740ms workspace=0 mathMode=0 waves=0.444444\n",
      "result 043 : algo={ Id=0, tileIdx=14 (32x128) splitK=0 reduc=0 swizzle=0 custom=0 stages=0} status 0 time 0.438108ms workspace=0 mathMode=0 waves=0.444444\n",
      "result 044 : algo={ Id=0, tileIdx=18 (128x64) splitK=0 reduc=0 swizzle=0 custom=0 stages=0} status 0 time 0.467180ms workspace=0 mathMode=0 waves=0.111111\n",
      "result 045 : algo={ Id=0, tileIdx=18 (128x64) splitK=0 reduc=0 swizzle=1 custom=0 stages=0} status 0 time 0.467323ms workspace=0 mathMode=0 waves=0.111111\n",
      "result 046 : algo={ Id=0, tileIdx=15 (64x64) splitK=0 reduc=0 swizzle=0 custom=0 stages=0} status 0 time 0.477491ms workspace=0 mathMode=0 waves=0.111111\n",
      "result 047 : algo={ Id=0, tileIdx=15 (64x64) splitK=0 reduc=0 swizzle=1 custom=0 stages=0} status 0 time 0.477542ms workspace=0 mathMode=0 waves=0.111111\n",
      "result 048 : algo={ Id=0, tileIdx=20 (128x128) splitK=0 reduc=0 swizzle=0 custom=0 stages=0} status 0 time 0.761897ms workspace=0 mathMode=0 waves=0.222222\n",
      "result 049 : algo={ Id=0, tileIdx=20 (128x128) splitK=0 reduc=0 swizzle=1 custom=0 stages=0} status 0 time 0.762358ms workspace=0 mathMode=0 waves=0.222222\n",
      "algo={ Id=6, tileIdx=15 (64x64) splitK=1 reduc=0 swizzle=1 custom=0 stages=18} status 0 time 0.038615ms workspace=0 mathMode=0 waves=0.888889\n",
      "***cublasLt Gemm Testing End***\n",
      "\n",
      "-----------------------------\n",
      "GEMM test 7: [M: 1, K: 2048, N: 4096] attr * output_kernel\n",
      "algo_99 costs 0.014ms \n",
      "algo_100 costs 0.013ms \n",
      "algo_101 costs 0.012ms \n",
      "algo_102 costs 0.012ms \n",
      "algo_103 costs 0.012ms \n",
      "algo_104 costs 0.012ms \n",
      "algo_105 costs 0.012ms \n",
      "algo_106 costs 0.012ms \n",
      "algo_107 costs 0.012ms \n",
      "algo_108 costs 0.012ms \n",
      "algo_109 costs 0.012ms \n",
      "algo_110 costs 0.012ms \n",
      "algo_111 costs 0.012ms \n",
      "algo_112 costs 0.012ms \n",
      "algo_113 costs 0.012ms \n",
      "algo_114 costs 0.012ms \n",
      "algo_115 costs 0.012ms \n",
      "fast_algo 111 costs 0.012 ms\n",
      "***cublasLt Gemm Testing Beign***\n",
      "AlgoCount: 5000\n",
      "return 8 and run heuristic algo\n",
      "workspacesize==0, run 245 algos\n",
      "result 000 : algo={ Id=31, tileIdx=15 (64x64) splitK=1 reduc=0 swizzle=0 custom=0 stages=17} status 0 time 0.012329ms workspace=0 mathMode=0 waves=0.296296\n",
      "result 001 : algo={ Id=6, tileIdx=18 (128x64) splitK=3 reduc=4 swizzle=0 custom=0 stages=16} status 0 time 0.013496ms workspace=24576 mathMode=0 waves=0.888889\n",
      "result 002 : algo={ Id=6, tileIdx=15 (64x64) splitK=1 reduc=0 swizzle=1 custom=0 stages=18} status 0 time 0.013629ms workspace=0 mathMode=0 waves=0.592593\n",
      "result 003 : algo={ Id=6, tileIdx=18 (128x64) splitK=4 reduc=4 swizzle=1 custom=0 stages=12} status 0 time 0.015770ms workspace=32768 mathMode=0 waves=0.592593\n",
      "result 004 : algo={ Id=6, tileIdx=18 (128x64) splitK=2 reduc=1 swizzle=1 custom=0 stages=12} status 0 time 0.016179ms workspace=128 mathMode=0 waves=0.296296\n",
      "result 005 : algo={ Id=21, tileIdx=21 (256x64) splitK=4 reduc=4 swizzle=0 custom=0 stages=10} status 0 time 0.016404ms workspace=32768 mathMode=0 waves=0.296296\n",
      "result 006 : algo={ Id=13, tileIdx=0 (UNDEF) splitK=32 reduc=2 swizzle=0 custom=7 stages=0} status 0 time 0.017080ms workspace=524304 mathMode=0 waves=0.948148\n",
      "result 007 : algo={ Id=6, tileIdx=18 (128x64) splitK=0 reduc=0 swizzle=1 custom=0 stages=15} status 0 time 0.017664ms workspace=0 mathMode=0 waves=0.148148\n",
      "result 008 : algo={ Id=6, tileIdx=18 (128x64) splitK=0 reduc=0 swizzle=0 custom=0 stages=15} status 0 time 0.017695ms workspace=0 mathMode=0 waves=0.148148\n",
      "result 009 : algo={ Id=6, tileIdx=18 (128x64) splitK=7 reduc=1 swizzle=0 custom=0 stages=16} status 0 time 0.025426ms workspace=128 mathMode=0 waves=2.074074\n",
      "result 010 : algo={ Id=6, tileIdx=20 (128x128) splitK=0 reduc=0 swizzle=1 custom=0 stages=15} status 0 time 0.030587ms workspace=0 mathMode=0 waves=0.296296\n",
      "result 011 : algo={ Id=6, tileIdx=20 (128x128) splitK=0 reduc=0 swizzle=0 custom=0 stages=15} status 0 time 0.030607ms workspace=0 mathMode=0 waves=0.296296\n",
      "result 012 : algo={ Id=6, tileIdx=20 (128x128) splitK=0 reduc=0 swizzle=0 custom=0 stages=11} status 0 time 0.030607ms workspace=0 mathMode=0 waves=0.148148\n",
      "result 013 : algo={ Id=6, tileIdx=20 (128x128) splitK=0 reduc=0 swizzle=1 custom=0 stages=11} status 0 time 0.030628ms workspace=0 mathMode=0 waves=0.148148\n",
      "result 014 : algo={ Id=6, tileIdx=18 (128x64) splitK=0 reduc=0 swizzle=0 custom=0 stages=0} status 0 time 0.039219ms workspace=0 mathMode=0 waves=0.074074\n",
      "result 015 : algo={ Id=6, tileIdx=18 (128x64) splitK=0 reduc=0 swizzle=1 custom=0 stages=0} status 0 time 0.039301ms workspace=0 mathMode=0 waves=0.074074\n",
      "result 016 : algo={ Id=5, tileIdx=20 (128x128) splitK=0 reduc=0 swizzle=1 custom=0 stages=0} status 0 time 0.053668ms workspace=0 mathMode=0 waves=0.148148\n",
      "result 017 : algo={ Id=5, tileIdx=20 (128x128) splitK=0 reduc=0 swizzle=0 custom=0 stages=0} status 0 time 0.053678ms workspace=0 mathMode=0 waves=0.148148\n",
      "result 018 : algo={ Id=6, tileIdx=23 (128x256) splitK=0 reduc=0 swizzle=0 custom=0 stages=15} status 0 time 0.054968ms workspace=0 mathMode=0 waves=0.296296\n",
      "result 019 : algo={ Id=6, tileIdx=23 (128x256) splitK=0 reduc=0 swizzle=1 custom=0 stages=15} status 0 time 0.054968ms workspace=0 mathMode=0 waves=0.296296\n",
      "result 020 : algo={ Id=6, tileIdx=23 (128x256) splitK=0 reduc=0 swizzle=1 custom=0 stages=0} status 0 time 0.055931ms workspace=0 mathMode=0 waves=0.296296\n",
      "result 021 : algo={ Id=6, tileIdx=23 (128x256) splitK=0 reduc=0 swizzle=0 custom=0 stages=0} status 0 time 0.055941ms workspace=0 mathMode=0 waves=0.296296\n",
      "result 022 : algo={ Id=5, tileIdx=21 (256x64) splitK=0 reduc=0 swizzle=0 custom=0 stages=0} status 0 time 0.056750ms workspace=0 mathMode=0 waves=0.074074\n",
      "result 023 : algo={ Id=5, tileIdx=21 (256x64) splitK=0 reduc=0 swizzle=1 custom=0 stages=0} status 0 time 0.056801ms workspace=0 mathMode=0 waves=0.074074\n",
      "result 024 : algo={ Id=1, tileIdx=13 (64x32) splitK=0 reduc=0 swizzle=0 custom=0 stages=0} status 0 time 0.056832ms workspace=0 mathMode=0 waves=0.296296\n",
      "result 025 : algo={ Id=1, tileIdx=13 (64x32) splitK=0 reduc=0 swizzle=1 custom=0 stages=0} status 0 time 0.056842ms workspace=0 mathMode=0 waves=0.296296\n",
      "result 026 : algo={ Id=5, tileIdx=21 (256x64) splitK=0 reduc=0 swizzle=0 custom=0 stages=7} status 0 time 0.056975ms workspace=0 mathMode=0 waves=0.074074\n",
      "result 027 : algo={ Id=5, tileIdx=21 (256x64) splitK=0 reduc=0 swizzle=1 custom=0 stages=7} status 0 time 0.056975ms workspace=0 mathMode=0 waves=0.074074\n",
      "result 028 : algo={ Id=5, tileIdx=20 (128x128) splitK=0 reduc=0 swizzle=0 custom=0 stages=7} status 0 time 0.057037ms workspace=0 mathMode=0 waves=0.148148\n",
      "result 029 : algo={ Id=5, tileIdx=20 (128x128) splitK=0 reduc=0 swizzle=1 custom=0 stages=7} status 0 time 0.057068ms workspace=0 mathMode=0 waves=0.148148\n",
      "result 030 : algo={ Id=1, tileIdx=11 (32x32) splitK=0 reduc=0 swizzle=1 custom=0 stages=0} status 0 time 0.061051ms workspace=0 mathMode=0 waves=0.296296\n",
      "result 031 : algo={ Id=1, tileIdx=11 (32x32) splitK=0 reduc=0 swizzle=0 custom=0 stages=0} status 0 time 0.061092ms workspace=0 mathMode=0 waves=0.296296\n",
      "result 032 : algo={ Id=5, tileIdx=23 (128x256) splitK=0 reduc=0 swizzle=0 custom=0 stages=7} status 0 time 0.066877ms workspace=0 mathMode=0 waves=0.296296\n",
      "result 033 : algo={ Id=5, tileIdx=23 (128x256) splitK=0 reduc=0 swizzle=1 custom=0 stages=7} status 0 time 0.066959ms workspace=0 mathMode=0 waves=0.296296\n",
      "result 034 : algo={ Id=5, tileIdx=24 (256x128) splitK=0 reduc=0 swizzle=0 custom=0 stages=0} status 0 time 0.068157ms workspace=0 mathMode=0 waves=0.148148\n",
      "result 035 : algo={ Id=5, tileIdx=24 (256x128) splitK=0 reduc=0 swizzle=1 custom=0 stages=0} status 0 time 0.068188ms workspace=0 mathMode=0 waves=0.148148\n",
      "result 036 : algo={ Id=5, tileIdx=23 (128x256) splitK=0 reduc=0 swizzle=1 custom=0 stages=0} status 0 time 0.068434ms workspace=0 mathMode=0 waves=0.296296\n",
      "result 037 : algo={ Id=5, tileIdx=23 (128x256) splitK=0 reduc=0 swizzle=0 custom=0 stages=0} status 0 time 0.068434ms workspace=0 mathMode=0 waves=0.296296\n",
      "result 038 : algo={ Id=1, tileIdx=16 (128x32) splitK=0 reduc=0 swizzle=0 custom=0 stages=0} status 0 time 0.105083ms workspace=0 mathMode=0 waves=0.296296\n",
      "result 039 : algo={ Id=1, tileIdx=16 (128x32) splitK=0 reduc=0 swizzle=1 custom=0 stages=0} status 0 time 0.105114ms workspace=0 mathMode=0 waves=0.296296\n",
      "result 040 : algo={ Id=0, tileIdx=16 (128x32) splitK=0 reduc=0 swizzle=0 custom=0 stages=0} status 0 time 0.106445ms workspace=0 mathMode=0 waves=0.074074\n",
      "result 041 : algo={ Id=0, tileIdx=16 (128x32) splitK=0 reduc=0 swizzle=1 custom=0 stages=0} status 0 time 0.106527ms workspace=0 mathMode=0 waves=0.074074\n",
      "result 042 : algo={ Id=0, tileIdx=14 (32x128) splitK=0 reduc=0 swizzle=1 custom=0 stages=0} status 0 time 0.201984ms workspace=0 mathMode=0 waves=0.296296\n",
      "result 043 : algo={ Id=0, tileIdx=14 (32x128) splitK=0 reduc=0 swizzle=0 custom=0 stages=0} status 0 time 0.201994ms workspace=0 mathMode=0 waves=0.296296\n",
      "result 044 : algo={ Id=0, tileIdx=18 (128x64) splitK=0 reduc=0 swizzle=0 custom=0 stages=0} status 0 time 0.215491ms workspace=0 mathMode=0 waves=0.074074\n",
      "result 045 : algo={ Id=0, tileIdx=18 (128x64) splitK=0 reduc=0 swizzle=1 custom=0 stages=0} status 0 time 0.215491ms workspace=0 mathMode=0 waves=0.074074\n",
      "result 046 : algo={ Id=0, tileIdx=15 (64x64) splitK=0 reduc=0 swizzle=1 custom=0 stages=0} status 0 time 0.220570ms workspace=0 mathMode=0 waves=0.074074\n",
      "result 047 : algo={ Id=0, tileIdx=15 (64x64) splitK=0 reduc=0 swizzle=0 custom=0 stages=0} status 0 time 0.220590ms workspace=0 mathMode=0 waves=0.074074\n",
      "result 048 : algo={ Id=0, tileIdx=20 (128x128) splitK=0 reduc=0 swizzle=1 custom=0 stages=0} status 0 time 0.384072ms workspace=0 mathMode=0 waves=0.148148\n",
      "result 049 : algo={ Id=0, tileIdx=20 (128x128) splitK=0 reduc=0 swizzle=0 custom=0 stages=0} status 0 time 0.384205ms workspace=0 mathMode=0 waves=0.148148\n",
      "algo={ Id=31, tileIdx=15 (64x64) splitK=1 reduc=0 swizzle=0 custom=0 stages=17} status 0 time 0.012329ms workspace=0 mathMode=0 waves=0.296296\n",
      "***cublasLt Gemm Testing End***\n",
      "\n",
      "-----------------------------\n",
      "GEMM test 8: [M: 1, K: 4096, N: 8192] ffn gemm 1\n",
      "algo_99 costs 0.050ms \n",
      "algo_100 costs 0.051ms \n",
      "algo_101 costs 0.051ms \n",
      "algo_102 costs 0.051ms \n",
      "algo_103 costs 0.050ms \n",
      "algo_104 costs 0.051ms \n",
      "algo_105 costs 0.051ms \n",
      "algo_106 costs 0.051ms \n",
      "algo_107 costs 0.051ms \n",
      "algo_108 costs 0.051ms \n",
      "algo_109 costs 0.050ms \n",
      "algo_110 costs 0.051ms \n",
      "algo_111 costs 0.051ms \n",
      "algo_112 costs 0.051ms \n",
      "algo_113 costs 0.051ms \n",
      "algo_114 costs 0.050ms \n",
      "algo_115 costs 0.051ms \n",
      "fast_algo 109 costs 0.050 ms\n",
      "***cublasLt Gemm Testing Beign***\n",
      "AlgoCount: 5000\n",
      "return 8 and run heuristic algo\n",
      "workspacesize==0, run 245 algos\n",
      "result 000 : algo={ Id=6, tileIdx=18 (128x64) splitK=0 reduc=0 swizzle=0 custom=0 stages=15} status 0 time 0.047585ms workspace=0 mathMode=0 waves=0.296296\n",
      "result 001 : algo={ Id=6, tileIdx=18 (128x64) splitK=0 reduc=0 swizzle=1 custom=0 stages=15} status 0 time 0.047606ms workspace=0 mathMode=0 waves=0.296296\n",
      "result 002 : algo={ Id=6, tileIdx=18 (128x64) splitK=1 reduc=0 swizzle=0 custom=0 stages=12} status 0 time 0.050125ms workspace=0 mathMode=0 waves=0.296296\n",
      "result 003 : algo={ Id=6, tileIdx=18 (128x64) splitK=1 reduc=0 swizzle=1 custom=0 stages=16} status 0 time 0.050125ms workspace=0 mathMode=0 waves=0.592593\n",
      "result 004 : algo={ Id=6, tileIdx=18 (128x64) splitK=2 reduc=1 swizzle=1 custom=0 stages=12} status 0 time 0.052234ms workspace=256 mathMode=0 waves=0.592593\n",
      "result 005 : algo={ Id=6, tileIdx=21 (256x64) splitK=3 reduc=4 swizzle=0 custom=0 stages=15} status 0 time 0.052562ms workspace=49152 mathMode=0 waves=0.888889\n",
      "result 006 : algo={ Id=6, tileIdx=21 (256x64) splitK=2 reduc=1 swizzle=1 custom=0 stages=15} status 0 time 0.054784ms workspace=128 mathMode=0 waves=0.592593\n",
      "result 007 : algo={ Id=6, tileIdx=20 (128x128) splitK=0 reduc=0 swizzle=0 custom=0 stages=15} status 0 time 0.055849ms workspace=0 mathMode=0 waves=0.592593\n",
      "result 008 : algo={ Id=6, tileIdx=20 (128x128) splitK=0 reduc=0 swizzle=1 custom=0 stages=15} status 0 time 0.055869ms workspace=0 mathMode=0 waves=0.592593\n",
      "result 009 : algo={ Id=6, tileIdx=20 (128x128) splitK=0 reduc=0 swizzle=1 custom=0 stages=11} status 0 time 0.055982ms workspace=0 mathMode=0 waves=0.296296\n",
      "result 010 : algo={ Id=6, tileIdx=20 (128x128) splitK=0 reduc=0 swizzle=0 custom=0 stages=11} status 0 time 0.055992ms workspace=0 mathMode=0 waves=0.296296\n",
      "result 011 : algo={ Id=21, tileIdx=21 (256x64) splitK=4 reduc=4 swizzle=0 custom=0 stages=10} status 0 time 0.058348ms workspace=65536 mathMode=0 waves=0.592593\n",
      "result 012 : algo={ Id=13, tileIdx=0 (UNDEF) splitK=1 reduc=0 swizzle=0 custom=103 stages=0} status 0 time 0.068137ms workspace=0 mathMode=0 waves=0.592593\n",
      "result 013 : algo={ Id=13, tileIdx=0 (UNDEF) splitK=32 reduc=2 swizzle=0 custom=7 stages=0} status 0 time 0.070932ms workspace=1048592 mathMode=0 waves=1.896296\n",
      "result 014 : algo={ Id=5, tileIdx=20 (128x128) splitK=0 reduc=0 swizzle=1 custom=0 stages=0} status 0 time 0.102707ms workspace=0 mathMode=0 waves=0.296296\n",
      "result 015 : algo={ Id=5, tileIdx=20 (128x128) splitK=0 reduc=0 swizzle=0 custom=0 stages=0} status 0 time 0.102758ms workspace=0 mathMode=0 waves=0.296296\n",
      "result 016 : algo={ Id=6, tileIdx=23 (128x256) splitK=0 reduc=0 swizzle=1 custom=0 stages=15} status 0 time 0.103455ms workspace=0 mathMode=0 waves=0.592593\n",
      "result 017 : algo={ Id=6, tileIdx=23 (128x256) splitK=0 reduc=0 swizzle=0 custom=0 stages=15} status 0 time 0.103496ms workspace=0 mathMode=0 waves=0.592593\n",
      "result 018 : algo={ Id=6, tileIdx=23 (128x256) splitK=0 reduc=0 swizzle=1 custom=0 stages=0} status 0 time 0.109445ms workspace=0 mathMode=0 waves=0.592593\n",
      "result 019 : algo={ Id=6, tileIdx=23 (128x256) splitK=0 reduc=0 swizzle=0 custom=0 stages=0} status 0 time 0.109507ms workspace=0 mathMode=0 waves=0.592593\n",
      "result 020 : algo={ Id=5, tileIdx=20 (128x128) splitK=0 reduc=0 swizzle=1 custom=0 stages=7} status 0 time 0.109691ms workspace=0 mathMode=0 waves=0.296296\n",
      "result 021 : algo={ Id=5, tileIdx=20 (128x128) splitK=0 reduc=0 swizzle=0 custom=0 stages=7} status 0 time 0.109824ms workspace=0 mathMode=0 waves=0.296296\n",
      "result 022 : algo={ Id=5, tileIdx=21 (256x64) splitK=0 reduc=0 swizzle=0 custom=0 stages=7} status 0 time 0.113480ms workspace=0 mathMode=0 waves=0.148148\n",
      "result 023 : algo={ Id=5, tileIdx=21 (256x64) splitK=0 reduc=0 swizzle=1 custom=0 stages=7} status 0 time 0.113521ms workspace=0 mathMode=0 waves=0.148148\n",
      "result 024 : algo={ Id=6, tileIdx=18 (128x64) splitK=0 reduc=0 swizzle=0 custom=0 stages=0} status 0 time 0.120033ms workspace=0 mathMode=0 waves=0.148148\n",
      "result 025 : algo={ Id=6, tileIdx=18 (128x64) splitK=0 reduc=0 swizzle=1 custom=0 stages=0} status 0 time 0.120095ms workspace=0 mathMode=0 waves=0.148148\n",
      "result 026 : algo={ Id=5, tileIdx=23 (128x256) splitK=0 reduc=0 swizzle=1 custom=0 stages=0} status 0 time 0.125972ms workspace=0 mathMode=0 waves=0.592593\n",
      "result 027 : algo={ Id=5, tileIdx=23 (128x256) splitK=0 reduc=0 swizzle=0 custom=0 stages=0} status 0 time 0.126013ms workspace=0 mathMode=0 waves=0.592593\n",
      "result 028 : algo={ Id=5, tileIdx=24 (256x128) splitK=0 reduc=0 swizzle=1 custom=0 stages=0} status 0 time 0.127519ms workspace=0 mathMode=0 waves=0.296296\n",
      "result 029 : algo={ Id=5, tileIdx=23 (128x256) splitK=0 reduc=0 swizzle=1 custom=0 stages=7} status 0 time 0.127570ms workspace=0 mathMode=0 waves=0.592593\n",
      "result 030 : algo={ Id=5, tileIdx=24 (256x128) splitK=0 reduc=0 swizzle=0 custom=0 stages=0} status 0 time 0.127590ms workspace=0 mathMode=0 waves=0.296296\n",
      "result 031 : algo={ Id=5, tileIdx=23 (128x256) splitK=0 reduc=0 swizzle=0 custom=0 stages=7} status 0 time 0.127652ms workspace=0 mathMode=0 waves=0.592593\n",
      "result 032 : algo={ Id=5, tileIdx=21 (256x64) splitK=0 reduc=0 swizzle=1 custom=0 stages=0} status 0 time 0.146319ms workspace=0 mathMode=0 waves=0.148148\n",
      "result 033 : algo={ Id=5, tileIdx=21 (256x64) splitK=0 reduc=0 swizzle=0 custom=0 stages=0} status 0 time 0.146381ms workspace=0 mathMode=0 waves=0.148148\n",
      "result 034 : algo={ Id=1, tileIdx=11 (32x32) splitK=0 reduc=0 swizzle=0 custom=0 stages=0} status 0 time 0.181279ms workspace=0 mathMode=0 waves=0.592593\n",
      "result 035 : algo={ Id=1, tileIdx=11 (32x32) splitK=0 reduc=0 swizzle=1 custom=0 stages=0} status 0 time 0.181299ms workspace=0 mathMode=0 waves=0.592593\n",
      "result 036 : algo={ Id=1, tileIdx=16 (128x32) splitK=0 reduc=0 swizzle=1 custom=0 stages=0} status 0 time 0.204636ms workspace=0 mathMode=0 waves=0.592593\n",
      "result 037 : algo={ Id=1, tileIdx=16 (128x32) splitK=0 reduc=0 swizzle=0 custom=0 stages=0} status 0 time 0.204739ms workspace=0 mathMode=0 waves=0.592593\n",
      "result 038 : algo={ Id=1, tileIdx=13 (64x32) splitK=0 reduc=0 swizzle=1 custom=0 stages=0} status 0 time 0.218911ms workspace=0 mathMode=0 waves=0.592593\n",
      "result 039 : algo={ Id=1, tileIdx=13 (64x32) splitK=0 reduc=0 swizzle=0 custom=0 stages=0} status 0 time 0.218921ms workspace=0 mathMode=0 waves=0.592593\n",
      "result 040 : algo={ Id=0, tileIdx=16 (128x32) splitK=0 reduc=0 swizzle=1 custom=0 stages=0} status 0 time 0.365015ms workspace=0 mathMode=0 waves=0.148148\n",
      "result 041 : algo={ Id=0, tileIdx=16 (128x32) splitK=0 reduc=0 swizzle=0 custom=0 stages=0} status 0 time 0.365056ms workspace=0 mathMode=0 waves=0.148148\n",
      "result 042 : algo={ Id=0, tileIdx=18 (128x64) splitK=0 reduc=0 swizzle=1 custom=0 stages=0} status 0 time 0.454011ms workspace=0 mathMode=0 waves=0.148148\n",
      "result 043 : algo={ Id=0, tileIdx=18 (128x64) splitK=0 reduc=0 swizzle=0 custom=0 stages=0} status 0 time 0.454318ms workspace=0 mathMode=0 waves=0.148148\n",
      "result 044 : algo={ Id=0, tileIdx=14 (32x128) splitK=0 reduc=0 swizzle=1 custom=0 stages=0} status 0 time 0.679905ms workspace=0 mathMode=0 waves=0.592593\n",
      "result 045 : algo={ Id=0, tileIdx=14 (32x128) splitK=0 reduc=0 swizzle=0 custom=0 stages=0} status 0 time 0.680171ms workspace=0 mathMode=0 waves=0.592593\n",
      "result 046 : algo={ Id=0, tileIdx=20 (128x128) splitK=0 reduc=0 swizzle=1 custom=0 stages=0} status 0 time 0.763300ms workspace=0 mathMode=0 waves=0.296296\n",
      "result 047 : algo={ Id=0, tileIdx=20 (128x128) splitK=0 reduc=0 swizzle=0 custom=0 stages=0} status 0 time 0.763556ms workspace=0 mathMode=0 waves=0.296296\n",
      "result 048 : algo={ Id=0, tileIdx=15 (64x64) splitK=0 reduc=0 swizzle=0 custom=0 stages=0} status 0 time 0.785859ms workspace=0 mathMode=0 waves=0.148148\n",
      "result 049 : algo={ Id=0, tileIdx=15 (64x64) splitK=0 reduc=0 swizzle=1 custom=0 stages=0} status 0 time 0.785900ms workspace=0 mathMode=0 waves=0.148148\n",
      "algo={ Id=6, tileIdx=18 (128x64) splitK=0 reduc=0 swizzle=0 custom=0 stages=15} status 0 time 0.047585ms workspace=0 mathMode=0 waves=0.296296\n",
      "***cublasLt Gemm Testing End***\n",
      "\n",
      "-----------------------------\n",
      "GEMM test 9: [M: 1, K: 8192, N: 4096] ffn gemm 2\n",
      "algo_99 costs 0.052ms \n",
      "algo_100 costs 0.052ms \n",
      "algo_101 costs 0.052ms \n",
      "algo_102 costs 0.052ms \n",
      "algo_103 costs 0.051ms \n",
      "algo_104 costs 0.052ms \n",
      "algo_105 costs 0.051ms \n",
      "algo_106 costs 0.052ms \n",
      "algo_107 costs 0.051ms \n",
      "algo_108 costs 0.051ms \n",
      "algo_109 costs 0.052ms \n",
      "algo_110 costs 0.052ms \n",
      "algo_111 costs 0.051ms \n",
      "algo_112 costs 0.052ms \n",
      "algo_113 costs 0.051ms \n",
      "algo_114 costs 0.051ms \n",
      "algo_115 costs 0.052ms \n",
      "fast_algo 113 costs 0.051 ms\n",
      "***cublasLt Gemm Testing Beign***\n",
      "AlgoCount: 5000\n",
      "return 8 and run heuristic algo\n",
      "workspacesize==0, run 245 algos\n",
      "result 000 : algo={ Id=31, tileIdx=15 (64x64) splitK=1 reduc=0 swizzle=0 custom=0 stages=17} status 0 time 0.047606ms workspace=0 mathMode=0 waves=0.296296\n",
      "result 001 : algo={ Id=6, tileIdx=15 (64x64) splitK=1 reduc=0 swizzle=1 custom=0 stages=18} status 0 time 0.049377ms workspace=0 mathMode=0 waves=0.592593\n",
      "result 002 : algo={ Id=6, tileIdx=18 (128x64) splitK=3 reduc=4 swizzle=0 custom=0 stages=16} status 0 time 0.051036ms workspace=24576 mathMode=0 waves=0.888889\n",
      "result 003 : algo={ Id=21, tileIdx=21 (256x64) splitK=4 reduc=4 swizzle=0 custom=0 stages=10} status 0 time 0.052613ms workspace=32768 mathMode=0 waves=0.296296\n",
      "result 004 : algo={ Id=6, tileIdx=18 (128x64) splitK=2 reduc=1 swizzle=1 custom=0 stages=12} status 0 time 0.052859ms workspace=128 mathMode=0 waves=0.296296\n",
      "result 005 : algo={ Id=6, tileIdx=18 (128x64) splitK=4 reduc=4 swizzle=1 custom=0 stages=12} status 0 time 0.053187ms workspace=32768 mathMode=0 waves=0.592593\n",
      "result 006 : algo={ Id=6, tileIdx=18 (128x64) splitK=7 reduc=1 swizzle=0 custom=0 stages=16} status 0 time 0.055962ms workspace=128 mathMode=0 waves=2.074074\n",
      "result 007 : algo={ Id=6, tileIdx=18 (128x64) splitK=0 reduc=0 swizzle=0 custom=0 stages=15} status 0 time 0.065843ms workspace=0 mathMode=0 waves=0.148148\n",
      "result 008 : algo={ Id=6, tileIdx=18 (128x64) splitK=0 reduc=0 swizzle=1 custom=0 stages=15} status 0 time 0.065884ms workspace=0 mathMode=0 waves=0.148148\n",
      "result 009 : algo={ Id=13, tileIdx=0 (UNDEF) splitK=108 reduc=2 swizzle=0 custom=7 stages=0} status 0 time 0.083763ms workspace=1769488 mathMode=0 waves=3.200000\n",
      "result 010 : algo={ Id=6, tileIdx=20 (128x128) splitK=0 reduc=0 swizzle=1 custom=0 stages=15} status 0 time 0.103137ms workspace=0 mathMode=0 waves=0.296296\n",
      "result 011 : algo={ Id=6, tileIdx=20 (128x128) splitK=0 reduc=0 swizzle=0 custom=0 stages=15} status 0 time 0.103199ms workspace=0 mathMode=0 waves=0.296296\n",
      "result 012 : algo={ Id=6, tileIdx=20 (128x128) splitK=0 reduc=0 swizzle=0 custom=0 stages=11} status 0 time 0.103578ms workspace=0 mathMode=0 waves=0.148148\n",
      "result 013 : algo={ Id=6, tileIdx=20 (128x128) splitK=0 reduc=0 swizzle=1 custom=0 stages=11} status 0 time 0.103608ms workspace=0 mathMode=0 waves=0.148148\n",
      "result 014 : algo={ Id=5, tileIdx=20 (128x128) splitK=0 reduc=0 swizzle=0 custom=0 stages=0} status 0 time 0.197827ms workspace=0 mathMode=0 waves=0.148148\n",
      "result 015 : algo={ Id=5, tileIdx=20 (128x128) splitK=0 reduc=0 swizzle=1 custom=0 stages=0} status 0 time 0.197847ms workspace=0 mathMode=0 waves=0.148148\n",
      "result 016 : algo={ Id=6, tileIdx=23 (128x256) splitK=0 reduc=0 swizzle=1 custom=0 stages=15} status 0 time 0.198072ms workspace=0 mathMode=0 waves=0.296296\n",
      "result 017 : algo={ Id=6, tileIdx=23 (128x256) splitK=0 reduc=0 swizzle=0 custom=0 stages=15} status 0 time 0.198072ms workspace=0 mathMode=0 waves=0.296296\n",
      "result 018 : algo={ Id=6, tileIdx=23 (128x256) splitK=0 reduc=0 swizzle=0 custom=0 stages=0} status 0 time 0.208077ms workspace=0 mathMode=0 waves=0.296296\n",
      "result 019 : algo={ Id=6, tileIdx=23 (128x256) splitK=0 reduc=0 swizzle=1 custom=0 stages=0} status 0 time 0.208128ms workspace=0 mathMode=0 waves=0.296296\n",
      "result 020 : algo={ Id=5, tileIdx=20 (128x128) splitK=0 reduc=0 swizzle=1 custom=0 stages=7} status 0 time 0.210452ms workspace=0 mathMode=0 waves=0.148148\n",
      "result 021 : algo={ Id=5, tileIdx=20 (128x128) splitK=0 reduc=0 swizzle=0 custom=0 stages=7} status 0 time 0.210452ms workspace=0 mathMode=0 waves=0.148148\n",
      "result 022 : algo={ Id=5, tileIdx=21 (256x64) splitK=0 reduc=0 swizzle=1 custom=0 stages=7} status 0 time 0.215378ms workspace=0 mathMode=0 waves=0.074074\n",
      "result 023 : algo={ Id=5, tileIdx=21 (256x64) splitK=0 reduc=0 swizzle=0 custom=0 stages=7} status 0 time 0.215480ms workspace=0 mathMode=0 waves=0.074074\n",
      "result 024 : algo={ Id=6, tileIdx=18 (128x64) splitK=0 reduc=0 swizzle=0 custom=0 stages=0} status 0 time 0.230042ms workspace=0 mathMode=0 waves=0.074074\n",
      "result 025 : algo={ Id=6, tileIdx=18 (128x64) splitK=0 reduc=0 swizzle=1 custom=0 stages=0} status 0 time 0.230062ms workspace=0 mathMode=0 waves=0.074074\n",
      "result 026 : algo={ Id=5, tileIdx=23 (128x256) splitK=0 reduc=0 swizzle=0 custom=0 stages=0} status 0 time 0.243640ms workspace=0 mathMode=0 waves=0.296296\n",
      "result 027 : algo={ Id=5, tileIdx=23 (128x256) splitK=0 reduc=0 swizzle=1 custom=0 stages=0} status 0 time 0.243640ms workspace=0 mathMode=0 waves=0.296296\n",
      "result 028 : algo={ Id=5, tileIdx=23 (128x256) splitK=0 reduc=0 swizzle=0 custom=0 stages=7} status 0 time 0.246231ms workspace=0 mathMode=0 waves=0.296296\n",
      "result 029 : algo={ Id=5, tileIdx=23 (128x256) splitK=0 reduc=0 swizzle=1 custom=0 stages=7} status 0 time 0.246262ms workspace=0 mathMode=0 waves=0.296296\n",
      "result 030 : algo={ Id=5, tileIdx=24 (256x128) splitK=0 reduc=0 swizzle=0 custom=0 stages=0} status 0 time 0.246323ms workspace=0 mathMode=0 waves=0.148148\n",
      "result 031 : algo={ Id=5, tileIdx=24 (256x128) splitK=0 reduc=0 swizzle=1 custom=0 stages=0} status 0 time 0.246374ms workspace=0 mathMode=0 waves=0.148148\n",
      "result 032 : algo={ Id=1, tileIdx=13 (64x32) splitK=0 reduc=0 swizzle=0 custom=0 stages=0} status 0 time 0.264233ms workspace=0 mathMode=0 waves=0.296296\n",
      "result 033 : algo={ Id=1, tileIdx=13 (64x32) splitK=0 reduc=0 swizzle=1 custom=0 stages=0} status 0 time 0.264274ms workspace=0 mathMode=0 waves=0.296296\n",
      "result 034 : algo={ Id=1, tileIdx=11 (32x32) splitK=0 reduc=0 swizzle=1 custom=0 stages=0} status 0 time 0.268831ms workspace=0 mathMode=0 waves=0.296296\n",
      "result 035 : algo={ Id=1, tileIdx=11 (32x32) splitK=0 reduc=0 swizzle=0 custom=0 stages=0} status 0 time 0.268943ms workspace=0 mathMode=0 waves=0.296296\n",
      "result 036 : algo={ Id=5, tileIdx=21 (256x64) splitK=0 reduc=0 swizzle=1 custom=0 stages=0} status 0 time 0.281477ms workspace=0 mathMode=0 waves=0.074074\n",
      "result 037 : algo={ Id=5, tileIdx=21 (256x64) splitK=0 reduc=0 swizzle=0 custom=0 stages=0} status 0 time 0.281600ms workspace=0 mathMode=0 waves=0.074074\n",
      "result 038 : algo={ Id=1, tileIdx=16 (128x32) splitK=0 reduc=0 swizzle=0 custom=0 stages=0} status 0 time 0.401183ms workspace=0 mathMode=0 waves=0.296296\n",
      "result 039 : algo={ Id=1, tileIdx=16 (128x32) splitK=0 reduc=0 swizzle=1 custom=0 stages=0} status 0 time 0.401316ms workspace=0 mathMode=0 waves=0.296296\n",
      "result 040 : algo={ Id=0, tileIdx=16 (128x32) splitK=0 reduc=0 swizzle=0 custom=0 stages=0} status 0 time 0.748513ms workspace=0 mathMode=0 waves=0.074074\n",
      "result 041 : algo={ Id=0, tileIdx=16 (128x32) splitK=0 reduc=0 swizzle=1 custom=0 stages=0} status 0 time 0.748616ms workspace=0 mathMode=0 waves=0.074074\n",
      "result 042 : algo={ Id=0, tileIdx=14 (32x128) splitK=0 reduc=0 swizzle=1 custom=0 stages=0} status 0 time 0.864983ms workspace=0 mathMode=0 waves=0.296296\n",
      "result 043 : algo={ Id=0, tileIdx=14 (32x128) splitK=0 reduc=0 swizzle=0 custom=0 stages=0} status 0 time 0.865290ms workspace=0 mathMode=0 waves=0.296296\n",
      "result 044 : algo={ Id=0, tileIdx=18 (128x64) splitK=0 reduc=0 swizzle=1 custom=0 stages=0} status 0 time 0.920463ms workspace=0 mathMode=0 waves=0.074074\n",
      "result 045 : algo={ Id=0, tileIdx=18 (128x64) splitK=0 reduc=0 swizzle=0 custom=0 stages=0} status 0 time 0.920709ms workspace=0 mathMode=0 waves=0.074074\n",
      "result 046 : algo={ Id=0, tileIdx=15 (64x64) splitK=0 reduc=0 swizzle=0 custom=0 stages=0} status 0 time 0.941005ms workspace=0 mathMode=0 waves=0.074074\n",
      "result 047 : algo={ Id=0, tileIdx=15 (64x64) splitK=0 reduc=0 swizzle=1 custom=0 stages=0} status 0 time 0.941199ms workspace=0 mathMode=0 waves=0.074074\n",
      "result 048 : algo={ Id=0, tileIdx=20 (128x128) splitK=0 reduc=0 swizzle=1 custom=0 stages=0} status 0 time 1.518111ms workspace=0 mathMode=0 waves=0.148148\n",
      "result 049 : algo={ Id=0, tileIdx=20 (128x128) splitK=0 reduc=0 swizzle=0 custom=0 stages=0} status 0 time 1.518500ms workspace=0 mathMode=0 waves=0.148148\n",
      "algo={ Id=31, tileIdx=15 (64x64) splitK=1 reduc=0 swizzle=0 custom=0 stages=17} status 0 time 0.047606ms workspace=0 mathMode=0 waves=0.296296\n",
      "***cublasLt Gemm Testing End***\n",
      "\n",
      "-----------------------------\n",
      "GEMM test 10: [M: 1, K: 4096, N: 25128] logits gemm\n",
      "algo_99 costs 0.135ms \n",
      "algo_100 costs 0.134ms \n",
      "algo_101 costs 0.134ms \n",
      "algo_102 costs 0.134ms \n",
      "algo_103 costs 0.134ms \n",
      "algo_104 costs 0.134ms \n",
      "algo_105 costs 0.134ms \n",
      "algo_106 costs 0.134ms \n",
      "algo_107 costs 0.134ms \n",
      "algo_108 costs 0.134ms \n",
      "algo_109 costs 0.134ms \n",
      "algo_110 costs 0.134ms \n",
      "algo_111 costs 0.134ms \n",
      "algo_112 costs 0.134ms \n",
      "algo_113 costs 0.134ms \n",
      "algo_114 costs 0.134ms \n",
      "algo_115 costs 0.134ms \n",
      "fast_algo 102 costs 0.134 ms\n",
      "***cublas Gemm Testing End***\n",
      "\n",
      "***GPT Gemm Testing End***\n"
     ]
    }
   ],
   "source": [
    "!CUDA_VISIBLE_DEVICES=1  ./bin/gpt_gemm 1 1 128 16 256 16384 50256 1 2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We remember from the first lab that we have 4 GPUs available on our server, so let's use 2 of them to start inference in TP=2 regime"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's configure `./FasterTransformer/examples/cpp/gptj/gptj_config.ini` to run GPT-J C++ inference case.</br>\n",
    "We need to open this file in JupyterLab and update these lines to make our inference case similar to the inference pipeline we had the previous time but with 2 GPUs:</br>\n",
    "* tensor_para_size=<b>2</b> #we run on 2xGPUs with Tensor-Parallelism mode\n",
    "\n",
    "All other parameters should be the same as in the previous run with FT.\n",
    " We will use GPUs 2 and 3 (numbers 1 and 2) because previous task (Notebook 02) was atarted onto GPU 0 and model is already there if you haven't stopped the Jupyter kernel. So we just avoiding OOM error. Let's run inference with optimized kernels now."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Total ranks: 2.\n",
      "Device Graphics Device\n",
      "P0 is runing with 0 GPU.\n",
      "[INFO] Setting tensor_para_size to 2 \n",
      "Device Graphics Device\n",
      "P1 is runing with 1 GPU.\n",
      "[INFO] Setting tensor_para_size to 2 \n",
      "[FT][WARNING] Device 1 peer access Device 0 is not avaiable. This may lead to peer access errors when doing tensor/pipeline parallel!\n",
      "[FT][WARNING] Device 0 peer access Device 1 is not avaiable. This may lead to peer access errors when doing tensor/pipeline parallel!\n",
      "after allocation, free 72.48 GB total 79.35 GB\n",
      "after allocation, free 64.46 GB total 79.35 GB\n",
      "Writing 128 elements\n",
      "  818   262   938  3155   286  1528    11   257 10089   517 \n",
      "zeroCount = 0\n",
      "[INFO] request_batch_size 1 beam_width 1 head_num 16 size_per_head 256 total_output_len 128 decoder_layers 28 vocab_size 50400 FT-CPP-decoding-beamsearch-time 779.50 ms\n",
      "[INFO] request_batch_size 1 beam_width 1 head_num 16 size_per_head 256 total_output_len 128 decoder_layers 28 vocab_size 50400 FT-CPP-decoding-beamsearch-time 779.50 ms\n"
     ]
    }
   ],
   "source": [
    "!CUDA_VISIBLE_DEVICES=1,2 mpirun -n 2 --allow-run-as-root ./bin/gptj_example"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We see that the inference time of the sentence with the same length is <b>1.25 seconds </b> which is <b>x5</b> times faster than the default PyTorch + Transformers pipeline! Of course, we used 2 GPUs here, but this demonstrates how we can achieve lower latency by adding more GPUs to inference.</br>\n",
    "Let's decode the `out` file and see what was generated by our neural network."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[INFO] batch 0: In the last couple of days, a handful more cases have been added to this list. Each time (or sometimes several times in one year) is usually seen as an \"uptick.\" In other words: while most people don't go nuts with buying things out right away when there's such high demand and many who would be affected by that price hike could hold on for some months before actually jumping at those prices...\n",
      "If you want my recommendation though it depends where are located because I'm not 100% sure whether their product might just work fine or even slightly better than similar quality products already available here.... So basically if its something used\n",
      "[[  818   262   938  3155   286  1528    11   257 10089   517  2663   423\n",
      "    587  2087   284   428  1351    13  5501   640   357   273  3360  1811\n",
      "   1661   287   530   614     8   318  3221  1775   355   281   366 37623\n",
      "    624   526   554   584  2456    25   981   749   661   836   470   467\n",
      "  14380   351  7067  1243   503   826  1497   618   612   338   884  1029\n",
      "   3512   290   867   508   561   307  5676   416   326  2756 18390   714\n",
      "   1745   319   329   617  1933   878  1682 14284   379   883  4536   986\n",
      "    198  1532   345   765   616 15602   996   340  8338   810   389  5140\n",
      "    780   314  1101   407  1802     4  1654  1771   511  1720  1244   655\n",
      "    670  3734   393   772  4622  1365   621  2092  3081  3186  1541  1695\n",
      "    994  1106  1406  6209   611   663  1223   973]]\n"
     ]
    }
   ],
   "source": [
    "!python3 ../examples/pytorch/gpt/utils/gpt_token_converter.py \\\n",
    "                               --vocab_file=../../models/gpt2-vocab.json  \\\n",
    "                               --bpe_file=../../models/gpt2-merges.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Results"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's compare the results. We get 3.5x acceleration using FasterTransformer in TP=1 mode in comparison with Transformers library based implementation and <b>5x</b> acceleration with two way tensor parallelism. We achieved a significant performance boost. You can use this C++ example as well as PyTorch/TF bindings in your inference pipelines.</br> \n",
    "At the same time, if you wish to have more unified inference engine/serving solution we will look at it in the next notebook."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h2 style=\"color:green;\">Congratulations!</h2>\n",
    "\n",
    "Please proceed on to [Inference of the GPT-J 6b model with Triton Inference server and FasterTransformer as a backend.](04_FTAndTritonRunRemoteInferenceOfTheGPT-J.ipynb)\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}