A Brief Study of LLAMA C++ (2) Starting from User Input

Li Feifei said, “Simple algorithms and a large amount of good data always win in the field of artificial intelligence, including in the future world model domain as well.”

Zihan, WeChat Official Account: HyperAI Super Neural. From dry cleaning to the Elizabeth Queen Engineering Award, Li Feifei reverses the Silicon Valley technology myth, focusing on the depersonalization risks of AI.

Using gdb for debugging, we continue learning LLAMA C++. First, set a breakpoint:

1       breakpoint     keep y   0x00007ffff7c12e92 in llama_batch_allocr::ubatch_add(std::vector<int, std::allocator<int> > const&, unsigned int, bool)                                                    at /home/zzy/Program/Torch/llama.cpp/src/llama-batch.cpp:702    stop only if batch.token[idxs[i]]==105

Following gdb, we arrive at

llama_batch_allocr::ubatch_add:702            udata->token[i] = batch.token[idxs[i]];

Here, batch.token[idxs[i]] contains the user input after simple processing.Then, I perform hardware breakpoint tracking on udata->token[i], and I arrive at the function:

ggml_compute_forward_get_rows_q

4500    static void ggml_compute_forward_get_rows_q(4501        const ggml_compute_params * params,4502              ggml_tensor * dst) {4503        const ggml_tensor * src0 = dst->src[0];4504        const ggml_tensor * src1 = dst->src[1];...4530        for (int64_t i = ir0; i < ir1; ++i) {4531            const int64_t i12 = i/(ne11*ne10);4532            const int64_t i11 = (i - i12*ne11*ne10)/ne10;4533            const int64_t i10 = (i - i12*ne11*ne10 - i11*ne10);4534            const int32_t i01 = *(int32_t *) ((char *) src1->data + i10*nb10 + i11*nb11 + i12*nb12);4535            GGML_ASSERT(i01 >= 0 && i01 < ne01);4538            dequantize_row_q(4539                (const void *) ((char *) src0->data + i01*nb01 + i11*nb02 + i12*nb03),4540                 (float *) ((char *)  dst->data + i10*nb1  + i11*nb2  + i12*nb3), nc);4541        }} 

src0 stores the quantized weights, which are read from the model file.src1 contains the processed user input.Therefore, the function’s purpose is to extract the corresponding rows from a quantized matrix src0 based on the row indices provided by src1 (as seen in line 4534), dequantize them into float, and write them to the output dst. It also supports multi-threaded block processing. The content in dst then serves as the input for the next layer of the large model inference, continuing the loop.f