Understanding the Overall Architecture of LLAMA C++

In that primordial ocean, during our ancient times, the ability to see and perceive the environment triggered a Cambrian explosion of interaction with other life forms. Today, that light is illuminating digital thinking. Spatial intelligence is enabling machines not only to interact with each other but also with humans and the three-dimensional world, whether real or virtual.

zhuxy, WeChat public account: Frontend Blackboard Report. With the help of spatial intelligence, artificial intelligence will understand the real world – “AI Matriarch” Li Feifei – (includes a mind map)

Starting from the input, to the output characters, and then feeding the output characters back into the reasoning framework of artificial intelligence. This week, I learned about the overall framework of LLAMA C++.At the AI prompt, if we input hello and press enter, LLAMA C++ will first encode our input and store it:

llama C++'s main function => llama_context::decode => llama_batch_allocr::ubatch_add

The encoded input is stored in a structure: mctx

llama_context::decode 1036 mctx = memory->init_batch(*balloc, cparams.n_ubatch, output_all);

Next is the construction of the transformer computation framework:

llama_context::decode => process_ubatch => build_graph => llm_build_gemma3_iswa

llm_build_gemma3_iswa is the main constructor of the transformer computation framework:

35 ggml_tensor * Qcur = build_lora_mm(model.layers[il].wq, cur);38 ggml_tensor * Kcur = build_lora_mm(model.layers[il].wk, cur);41 ggml_tensor * Vcur = build_lora_mm(model.layers[il].wv, cur);71 cur = build_attn(inp_attn,72 model.layers[il].wo, NULL,73 Qcur, Kcur, Vcur, nullptr, nullptr, nullptr, 1.0f, il);

In constructing Q, K, and V, the interdependencies between calculations form a graph relationship, for example inggml_rms_norm_impl:

2989 result->src[0] = a;

Then in theggml_visit_parents function, these dependencies are organized into a graph:

6546 cgraph->nodes[cgraph->n_nodes] = node;6547 cgraph->n_nodes++;

Next, in theggml_backend_sched_compute_splits function, this graph is split into tasks for parallelization.

Then in theggml_graph_compute_thread function, the split graph is computed:

2884 for (int node_n = 0; node_n < cgraph->n_nodes && atomic_load_explicit(&tp->abort, memory_order_relaxed) != node_n; node_n++) {2885 struct ggml_tensor * node = cgraph->nodes[node_n];2886 ggml_compute_forward(&params, node);

The specific computation is executed by theggml_compute_forward function, such as matrix calculations, and the results are passed to the next layer’s nodes for further computation.

When the transformer completes all levels of attention inference, the results are placed in the data structure ctx->logits:

976 int llama_context::decode1144 auto * t_logits = res->get_logits();

1157 float * logits_out = logits + n_outputs_prev*n_vocab;1158 1159 if (n_outputs) {1160 GGML_ASSERT( n_outputs_prev + n_outputs <= n_outputs_all);1161 GGML_ASSERT((n_outputs_prev + n_outputs)*n_vocab <= (int64_t) logits_size);1162 ggml_backend_tensor_get_async(backend_res, t_logits, logits_out, 0, n_outputs*n_vocab*sizeof(float));1163 }

In the next round of the main loop of llama C++, this output will be used as the input for the next round:

86 int main(int argc, char ** argv) {702 const llama_token id = common_sampler_sample(smpl, ctx, -1);

339 llama_token common_sampler_sample(struct common_sampler * gsmpl, struct llama_context * ctx, int idx, bool grammar_first) {340 gsmpl->set_logits(ctx, idx);

llama_context::get_logits_ith590 return logits + j*model.vocab.n_tokens();