Introduction The previous article “Implementing LLM from Scratch with 200 Lines of Python” created a “poetry generator” starting from a “probabilistic” implementation, ultimately using PyTorch to realize a classic Bigram model. In the Bigram model, each character is only related to the previous character. Despite this, our <span>babygpt_v1.py</span> also outputs sentences like “Gradually realizing the … Read more
In the Transformer model,the Multi-head Attention Mechanism is indeed a key extension of the Self-Attention mechanism, with the core purpose of enhancing the model’s ability to capture different aspects of information in the input sequence by learning multiple sets of independent attention weights in parallel. Below is a detailed analysis from principles, implementation to advantages: … Read more
As mentioned in previous articles, LoRA fine-tuning primarily targets the weight matrices of linear layers, such as the Q, K, and V projection matrices in the attention mechanism, as well as the weight matrices in the feedforward network (FFN). So, when fine-tuning a model with a Transformer architecture using LoRA, which weight matrices should we … Read more
(1) The drive waveform of soft switch welding machines rises and falls at an angle, as shown in the figure below. The waveform of hard switch machines is like a single tube wave, rising and falling straight. Issues with absorption can cause spikes in the waveform. During maintenance, some waveforms may be unfamiliar; if the … Read more
Recently, a colleague at the company asked me: “Can an oscilloscope be directly connected to the 220V voltage between the live wire and the neutral wire? Will it blow up?” Because it indeed exploded once before, it left a lingering fear~ I believe many of you have similar questions, so today I will summarize this … Read more