Remember to star the public account ⭐️ to receive notifications promptly.
When it comes to artificial intelligence infrastructure, Tenstorrent may not be the first name that comes to mind. However, unlike many AI chip startups competing for venture capital and Nvidia’s market share, Tenstorrent’s chips have actually left the lab.
If you are a bit unconventional and want to break the mold, Tenstorrent’s components and systems are available for anyone in need. In fact, you might be surprised to find that the company has launched three generations of RISC-V-based accelerators aimed at empowering the open-source community.
Their philosophy is to provide reasonably performing accelerators that can efficiently scale from a single card to a 32-chip system or even larger, at a cost that is only a fraction of similar GPU devices from AMD or Nvidia.
El Reg recently had the privilege of experiencing one of the startup’s most powerful systems to date—a liquid-cooled AI workstation, the Blackhole QuietBox, priced at $11,999. This 80-pound (36 kg) machine is essentially a streamlined version of the upcoming Blackhole Galaxy server from the chip startup, which is expected to be released next year. Its intended use is as a development platform for learning architectures, porting existing codebases, and optimizing model kernels before deploying them to production systems.
Moreover, since this machine uses the same chips, memory, and interconnect technology as the Tenstorrent Galaxy server, theoretically, its performance can seamlessly scale to the entire system—something that is not common in most AI workstations today. Of course, you could also stuff four RTX 5000 ADA or Radeon Pro AI R9700 graphics cards into a similarly configured workstation to achieve performance close to that of the QuietBox, but the performance characteristics of those systems would be entirely different from the GB200 rack or MI350 server where your code ultimately runs.
This is one of the reasons Nvidia regularly releases products like the DGX Station, which cram data center-level CPUs and GPUs into cases more suitable for office environments. That said, we fully expect the price of Nvidia’s next-generation DGX Station based on Blackwell Ultra to be several times that of the Tenstorrent QuietBox.
If you are looking for a tool capable of running local AI inference or fine-tuning medium to small models, then Tenstorrent’s QuietBox may not be suitable for you. We expect this situation to change, but for now, the company’s software stack is not mature enough for most local AI enthusiasts.
However, for machine learning software developers interested in exploring Tenstorrent’s hardware architecture and software stack, or those looking to deploy the startup’s chips in production, systems like the Blackhole QuietBox provide a relatively low-cost entry point into the company’s RISC-V-based accelerator ecosystem.
TT-QuietBox Unboxing
Tenstorrent’s QuietBox is not only beautifully designed but also visually striking. The machine runs quietly, but its eye-catching blue stripe paint job is anything but subtle. Tenstorrent has created a highly recognizable machine that deserves full marks.
Whether it’s the Blackhole version we see here or the earlier wormhole-based version, the QuietBox features a custom chassis, with the placement of the heat sinks and front water tank reminiscent of Lian Li’s O11 Dynamic chassis.
The interior of the chassis resembles a chimney, drawing in cool air through a 400 mm heat sink at the bottom and expelling it through another 400 mm heat sink at the top. This may sound a bit exaggerated, but these heat sinks need to handle nearly 1200 watts of accelerator power, not to mention the heat generated by the CPU, system memory, and storage devices that also need to be dissipated.
This design only requires four Noctua 200 mm fans to cool the entire machine, effectively maintaining high performance while controlling noise levels. This is precisely the intention behind the system’s design. Tenstorrent aims to create a high-performance computing platform that users will be happy to place on their desks.
It is indeed quiet, but not completely silent. The fan noise ranges from a gentle hum to a sound similar to a desktop electric heater, which is a fitting description considering the internal construction of the QuietBox. In our tests, the temperature of the accelerators never exceeded 70 degrees Celsius, primarily because all 1300 watts of heat were dissipated into the room.
Between the heat sinks sits an Epyc server motherboard from ASRock Rack, which provides connectivity for the accelerators and several 10 Gigabit Ethernet connections for local network access.
Tenstorrent did not opt for typical workstation CPUs like Intel Xeon-W or AMD Threadripper, but instead chose the Epyc Siena 8124P based on the Zen architecture. This 125-watt chip features 16 Zen4C cores—”C” stands for compact—and can boost up to 3 GHz under load.
The CPU is powered by eight 64 GB DDR5 4800 RDIMM memory modules (with one channel in each of the six channels running in a mode of two DIMMs per channel), totaling 512 GB, with memory bandwidth slightly above 200 GB/s.
Under the CPU heat sink, we can see the main highlight of the QuietBox: four Tenstorrent Blackhole P150 accelerators. These chips collectively boast over 3 petaFLOPS of FP8 dense computing performance.
At the 2024 HotChips conference, we closely examined Tenstorrent’s Blackhole architecture. In short, each 300-watt chip integrates 752 “mini” RISC-V processor cores, which make up the chip’s 140 Tensix processor cores (T) to memory (D), PCIe (P), and Ethernet (E) controllers.
The chip also features 16 SiFive Intelligence x280 cores, which can handle operations that are difficult to parallelize and run Linux. We were unable to test this, but theoretically, this should allow the Blackhole to operate as a standalone computer.
The Tensix cores of the chip work together to provide approximately 774 trillion floating-point operations per second (TFLOPS) of dense FP8 computing capability, or 387 trillion floating-point operations per second (TFLOPS) when using Tenstorrent’s four or eight-bit block floating-point data types. This computing power is matched with 32GB of GDDR6 memory, providing 512GB/s of memory bandwidth. There are four such chips in the system. However, as you will see, actually leveraging such high performance and bandwidth is not an easy task.
If you think spending nearly $12,000 on a complete system like the QuietBox is a bit beyond your budget, Tenstorrent also sells a $1,399 air-cooled (active and passive) version of the graphics card. Alternatively, if you only need a single graphics card, you can opt for a cheaper, slightly lower-performing version priced at $999, which lacks inter-chip networking capabilities and has only 28 GB of memory.
Scalability
The P150 does not rely on proprietary interconnect technologies like PCIe 5.0 or Nvidia’s NVLink, but instead is equipped with four QSFP-DD slots for high-speed 800 Gbps Ethernet connections to other graphics cards in the system. In a sense, it is not a standard Ethernet interface. Tenstorrent warns that these interfaces are only for inter-chip communication and cannot be directly connected to a switch.
Typically, Ethernet speeds are insufficient for scaling networks, but each P150 accelerator has a total bandwidth of 3,200 Gbps, with interconnect bandwidth nearly comparable to Nvidia’s H100 (400 GB/s vs. 450 GB/s).
For the QuietBox, Tenstorrent recommends a 2D ring topology as shown in the diagram below, using the eight 0.5-meter direct copper cables included with the system.
These cables are elegantly designed but come at a high price, costing about $200 each. We can’t help but wonder if using NVLink-style bridge connectors would have been cheaper. However, the QSFP-DD interfaces mean that these network cards can easily scale to 8, 16, or even more. Want to connect several QuietBox network cards? No problem at all.
More importantly, this architecture means that any code developed on the P150 or QuietBox can be directly applied to larger clusters through varying degrees of pipelining, tensor, data, and/or expert parallelism.
In fact, if you closely observe the core layout of the P150, you will notice that only 8 of the 14 Ethernet cores on the chip are activated. We speculate that future versions of the chip will support larger topologies. For example, Tenstorrent’s Blackhole Galaxy will arrange 32 accelerators in a 4×8 mesh structure.
The Blackhole Galaxy is expected to provide approximately 25 petaFLOPS of dense FP8 performance, 1TB of GDDR6 memory, and 16TB/s of total bandwidth. To put this into perspective, the DGX H100 equipped with eight GPUs has a dense FP8 performance slightly below 16 petaFLOPS, with 640GB of HBM3e memory, but a memory bandwidth of 26.8TB/s. Nvidia’s Blackwell and AMD’s MI350 series systems are on another level, and their prices reflect that.
But this refers to a single node. Tenstorrent’s interconnect architecture means it can scale to racks or even larger scales. In a rack-level configuration, we can envision a system containing 192 accelerators. If we add some optical transceivers to enhance transmission distance, theoretically, this platform could scale to multiple racks, accommodating thousands of accelerators.
This is the true meaning of the startup’s claim of “Blackhole infinite scalability.” In fact, its architecture is closer to how Google and Amazon build TPU and Trainium clusters.
Initial Setup
The setup of the QuietBox is slightly different from that of a typical desktop or workstation.
First, it does not have a dedicated graphics card—at least not in the traditional sense. Instead, you can choose to use the motherboard’s integrated graphics (via the included VGA to HDMI adapter) or control the system remotely through its IPMI interface over the network.
Additionally, you need to connect the accelerator lines as shown in the diagram below.
Once everything is ready, we can power on the system. Tenstorrent notes that the first boot may take over 10 minutes. This is normal for Epyc systems, especially those with large memory capacities, but it may catch newcomers off guard, so Tenstorrent’s warning is reasonable.
This machine comes pre-installed with Ubuntu Desktop 22.04 LTS. Since we know that all interactions with the system will be through the terminal, we chose to install OpenSSH for remote access over the network.
Tenstorrent provides an automated installation script that ensures all dependencies are installed, device firmware is fully updated, and the Ethernet mesh network connecting the network cards is functioning properly.
The script mainly involves answering a few “yes/no” questions, most of which you will answer “yes”. Unfortunately, in our initial setup, recent changes to package naming caused the script to fail.
This turned out to foreshadow the direction of Tenstorrent’s remaining software stack development.
In summary, after rebooting (which took about eight minutes), we ran Tenstorrent’s diagnostic tool tt-smi and confirmed that all cards were detected correctly.
All four graphics cards displayed as expected, but evidently `tt-smi` did not accurately recognize our Blackhole graphics cards. Aside from being identified as Tenstorrent’s passive cooling P150b chip, many fields in TT-SMI were either blank or missing. This is a bit concerning, but evidently, it is a normal occurrence and not a cause for alarm.
Model Demonstration
The Blackhole is not a GPU. This means you cannot directly launch models in programs like Ollama or Llama.cpp as you would normally. Fortunately, Tenstorrent’s getting started guide provides many demonstrations, including examples of running LLMs in the TT-Inference-Server.
This process is quite straightforward, and with the help of several convenient auxiliary scripts, it can automatically select the appropriate hardware model and flags.
The entire process took about 45 minutes. Most of the wait time was spent downloading approximately 140 GB of model files and converting them into Tenstorrent’s four-bit and eight-bit block floating-point data types for use.
Ultimately, we obtained an API endpoint compatible with OpenAI that we could point to the frontend (such as Open WebUI) and start issuing commands to the system.
Since LLM inference is one of the primary use cases for such hardware, it is encouraging to see that Tenstorrent has prioritized the documentation for TT-Inference-Server, even if its performance has not yet fully met expectations.
In addition to the LLM service demonstration, Tenstorrent also provides a containerized environment for running `tt-metallium-demos` to launch model demonstrations.
This saves the hassle of downloading and processing the dependencies required to run. After some effort, we successfully ran various models, including ResNet50, BERT, Stable Diffusion 1.4, and the Boltz 2 structural biology base model (which is designed to predict protein structures and binding properties).
Unfortunately, finding these demos requires a laborious search through Tenstorrent’s TT-Metal GitHub repository. Even if you find them, whether they can run is a matter of luck, and there is no documentation to help us figure out why.
We would very much like to see more operational manuals and tutorials like Tenstorrent’s LLM service guide for popular genAI workloads, such as image generation, text and image classification, object detection, speech transcription, and fine-tuning, etc.
New users should not have to struggle to search through GitHub repositories or interpret code comments to run demo programs. A few simple and clear tutorials could greatly attract the interest of developers and students. This would also give Tenstorrent the opportunity to highlight the advantages of its hardware, which are not always apparent given the overall immaturity of the software stack.
Tenstorrent’s Software Philosophy
Over the past few years, one thing has become very clear: no matter how great your GPU or AI ASIC looks on paper, if no one can or wants to program it, it means nothing.
In response to this challenge, Tenstorrent has taken a multi-pronged approach. First, its software stack is entirely open-source. Second, it is developing a low-level API interface similar to Nvidia CUDA, as well as a high-level compiler for running existing PyTorch, JAX, or Onnx models.
This sets Tenstorrent apart from many AI chip startups that may have initially aspired to have such a comprehensive software stack but ultimately delivered only an LLM inference server or API service.
The lowest layer of Tenstorrent’s software stack is its low-level kernel environment (TT-LLK), which is almost as close to programming on bare metal as possible.
Above that is TT-Metalium (abbreviated TT-Metal), which provides a low-level API for writing custom kernels for Tenstorrent hardware in C or C++. You can think of TT-Metal as the equivalent of Nvidia’s CUDA or AMD’s HIP. However, while TT-Metal provides low-level access to hardware features, it also introduces a new programming paradigm.
On top of TT-Metal is TT-NN, a library of supported neural network operations made available to users without requiring them to delve into the underlying hardware. These libraries are compatible with both standard Python and C++, providing a higher-level programming environment for running AI models.
As far as we know, most of Tenstorrent’s model implementations are based on the TT-Metal and TT-NN frameworks. For example, Tenstorrent’s Transformers and vLLM implementations run on top of TT-NN.
Undoubtedly, the programming complexity at these levels is one reason why adding support for new models takes so long, as each model requires custom kernel code to be written manually.
These challenges are not unique to TensorRent. This is also one of the reasons why PyTorch, TensorFlow, and JAX have become so popular in recent years. They provide an abstraction layer that is hardware-agnostic for accelerated computing.
However, since Blackhole is completely different from modern GPUs, using these same frameworks means Tenstorrent needs a compiler.
The company is developing a compiler called Forge, based on multi-level intermediate representation (TT-MLIR). The idea is that TT-Forge can convert PyTorch, JAX, or other models into intermediate representation. Then, TT-Metal can use this intermediate representation to compile compatible kernels for the underlying hardware.
Forge is currently in testing and is evidently under active development. If all goes well, TT-MLIR and Forge will allow users to support new models without having to manually write custom kernels. As we have seen from similar projects, its performance may not match that of TT-NN or TT-Metal, but it will greatly benefit the potential market for Tenstorrent.
AI Performance
Currently, the Tenstorrent Blackhole accelerators on the market, including the QuietBox, are development kits, which makes performance comparisons somewhat tricky. The purpose of these hardware is not to compete directly with Nvidia or AMD GPUs (at least not at this time), but to enable users to write software for Tenstorrent hardware.
This startup’s software stack is evolving rapidly, with performance improvements being submitted to GitHub almost daily. As we previously saw with AMD’s ROCm 6.0 library, the impact of software on inference and training performance far exceeds that of the hardware itself. In just a year and a half, AMD has tripled the inference performance of the MI300X. We are fully confident that over time, Tenstorrent can deliver similar performance improvements.
But this means that any benchmark results we share here will become outdated. Therefore, the following data should be viewed as a snapshot of the performance of the Blackhole P150 and QuietBox as of November 2025, rather than a definitive conclusion on the final performance of these accelerators.
LLM Inference Performance
For the Blackhole accelerators, we tested LLM performance using the TT-Transformers library and Tenstorrent’s vLLM branch. We also tried the TT-Inference-Server, but found that it did not offer significant advantages compared to using vLLM.
For most LLM inference workloads, Tenstorrent uses 4-bit or 8-bit block floating-point data types, which are very similar to OCP’s MXFP4 and MXFP8 or Nvidia’s NVFP4, which we previously discussed.
Unfortunately, the implementation of these data types is not standard. We cannot directly extract NVFP4 quantization data from Hugging Face and expect it to work. Instead, we need to quantize higher precision models to run on the QuietBox’s P150 processors. In AI, quantization refers to compressing model weights from one precision to another, thereby reducing the model’s precision.
The specific combinations of FP8, BF16, BFP8, and BFP4 used by Tenstorrent vary by model to balance performance and accuracy.
To maintain consistency, we chose to quantize the test models (including Llama 3.1 8B and Llama 3.3 70B) to BFP8. In practical applications, we recommend sticking to Tenstorrent’s default settings.
If you’re curious, we used Llama instead of newer, more powerful models like gpt-oss-20B or 120B. The reason is simple: among the supported LLM models, these two Llama models seem to be the best optimized for Blackhole.
Performance of Batch 1
We first tested the single-batch performance of the P150 under single, dual, and quad tensor parallel configurations.
Tensor parallelism is a method of distributing model weights and inference workloads across multiple accelerators. It typically scales better than pipeline parallelism while being more memory-efficient than data parallelism.
The performance of decoding and pre-filling is displayed in blue and green on the left side, respectively. If you are not familiar, decoding performance represents the speed at which the hardware generates tokens, while pre-filling performance measures the time taken to process prompts.
On the right side, we present a chart showing how efficiently the inference engine utilizes the memory bandwidth of the graphics cards to achieve a percentage of its theoretical peak performance.
Even with larger batch sizes, performance was far below expectations. At a batch size of 32, the results showed that increasing from a single card to dual cards, and from dual cards to quad cards, reduced end-to-end latency by approximately 25%. Meanwhile, the overall throughput increased by 36% when going from a single P150 to two, and by 27% when going from two to four.
We also observed similar scalability in online service benchmarking. The quad-card configuration could handle 1.78 times the number of requests as a single P150, while the dual-card configuration’s performance was roughly in between.
Assuming four requests are processed per second, the QuietBox should be able to handle over 14,000 requests per hour while maintaining reasonable response times (TTFT) and interactivity. The performance is not bad, just less than half of what we expected from hardware of this caliber.
The relatively uniform spacing in both tests is reassuring, but the performance is far from it. Given the bandwidth of up to 12.8 Tbps connecting these graphics cards, we had expected better scalability.
To better understand these performance data, we tested and found that the performance of a single P150 is almost identical to that of the Nvidia DGX Spark. In terms of speed and data flow, the Blackhole graphics card should be able to achieve 2-3 times the performance of the latter in FP8 mode when serving LLMs.
It is puzzling that Tenstorrent’s own documentation shows that upgrading from the 8-chip Wormhole QuietBox to the 32-chip Galaxy node actually results in superlinear growth in decoding performance. Meanwhile, for pre-filling, the prompt processing time was reduced by about two-thirds when upgrading from 8 accelerators to 32. This indicates that a lack of optimization is limiting the P150 and QuietBox from realizing their full potential.
Understanding the Half-Baked Performance of Blackhole
So why is the performance of Tenstorrent’s latest accelerators so disappointing? From our observations, the issue seems to be that all the models we tested appear to be using kernels written for their earlier Wormhole accelerators.
These models are forward compatible with Blackhole, meaning they can run, but they cannot leverage the higher core counts of the new chips.
Wormhole has 80 Tensix cores, but on the N150 and N300 chips, only 72 or 128 are actually enabled (64 per ASIC). We suspect this is to maximize compatibility, so most models are ultimately optimized for 64 Tensix cores. Unfortunately, this means that when running kernels written for Wormhole on Blackhole, 76 of the 140 Tensix cores on the chip end up idle.
The lack of kernel optimization also seems to be the reason why Blackhole’s token generation speed is far below expectations. The kernels are not optimized for the additional memory bandwidth, so they cannot utilize it effectively. The decoding performance we observed suggests that the card’s bandwidth is artificially limited to 288 GB/s, which happens to be the bandwidth ceiling of Wormhole.
If our judgment is correct, this is indeed a missed opportunity for Tenstorrent. We understand this is a bit like the chicken or the egg problem. Without the corresponding hardware, kernels cannot be written. But even just one optimized model would be enough to showcase the architectural improvements of Blackhole.
Instead, what we got was an accelerator that seems to only show slight performance improvements over the previous generation in testing, but with power consumption about twice that of the previous generation.
From this, we clearly feel that in the rush to bring the product to market, Tenstorrent’s marketing team may have jumped ahead of the software engineers.
Conclusion
With the Blackhole QuietBox, Tenstorrent has built a powerful, quiet, and relatively affordable development platform for its latest generation of accelerators.
On paper, the four Blackhole P150 chips in this system are expected to achieve a good balance between computing power, memory, and bandwidth, while also providing scalability that similar-priced GPUs cannot match. The application of NVLink in consumer and workstation platforms is already outdated, and the scalability of PCIe is ultimately limited. With 3.2 Tbps of bandwidth per chip, this platform built by Tenstorrent should provide the same linear scalability as its previous generation Wormhole graphics cards.
While some may hesitate at the $11,999 price tag of this machine, to get close to performance, memory, and networking, you would need to spend at least that much, if not more.
The larger issue Tenstorrent faces is that, despite the scalability of those competing platforms being inferior to Tenstorrent’s, they are still more useful and valuable due to their more mature software stacks.
The open-source community (which can be said to be the target market for the P150 and QuietBox) is able and is helping to address this issue, but without clear examples to demonstrate the potential of the Blackhole architecture, it is difficult to market it.
Imagine if Nvidia released a new GPU with three times the performance and nearly double the memory bandwidth, but to fully utilize its performance, it meant rewriting codebases, with no guarantee of actually achieving the claimed performance improvements. That would be asking too much. However, this is essentially the situation we face when using Blackhole.
The lack of optimized kernels for LLM inference (the most important, or at least the most in-demand workload in the world today) is a particularly unforgivable mistake that Tenstorrent should not waste time addressing.
Even just one optimized model, such as OpenAI’s gpt-oss, could greatly enhance confidence and, more importantly, build momentum for the Blackhole architecture.
On the software side, we also hope Tenstorrent can better integrate its documentation. Currently, the company’s documentation is scattered across multiple dedicated websites or buried in dozens of poorly indexed GitHub repositories.
We particularly believe that Tenstorrent could benefit greatly from increasing the number and quality of its “getting started” guides. If members of Tenstorrent’s software team need some inspiration, they might look at the documentation Nvidia provides for DGX Spark.
The more potential customers know about the capabilities that can run and execute on Tenstorrent hardware, the faster the company can build momentum for its products, and the easier it will be to sell accelerators and core IP.
Production servers based on the Tenstorrent Blackhole architecture have yet to begin shipping, so the startup still has time to refine its software platform and make it more robust.
(Source: Content compiled from theregister)*Disclaimer: This article is original by the author. The content of the article reflects the author’s personal views, and Semiconductor Industry Observation reprints it only to convey a different perspective, which does not represent Semiconductor Industry Observation’s endorsement or support of this view. If there are any objections, please feel free to contact Semiconductor Industry Observation.
END
This is the 4240th issue shared by Semiconductor Industry Observation. Welcome to follow.
Recommended Reading
★A Chip That Changed the World
★U.S. Secretary of Commerce: Huawei’s Chips Are Not That Advanced
★“ASML’s New Lithography Machine, Too Expensive!”
★The Quiet Rise of Nvidia’s New Competitors
★Chip Prices Plummet, All Blame Trump
★New Solutions Announced to Replace EUV Lithography!
★Semiconductor Equipment Giants, Salaries Soar by 40%
★Foreign Media: The U.S. Will Propose Banning Software and Hardware Made in China for Cars
Star it ⭐️ to receive notifications promptly, the small account to prevent loss
Seeking likes
Seeking shares
Seeking recommendations
