Accelerating IPFS DHT’s HTTP API: Someguy Optimizes Full Resolution

In the IPFS ecosystem, the content retrieval efficiency of resource-constrained devices such as browsers and mobile phones has long been limited by the inability to run a complete DHT (Distributed Hash Table) client. In 2024, the IPFS team launched an optimization plan for the HTTP delegated routing API targeting Amino DHT and IPNI — Someguy, which significantly improves address return rates through address book caching and proactive peer discovery technology, thereby enhancing the speed of peer-to-peer content retrieval for lightweight devices. This solution has been integrated into Someguy v0.7.0 and subsequent versions, becoming a core optimization direction for lightweight access to IPFS.

1. Someguy: The “Routing Mediator” for Lightweight IPFS Access

1. Core Positioning and Functionality

Someguy is essentially a delegated routing HTTP API, serving as a “routing mediator”: it proxies routing requests from the IPFS client to the routing systems of Amino DHT, IPNI, or other compatible APIs, ultimately returning results that include the “provider peers” corresponding to the CID (Content Identifier) and their network addresses, providing services via an HTTP interface.

2. Core Pain Points Addressed

The Amino DHT client needs to maintain hundreds of network connections to manage the routing table and find content providers, which is extremely unfriendly to devices like browsers and mobile phones — these devices have limited network transmission methods, a small number of simultaneous connections, and are constrained by battery capacity and bandwidth, making it impossible to support the operation of a complete DHT client.

The delegated routing mechanism of Someguy precisely solves this problem: devices do not need to maintain complex DHT connections themselves and can query content providers in the DHT with just a single HTTP request. To facilitate developer use, the IPFS Foundation also provides a public delegated routing endpoint: <span><span>https://delegated-ipfs.dev/routing/v1</span></span>, which Helia (one of the core implementations of IPFS) uses by default to accelerate content retrieval.

2. The “Three-Step Process” of Someguy Content Retrieval

When Helia or <span><span>helia/verified-fetch</span></span> retrieves content from the IPFS network, Someguy completes routing and matching according to the following process:

1. Initiate Request: Helia sends a GET request to Someguy with a streaming response header (Accept: application/x-ndjson), specifying the target CID.
2. Traversal and Response: Someguy traverses the Amino DHT, filtering out providers that have the target content, and returns results containing their network addresses, peer IDs, and other information.
3. Direct Connection to Retrieve Content: The browser or mobile application, upon receiving the provider records, directly establishes connections with these peers and attempts to quickly retrieve content through parallel connections.

In short, the response speed of Someguy to “valid addresses” directly determines the efficiency of the client in initiating content retrieval — every millisecond saved in routing queries translates to an increase in content delivery speed.

3. Pain Points of Old Versions: Invalid Responses of “ID without Address”

Before Someguy v0.7, a core issue severely hampered retrieval efficiency: the returned provider records often “lacked substance” — containing only peer IDs but missing critical network addresses.

1. Problem Manifestation

Invalid responses, with the <span><span>Addrs</span></span> field empty, prevent the client from directly connecting to peers:

This forces the client to send additional requests (<span><span>/routing/v1/peers/{peerid}</span></span>) to obtain addresses, which not only increases the number of network requests but also significantly prolongs content retrieval time.

2. Root Cause of the Problem: Mismatched Cache Times

The problem stems from three core constants controlling cache time in the go-libp2p and go-libp2p-kad-dht libraries, where the “time difference” among them leads to address expiration:

• DefaultProvideValidity (48 hours): The “association record” (provider record) between CID and peer ID is retained in memory for 48 hours;
• DefaultProviderAddrTTL (24 hours): The network address of the provider is retained for only 24 hours, and must be re-searched after expiration;
• RecentlyConnectedAddrTTL (15 minutes): The address of a disconnected peer is retained in memory for only 15 minutes.

This means that within 24-48 hours after the provider record is published, the DHT server will return records of “ID without address” (to avoid returning outdated addresses). Although the theoretical “re-provisioning” mechanism every 24 hours should update addresses, in practice, the process is complex, and the address absence issue frequently occurs.

4. Optimization Plan: Three Mechanisms to Ensure “Address Availability”

To address the “address absence” issue, the team introduced three core mechanisms through PR 90 to ensure that Someguy either returns provider records with valid addresses or does not return records at all, completely avoiding invalid requests from the client.

1. Address Book Caching: Efficient Storage of “Available Addresses”

Based on the <span><span>memoryAddrBook</span></span> encapsulation of go-libp2p, an address caching system with both performance and capacity has been created, featuring core characteristics including:

• 48-hour caching cycle: Synchronized with the expiration time of provider records (48 hours), ensuring that addresses and records “live and die together”;
• 1 million peer capacity: Setting a memory usage limit to support large-scale peer node management while avoiding memory overflow;
• LRU eviction strategy: Prioritizing the retention of recently accessed peer addresses to enhance cache utilization;
• Event-driven caching: By subscribing to the libp2p event bus, addresses are cached only after successful peer authentication, eliminating the need for active polling of the DHT and reducing resource consumption.

2. Background Active Probing: Proactively Validating “Address Validity”

Someguy no longer passively waits for address requests but actively verifies the availability of cached addresses in the background, with core logic including:

• 15-minute periodic checks: Every 15 minutes, connection tests are performed on cached addresses to promptly eliminate invalid addresses;
• Exponential backoff strategy: For peers that remain offline, the probing frequency is gradually reduced to avoid resource waste;
• 20 concurrent tests: Validating up to 20 peer connections simultaneously to enhance probing efficiency;
• Selective probing: Testing only those peers that have not been verified recently to reduce redundant work.

3. Cached Router: Optimizing “Response Logic”

In <span><span>server_cached_router.go</span></span>, the response process has been restructured using a non-blocking iterator to make result returns more efficient:

• Cache priority: Prioritizing the return of verified cached addresses to reduce waiting time;
• Background completion: If no address is found in the cache, new addresses are asynchronously searched without blocking the current response;
• Streaming return: Once a valid address is found, it is sent immediately without waiting for all results to be gathered;
• Filtering invalid:

The above optimizations are enabled by default in Someguy v0.7.0 and subsequent versions, and can be manually disabled via the environment variable <span><span>SOMEGUY_CACHED_ADDR_BOOK</span></span>.

5. Optimization Results: Data Witnessing Performance Leap

To validate the optimization value, the team deployed two sets of Someguy instances (one set with caching and active probing enabled, and the other set with them disabled) and quantified the effects through real-world scenario testing.

1. Test Basic Configuration

• Cache warming: The address book needs 12 hours to complete “warming up”, stabilizing at about 30,000 peers, and finally stabilizing around 60,000 (matching the number of DHT servers measured by ProbeLab in Q3 2025);
• Test traffic: At a rate of 100 requests/second, 500,000 CID requests from the public ipfs.io gateway were sent to the /routing/v1/providers/[CID] endpoint of both instances (not deduplicated, simulating high-frequency access to popular content in real scenarios);
• Core metrics: Cache hit rate, HTTP request latency (P95 percentile, reflecting the performance of the vast majority of requests), HTTP success rate.

2. Key Metric Performance

(1) Cache Validity: 83% Reduction in Additional Requests

Out of a total of 3,742,739 lookups:

• 65.6% (2,455,120 requests): DHT directly returned records with addresses, without relying on the cache;
• 34.4% (1,287,619 requests): DHT returned records without addresses, requiring supplementation from the cache:
• Cache hit rate reached 83%: 83% of requests directly obtained valid addresses from the cache;
• Only 17% required additional lookups: significantly reducing invalid requests from the client.

(2) Latency and Success Rate: Dual Improvement

Scenario	200 Response P95 Latency	404 Response P95 Latency	Success Rate	Latency Improvement
Cache Disabled (Baseline)	1.91 seconds	7.35 seconds	52.0%	–
Cache Enabled (Warmed Up)	1.35 seconds	7.46 seconds	57.2%	560 milliseconds (29%) faster

Core Conclusion:

• Latency for successful responses decreased by 29%, from 1.91 seconds to 1.35 seconds;
• Success rate increased by 5.2 percentage points, from 52.0% to 57.2%;
• Active probing validated peer connections in advance, allowing repeated CID requests to directly reuse the cache, reducing DHT traversal times and indirectly enhancing overall performance.

6. Additional Optimization: HTTP-Level Caching Reduces Server Load

In addition to peer address caching, Someguy also implements HTTP-level caching through the <span><span>Cache-Control</span></span> header, forming a “dual-layer caching system” that further enhances efficiency.

1. Caching Strategy Breakdown

• Valid Responses (with Providers): Cached for 5 minutes — capturing newly online providers in a timely manner while reducing repeated DHT lookups;
• Empty Responses (without Providers): Cached for 15 seconds — short validity ensures that content can be quickly discovered after going online;
• Reusing Expired Data: Through the stale-while-revalidate mechanism, clients are allowed to continue using expired data within 48 hours while updating data in the background, avoiding waiting.

2. Synergistic Value of Dual-Layer Caching

• Peer address caching: Ensures that returned records “have available addresses”;
• HTTP caching: Reduces repeated requests from different clients for the same CID;
• Supports CDN acceleration: CDNs can directly cache routing responses for popular content without needing to return to Someguy, further reducing server pressure.

7. Configuration and Monitoring: Flexibly Adapting to Business Needs

1. Core Configuration Items

Through environment variables, caching and probing strategies can be flexibly adjusted, with key parameters as follows (detailed explanations can be found in the official documentation):

• <span><span>SOMEGUY_CACHED_ADDR_BOOK: Enable/Disable address caching;</span></span>
• <span><span>SOMEGUY_CACHED_ADDR_BOOK_ACTIVE_PROBING: Enable/Disable background active probing;</span></span>
• <span><span>SOMEGUY_CACHED_ADDR_BOOK_RECENT_TTL: Caching time for recently connected peer addresses.</span></span>

2. Monitoring Metrics

After enabling optimizations, the following core metrics can be monitored via Prometheus (details can be found in the official monitoring documentation):

• Number of peers in the cached address book;
• Success rate of active probing and the proportion of invalid addresses;
• Cache hit rate and distribution of HTTP request latency.

8. Conclusion and Outlook

The optimization of Someguy is a significant breakthrough for lightweight access to IPFS: through address caching, active probing, and HTTP dual-layer caching, it not only addresses the core pain point of “address absence” but also achieves an 83% reduction in additional requests, a 29% decrease in latency, and a 5.2% increase in success rate, greatly improving the IPFS content retrieval experience for devices such as browsers and mobile phones.

Currently, these optimizations have been implemented in Someguy v0.7.0 and above, and the public endpoint <span><span>https://delegated-ipfs.dev/routing/v1/providers</span></span> is fully enabled, allowing developers to use it directly or deploy private instances to provide delegated routing services for their own businesses.

In the future, the team will continue to explore more optimization directions, such as dynamically adjusting caching strategies and optimizing DHT traversal efficiency, further enhancing the availability and performance of IPFS on various devices, and promoting the popularization of decentralized content access.