How to Maximize Concurrency and Performance When Making HTTP Requests with HttpWebRequest in C#

Introduction

When usingHttpWebRequest to initiate HTTP requests in C#, you can improve concurrency and performance in the following ways:

1. ServicePointManager Settings

ServicePointManager is a static class that provides properties and methods for managing HTTP connections. To enhance concurrent performance, you need to adjust the following key properties:

DefaultConnectionLimit: By default, the .NET Framework’sServicePointManager limits the number of concurrent connections to the same domain (usually 2). You can increase this limit to allow for more concurrent connections.

ServicePointManager.DefaultConnectionLimit = 100; // A suitable value, e.g., 100

Expect100Continue: When you send a POST request, .NET first sends a request with theExpect: 100-continue header to ask the server if it is willing to accept the data. Disabling this option may improve performance.

ServicePointManager.Expect100Continue = false;

ReusePort: This is a setting in .NET Core. If you are using .NET Core, enabling this setting allows different HTTP requests to reuse the same local port.

ServicePointManager.ReusePort = true;

2. Asynchronous Programming Model

Using asynchronous methods ofHttpWebRequest, such asBeginGetResponse and EndGetResponse or GetResponseAsync, allows your application to remain non-blocking while waiting for HTTP responses, which is crucial for improving concurrent performance.

3. Resource Utilization and Release

Ensure that you promptly releaseHttpWebResponse objects and other resources after the request is completed to avoid unnecessary resource consumption and memory leaks.

using (HttpWebResponse response = (HttpWebResponse)request.GetResponse()) { // Process response }

4. Parallel Processing

In .NET, you can use theParallel class or the Task class to send multiple requests in parallel.

var tasks = urls.Select(url => Task.Run(() => { 
// Send request using HttpWebRequest })).ToArray(); 
Task.WaitAll(tasks); // Wait for all requests to complete

5. Use HttpClient

If possible, consider using theHttpClient class instead ofHttpWebRequest.

HttpClient is a more modern HTTP client that provides a cleaner API, better asynchronous support, and is configured by default for higher concurrent connection limits.

using (var client = new HttpClient()) { // Send request }

6. System-Level Configuration

Sometimes, operating system-level settings can also affect HTTP concurrent performance. For example, on Windows, you may need to adjust theMaxUserPort and TcpTimedWaitDelay values in the registry to increase the number of available ports and reduce the wait time before ports are released.

In Windows,MaxUserPort is a registry entry that determines the maximum user port number available. By default, the value ofMaxUserPort is typically set to 5000, meaning the TCP/IP stack will use port numbers between 1024 and 5000 for user TCP/UDP connections.

If you need to adjust the value ofMaxUserPort (for example, if you want to allow more concurrent network connections), you can modify it using the registry editor (regedit). Typically, the value ofMaxUserPort is located at the following registry path:

HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\Tcpip\Parameters

After modifying the value ofMaxUserPort, you usually need to restart the Windows operating system for the changes to take effect, as the TCP/IP stack needs to reload the configuration parameters.

For high-concurrency applications, theMaxUserPort value can be set higher to allow the system to open more dynamic ports. Theoretically, this value can be set to 65534 (since 0 to 1023 are reserved ports, and 65535 is the maximum port number), but in practice, the recommended maximum value is usually lower than this theoretical limit.

In practice, many Windows server administrators may set theMaxUserPort between 10000 and 60000, depending on application needs and system and network environments. Microsoft’s official documentation has suggested settingMaxUserPort to 32768, but this is not a hard limit, and actual applications should be set according to specific circumstances.

Be cautious when settingMaxUserPort, as very high values may lead to increased consumption of system resources (such as memory and handles). Additionally, this change may affect network security policies. Therefore, it is best to evaluate the actual needs of the application before adjusting this value and conduct thorough testing in a test environment.

After modifyingMaxUserPort, you need to restart the system for the settings to take effect. It is also recommended to adjust theTcpTimedWaitDelay registry entry together, which can help recover ports in the TIME_WAIT state more quickly, allowing the system to reuse these ports. By default, the value ofTcpTimedWaitDelay is 240 seconds, but it can be reduced to 30-60 seconds, especially in high-concurrency environments, which can help reduce connection issues caused by port exhaustion.

TcpTimedWaitDelay is an entry in the Windows registry that controls the time a port remains in the TIME_WAIT state after a TCP connection is closed. The TIME_WAIT state is a holding state after a TCP connection is terminated, ensuring that the final acknowledgment packet can arrive. This time is set to 240 seconds by default.

Reducing the value ofTcpTimedWaitDelay can speed up port recovery, which is beneficial for high-concurrency applications that need to handle a large number of short connections. Typical settings forTcpTimedWaitDelay range between 30 seconds and 120 seconds.

To setTcpTimedWaitDelay, follow these steps:

1. Open the registry editor (regedit). This can be done by searching for “regedit” in the start menu or by pressing Win + R to open the run window, then typing regedit and pressing enter.

2. Navigate to the following path:

HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\Tcpip\Parameters

3. Look for theTcpTimedWaitDelay key value. If it does not exist, you need to create it:

4. Right-click on theParameters directory, select New -> DWORD (32-bit) Value.

5. Name the newly created key value as TcpTimedWaitDelay.

6. Double-click theTcpTimedWaitDelay key value, then enter the number of seconds you wish to set in the “Value data” box (make sure you select decimal instead of hexadecimal).

7. Click “OK” to save the changes.

8. Close the registry editor.

9. To make the changes effective, you need to restart your computer.

Before adjustingTcpTimedWaitDelay, ensure you understand the impact of the changes and conduct thorough testing in a test environment before making changes in a production environment. Improper settings may lead to unexpected behavior, such as potential network issues or performance degradation.

7. Server Settings

The improvement of client performance also depends on server-side configuration. Ensure that the server can handle high concurrent connections and requests.

8. Performance Testing

Use stress testing tools (such as JMeter or LoadRunner) to perform stress tests on your application to determine the optimal concurrent settings. Testing can reveal performance bottlenecks, allowing you to adjust settings accordingly.

Notes

1. SettingServicePointManager.DefaultConnectionLimit too high may increase server pressure and even lead to denial of service; it should be set cautiously based on actual conditions.

2. In high-concurrency scenarios,HttpClient is usually a better choice thanHttpWebRequest.

3. When using asynchronous programming patterns, ensure you understand the async and await keywords to avoid common pitfalls, such as deadlocks.

Source: Eric Zhou

Link: cnblogs.com/tianqing/p/17868395.html

How to Maximize Concurrency and Performance When Making HTTP Requests with HttpWebRequest in C# Reply [Close]Learn to permanently close app splash adsReply [Delete]Learn to automatically detect which WeChat friends have been deleted or blockedReply [Manual]Get a 30,000-word .NET, C# engineer interview manualReply [Help]Get 100+ commonly used C# helper librariesReply [Join Group]Join the DotNet learning exchange group

2. Asynchronous Programming Model

3. Resource Utilization and Release

4. Parallel Processing

5. Use HttpClient

6. System-Level Configuration

Related posts

Leave a Comment Cancel reply