The Life of a Network Packet in Linux

A practical guide from write() to recv().

You ran curl http://example.com, and now you see some HTML in the terminal, but what actually happened? Linux takes your bytes through a series of defined steps: selecting a path, finding the neighbor’s MAC address, queuing the packet for the network interface card (NIC) to send, and then performing the reverse operation on the other end.

This article aims to explain this path as simply as possible. If you have used Linux, run curl, or tried ip addr, you will be able to understand this article without needing a deep background.

Note: When I refer to the “kernel” in this article, I am actually referring to the “Linux kernel and its network stack,” which is the part of the kernel that runs and moves packets.

Topics Covered

Here is the simplified path we will discuss:

your app
  ↓ write()/send()
TCP (segments your bytes)
  ↓
IP (chooses where to send them)
  ↓
Neighbor/ARP (find the next-hop MAC)
  ↓
qdisc (queueing, pacing)
  ↓
driver/NIC (DMA to hardware)
  ↓
wire / Wi‑Fi / fiber
  ↓
NIC/driver (other host)
  ↓
IP (checks, decides it's for us)
  ↓
TCP (reassembles, ACKs)
  ↓
server app

Part One: Transmission – From `write()` to the Ethernet

Step 1: Your application passes bytes to the kernel

You call send() or write() on a TCP socket, and the kernel accepts your buffer and sends it in order.

TCP will split large buffers into **segments** that fit the path size. The two communicating parties announce their **Maximum Segment Size (MSS)** during the TCP handshake, and the sender will limit its segment size to the MSS announced by the other party, while also being constrained by the current Path MTU and any IP/TCP options (e.g., timestamps).
It will label each segment with a **sequence number** so that the receiver can reassemble them in the correct order.

Socket 🔌Socket is just a communication endpoint for your program. For TCP, the kernel maintains state information for each socket, including:sequence numbers, congestion window, and timers, etc.

TCP Handshake 🤝TCP Handshake is a quick three-step setup before any write() reaches the other end: 1) Client -> Server: SYN, with options (MSS, SACK allowed, window scale, timestamps, ECN). 2) Server -> Client: SYN-ACK with its options. 3) Client -> Server: ACK. Both parties agree on the initial sequence number and options; the state is established. TLS Note: For HTTPS, the TLS handshake runs after TCP is established.

Try it out Run ss -tni while downloading something. You will see the sizes of the TCP send and receive queues fluctuate as data is transmitted over the line and consumed by the application.

Step 2: The kernel decides where to send the data (routing)

The kernel looks at the destination IP and selects the most appropriate route. On a typical host, the question boils down to: Is this IP on my local network, or should I hand it off to the gateway?

If the address is on a directly connected network, it will be sent through that interface.
Otherwise, it will connect to your default gateway (usually a router).

Try it out > ip route get 192.0.2.10 It will print the interface, next hop (if any), and the source IP the kernel will use.

Policy Routing The kernel can use ip rule to query multiple routing tables (e.g., selecting routes by source address or mark). Most laptops and servers use the main routing table.

Step 3: The kernel learns the next-hop MAC (Neighbor/ARP)

IP routing selects the next hop. To actually send an Ethernet frame, the kernel needs the MAC address of that hop.

If the kernel already knows the next hop (in the neighbor/ARP cache), that’s great.
If not, it will send a broadcast ARP request: “Who has 10.0.0.1? Tell me your MAC.” The reply will be cached.

Try it out > ip neigh show You will see entries like 10.0.0.1 lladdr 00:11:22:33:44:55 REACHABLE.

ARP vs NDP IPv4 uses ARP (broadcast). IPv6 uses NDP (multicast). The principle is the same: find the link-layer address for an IP in your network.

Step 4: The packet waits its turn (qdisc)

Before the NIC sends anything, the packet enters the queuing discipline (qdisc). You can think of it as a small waiting line plus a traffic cop, where the kernel can:

Smooth burst traffic to avoid link flooding and bufferbloat (large queues -> high latency),
Fairly share bandwidth among different flows,
If you have configured shaping/rate-limiting rules, enforce them.

Try it outtc qdisc show dev eth0 > tc -s qdisc show dev eth0 # same, but with counters/stats Replace eth0 with your actual interface name (e.g., enp3s0, wlp2s0).

MTU vs MSS MTU is the maximum L2 payload that the link can carry (typical Ethernet is 1500 bytes). MSS is the maximum TCP payload within a segment, just below the IP + TCP headers and options. During the TCP handshake, both parties announce the MSS they can receive, and the sender will not send segments larger than the MSS announced by the other party, while also adhering to the path MTU (PMTU). In common cases without options in IPv4, MSS ≈ MTU−40 bytes. Options further reduce MSS.

Step 5: The NIC driver and NIC do the heavy lifting

The kernel’s network driver hands your packet to the NIC and places it in a small transmit queue from which the card reads. The NIC then:

Directly extracts bytes from memory (using DMA) and converts them into a bitstream on the link, tiny voltage changes on copper cables, light pulses on fiber, or radio waves if you’re using Wi-Fi.

That’s the real “wiring” moment: data in memory becomes signals on the network.

Try it out > ip -s link show dev eth0 > ethtool -S eth0 # NIC stats > ethtool -k eth0 # offloads enabled Replace eth0 with your actual interface name.

Offloads TSO/GSO: Allows the NIC or stack to split large buffers into frames of MTU size. Checksum offload: During transmission, the NIC fills in the IP/TCP checksum after the kernel submits the packet, and before sending, the receiving NIC can verify the checksum and inform the kernel of the result. GRO (Receive Side): Merges many small packets into larger chunks to save CPU.

DMA Direct Memory Access (DMA) allows the NIC to read and write your data in RAM directly over the bus (e.g., PCIe) without CPU copying bytes. This is why NICs can efficiently pull frames from the transmit ring (and place received frames).

Step 6: Going live

On Ethernet, the NIC sends a frame that looks like this:

[ dst MAC | src MAC | EtherType (IPv4) | IP header | TCP header | payload | FCS ]

Switches care about the Ethernet header: they look at the destination MAC address and forward the frame to the correct port.

Routers look at the IP header, decrement the TTL / Hop Limit, and after updating the header checksum (in IPv4), forward the packet to the next hop.

Each switch and router repeats this process hop by hop until the router finally obtains a route directly to the destination network and delivers the packet to the server’s local area network.

Frame vs PacketFrame vs Packet A packet is an IP-level unit (IP header + TCP/UDP + payload). A frame refers to how that packet is transmitted over a specific link (e.g., Ethernet) using src/dst MAC and checksum.

Part Two: Reception – From the line back to your application

Step 7: The NIC passes data to the kernel (NAPI)

On the server side, the NIC writes the received frames into receive rings (small queues in memory). The Linux kernel then efficiently pulls packets using NAPI: quickly switching from interrupts to polling to handle a batch of packets at once.

NAPI If every packet triggered a full interrupt, a busy NIC could overwhelm the CPU. The trick of NAPI is:

Trigger one interrupt,

Temporarily switch to polling to drain a large number of packets,

Then re-enable interrupts.

Fewer interrupts lead to better throughput.

Step 8: IP checks the packet and decides the next action

The kernel verifies the IP header (version, checksum, TTL, etc.) and then asks, “Is this packet for me?”

If the destination IP matches one of the server’s addresses, that IP is local and moves up the stack.
If not, and IP forwarding is enabled, the kernel may forward it, acting like a Linux router.
Otherwise, the packet will be dropped.

If you are using a firewall, hooks like PREROUTING and INPUT (nftables/iptables) can filter, log, or DNAT traffic before it is sent to the local socket. SNAT masquerading may occur in POSTROUTING. For locally generated packets, DNAT may also appear in OUTPUT.

Try it out > sudo nft list ruleset > # or, with iptables: > sudo iptables -L -n -v > sudo iptables -t nat -L -n -v

Step 9: TCP reassembles, acknowledges, and wakes the application

The TCP stack will order the segments, check for missing parts, and send ACKs. When the data is ready, it will wake the process waiting in recv().

Try it out > ss -tni 'sport = :80 or dport = :80' As the application reads, the receive queue (Recv-Q) will grow and shrink.

Short Practical Notes

Loopback is special (and fast)

Packets sent to 127.0.0.1 never reach the physical NIC. Routing still occurs, but everything stays within the memory of the purely software <code>lo interface.

Bridging vs Routing (Same box, different roles)

If the box is a bridge (e.g., with br0), it forwards frames at Layer 2 without changing the TTL. If it is routing, it forwards at Layer 3, decrementing the TTL by one hop.

NAT Hairpin (Why internal clients access external IP)

Accessing services via the router’s public IP from the same local area network requires “hairpin NAT.” If connections reset in this case, check the PREROUTING and <code>POSTROUTING NAT rules.

IPv6

Replace ARP with NDP. Otherwise, the path is the same:

ip -6 route
ip -6 neigh

UDP is intentionally different

UDP does not do ordering, retransmission, or congestion control. The sending path uses udp_sendmsg, and the receiving path delivers the complete datagram. Your application is responsible for handling data loss.

Take a look in person (10 quick commands)

# 1) Where would the kernel send a packet?
ip route get 192.0.2.10

# 2) What routes and rules exist?
ip route; ip rule

# 3) Who's my next hop?
ip neigh show

# 4) What's my firewall/NAT doing?
sudo nft list ruleset
# or:
sudo iptables -L -n -v
sudo iptables -t nat -L -n -v

# 5) Which sockets are active?
ss -tni

# 6) What's on the wire (swap eth0/host as needed)?
sudo tcpdump -ni eth0 -e -vvv 'host 192.0.2.10 and tcp port 80'

# 7) Are my queues healthy?
tc -s qdisc show dev eth0

# 8) Is my NIC happy?
ip -s link show dev eth0
ethtool -S eth0

# 9) Are counters hinting at a problem?
nstat -a | grep -E 'InErrors|OutErrors|InNoRoutes|InOctets|OutOctets'
# (Use `-z` instead of `-a` if you explicitly want to zero the counters.)

# 10) Is the path MTU safe?
tracepath 192.0.2.10   # discovers PMTU via ICMP: IPv4 "Fragmentation Needed" (Type 3, Code 4) / IPv6 "Packet Too Big" (Type 2)

ARP/Neighbor Issues

IP neigh shows failures or constantly switching states -> L2 reachability, VLAN tagging, or switch filtering issues.

MTU / PMTU Black Hole

Small pings work fine, large transfers stall -> MTU mismatch or ICMP being blocked.

Allow PMTU signals through the firewall (IPv4: ICMP Type 3 Code 4 “Fragmentation Needed”, IPv6: ICMPv6 Type 2 “Packet Too Big”) or fix the MTU.

Reverse Path Filter Pain Points

Asymmetric routing + rp_filter=1 will drop return traffic. Use rp_filter=2 (loose) or make routing symmetric.

NAT Surprises

SNAT/MASQUERADE incorrectly rewrites the source, so replies are invalid. Check NAT rules and conntrack -L.

Backlog/Accept Pressure

New connections reset under heavy load -> increase application backlog and <code>net.core.somaxconn, ensuring the application can process <code>accept in a timely manner.

Bufferbloat from Bursts

If you encounter issues with large queues and severe latency spikes, choose **fq_codel<code>** (or **<code>fq**) as the **queuing discipline (qdisc)**, and if the application supports it, enable **packet pacing**.

The kernel call path (if you’re interested) for sending (typical TCP path):

tcp_sendmsg
  -> tcp_push_pending_frames
    -> __tcp_transmit_skb
      -> ip_queue_xmit
        -> ip_local_out / ip_output
          -> ip_finish_output
            -> neigh_output
              -> dev_queue_xmit
                -> qdisc / sch_direct_xmit
                  -> ndo_start_xmit (driver)

Receiving (typical IPv4 TCP path):

napi_gro_receive / netif_receive_skb
  -> __netif_receive_skb_core
    -> ip_rcv
      -> ip_rcv_finish
        -> ip_local_deliver
          -> ip_local_deliver_finish
            -> tcp_v4_rcv
              -> tcp_v4_do_rcv
                -> tcp_data_queue (wake reader)

A small checklist to keep handy

Socket – Your program’s handle for network I/O.
MTU / MSS – Max link payload / max TCP payload.
ARP / NDP – Find the link-layer address (IPv4 / IPv6).
qdisc – Per-device queuing policy (fairness, shaping).
NAPI – Efficient receive: interrupt, then poll a batch.
TSO/GSO/GRO – Offloads to split/merge packets and save CPU.
Conntrack – Kernel’s flow table (used by NAT and filtering).
PREROUTING/INPUT/OUTPUT/POSTROUTING – Firewall hook points.
DMA (Direct Memory Access) – Hardware reads/writes RAM without CPU copies, NICs use this for TX/RX rings.
TTL / Hop Limit – Per‑packet counter decremented by each router (TTL in IPv4, Hop Limit in IPv6). When it hits zero, the packet is dropped.
FCS (Frame Check Sequence) – Link‑layer CRC at the end of an Ethernet frame, used to detect bit errors on the wire.

Translated from:https://www.0xkato.xyz/life-of-a-packet-in-the-linux-kernel/

The Life of a Network Packet in Linux

Topics Covered

Part One: Transmission – From `<span>write()</span>` to the Ethernet

Step 1: Your application passes bytes to the kernel

Step 2: The kernel decides where to send the data (routing)

Step 3: The kernel learns the next-hop MAC (Neighbor/ARP)

Step 4: The packet waits its turn (qdisc)

Step 5: The NIC driver and NIC do the heavy lifting

Step 6: Going live

Part Two: Reception – From the line back to your application

Step 7: The NIC passes data to the kernel (NAPI)

Step 8: IP checks the packet and decides the next action

Step 9: TCP reassembles, acknowledges, and wakes the application

Short Practical Notes

Loopback is special (and fast)

Bridging vs Routing (Same box, different roles)

NAT Hairpin (Why internal clients access external IP)

IPv6

UDP is intentionally different

Take a look in person (10 quick commands)

ARP/Neighbor Issues

MTU / PMTU Black Hole

Reverse Path Filter Pain Points

NAT Surprises

Backlog/Accept Pressure

Bufferbloat from Bursts

A small checklist to keep handy

Leave a Comment Cancel reply

Topics Covered

Part One: Transmission – From <span>write()</span> to the Ethernet

Step 1: Your application passes bytes to the kernel

Step 2: The kernel decides where to send the data (routing)

Step 3: The kernel learns the next-hop MAC (Neighbor/ARP)

Step 4: The packet waits its turn (qdisc)

Step 5: The NIC driver and NIC do the heavy lifting

Step 6: Going live

Part Two: Reception – From the line back to your application

Step 7: The NIC passes data to the kernel (NAPI)

Step 8: IP checks the packet and decides the next action

Step 9: TCP reassembles, acknowledges, and wakes the application

Short Practical Notes

Loopback is special (and fast)

Bridging vs Routing (Same box, different roles)

NAT Hairpin (Why internal clients access external IP)

IPv6

UDP is intentionally different

Take a look in person (10 quick commands)

ARP/Neighbor Issues

MTU / PMTU Black Hole

Reverse Path Filter Pain Points

NAT Surprises

Backlog/Accept Pressure

Bufferbloat from Bursts

A small checklist to keep handy

Related posts

Leave a Comment Cancel reply

Part One: Transmission – From `<span>write()</span>` to the Ethernet