In the context of the Internet of Things (IoT), video streaming involves numerous parameters and selections from acquisition, encoding, transmission to playback. This article outlines the core concepts of video streaming, parameter selection, and low-latency optimization suggestions, laying the groundwork for subsequent protocols, platforms, and engineering practices.
II. Full Link Process of Video Streaming
Original Video Frame ↓ Compression Encoding (H.264/H.265/AV1) ↓ Packaging (MP4/FLV/TS/Raw RTP) ↓ Transmission (RTSP/RTMP/HLS/WebRTC/SRT) ↓ Player DecodingCore Concepts of Each Link in the Full Process
1. Original Video Frame (Acquisition)
- Resolution: Such as 1920×1080, 1280×720, determines image clarity.
- Frame Rate (fps): How many frames are captured per second, commonly 15/25/30 fps.
- Color Space/Pixel Format: Such as YUV, RGB, affects subsequent encoding efficiency.
- Bit Depth: Such as 8bit, 10bit, determines color accuracy.
- Acquisition Method: Camera, screen capture, file reading, etc.
- Engineering Focus: Acquisition end capabilities, latency, frame loss, audio-video synchronization.
2. Compression Encoding (H.264/H.265/AV1, etc.)
- Encoder Type: H.264 (mainstream), H.265, AV1, etc.
- Frame Type: I-frames (key frames), P-frames, B-frames.
- GOP (Group of Pictures): Interval of I-frames, affects startup latency and compression efficiency.
- Bitrate Control: CBR (Constant), VBR (Variable), ABR (Adaptive).
- Pixel Format: Such as yuv420p (most common), yuv422p.
- Profile/Level: Such as baseline, main, high, affects compatibility.
- Frame Rate: Frame rate output from encoding (may differ from acquisition).
- Preset/Tuning: Such as veryfast, zerolatency, affects encoding speed and latency.
- Engineering Focus: Parameter selection balances image quality, latency, and bandwidth; in low-latency scenarios, disable B-frames and shorten GOP.
3. Packaging (MP4/FLV/TS/Raw RTP, etc.)
- Packaging Format: Determines the way audio and video streams are packaged and their compatibility.
- MP4: Friendly for on-demand, supports fast forward and rewind.
- FLV: Commonly used for live streaming, low latency.
- TS: Commonly used for HLS, fragmented transmission.
- Raw RTP: Real-time transmission, commonly used in WebRTC/national standards.
- Timestamp/Synchronization Information: Ensures audio-video synchronization.
- Stream Information: Metadata such as frame rate, resolution, bitrate, etc.
- Engineering Focus: Format selection, synchronization, information integrity.
4. Transmission (RTSP/RTMP/HLS/WebRTC/SRT, etc.)
- Protocol Type:
- RTSP: Low latency, suitable for local area networks/monitoring.
- RTMP: Commonly used for live streaming, pushing/pulling streams.
- HLS: HTTP fragmentation, suitable for large-scale distribution, higher latency.
- WebRTC: Ultra-low latency, peer-to-peer/interactivity.
- SRT: Secure and reliable, optimized for weak networks.
- Bandwidth and Bitrate: Directly affects smoothness and image quality.
- Packet Loss/Retransmission/Buffering: Affects stability and latency.
- Multiplexing/Distribution: CDN, forwarding, transcoding, etc.
- Engineering Focus: Protocol selection, adaptation to weak networks, end-to-end latency control.
5. Player Decoding
- Decoder Support: H.264/H.265/AV1, etc., software/hardware decoding capabilities.
- Buffering Strategy: Affects startup speed and stuttering rate.
- Synchronization Mechanism: Audio-video synchronization, clock calibration.
- Compatibility: Support status across different terminals/browsers/apps.
- Rendering Frame Rate: Actual playback frame rate, may drop due to performance/network.
- Engineering Focus: Terminal decoding capabilities, startup latency, stuttering, compatibility adaptation.
Influencing Factors: Resolution, frame rate, GOP, bitrate control, B-frames, low latency, network jitter.
III. Detailed Explanation of Basic Concepts
1. Frame Types (I/P/B Frames)
- I-Frame (Key Frame): Can be independently decoded, largest size, determines startup latency.
- P-Frame: References the previous frame, medium size.
- B-Frame: References both previous and next frames, high compression rate but introduces latency.
- Engineering Suggestion: In low-latency scenarios, disable/reduce B-frames, shorten GOP.
2. GOP (Group of Pictures)
- Definition: Length of the frame group between two I-frames, GOP = frame rate × seconds.
- Impact: Longer GOP increases compression efficiency but results in greater startup/seek latency.
- Suggestion: For monitoring/inspection, keep GOP at 0.5–2 seconds; for real-time interaction, shorter is better.
3. Resolution and Frame Rate
| Resolution | Typical Scenario |
|---|---|
| 1920×1080 | HD live streaming/recording |
| 1280×720 | Mainstream live streaming/conferencing |
| 640×480 | Monitoring/national standard/weak network |
| 320×240 | IoT/preview stream |
- Frame Rate: 15/20/25/30 fps, monitoring/inspection at 15–25 fps is sufficient.
4. Bitrate and Bitrate Control
- CBR (Constant Bitrate): Bandwidth controllable, image quality fluctuates.
- VBR (Variable Bitrate): Stable image quality, peaks high.
- ABR (Adaptive Bitrate): Multi-bitrate switching, suitable for HLS/DASH.
- Experience Values:
- 1080p@25fps: 2–4 Mbps
- 720p@25fps: 1–2 Mbps
- 480p@15–20fps: 0.5–1 Mbps
5. Pixel Format and Chroma Sampling
| Format | Description | Compatibility | Scenario |
|---|---|---|---|
| yuv420p | 4:2:0 mainstream | Best | Live streaming/mobile |
| yuv422p | 4:2:2 high color | Average | Professional recording |
| uyvy422 | Device native | Average | Capture card/post-production |
6. Codec Selection
- H.264 (AVC): Widely supported, good hardware support, first choice for IoT.
- H.265 (HEVC): High compression efficiency, compatibility needs assessment.
- AV1/H.266: Emerging, cloud/browser support is increasing.
- Suggestion: Prioritize H.264 for IoT; assess H.265/AV1 under high bandwidth pressure.
IV. Low Latency and Weak Network Optimization
- Disable/reduce B-frames, bframes=0
- Shorten GOP (0.5–1 second)
- Reduce resolution/frame rate before blindly increasing bitrate
- Prioritize CBR or tighten VBR peak
- Optimize end-to-end buffering at the playback/link end
V. Parameter Selection and Scenario Comparison Table
| Scenario | Resolution | Bitrate | Pixel Format | Encoding Profile | B-Frames |
|---|---|---|---|---|---|
| Live Streaming/Pushing | 1280×720 | 1500-3000 | yuv420p | baseline | 0 |
| Mobile Live Streaming | 854×480 | 800-1500 | yuv420p | baseline | 0 |
| HD Recording | 1920×1080 | 4000-8000 | yuv420p | high | 2~4 |
| Professional Post-Production | 1920×1080 | 8000+ | yuv422p | high422 | 2~4 |
| National Standard GB28181 | 640×480 | 300-800 | yuv420p | baseline | 0 |
VI. Engineering Practices and Checklist
1. Typical ffmpeg Commands
# Local transcoding and bitrate reductionffmpeg -re -i input.mp4 -c:v libx264 -preset veryfast -tune zerolatency \ -x264-params keyint=25:min-keyint=25:scenecut=0 -b:v 2500k -maxrate 3000k -bufsize 2500k \ -r 25 -c:a aac -b:a 128k output_1080p25.mp4
# Push RTMPffmpeg -re -i input.mp4 -c copy -f flv rtmp://127.0.0.1:1935/live/cam012. Engineering Checklist
- [ ] Encoding: H.264 baseline/main, disable/few B-frames, reasonable GOP
- [ ] Bitrate Control: CBR or limited VBR, clear peak and buffering
- [ ] Resolution/Frame Rate: Set according to business and network
- [ ] Compatibility: End-to-end decoding capabilities, mobile power consumption
- [ ] Monitoring: Acquisition end bitrate/frame loss/buffering, platform end startup delay/stuttering rate
– END –