Encapsulating HTTP Requests and Logging in Go

Hello everyone! I am Lao Kou! Let’s learn how to encapsulate HTTP requests and logging together.

HTTP Requests

To encapsulate HTTP requests, you can directly use net/http. There are two main points to note: How to disable HTTPS verification and Client file upload.

How to Disable HTTPS Verification

// Skip TLS certificate verification
client := &http.Client{
    Transport: &http.Transport{
       TLSClientConfig: &tls.Config{InsecureSkipVerify: true},
    },
}

Client File Upload

File uploads must comply with the RFC 1867^[1] standard, so the request header Content-Type should be set to multipart/form-data;boundary=xxx.

http.go

import (
    "bytes"
    "crypto/tls"
    "errors"
    "io"
    "mime/multipart"
    "net/http"
)

func SendRequest(method, url string, param io.Reader, header map[string]string) (*http.Response, error) {
    request, err := http.NewRequest(method, url, param)
    if err != nil {
       return nil, errors.New("Failed to create Request, error: " + err.Error())
    }
    if header != nil {
       for k, v := range header {
          request.Header.Set(k, v)
       }
    }
    return sendHttpRequest(request)
}

func GetFormFile(fileName string, buf []byte) (io.Reader, string, error) {
    body := &bytes.Buffer{}
    writer := multipart.NewWriter(body)
    part, err := writer.CreateFormFile("file", fileName)
    if err != nil {
       return nil, "", errors.New("Failed to create FormFile, error: " + err.Error())
    }
    _, err = io.Copy(part, bytes.NewReader(buf))
    if err != nil {
       return nil, "", errors.New("Failed to copy byte array, error: " + err.Error())
    }
    err = writer.Close()
    if err != nil {
       return nil, "", errors.New("Failed to close Writer, error: " + err.Error())
    }
    return body, writer.FormDataContentType(), nil
}

func SendRequestAndGetBody(method, url string, param io.Reader, header map[string]string) ([]byte, error) {
    response, err := SendRequest(method, url, param, header)
    if err != nil {
       return nil, err
    }
    body, err := io.ReadAll(response.Body)
    if err != nil {
       return nil, errors.New("Failed to read response, error: " + err.Error())
    }
    defer response.Body.Close()
    return body, nil
}

func sendHttpRequest(request *http.Request) (*http.Response, error) {
    client := &http.Client{
       Transport: &http.Transport{
          TLSClientConfig: &tls.Config{InsecureSkipVerify: true},
       },
    }
    response, err := client.Do(request)
    if err != nil {
       return nil, errors.New("Failed to send HTTP request, error: " + err.Error())
    }
    return response, nil
}

http_test.go

func Test_Http(t *testing.T) {
   // GET request
    header = map[string]string{
        "Cookie": 'token=123',
    }
    _, _ = SendRequest(http.MethodGet, "https://localhost/api/test", nil ,header)
   
   // POST request
   _, _ = SendRequest(http.MethodPost, "https://localhost/api/test", strings.NewReader("username=a&password=123") ,header)
   
   // POST file upload【form】
   // For demonstration purposes, define a byte array
    buf := []byte('123')
    formFile, contentType, err := core.GetFormFile("image.jpg", buf)
    if err != nil {
       fmt.Println(err.Error())
       return
    }
    header = map[string]string{
        "Content-Type": contentType,
    }
    // Upload file
    _, _ = SendRequest(http.MethodPost, "https://localhost/api/upload", formFile, header)
}

Logging

Logging is very important for a project, so it is necessary to choose based on business needs. Common logging packages in Go include glog, logrus, and zap.

Dimension	glog (Google)	logrus (Community Maintained)	zap (Uber)
Project Background	Commonly used in infrastructure projects like Kubernetes, lightweight	Early mainstream in the Go community, no longer updated (only maintained)	High-performance industrial-grade practice, actively maintained
Performance	Medium (optimized based on standard library)	Lower (reflection serialization, many memory allocations)	Extremely high (pre-allocated memory, zero reflection, millions of logs per second)
Log Levels	4 levels (Info/Warning/Error/Fatal)	6 levels (Trace/Debug/Info/Warn/Error/Fatal)	7 levels (Debug/Info/Warn/Error/Dpanic/Panic/Fatal)
Structured Logging	❌ Not supported (only text format)	✅ Supported (`<span>WithFields</span>` dynamic type)	✅ Strong type (`<span>zap.String()</span>`), `<span>SugaredLogger</span>` compatible with unstructured
API Style	Similar to standard library (e.g., `<span>glog.Info("msg")</span>`)	Chained calls (`<span>logrus.WithField().Info()</span>`)	Explicit type (`<span>zap.Info("msg", zap.String("k","v"))</span>`)
Log Rotation	❌ Must be implemented manually or with third-party libraries	❌ Depends on plugins (e.g., `<span>file-rotatelogs</span>`)	✅ Native support (integrated with `<span>lumberjack</span>`)
Memory Allocation	Less	More (reflection + dynamic fields)	Very little (pre-allocated + non-reflection)
Dynamic Level Adjustment	❌ Not supported	✅ Must be implemented manually	✅ Native support (`<span>AtomicLevel</span>`)
Learning Curve	Low (similar to standard library)	Low (similar to `<span>fmt</span>`)	Medium (structured API is slightly complex, `<span>SugaredLogger</span>` simplifies)
Applicable Scenarios	Lightweight tools, compatible with K8s ecosystem	Old project maintenance, plugin-dependent applications	High concurrency services, cloud-native, performance-sensitive scenarios

Considering gateway devices, to save memory space and achieve extreme performance, we use zap encapsulation. Since zap does not support log rotation, we use a third-party component timberjack to enhance it, supporting log rotation, log retention days, and log levels.

log.go

import (
    "errors"
    "github.com/DeRuina/timberjack"
    "go.uber.org/zap"
    "go.uber.org/zap/zapcore"
    "log"
    "time"
)

const (
    PROD = "prod"
)

type LogConfig struct {
    // Log level
    Level string `yaml:"level"`
    // Environment
    Profile string `yaml:"profile"`
    // Log format
    Pattern string `yaml:"pattern"`
    // Log file path
    FilePath string `yaml:"file-path"`
    // Max log size (MB)
    MaxSize int `yaml:"max-size"`
    // Max number of logs
    MaxBackups int `yaml:"max-backups"`
    // Log retention time (days)
    MaxAge int `yaml:"max-age"`
    // Whether to compress
    Compress bool `yaml:"compress"`
    // Local time, default: false (use UTC)
    LocalTime bool `yaml:"local-time"`
    // JSON format
    JsonFormat bool `yaml:"json-format"`
    // Rotation frequency
    RotationInterval time.Duration `yaml:"rotation-interval"`
    // Rotation time (minutes)
    RotateAtMinutes []int `yaml:"rotate-at-minutes"`
    // Log time format
    BackupTimeFormat string `yaml:"backup-time-format"`
}

func (c *LogConfig) InitLogger() (*zap.Logger, error) {
    timberjackLogger := &timberjack.Logger{
       Filename:         c.FilePath,
       MaxSize:          c.MaxSize,
       MaxBackups:       c.MaxBackups,
       MaxAge:           c.MaxAge,
       Compress:         c.Compress,
       LocalTime:        c.LocalTime,
       RotationInterval: c.RotationInterval,
       RotateAtMinutes:  c.RotateAtMinutes,
       BackupTimeFormat: c.BackupTimeFormat,
    }
    log.SetOutput(timberjackLogger)
    defer timberjackLogger.Close()
    writeSyncer := zapcore.AddSync(timberjackLogger)
    // Configure log level
    levelConfig := zap.NewAtomicLevel()
    level, err := zapcore.ParseLevel(c.Level)
    if err != nil {
       return nil, errors.New("Log level does not exist, please reconfigure, error: " + err.Error())
    }
    levelConfig.SetLevel(level)
    // Configure environment
    var encoderConfig zapcore.EncoderConfig
    switch c.Profile {
    case PROD:
       encoderConfig = zap.NewProductionEncoderConfig()
    default:
       encoderConfig = zap.NewDevelopmentEncoderConfig()
    }
    // Set time format
    encoderConfig.EncodeTime = c.customTimeEncoder
    var encoder zapcore.Encoder
    // JSON format
    if c.JsonFormat {
       encoder = zapcore.NewJSONEncoder(encoderConfig)
    } else {
       encoder = zapcore.NewConsoleEncoder(encoderConfig)
    }
    core := zapcore.NewCore(encoder, writeSyncer, levelConfig)
    return zap.New(core, zap.AddCaller()), nil
}

func (c *LogConfig) customTimeEncoder(t time.Time, enc zapcore.PrimitiveArrayEncoder) {
    enc.AppendString(t.Format(c.Pattern))
}

application.yaml

log:
  # Level
  level: error
  # Environment
  profile: prod
  # Log format
  pattern: 2006-01-02 15:04:05.000
  # Log file path
  file-path: test.log
  # Max log size (MB)
  max-size: 500
  # Max number of logs
  max-backups: 10
  # Log retention time (days)
  max-age: 30
  # Whether to compress
  compress: true
  # Local time, default: false (use UTC)
  local-time: true
  # JSON format, true for JSON logs
  json-format: false
  # Rotation frequency
  rotation-interval: 24h
  # Rotation time (minutes)
  rotate-at-minutes:
    - 0
    - 15
    - 30
    - 45
  # Log time format
  backup-time-format: 2006-01-02-15-04-05

config.go

import (
    "errors"
    "gopkg.in/yaml.v3"
    "os"
)

type SystemConfig struct {
    Log LogConfig `yaml:"log"`
}

func GetSystemConfig(path string) (*SystemConfig, error) {
    data, err := os.ReadFile(path)
    if err != nil {
       return nil, errors.New("Failed to read configuration file, error: " + err.Error())
    }
    sysConfig := &SystemConfig{}
    err = yaml.Unmarshal(data, sysConfig)
    if err != nil {
       return nil, errors.New("Failed to deserialize configuration file, error: " + err.Error())
    }
    return sysConfig, nil
}

log_test.go

import (
    "testing"
)

func Test_Log(t *testing.T) {
    config, err := GetSystemConfig("application.yaml")
    if err != nil {
       t.Error(err.Error())
       return
    }
    logger, err := config.Log.InitLogger()
    if err != nil {
       t.Error(err.Error())
       return
    }
    logger.Info("Information")
    logger.Warn("Warning")
    logger.Error("Error")
    t.Log("Log test passed")
}

Note: When deploying to the gateway, logs cannot be printed. You need to increase log permissions and cannot run with sudo.

# Increase log permissions
sudo chmod -R 7777 /home/a/app/logs
# Run the project, do not run the project with sudo, otherwise logs cannot be printed, this is a pit I encountered
nohup /home/a/app/log > /dev/null 2>&1 &

I am Lao Kou, see you next time!

References[1]

RFC 1867: https://tools.ietf.org/html/rfc1867

HTTP Requests

Logging

Leave a Comment Cancel reply