Chaos Engineering and Fault Injection in Ruby Applications

Today, I want to talk to you about a very interesting topic, especially in the context of modern distributed systems—Chaos Engineering and Fault Injection. This may sound a bit profound, but let me explain it in a simple way and practice it with Ruby code.

The core idea of Chaos Engineering is: to actively create “chaos” (i.e., faults) in the system to validate its robustness and resilience. It’s like intentionally turning off the power in a house to see if the emergency lights will turn on, ensuring that our system won’t crash when a real incident occurs.

1. What are Chaos Engineering and Fault Injection?
2. How to perform simple fault injection using Ruby?
3. Simulating typical scenarios: Network latency and service exceptions.
4. Considerations and improvement suggestions in practice.

Chaos Engineering is an engineering practice aimed at understanding how a system behaves under unpredictable events (such as server crashes, network latency, service unavailability, etc.). By doing this, we can discover potential problems in advance and optimize the system’s design.

Fault Injection is the core tool of Chaos Engineering. It tests the system’s fault tolerance by artificially creating faults (like throwing exceptions, delaying responses, etc.).

In Chaos Engineering, we need to inject some faults. For example, simulating service unavailability, delayed responses, random failures, etc. Below is a simple Ruby example where we randomly inject faults to simulate service issues.

def simulated_service
case rand(13)
when 1
when 2
“Service returns result after delay”
else
“Service is running normally”
end
end

begin
puts simulated_service
rescue => e
end

1. **rand(13)**: Randomly generates a number from 1 to 3 to decide which fault (or normal operation) to simulate.
2. **raise**: Intentionally throws an exception to simulate the service being unavailable.
3. **sleep(3)**: Pauses the program for 3 seconds to simulate service delay.
4. Exception Handling: Captures errors through the begin-rescue block and logs the exceptions.

• If you randomly get 1, you will see "Caught exception: Service unavailable!".
• If you randomly get 2, it will output "Service returns result after delay" after a 3-second delay.
• If you randomly get 3, it will directly output "Service is running normally".

Assuming we have a service that needs to fetch data from a remote API, we use NetHTTP to simulate the network request and inject faults.

ruby copy

require 'net/http'

def fetch_data_from_api


  case rand(14)
  when 1

  when 2
    "Successfully fetched data after delay"
  when 3

  else

  end
end


begin
  response = fetch_data_from_api
  if response.nil?
    puts "Service returned empty data!"
  else

  end
rescue => e

end

1. **NetHTTP.get**: Used to simulate a normal HTTP GET request.
2. Injecting Faults:

   • raise simulates a network timeout.
   • sleep simulates a delay.
   • return nil simulates the service returning empty data.
3. Fault Handling: Captures exceptions through begin-rescue and handles nil data.

• Test how the frontend behaves when the API times out.
• Verify how well the system tolerates delayed responses (e.g., whether it will retry on timeout).
• Check if there are reasonable default values or error messages when the service returns empty data.

1. Do not inject faults directly in production: Chaos Engineering is typically conducted in a testing environment to ensure that there is no impact on actual users.

2. Control the scope of injection: You can use configuration files or parameters to control which services inject faults.

3. Monitoring and Logging: When implementing Chaos Engineering, it is essential to monitor the system’s operational status and log all fault behaviors.

4. Use dedicated tools: The Ruby community has some specialized tools to help implement Chaos Engineering, such as [chaos_monkey](https://github.com/Netflix/chaosmonkey).

5. Combine with automated testing: Integrate fault injection with automated testing to regularly run chaos tests.

6. Progressive Injection: Start testing on a small scale and gradually expand the range of fault injection.

• The basic concepts of Chaos Engineering.
• How to inject faults using Ruby to simulate service exceptions and network latency.
• Issues to be aware of and directions for improvement in practice.

Chaos Engineering is not about breaking the system, but rather discovering problems and improving the system in a “controlled chaos” environment. Give it a try! Write some of your own code and see if your system is robust enough to handle faults.

（）