Exploring the Principles of Android WatchDog

1. Starting the WatchDog

The WatchDog is created and started in the startBootstrapServices() method of the SystemServer process.

//[frameworks/base/services/java/com/android/server/SystemServer.java]public final class SystemServer implements Dumpable {  private void startBootstrapServices(@NonNull TimingsTraceAndSlog t) {        t.traceBegin("startBootstrapServices");      // Start the watchdog as early as possible so we can crash the system server      // if we deadlock during early boot        t.traceBegin("StartWatchdog");        final Watchdog watchdog = Watchdog.getInstance();        watchdog.start();        t.traceEnd();      //...    }}

2. Initializing the WatchDog

Exploring the Principles of Android WatchDog

//[frameworks/base/services/core/java/com/android/server/Watchdog.java]public class Watchdog {  private static Watchdog sWatchdog;  public static Watchdog getInstance() {    if (sWatchdog == null) {                sWatchdog = new Watchdog();    }    return sWatchdog;  }    private Watchdog() {      mThread = new Thread(this::run, "watchdog");      // Initialize handler checkers for each common thread we want to check.  Note      // that we are not currently checking the background thread, since it can      // potentially hold longer running operations with no guarantees about the timeliness      // of operations there.      // The shared foreground thread is the main checker.  It is where we      // will also dispatch monitor checks and do other work.      mMonitorChecker = new HandlerChecker(FgThread.getHandler(),  "foreground thread", DEFAULT_TIMEOUT);      mHandlerCheckers.add(mMonitorChecker);      // Add checker for main thread.  We only do a quick check since there      // can be UI running on the thread.      mHandlerCheckers.add(new HandlerChecker(new Handler(Looper.getMainLooper()),  "main thread", DEFAULT_TIMEOUT));      // Add checker for shared UI thread.      mHandlerCheckers.add(new HandlerChecker(UiThread.getHandler(),  "ui thread", DEFAULT_TIMEOUT));      // And also check IO thread.      mHandlerCheckers.add(new HandlerChecker(IoThread.getHandler(),  "i/o thread", DEFAULT_TIMEOUT));      // And the display thread.      mHandlerCheckers.add(new HandlerChecker(DisplayThread.getHandler(),  "display thread", DEFAULT_TIMEOUT));      // And the animation thread.      mHandlerCheckers.add(new HandlerChecker(AnimationThread.getHandler(),  "animation thread", DEFAULT_TIMEOUT));      // And the surface animation thread.      mHandlerCheckers.add(new HandlerChecker(SurfaceAnimationThread.getHandler(),  "surface animation thread", DEFAULT_TIMEOUT));      // Initialize monitor for Binder threads.      addMonitor(new BinderThreadMonitor());      mInterestingJavaPids.add(Process.myPid());      // See the notes on DEFAULT_TIMEOUT.      assert DB ||                  DEFAULT_TIMEOUT > ZygoteConnectionConstants.WRAPPED_PID_TIMEOUT_MILLIS;          mTraceErrorLogger = new TraceErrorLogger();  }  /**       * Called by SystemServer to cause the internal thread to begin execution.       */  public void start() {        mThread.start();  }}

The WatchDog thread will detect if the binder thread pool has run out by calling IPCThreadState::self()->blockUntilThreadAvailable().

It will block the current thread until the number of threads executing Binder commands is less than the maximum number of binder threads set. The default value for mMaxThreads is DEFAULT_MAX_BINDER_THREADS (15), but SystemServer modifies its process to 32 threads.

How is the number of threads executing Binder commands calculated?

In the IPCThreadState::getAndExecuteCommand() function, each time talkWithDriver() receives a Binder command, it increments mExecutingThreadsCount, and after processing the command, it decrements mExecutingThreadsCount.

Question: Does mExecutingThreadsCount include the Binder main thread?

Implementation of IPCThreadState::blockUntilThreadAvailable():

//[frameworks/native/libs/binder/IPCThreadState.cpp]void IPCThreadState::blockUntilThreadAvailable(){    pthread_mutex_lock(& mProcess-> mThreadCountLock);    mProcess-> mWaitingForThreads++;    while (mProcess-> mExecutingThreadsCount >= mProcess-> mMaxThreads) {        ALOGW("Waiting for thread to be free. mExecutingThreadsCount=%lu mMaxThreads=%lu\n",                static_cast<unsigned long>(mProcess-> mExecutingThreadsCount),                static_cast<unsigned long>(mProcess-> mMaxThreads));        pthread_cond_wait(& mProcess-> mThreadCountDecrement, & mProcess-> mThreadCountLock);    }    mProcess-> mWaitingForThreads--;    pthread_mutex_unlock(& mProcess-> mThreadCountLock);}

3. What does the monitor() method in WatchDog monitor?

In system-level services, the monitor() method just tries to acquire a lock and does nothing; if it can acquire the lock, it proves that the current thread is not deadlocked.

public class Watchdog {    ...    public interface Monitor {        void monitor();    }    ...}

Implementation in ActivityManagerService:

public class ActivityManagerService extends IActivityManager.Stub        implements Watchdog.Monitor, BatteryStatsImpl.BatteryCallback, ActivityManagerGlobalLock {    ...    /** In this method we try to acquire our lock to make sure that we have not deadlocked */    public void monitor() {        synchronized (this) { }    }}

Implementation in WindowManagerService:

public class WindowManagerService extends IWindowManager.Stub        implements Watchdog.Monitor, WindowManagerPolicy.WindowManagerFuncs {    ...    // Called by the heartbeat to ensure locks are not held indefinitely (for deadlock detection).    @Override    public void monitor() {        synchronized (mGlobalLock) { }    }    ...}

Listening in the FgThread thread to see if the number of executing Binder threads exceeds the limit;

Binder threads can be divided into three types:

1. Binder main thread: Threads created in the current process via ProcessState::startThreadPool().

//[frameworks/native/libs/binder/ProcessState.cpp]ProcessState::startThreadPool()    --> ProcessState::spawnPooledThread(true)        //[frameworks/native/libs/binder/IPCThreadState.cpp]        --> IPCThreadState::self()->joinThreadPool(true)

2. Binder ordinary thread: When all threads in the Binder thread pool are busy and the number of requested threads is less than the maximum number of Binder threads, the Binder driver issues a request (BR_SPAWN_LOOPER) to user space to create new Binder threads.

static int binder_thread_read(struct binder_proc *proc,            struct binder_thread *thread,            binder_uintptr_t binder_buffer, size_t size,            binder_size_t *consumed, int non_block){    //...  done:  *consumed = ptr - buffer;  binder_inner_proc_lock(proc);  if (proc->requested_threads == 0 &&      list_empty(&thread->proc->waiting_threads) &&      proc->requested_threads_started < proc->max_threads &&&      (thread->looper && (BINDER_LOOPER_STATE_REGISTERED |       BINDER_LOOPER_STATE_ENTERED)) /* the user-space code fails to */       /*spawn a new thread if we leave this out */) {    proc->requested_threads++;    binder_inner_proc_unlock(proc);    binder_debug(BINDER_DEBUG_THREADS,           "%d:%d BR_SPAWN_LOOPER\n",           proc->pid, thread->pid);    if (put_user(BR_SPAWN_LOOPER, (uint32_t __user *)buffer))      return -EFAULT;    binder_stat_br(proc, thread, BR_SPAWN_LOOPER);  } else    binder_inner_proc_unlock(proc);  return 0;}status_t IPCThreadState::executeCommand(int32_t cmd)    BBinder* obj;    RefBase::weakref_type* refs;    status_t result = NO_ERROR;    switch ((uint32_t)cmd) {    ...    case BR_SPAWN_LOOPER:        mProcess->spawnPooledThread(false);        break;    }    default:        ALOGE("*** BAD COMMAND %d received from Binder driver\n", cmd);        result = UNKNOWN_ERROR;        break;    }    ...    return result;}

3. Binder other threads: Directly call IPCThreadState::self()->joinThreadPool() to add the current thread as a Binder main thread into the Binder thread pool.

joinThreadPool() never exits until an error occurs.

IPCThreadState::self()->joinThreadPool(); // Add the current thread as a Binder main thread to Binder

Leave a Comment

×