1. Starting the WatchDog
The WatchDog is created and started in the startBootstrapServices() method of the SystemServer process.
//[frameworks/base/services/java/com/android/server/SystemServer.java]public final class SystemServer implements Dumpable { private void startBootstrapServices(@NonNull TimingsTraceAndSlog t) { t.traceBegin("startBootstrapServices"); // Start the watchdog as early as possible so we can crash the system server // if we deadlock during early boot t.traceBegin("StartWatchdog"); final Watchdog watchdog = Watchdog.getInstance(); watchdog.start(); t.traceEnd(); //... }}
2. Initializing the WatchDog
//[frameworks/base/services/core/java/com/android/server/Watchdog.java]public class Watchdog { private static Watchdog sWatchdog; public static Watchdog getInstance() { if (sWatchdog == null) { sWatchdog = new Watchdog(); } return sWatchdog; } private Watchdog() { mThread = new Thread(this::run, "watchdog"); // Initialize handler checkers for each common thread we want to check. Note // that we are not currently checking the background thread, since it can // potentially hold longer running operations with no guarantees about the timeliness // of operations there. // The shared foreground thread is the main checker. It is where we // will also dispatch monitor checks and do other work. mMonitorChecker = new HandlerChecker(FgThread.getHandler(), "foreground thread", DEFAULT_TIMEOUT); mHandlerCheckers.add(mMonitorChecker); // Add checker for main thread. We only do a quick check since there // can be UI running on the thread. mHandlerCheckers.add(new HandlerChecker(new Handler(Looper.getMainLooper()), "main thread", DEFAULT_TIMEOUT)); // Add checker for shared UI thread. mHandlerCheckers.add(new HandlerChecker(UiThread.getHandler(), "ui thread", DEFAULT_TIMEOUT)); // And also check IO thread. mHandlerCheckers.add(new HandlerChecker(IoThread.getHandler(), "i/o thread", DEFAULT_TIMEOUT)); // And the display thread. mHandlerCheckers.add(new HandlerChecker(DisplayThread.getHandler(), "display thread", DEFAULT_TIMEOUT)); // And the animation thread. mHandlerCheckers.add(new HandlerChecker(AnimationThread.getHandler(), "animation thread", DEFAULT_TIMEOUT)); // And the surface animation thread. mHandlerCheckers.add(new HandlerChecker(SurfaceAnimationThread.getHandler(), "surface animation thread", DEFAULT_TIMEOUT)); // Initialize monitor for Binder threads. addMonitor(new BinderThreadMonitor()); mInterestingJavaPids.add(Process.myPid()); // See the notes on DEFAULT_TIMEOUT. assert DB || DEFAULT_TIMEOUT > ZygoteConnectionConstants.WRAPPED_PID_TIMEOUT_MILLIS; mTraceErrorLogger = new TraceErrorLogger(); } /** * Called by SystemServer to cause the internal thread to begin execution. */ public void start() { mThread.start(); }}
The WatchDog thread will detect if the binder thread pool has run out by calling IPCThreadState::self()->blockUntilThreadAvailable().
It will block the current thread until the number of threads executing Binder commands is less than the maximum number of binder threads set. The default value for mMaxThreads is DEFAULT_MAX_BINDER_THREADS (15), but SystemServer modifies its process to 32 threads.
How is the number of threads executing Binder commands calculated?
In the IPCThreadState::getAndExecuteCommand() function, each time talkWithDriver() receives a Binder command, it increments mExecutingThreadsCount, and after processing the command, it decrements mExecutingThreadsCount.
Question: Does mExecutingThreadsCount include the Binder main thread?
Implementation of IPCThreadState::blockUntilThreadAvailable():
//[frameworks/native/libs/binder/IPCThreadState.cpp]void IPCThreadState::blockUntilThreadAvailable(){ pthread_mutex_lock(& mProcess-> mThreadCountLock); mProcess-> mWaitingForThreads++; while (mProcess-> mExecutingThreadsCount >= mProcess-> mMaxThreads) { ALOGW("Waiting for thread to be free. mExecutingThreadsCount=%lu mMaxThreads=%lu\n", static_cast<unsigned long>(mProcess-> mExecutingThreadsCount), static_cast<unsigned long>(mProcess-> mMaxThreads)); pthread_cond_wait(& mProcess-> mThreadCountDecrement, & mProcess-> mThreadCountLock); } mProcess-> mWaitingForThreads--; pthread_mutex_unlock(& mProcess-> mThreadCountLock);}
3. What does the monitor() method in WatchDog monitor?
In system-level services, the monitor() method just tries to acquire a lock and does nothing; if it can acquire the lock, it proves that the current thread is not deadlocked.
public class Watchdog { ... public interface Monitor { void monitor(); } ...}
Implementation in ActivityManagerService:
public class ActivityManagerService extends IActivityManager.Stub implements Watchdog.Monitor, BatteryStatsImpl.BatteryCallback, ActivityManagerGlobalLock { ... /** In this method we try to acquire our lock to make sure that we have not deadlocked */ public void monitor() { synchronized (this) { } }}
Implementation in WindowManagerService:
public class WindowManagerService extends IWindowManager.Stub implements Watchdog.Monitor, WindowManagerPolicy.WindowManagerFuncs { ... // Called by the heartbeat to ensure locks are not held indefinitely (for deadlock detection). @Override public void monitor() { synchronized (mGlobalLock) { } } ...}
Listening in the FgThread thread to see if the number of executing Binder threads exceeds the limit;
Binder threads can be divided into three types:
1. Binder main thread: Threads created in the current process via ProcessState::startThreadPool().
//[frameworks/native/libs/binder/ProcessState.cpp]ProcessState::startThreadPool() --> ProcessState::spawnPooledThread(true) //[frameworks/native/libs/binder/IPCThreadState.cpp] --> IPCThreadState::self()->joinThreadPool(true)
2. Binder ordinary thread: When all threads in the Binder thread pool are busy and the number of requested threads is less than the maximum number of Binder threads, the Binder driver issues a request (BR_SPAWN_LOOPER) to user space to create new Binder threads.
static int binder_thread_read(struct binder_proc *proc, struct binder_thread *thread, binder_uintptr_t binder_buffer, size_t size, binder_size_t *consumed, int non_block){ //... done: *consumed = ptr - buffer; binder_inner_proc_lock(proc); if (proc->requested_threads == 0 && list_empty(&thread->proc->waiting_threads) && proc->requested_threads_started < proc->max_threads &&& (thread->looper && (BINDER_LOOPER_STATE_REGISTERED | BINDER_LOOPER_STATE_ENTERED)) /* the user-space code fails to */ /*spawn a new thread if we leave this out */) { proc->requested_threads++; binder_inner_proc_unlock(proc); binder_debug(BINDER_DEBUG_THREADS, "%d:%d BR_SPAWN_LOOPER\n", proc->pid, thread->pid); if (put_user(BR_SPAWN_LOOPER, (uint32_t __user *)buffer)) return -EFAULT; binder_stat_br(proc, thread, BR_SPAWN_LOOPER); } else binder_inner_proc_unlock(proc); return 0;}status_t IPCThreadState::executeCommand(int32_t cmd) BBinder* obj; RefBase::weakref_type* refs; status_t result = NO_ERROR; switch ((uint32_t)cmd) { ... case BR_SPAWN_LOOPER: mProcess->spawnPooledThread(false); break; } default: ALOGE("*** BAD COMMAND %d received from Binder driver\n", cmd); result = UNKNOWN_ERROR; break; } ... return result;}
3. Binder other threads: Directly call IPCThreadState::self()->joinThreadPool() to add the current thread as a Binder main thread into the Binder thread pool.
joinThreadPool() never exits until an error occurs.
IPCThreadState::self()->joinThreadPool(); // Add the current thread as a Binder main thread to Binder