1. What is SPI

1.1 Introduction to SPI

SPI, which stands for Service Provider Interface, is a service discovery mechanism. It automatically loads the classes defined in files located in the META-INF/services directory on the ClassPath. This mechanism enables extensibility for many frameworks, such as Dubbo and JDBC. Let’s start with a simple example to see how it works.

In simple terms, SPI is an extension mechanism that configures services without modifying the core code. By loading services from configuration files and determining the logic to execute based on passed parameters, it adheres to the open-closed principle: open for extension, closed for modification.

1.2 Application Scenarios of SPI

When writing core code, if a certain point involves different logic based on varying parameters, without SPI, you might end up writing a lot of if-else statements, making the code inflexible. If a new logic is added one day, the code must be modified, which violates the open-closed principle. The emergence of SPI addresses this extension issue; you can configure all implementation classes in a configuration file, and in the core code, simply load the configuration file and match the loaded classes based on input parameters. If a match is found, that logic is executed. Thus, if new logic is added in the future, the core code remains unchanged; only the configuration file and new classes in your project need to change, complying with the open-closed principle.

2. SPI Mechanism in JDK

Having introduced what SPI is and where it is used, we will now focus on its implementation in the JDK.

2.1 Example

2.1.1 Top-Level Interface

public interface KLog {
   boolean support(String type);
   void debug();
   void info();
}

2.1.2 Interface Implementation

public class Log4j implements KLog {
    @Override
    public boolean support(String type) {
        return "log4j".equalsIgnoreCase(type);
    }
    @Override
    public void debug() {
        System.out.println("====log4j.debug======");
    }
    @Override
    public void info() {
        System.out.println("====log4j.info======");
    }
}
public class Logback implements KLog {
    @Override
    public boolean support(String type) {
        return "Logback".equalsIgnoreCase(type);
    }
    @Override
    public void debug() {
        System.out.println("====Logback.debug======");
    }
    @Override
    public void info() {
        System.out.println("====Logback.info======");
    }
}
public class Slf4j implements KLog {
    @Override
    public boolean support(String type) {
        return "Slf4j".equalsIgnoreCase(type);
    }
    @Override
    public void debug() {
        System.out.println("====Slf4j.debug======");
    }
    @Override
    public void info() {
        System.out.println("====Slf4j.info======");
    }
}

2.1.3 File Configuration

Create a file in the resources/META-INF/services directory, and the file name must match the fully qualified name of the interface. As shown in the image:

This interface file configures the fully qualified names of all implementation classes for that interface, as shown in the image:

Now, based on the input parameters, we can decide whether to execute the logic for Log4j or Logback.

2.1.4 Unit Test

public class MyTest {
    // Different input parameters correspond to different logic calls
    public static void main(String[] args) {
        // This is the core business code, which will decide which instance to call based on external parameters
        // It can read the properties configuration file to determine which instance to call
        // JDK API loads the configuration file to configure instances
        ServiceLoader<KLog> all = ServiceLoader.load(KLog.class);
        Iterator<KLog> iterator = all.iterator();
        Scanner scanner = new Scanner(System.in);
        String s = scanner.nextLine();
        while (iterator.hasNext()) {
            KLog next = iterator.next();
            // Check if this instance is the one we need to call
            // Strategy pattern: check if the current instance matches the input parameter
            if (next.support(s)) {
                next.debug();
            }
        }
    }
}

2.2 Summary

We can view this unit test code as the core code of our business, which does not need to be modified. The only extensions added are the implementation classes corresponding to the interface. In the core code, we only need to call different logic based on the input parameters. This is the charm of SPI extension: the core code remains unchanged.

From the above test code, we also learn that the SPI in the JDK is a way to obtain class instances, and when combined with the strategy pattern, it allows for instance selection based on parameters.

3. SPI Mechanism in Dubbo

The SPI mechanism in Dubbo is generally similar to that in the JDK, but there are many differences in details. Let’s take a closer look at the SPI mechanism in Dubbo.

3.1 Example

3.1.1 Top-Level Interface

// This is mandatory
@SPI("spring")
public interface ActivateApi {
    @Adaptive
    String todo(String param, URL url);
}

3.1.2 Interface Implementation

@Adaptive
public class DubboActivate implements ActivateApi {
    @Override
    public String todo(String param, URL url) {
        return param;
    }
}
public class MybatisActivate implements ActivateApi {
    private ActivateApi activateApi;
    @Override
    public String todo(String param, URL url) {
        return param;
    }
    public void setActivateApi(ActivateApi activateApi) {
        this.activateApi = activateApi;
        System.out.println(activateApi);
    }
}
public class Rabbitmq1Activate implements ActivateApi {
    @Override
    public String todo(String param, URL url) {
        return param;
    }
}
public class Rabbitmq2Activate implements ActivateApi {
    @Override
    public String todo(String param, URL url) {
        return param;
    }
}
public class RabbitmqActivate implements ActivateApi {
    @Override
    public String todo(String param, URL url) {
        return param;
    }
}
public class SpringActivate implements ActivateApi {
    @Override
    public String todo(String param, URL url) {
        return param;
    }
}
public class SpringCloudActivate implements ActivateApi {
    @Override
    public String todo(String param, URL url) {
        return param;
    }
}

3.1.3 File Configuration

Configure the interface file under resources/META-INF/dubbo, and the file name must match the fully qualified name of the interface, as shown:The content configured in the interface file is the fully qualified names of the implementation classes for that interface. Unlike the JDK configuration, Dubbo can include keys, which are the mapping keys for the implementation classes, allowing retrieval of the corresponding class instance based on this key, as shown:

3.1.4 Unit Test

@Test
public void adaptive() {
  ActivateApi adaptiveExtension =
ExtensionLoader.getExtensionLoader(ActivateApi.class).getAdaptiveExtension();
  System.out.println(adaptiveExtension.getClass());
}

Next, we will introduce several very important SPI APIs in Dubbo.

3.2 getAdaptiveExtension()

3.2.1 Method Functionality

This method retrieves an implementation class of an interface, with the following retrieval methods:

1. Retrieve an instance of the class annotated with @Adaptive.2. If none of the interface’s implementation classes have the @Adaptive annotation, Dubbo dynamically generates one.

3.2.2 Method Usage

// Specify the interface type to retrieve its implementation class
ActivateApi adaptiveExtension =
ExtensionLoader.getExtensionLoader(ActivateApi.class).getAdaptiveExtension();

3.2.3 Source Code Analysis

3.2.3.1 ExtensionLoader

public static <T> ExtensionLoader<T> getExtensionLoader(Class<T> type) {
    if (type == null) {
        throw new IllegalArgumentException("Extension type == null");
    }
    if (!type.isInterface()) {
        throw new IllegalArgumentException("Extension type (" + type + ") is not an interface!");
    }
    // If the interface does not have the @SPI annotation, throw an error
    if (!withExtensionAnnotation(type)) {
        throw new IllegalArgumentException("Extension type (" + type + ") is not an extension, because it is NOT annotated with @" + SPI.class.getSimpleName() + "!");
    }
    // Retrieve from cache
    ExtensionLoader<T> loader = (ExtensionLoader<T>) EXTENSION_LOADERS.get(type);
    if (loader == null) {
        // Each @SPI interface type corresponds to an ExtensionLoader object
        // Note that one interface type corresponds to one ExtensionLoader object
        EXTENSION_LOADERS.putIfAbsent(type, newExtensionLoader<T>(type));
        loader = (ExtensionLoader<T>) EXTENSION_LOADERS.get(type);
    }
    return loader;
}

3.2.3.2 getAdaptiveExtension

The core idea is to first retrieve the instance from the cache; if it does not exist, call createAdaptiveExtension() to create an instance. Once created, it is stored in the cache. The cache is a Holder object in ExtensionLoader, such as: Holder cachedAdaptiveInstance = new Holder<>();

public T getAdaptiveExtension() {
    // First retrieve the instance from the cache
    Object instance = cachedAdaptiveInstance.get();
    // DCL principle
    if (instance == null) {
        if (createAdaptiveInstanceError != null) {
            throw new IllegalStateException("Failed to create adaptive instance: " + createAdaptiveInstanceError.toString(), createAdaptiveInstanceError);
        }
        synchronized (cachedAdaptiveInstance) {
            instance = cachedAdaptiveInstance.get();
            if (instance == null) {
                try {
                    // Core method for creating interface instances
                    instance = createAdaptiveExtension();
                    cachedAdaptiveInstance.set(instance);
                } catch (Throwable t) {
                    createAdaptiveInstanceError = t;
                    throw new IllegalStateException("Failed to create adaptive instance: " + t.toString(), t);
                }
            }
        }
    }
    return (T) instance;
}

3.2.3.3 createAdaptiveExtension

This method is responsible for creating instances and performing IOC property dependency injection on them.

private T createAdaptiveExtension() {
    try {
        // injectExtension is Dubbo's IOC logic for assigning values to properties
        return injectExtension((T) getAdaptiveExtensionClass().newInstance());
    } catch (Exception e) {
        throw new IllegalStateException("Can't create adaptive extension " + type + ", cause: " + e.getMessage(), e);
    }
}

3.2.3.4 getAdaptiveExtensionClass

Retrieve and return the Class object of the class to be instantiated.

private Class<?> getAdaptiveExtensionClass() {
    // Core method, focus here. Establish the mapping between names and classes
    getExtensionClasses();
    // If there is a class with the @Adaptive annotation, return that class
    if (cachedAdaptiveClass != null) {
        return cachedAdaptiveClass;
    }
    // Dynamically assemble the class, dynamically compile and generate it
    return cachedAdaptiveClass = createAdaptiveExtensionClass();
}

3.2.3.5 getExtensionClasses

This method is a very core method; many APIs in Dubbo SPI require calling this method first to establish the mapping between keys and classes. The process of obtaining the mapping is also to first retrieve from the cache; if not present, read the configuration file to establish the mapping and cache it.

The purpose of this method:

1. Establish the mapping between keys and classes.2. Assign values to many global variables in ExtensionLoader that are used in Dubbo SPI APIs.

private Map<String, Class<?>> getExtensionClasses() {
    // First retrieve from cache
    Map<String, Class<?>> classes = cachedClasses.get();
    // DCL
    if (classes == null) {
        synchronized (cachedClasses) {
            classes = cachedClasses.get();
            if (classes == null) {
                // Load the key-class relationship from local files
                classes = loadExtensionClasses();
                // Cache the loaded mapping
                cachedClasses.set(classes);
            }
        }
    }
    return classes;
}

3.2.3.6 loadExtensionClasses

This method reads configuration files under resources/META-INF/dubbo and META-INF/dubbo/internal, parses the configuration files, establishes the mapping between keys and classes, assigns values to global variables in ExtensionLoader, and more.

private Map<String, Class<?>> loadExtensionClasses() {
    // Get the default implementation class name and cache it
    cacheDefaultExtensionName();
    Map<String, Class<?>> extensionClasses = new HashMap<>();
    // Retrieve LoadingStrategy instances from JDK's SPI mechanism
    for (LoadingStrategy strategy : strategies) {
        // Load files in the directory to establish the mapping between names and classes. Core logic
        loadDirectory(extensionClasses, strategy.directory(), type.getName(),
        strategy.preferExtensionClassLoader(), strategy.overridden(), strategy.excludedPackages());
        loadDirectory(extensionClasses, strategy.directory(),
        type.getName().replace("org.apache", "com.alibaba"), strategy.preferExtensionClassLoader(),
        strategy.overridden(), strategy.excludedPackages());
    }
    return extensionClasses;
}

3.2.3.6.1 cacheDefaultExtensionName

Sets the value of the global variable cachedDefaultName in ExtensionLoader, which is derived from the value of the @SPI(“spring”) annotation in the interface.

private void cacheDefaultExtensionName() {
    // Get the @SPI annotation on the class
    final SPI defaultAnnotation = type.getAnnotation(SPI.class);
    if (defaultAnnotation == null) {
        return;
    }
    String value = defaultAnnotation.value();
    // If there is a value in the @SPI annotation
    if ((value = value.trim()).length() > 0) {
        String[] names = NAME_SEPARATOR.split(value);
        if (names.length > 1) {
            throw new IllegalStateException("More than 1 default extension name on extension " + type.getName() + ": " + Arrays.toString(names));
        }
        // Set the value to cachedDefaultName, which is the default implementation class
        if (names.length == 1) {
            cachedDefaultName = names[0];
        }
    }
}

3.2.3.6.2 loadDirectory

Loads all files in the directory, establishes the mapping between names and classes, and sets the values of global variables. It calls loadResource in a loop.

while (urls.hasMoreElements()) {
    java.net.URL resourceURL = urls.nextElement();
    // Core method for loading files corresponding to the interface
    loadResource(extensionClasses, classLoader, resourceURL, overridden, excludedPackages);
}

3.2.3.6.3 loadResource

This method processes each file, establishing the mapping between names and classes and setting the values of global variables. It processes each line of data read from the file.

private void loadResource(Map<String, Class<?>> extensionClasses, ClassLoader classLoader,
java.net.URL resourceURL, boolean overridden, String... excludedPackages) {
    try {
        try (BufferedReader reader = new BufferedReader(new InputStreamReader(resourceURL.openStream(), StandardCharsets.UTF_8))) {
            String line;
            String clazz = null;
            // Read a line of data
            while ((line = reader.readLine()) != null) {
                final int ci = line.indexOf('#');
                // If there is a # in the line, only keep the part before it; the rest can be comments
                if (ci >= 0) {
                    line = line.substring(0, ci);
                }
                line = line.trim();
                if (line.length() > 0) {
                    try {
                        String name = null;
                        // Split by =
                        int i = line.indexOf('=');
                        if (i > 0) {
                            // The part before = is the key
                            name = line.substring(0, i).trim();
                            // The part after = is the class
                            clazz = line.substring(i + 1).trim();
                        } else {
                            clazz = line;
                        }
                        if (StringUtils.isNotEmpty(clazz) && !isExcluded(clazz, excludedPackages)) {
                            // Core logic for loading the class
                            loadClass(extensionClasses, resourceURL, Class.forName(clazz, true, classLoader), name, overridden);
                        }
                    } catch (Throwable t) {
                        IllegalStateException e = new IllegalStateException("Failed to load extension class (interface: " + type + ", class line: " + line + ") in " + resourceURL + ", cause: " + t.getMessage(), t);
                        exceptions.put(line, e);
                    }
                }
            }
        }
    } catch (Throwable t) {
        logger.error("Exception occurred when loading extension class (interface: " + type + ", class file: " + resourceURL + ") in " + resourceURL, t);
    }
}

3.2.3.6.4 loadClass

This method follows three main logic paths:

1. If the class has the @Adaptive annotation.2. If the class is a wrapper class.3. If it is neither an @Adaptive class nor a wrapper class.

private void loadClass(Map<String, Class<?>> extensionClasses, java.net.URL resourceURL,
Class<?> clazz, String name, boolean overridden) throws NoSuchMethodException {
    // If the class type does not match the interface type, throw an error
    if (!type.isAssignableFrom(clazz)) {
        throw new IllegalStateException("Error occurred when loading extension class (interface: " + type + ", class line: " + clazz.getName() + "), class " + clazz.getName() + " is not subtype of interface.");
    }
    // If the class has the @Adaptive annotation
    if (clazz.isAnnotationPresent(Adaptive.class)) {
        // Assign value to cachedAdaptiveClass variable in this method
        cacheAdaptiveClass(clazz, overridden);
    } else if (isWrapperClass(clazz)) {
        // If it is a wrapper class, it must hold a reference to the target interface with a corresponding constructor
        // Assign value to cachedWrapperClasses variable
        cacheWrapperClass(clazz);
    } else {
        // Get the no-argument constructor of the class; if it is a wrapper class, this will throw an error, but wrapper classes are handled first
        clazz.getConstructor();
        // If no key is configured
        if (StringUtils.isEmpty(name)) {
            // If the class has an Extension annotation, it is the value of the annotation
            // If there is no annotation, the lowercase class name is used as the name
            name = findAnnotationName(clazz);
            if (name.length() == 0) {
                throw new IllegalStateException("No such extension name for the class " + clazz.getName() + " in the config " + resourceURL);
            }
        }
        String[] names = NAME_SEPARATOR.split(name);
        if (ArrayUtils.isNotEmpty(names)) {
            // If the class has an @Activate annotation, establish the mapping between names and annotations
            // Assign value to cachedActivates global variable
            cacheActivateClass(clazz, names[0]);
            for (String n : names) {
                cacheName(clazz, n);
                // Here, extensionClasses establishes the mapping between key and class
                saveInExtensionClass(extensionClasses, clazz, n, overridden);
            }
        }
    }
}

3.2.3.7 createAdaptiveExtensionClass

1. Automatically generating the class string based on annotations and parameter configurations in the interface.2. Dynamically compiling and generating bytecode files to load into the JVM.

Rules defined by the interface:

1. At least one method in the interface must have the @Adaptive annotation; otherwise, an error will be thrown.2. The method parameters must include a URL parameter; otherwise, an error will be thrown.

Why is this designed this way?

Because this way, Dubbo will automatically generate a class, which is essentially a proxy class. However, Dubbo does not know which logic to execute, meaning this proxy class does not know which implementation class to use for the interface. Therefore, when invoking methods on the proxy object, we must inform Dubbo which implementation class to use based on the method’s input parameters, and the parameter for selecting the implementation class is in the URL parameter. Thus, the URL parameter must be included. Below is the proxy class generated by Dubbo:

import org.apache.dubbo.common.extension.ExtensionLoader;
public class ActivateApi$Adaptive implements cn.enjoy.dubbospi.ActivateApi {
    public java.lang.String todo(java.lang.String arg0, org.apache.dubbo.common.URL arg1) {
        if (arg1 == null) throw new IllegalArgumentException("url == null");
        org.apache.dubbo.common.URL url = arg1;
        String extName = url.getParameter("activate.api", "spring");
        if (extName == null) throw new IllegalStateException("Failed to get extension (cn.enjoy.dubbospi.ActivateApi) name from url (" + url.toString() + ") use keys([activate.api])");
        cn.enjoy.dubbospi.ActivateApi extension = (cn.enjoy.dubbospi.ActivateApi) ExtensionLoader.getExtensionLoader(cn.enjoy.dubbospi.ActivateApi.class).getExtension(extName);
        return extension.todo(arg0, arg1);
    }
}

From the code, it can be seen that the instance is selected based on the URL input parameter, and this instance is then used to invoke the method. The class ActivateApi$Adaptive is merely a proxy layer without substantive business logic; it is a process of selecting an instance based on parameters.

3.2.3.8 injectExtension

From our previous analysis, we have already generated an instance. This method is responsible for performing property dependency injection on that instance, essentially invoking the setXXX methods of that instance to assign parameter values.

private T injectExtension(T instance) {
    if (objectFactory == null) {
        return instance;
    }
    try {
        // Perform property injection via setXXX methods on the instance
        for (Method method : instance.getClass().getMethods()) {
            if (!isSetter(method)) {
                continue;
            }
            /**
            * Check {@link DisableInject} to see if we need auto injection for this property
            */
            // If there is a DisableInject annotation, do not inject
            if (method.getAnnotation(DisableInject.class) != null) {
                continue;
            }
            Class<?> pt = method.getParameterTypes()[0];
            if (ReflectUtils.isPrimitives(pt)) {
                continue;
            }
            try {
                // Get the property name from the method name
                String property = getSetterProperty(method);
                // Get the value to be injected
                Object object = objectFactory.getExtension(pt, property);
                if (object != null) {
                    // Reflectively assign the value
                    method.invoke(instance, object);
                }
            } catch (Exception e) {
                logger.error("Failed to inject via method " + method.getName() + " of interface " + type.getName() + ": " + e.getMessage(), e);
            }
        }
    } catch (Exception e) {
        logger.error(e.getMessage(), e);
    }
    return instance;
}

However, it is important to emphasize the sources of the parameter values for invoking the setXXX methods:

1. Obtained through Dubbo’s SPI method.2. Obtained through Spring container’s getBean method.

Using Dubbo SPI to Obtain Values

This method essentially calls the getAdaptiveExtension method to obtain the instance.

public class SpiExtensionFactory implements ExtensionFactory {
    @Override
    public <T> T getExtension(Class<T> type, String name) {
        if (type.isInterface() && type.isAnnotationPresent(SPI.class)) {
            ExtensionLoader<T> loader = ExtensionLoader.getExtensionLoader(type);
            if (!loader.getSupportedExtensions().isEmpty()) {
                return loader.getAdaptiveExtension();
            }
        }
        return null;
    }
}

Using Spring Container to Obtain Values

This method retrieves instances from the container using getBean.

public class SpringExtensionFactory implements ExtensionFactory, Lifecycle {
    private static final Logger logger = LoggerFactory.getLogger(SpringExtensionFactory.class);
    private static final Set<ApplicationContext> CONTEXTS = new ConcurrentHashSet<ApplicationContext>();
    public static void addApplicationContext(ApplicationContext context) {
        CONTEXTS.add(context);
        if (context instanceof ConfigurableApplicationContext) {
            ((ConfigurableApplicationContext) context).registerShutdownHook();
            // see https://github.com/apache/dubbo/issues/7093
            DubboShutdownHook.getDubboShutdownHook().unregister();
        }
    }
    public static void removeApplicationContext(ApplicationContext context) {
        CONTEXTS.remove(context);
    }
    public static Set<ApplicationContext> getContexts() {
        return CONTEXTS;
    }
    // currently for test purpose
    public static void clearContexts() {
        CONTEXTS.clear();
    }
    @Override
    @SuppressWarnings("unchecked")
    public <T> T getExtension(Class<T> type, String name) {
        // SPI should be obtained from SpiExtensionFactory
        if (type.isInterface() && type.isAnnotationPresent(SPI.class)) {
            return null;
        }
        for (ApplicationContext context : CONTEXTS) {
            T bean = BeanFactoryUtils.getOptionalBean(context, name, type);
            if (bean != null) {
                return bean;
            }
        }
        // logger.warn("No spring extension (bean) named:" + name + ", try to find an extension (bean) of type " + type.getName());
        return null;
    }
    @Override
    public void initialize() throws IllegalStateException {
        clearContexts();
    }
    @Override
    public void start() throws IllegalStateException {
        // no op
    }
    @Override
    public void destroy() {
        clearContexts();
    }
}

3.2.4 Summary

The getAdaptiveExtension method is a core method in Dubbo SPI that performs two main functions: obtaining instances and assigning values to global variables in ExtensionLoader, which will be used in other APIs. Instance retrieval can be divided into two types: obtaining instances of classes annotated with @Adaptive and obtaining proxy instances generated by Dubbo, which are generated based on interface configurations. The interface methods must have at least one method annotated with @Adaptive, and the method parameters must include a URL parameter.