The image on the cover is generated by a large model and may not be very reliable; please do not take it at face value.
The term “multi-modal” can be ambiguous. Those familiar with databases might interpret it as multiple data models (Multi-model), while those acquainted with artificial intelligence might understand it as multiple data modalities (Multi-modal). This reflects the first problem that needs to be addressed when combining data with AI: the alignment of concepts and semantics; otherwise, it could lead to misunderstandings. Additionally, in different contexts, the term “multi-modal database” may not refer to the same thing. In everyday communication, “multi-modal” is often used to indicate various types of data (such as text, audio, video, etc.). Similarly, there are issues with distinguishing between memory (Memory) and recall (Memory), as well as key-value (KV) and large model KV. Given that the English words for the latter two examples are the same, it is crucial to differentiate them based on context.
In the field of artificial intelligence, the data to be processed includes not only structured and semi-structured data but also a significant amount of unstructured data. Traditional databases excel at handling structured data, struggle with semi-structured data, and are even more challenged by unstructured data. From machine learning to the current large model training, the AI field predominantly uses programming languages like Python to facilitate data processing, developing various libraries and frameworks for this purpose. DuckDB can be considered a typical case in the database field aiming to leverage its strengths, hoping to accelerate daily data processing through a simple and user-friendly SQL language and high-performance system implementation. In the realm of large-scale structured and semi-structured data processing, big data systems, data warehouse systems, and data lake systems play a more significant role.
In facing the challenges of multi-modal data analytics, the most natural idea in the data field is to enhance or design new systems like data lakes to improve the handling of unstructured data, potentially allowing for the processing of all data types, including text, images, audio, and video, within a single system. Thus, architects still need to address several traditional issues: 1. Data storage (where to store it) 2. Data format (how to store it) 3. Data querying (how to compute it). From the perspective of large-scale data processing, there is no choice for data storage; object storage is prioritized. From the perspective of the large-scale data processing ecosystem, data format is also manageable, with open-source formats being prioritized to ensure interoperability. Data query processing, in addition to large-scale parallel computing, must also implement various multi-modal operators, introducing new cost models and query optimization techniques. From the perspective of system deployment and operational monitoring, choices need to be made regarding the runtime environment (preferably containers or lightweight virtual machines), processor chips (CPU, NPU, GPU, etc.), and operational monitoring systems (preferably integrating with existing systems).
Besides large-scale data processing requiring multi-modal data analytics, everyday data processing, which may not be as extensive, will also need multi-modal data analytics. For instance, every individual’s smartphone, tablet, laptop, and PC devices store a significant amount of multi-modal data, which should be deeply integrated to create more user-friendly query and analysis applications.
However, there are significant differences between multi-modal data analytics and traditional big data processing. A piece of structured data takes up very few bytes and can be processed quickly and economically by a CPU. In contrast, a piece of unstructured data is often a large file, and processing it with a GPU is not as fast. When both types of data are processed simultaneously in a single database record, it is evident that the latter will consume more resources and time. The ratio of CPU to GPU in servers is also challenging to match perfectly for these two different workloads, and data must flow not only across cards but also across machines. Even relatively simple multi-modal data vectorization (Embedding) for similarity retrieval, whether in batch or incremental data, is slow. Therefore, we can reasonably infer that multi-modal data analytics systems must first effectively address the issues of small amounts of multi-modal data before they can tackle the challenges of massive multi-modal data.