Understanding Cloud Computing, Big Data, and Artificial Intelligence

Source: Liu Chao's Simplified Cloud Computing, Big Data and Machine Learning Digest

This article is about 14,200 words long and is recommended to be read in over 20 minutes.
This article provides a detailed introduction to cloud computing, big data, and artificial intelligence.

Today, let’s talk about cloud computing, big data, and artificial intelligence. Why discuss these three topics? Because they are currently very popular and seem to be interconnected: when discussing cloud computing, big data is often mentioned; when talking about artificial intelligence, big data is also referenced; and when discussing artificial intelligence, cloud computing comes up as well… It feels like these three are mutually reinforcing and inseparable. However, for non-technical personnel, it may be difficult to understand the interrelationship among these three, so it is necessary to explain.

1. The Original Goal of Cloud Computing

Let’s first talk about cloud computing. The original goal of cloud computing was resource management, primarily managing three aspects: computing resources, network resources, and storage resources.

1. Managing Data Centers Like Setting Up Computers

What are computing, network, and storage resources? For example, if you want to buy a laptop, you would be concerned about the type of CPU it has and how much memory it has. These two are referred to as computing resources.

This laptop needs to connect to the internet, which requires a port for an Ethernet cable or a wireless network card that connects to your home router. You also need to activate a network with a service provider like China Unicom, China Mobile, or China Telecom, such as a 100M bandwidth. A technician will run a cable to your home and may help you configure your router to connect to their network. This way, all the computers, phones, and tablets in your home can access the internet through your router. This is network resources.

You may also ask how large the hard drive is? In the past, hard drives were small, around 10G; later, 500G, 1T (1T is 1000G), and 2T hard drives became common. This is storage resources.

For a single computer, it’s like this; for a data center, it’s the same. Imagine you have a very large server room filled with many servers, which also have CPUs, memory, and hard drives, and are connected to the internet through similar devices to routers. The question is: how do the people managing the data center uniformly manage these devices?

2. Flexibility Means Availability Anytime and Anywhere

The goal of management is to achieve flexibility in two aspects. What are these two aspects?

Let’s use an example to understand: if someone needs a very small computer with only one CPU, 1G of memory, 10G of hard drive, and one megabit of bandwidth, can you provide it? Such a small specification computer is outperformed by any modern laptop, and any home broadband connection is at least 100M. However, if they want this resource on a cloud computing platform, they can obtain it with just a click.

In this case, it can achieve flexibility in two aspects:

• Time Flexibility: You can request resources whenever you want; they are available immediately when needed.
• Space Flexibility: You can have as much as you need. If a small computer is required, it can be provided; if a very large space is needed, such as a cloud disk, it can allocate a very large space to everyone, allowing for uploads without running out of space.

Time flexibility and space flexibility are what we often refer to as the elasticity of cloud computing. The development of this elasticity has undergone a long evolution.

3. Physical Devices Lack Flexibility

The first stage is the physical device period. In this period, if a customer needs a computer, we buy one and place it in the data center.

Physical devices, of course, are becoming increasingly powerful, such as servers with hundreds of GB of memory; for example, network devices with a port bandwidth of tens of G or even hundreds of G; for storage, at least in the data center, it is at the PB level (1P is 1000T, 1T is 1000G).

However, physical devices cannot achieve good flexibility:

• First, it lacks time flexibility. It cannot meet the requirement of wanting resources whenever needed. For instance, buying a server or a computer requires procurement time. If a user suddenly tells a cloud vendor they want to start a physical server, it can be difficult to procure it immediately. If the relationship with the supplier is good, it may take a week; if the relationship is average, it may take a month. The user waits a long time for the computer to arrive, and then they have to log in and slowly start deploying their applications. Time flexibility is very poor.
• Secondly, its space flexibility is also lacking. For example, if the aforementioned user needs a very small computer, where can you find such a small model? You cannot just buy a tiny machine just to meet the demand for 1G of memory and 80G of hard drive. However, if you buy a larger one, you will have to charge the user more, which feels unfair since they only need such a small specification.

4. Virtualization Offers More Flexibility

Someone figured out a way. The first solution is virtualization. If the user only needs a very small computer, the physical devices in the data center are powerful enough; I can virtualize a small portion of the physical CPU, memory, and hard drive for the customer, while also virtualizing a small portion for other customers. Each customer can only see their small portion, but in reality, they are all using a small part of the entire large device.

The technology of virtualization allows different customers’ computers to appear isolated. That is, it looks like this disk is mine, and that disk is yours, but in reality, my 10G and your 10G may reside on the same very large storage. Moreover, if the physical devices are prepared in advance, the virtualization software can create a computer very quickly, basically within minutes. Therefore, creating a computer on any cloud takes just a few minutes, and this is how it works.

Thus, space flexibility and time flexibility are basically resolved.

5. Making Money and Passion in the Virtual World

During the virtualization stage, the most powerful company was VMware. It was one of the earliest companies to implement virtualization technology, achieving virtualization of computing, networking, and storage. This company was very successful, producing excellent performance, selling virtualization software very well, and making a lot of money, which later led to its acquisition by EMC (the world’s top 500, the number one brand in storage manufacturing).

However, there are still many passionate people in the world, especially among programmers. What do passionate people like to do? Open source.

In this world, many software have both closed-source and open-source versions. Closed-source means that the code of a well-developed software is kept secret; only my company knows it, and others do not. If others want to use this software, they must pay me, which is called closed-source.

But there are always some great individuals who cannot stand the situation where all the money goes to one company. They believe that if you can develop it, so can I; if I develop it, I won’t charge for it, and I will share the code with everyone. Everyone in the world can benefit from it, which is called open source.

For example, Tim Berners-Lee is a very passionate person. In 2017, he received the Turing Award for “inventing the World Wide Web, the first browser, and the fundamental protocols and algorithms that made the web expand.” However, what is most admirable about him is that he contributed the World Wide Web technology, which we commonly refer to as WWW, to the world for free. We should all thank him for our online behavior; if he had charged for this technology, he would likely be as wealthy as Bill Gates.

There are many examples of open source and closed source:

For instance, in the closed-source world, there is Windows, which everyone must pay Microsoft to use; in the open-source world, there is Linux. Bill Gates made a lot of money from closed-source software like Windows and Office, becoming the world’s richest person, while another great individual developed another operating system called Linux. Many people may not have heard of Linux, but many backend programs run on Linux, such as those that support the shopping systems of Taobao, JD, and Kaola during the Double Eleven shopping festival.

Similarly, for Apple, there is Android. Apple has a high market value, but we cannot see the code of the Apple system. Therefore, some great individuals developed the Android operating system. As a result, almost all other phone manufacturers install the Android system because the Apple system is not open source, while the Android system is available for everyone to use.

In virtualization software, there is also VMware, which is very expensive. So, some great individuals developed two open-source virtualization software, called Xen and KVM. If you are not technical, you can ignore these two names, but they will be mentioned later.

6. Semi-Automatic Virtualization and Fully Automatic Cloud Computing

While virtualization software solves the flexibility problem, it is not entirely correct. This is because creating a virtual computer typically requires manual specification of which physical machine the virtual computer will be placed on. This process may also require complex manual configurations. Therefore, using VMware’s virtualization software requires passing a very challenging certification, and those who obtain this certification have quite high salaries, indicating the complexity involved.

Thus, the scale of physical machine clusters that can be managed by virtualization software is not particularly large, generally just a few dozen or at most a hundred machines.

This affects time flexibility: although the time to create a virtual computer is short, as the scale of the cluster increases, the manual configuration process becomes increasingly complex and time-consuming. It also affects space flexibility: when the number of users increases, this cluster scale cannot meet the demand for “as much as needed,” and resources may run out quickly, requiring procurement.

As the scale of the cluster increases, it generally starts from thousands of machines, easily reaching tens of thousands or even hundreds of thousands of machines. If you check the number of servers for companies like BAT, including NetEase, Google, and Amazon, the numbers are astonishing. Managing so many machines by manually selecting a location for each virtual computer and making the corresponding configuration is almost impossible and requires machines to do this.

Various algorithms have been invented to address this issue, called scheduling. Simply put, there is a scheduling center where thousands of machines are pooled together; regardless of how many CPUs, memory, or hard drives a user needs for their virtual computer, the scheduling center will automatically find a suitable place in the pool to meet the user’s needs, start the virtual computer, and configure it so that the user can use it directly. This stage is referred to as pooling or clouding. At this stage, it can be called cloud computing; prior to this, it can only be referred to as virtualization.

7. Private and Public Cloud Computing

Cloud computing is roughly divided into two types: private cloud and public cloud, and some people connect private and public clouds and call it hybrid cloud, which we will not discuss here.

• Private Cloud: This involves deploying virtualization and cloud software in someone else’s data center. Users of private clouds are often wealthy, buying land to build their own data centers and purchasing their own servers, then having cloud vendors deploy them there. VMware later introduced cloud computing products in addition to virtualization and made a significant profit in the private cloud market.
• Public Cloud: This involves deploying virtualization and cloud software in the cloud vendor’s data center, where users do not need to make a large investment; they simply register an account and can create a virtual computer with a click on a webpage. For example, AWS is Amazon’s public cloud; in China, there are Alibaba Cloud, Tencent Cloud, NetEase Cloud, etc.

Why did Amazon create a public cloud? We know that Amazon was originally a major e-commerce company abroad. When it was doing e-commerce, it surely encountered situations similar to Double Eleven: at a certain moment, everyone rushes to buy things. When everyone rushes to buy, there is a strong need for the time and space flexibility of the cloud. They cannot have all resources prepared all the time, as that would be a waste. However, they also cannot have nothing prepared and watch so many users wanting to buy but unable to log in. Therefore, they need to create a large number of virtual computers to support e-commerce applications during Double Eleven and release those resources afterward for other uses. Thus, Amazon needed a cloud platform.

However, commercial virtualization software is too expensive, and Amazon cannot spend all the money it earns from e-commerce on virtualization vendors. Therefore, Amazon developed its own cloud software based on open-source virtualization technologies, such as Xen or KVM. Unexpectedly, Amazon’s e-commerce business became increasingly successful, and its cloud platform also thrived.

Because its cloud platform needed to support its e-commerce applications, and traditional cloud computing vendors mostly originated from IT companies with no applications of their own, Amazon’s cloud platform became more application-friendly, rapidly developing into the number one brand in cloud computing and earning a lot of money.

Before Amazon released its cloud platform’s financial report, people speculated whether Amazon’s e-commerce was profitable and whether its cloud services were profitable as well. Later, when the financial report was released, it turned out that it was not just profitable. Last year alone, Amazon AWS’s annual revenue reached $12.2 billion, with an operating profit of $3.1 billion.

8. Making Money and Passion in Cloud Computing

The leading public cloud provider, Amazon, is doing very well, while the second place, Rackspace, is doing just average. This is the cruel nature of the internet industry, where often the winner takes all. Therefore, if the second place is not in the cloud computing industry, many people may not have heard of it.

So the second place thought, how can I compete with the leader? Let’s go open source. As mentioned before, although Amazon used open-source virtualization technology, the cloud code is closed source. Many companies that want to build a cloud platform but cannot do so can only watch Amazon make big money. Rackspace decided to open its source code, allowing the entire industry to collaboratively improve the platform, and everyone can join in to compete with the leader.

Thus, Rackspace collaborated with NASA to establish the open-source software OpenStack. As shown in the architecture diagram of OpenStack, those who are not in the cloud computing industry do not need to understand this diagram, but they can see three keywords: Compute, Networking, Storage. It is still a cloud management platform for computing, networking, and storage.

Of course, the technology of the second place is also very impressive. With OpenStack, as Rackspace hoped, all major IT companies that you can think of went crazy wanting to build cloud platforms: IBM, HP, Dell, Huawei, Lenovo, etc.

It turned out that everyone wanted to build a cloud platform. Seeing that Amazon and VMware made so much money, they felt that it was quite difficult to create one. Now, with the open-source cloud platform OpenStack, all IT vendors joined this community to contribute to this cloud platform, packaging it into their own products and selling it along with their hardware. Some created private clouds, some created public clouds, and OpenStack has become the de facto standard for open-source cloud platforms.

9. IaaS, Flexibility at the Resource Level

As OpenStack’s technology matured, the scale it could manage also increased, and multiple OpenStack clusters could be deployed. For example, one set could be deployed in Beijing, two sets in Hangzhou, and one set in Guangzhou, allowing for unified management. This way, the overall scale becomes larger.

At this scale, for ordinary users, it can basically achieve the ability to request resources whenever needed and have as much as required.

Taking cloud storage as an example, each user is allocated 5T or even larger space. If there are 100 million users, how large is that space?

In fact, the underlying mechanism is as follows: when allocating your space, you may have only used a small portion of it. For instance, if it allocated you 5T, this large space is only what you see, not what is actually allocated. If you have only used 50G, then the real allocation is just 50G, and as you keep uploading files, the allocated space will increase.

When everyone uploads and the cloud platform finds that it is nearing capacity (for example, 70% usage), it will procure more servers to expand the underlying resources. This process is transparent and invisible to users. From a perception standpoint, it achieves the elasticity of cloud computing. It is somewhat like a bank: the feeling for depositors is that they can withdraw money whenever they want; as long as there is no run on the bank, it will not collapse.

10. Conclusion

At this stage, cloud computing has basically achieved time flexibility and space flexibility; it has achieved elasticity in computing, networking, and storage resources. We often refer to computing, networking, and storage as infrastructure. Therefore, the elasticity at this stage is called resource-level elasticity. The cloud platform that manages resources is known as Infrastructure as a Service, or IaaS.

II. Cloud Computing Not Only Manages Resources but Also Applications

Is achieving resource-level elasticity enough with IaaS? Obviously not; there is also a need for application-level elasticity.

Let’s take an example: for an e-commerce application, ten machines are usually sufficient, but during Double Eleven, one hundred machines are needed. You might think this is easy to handle; with IaaS, you can simply create ninety new machines. However, the ninety machines created are empty; the e-commerce application has not been installed on them, and the operations personnel of the company must set it up one by one, which takes a long time to complete.

Although resource-level elasticity has been achieved, without application-level elasticity, flexibility remains insufficient. Is there a way to solve this problem?

People added another layer on top of the IaaS platform to manage the application elasticity issue, which is usually referred to as PaaS (Platform As A Service). This layer is often difficult to understand and is roughly divided into two parts: one part I call “automated installation of your own application,” and the other part I call “general applications that do not require installation.”

• Your Own Application Automated Installation: For example, if the e-commerce application is developed by you, no one else knows how to install it. During installation, you need to configure your Alipay or WeChat account to ensure that payments made by others on your e-commerce site go into your account. No one else knows how to do this. Therefore, the platform cannot assist with the installation process, but it can help automate it. You need to do some work to integrate your configuration information into the automated installation process. For instance, in the above example, if the ninety new machines created during Double Eleven are empty, if a tool can be provided to automatically install the e-commerce application on these new machines, it would achieve true application-level elasticity. Technologies such as Puppet, Chef, Ansible, and Cloud Foundry can do this, and the latest container technology, Docker, can do it even better.
• General Applications That Do Not Require Installation: General applications refer to complex applications that are commonly used, such as databases. Almost all applications will use databases, but database software is standard; although installation and maintenance can be complex, anyone can install it the same way. Such applications can be turned into standard PaaS applications and placed on the cloud platform interface. When a user needs a database, they can obtain it with just a click and start using it. Some may ask, since anyone can install it, why not just do it myself instead of paying the cloud platform? The answer is no; databases are very complex. For example, Oracle makes a lot of money from databases, and purchasing Oracle can be quite expensive.

However, most cloud platforms will provide open-source databases like MySQL, which do not require as much expenditure. But maintaining this database requires a large team, and optimizing it to support Double Eleven cannot be achieved in just one or two years.

For example, if you are running a bicycle business, you certainly do not need to hire a very large database team to handle this, as the cost would be too high. Instead, you should leave this task to the cloud platform, which has specialized personnel maintaining the system, allowing you to focus on your bicycle application.

In summary, whether it is through automated deployment or not needing deployment, the application layer should also require less concern; this is the important role of the PaaS layer.

While scripts can solve the deployment problem of your own applications, different environments can vary greatly, and a script that runs correctly in one environment may not work in another.

Containers can better solve this problem.

Containers refer to a packaging method for software delivery. The characteristics of containers are: encapsulation and standardization.

In the era before containers, if you wanted to transport goods from A to B, you had to go through three docks and change ships three times. Each time, you had to unload the goods from the ship, causing disarray, and then reload them onto the new ship. Therefore, without containers, every time you change ships, the crew would have to stay on shore for a few days before departing.

With containers, all goods are packed together, and all containers are of consistent size, so every time you change ships, you can simply move the whole container over, completing the process in a matter of hours, and the crew no longer has to delay on shore.

This demonstrates the application of the two main characteristics of containers: encapsulation and standardization.

So how do containers package applications? We still need to learn from containers. First, there must be a closed environment to encapsulate the goods, ensuring that the goods do not interfere with each other and are isolated, making loading and unloading convenient. Fortunately, LXC technology in Ubuntu can already achieve this.

The closed environment mainly uses two types of technologies: one is the isolation technology, called Namespace, which means that applications within each Namespace see different IP addresses, user spaces, process numbers, etc. The other is resource isolation technology, called Cgroups, which means that while the entire machine has many CPUs and memory, one application can only use a portion of them.

The so-called image is the moment you finish packing your container, saving the state of the container, much like the Monkey King saying, “Stop!” At that moment, the container is fixed, and the state at that moment is saved as a series of files. These files are standardized, allowing anyone who sees them to restore the state at that moment. The process of restoring the image to runtime (reading the image file and restoring that moment) is the process of running the container.

With containers, the PaaS layer for users’ applications becomes fast and elegant.

III. Big Data Embraces Cloud Computing

In the PaaS layer, a complex general application is the big data platform. How does big data gradually integrate into cloud computing?

1. Data is not large but contains wisdom

Initially, big data was not that big. How much data was there originally? Nowadays, everyone reads e-books and browses news online. When we were kids in the 80s, the amount of information was not that large; we would just read books and newspapers. How much text could be in a week’s worth of newspapers? If you are not in a big city, an ordinary school library may have only a few shelves of books. It was only with the advent of information technology that the amount of information increased.

First, let’s look at the types of data in big data, which can be divided into three types: structured data, unstructured data, and semi-structured data.

• Structured Data: This refers to data with a fixed format and limited length. For example, filling out a form is structured data, such as nationality: People’s Republic of China, ethnicity: Han, gender: male; all of these are structured data.
• Unstructured Data: Unstructured data is increasingly common; it has no fixed format and varies in length, such as web pages, which can be very long or just a few sentences; audio and video are also unstructured data.
• Semi-Structured Data: This includes data in formats like XML or HTML, which may not be understood by non-technical individuals, but that’s okay.

In fact, data itself is not useful; it must go through certain processing. For example, the data collected by a fitness tracker when you run is also data, and the numerous web pages online are also data, which we refer to as Data. Data itself has little value, but it contains something very important called information (Information).

Data is very chaotic, and it must be sorted and cleaned to be considered information. Information contains many patterns, and we need to summarize the patterns from the information, which we call knowledge (Knowledge), and knowledge changes destiny. There is a lot of information, but some people see it as merely information, while others see the future of e-commerce or the future of live streaming from the information, making them successful. If you do not extract knowledge from information, you will remain a bystander in the torrent of the internet.

With knowledge, you can apply it in practice; some people do very well, and this is called wisdom (Intelligence). Having knowledge does not necessarily mean having wisdom; for example, many scholars possess knowledge and can analyze past events from various perspectives but cannot translate that into wisdom. Many entrepreneurs are great because they apply the knowledge they acquire in practice and end up building great businesses.

Therefore, the application of data can be summarized in four steps: Data, Information, Knowledge, Wisdom.

The final stage is what many businesses desire. They collect a lot of data and wonder if they can use this data to help them make decisions and improve their products. For example, when users watch videos, ads pop up next to them for products they want to buy; or when users listen to music, they are recommended other songs they would love to hear.

Any action users take on my application or website, such as clicking the mouse or typing in text, generates data for me. I want to extract certain elements from this data, guide practice, and form wisdom, ensuring that users are so engaged in my application that they cannot extricate themselves. They will keep clicking and buying.

Many people say they want to cut off the internet during Double Eleven because their wives keep buying, purchasing A and then recommending B. The wife says, “Oh, B is something I like too; I want to buy it, dear.” How does this program know so much? How does it understand me better than I do? How is this achieved?

2. How Data Transforms into Wisdom

The processing of data involves several steps before achieving wisdom.

The first step is data collection. First, there must be data, and there are two ways to collect data:

• The first method is to capture, which is technically called scraping or crawling. For example, this is what search engines do: they download all the information from the internet to their data centers, allowing you to search for it later. For instance, when you search, the results are a list; why is this list in the search engine’s company? It’s because they captured the data, but when you click on a link, the page is no longer in the search engine’s company.
• The second method is pushing, where many terminals can help collect data. For example, a Xiaomi fitness tracker can upload your daily running data, heart rate data, and sleep data to the data center.

The second step is data transmission. Generally, this is done through a queuing method because the data volume is enormous, and it must be processed to be useful. If the system cannot handle it, it must queue up for slow processing.

The third step is data storage. Data is money now; having data is equivalent to having money. Otherwise, how would websites know what you want to buy? Because they have your historical transaction data, this information cannot be shared with others and is extremely valuable, so it needs to be stored.

The fourth step is data processing and analysis. The stored data is raw data, which is often chaotic and contains a lot of garbage data; thus, it needs to be cleaned and filtered to obtain high-quality data. For high-quality data, analysis can be conducted to classify the data or discover the relationships between data, leading to knowledge.

For example, the famous story of Walmart’s beer and diapers was discovered by analyzing people’s purchasing data, revealing that men often buy beer when they purchase diapers. This led to the discovery of the relationship between beer and diapers, which was then applied in practice by placing the beer and diaper counters close together, thereby achieving wisdom.

The fifth step is data retrieval and mining. Retrieval refers to searching; when it comes to external matters, we consult Google, and for internal matters, we consult Baidu. Both major search engines place analyzed data into their engines, so when people want to find information, they can search and obtain results.

Additionally, mining is necessary; merely searching does not satisfy people’s requirements. It is also essential to extract mutual relationships from the information. For instance, in financial searches, when searching for a company’s stock, should the company’s executives also be extracted? If you only search for a company’s stock and find it is performing well, then decide to buy it, but the executives have made a statement that negatively affects the stock, it could be detrimental to investors. Therefore, using various algorithms to mine relationships in the data and form a knowledge base is crucial.

Therefore, what is big data? Simply put, it means that one machine cannot handle it alone, so everyone must work together. However, as the data volume increases, many small companies need to process a significant amount of data, but these small companies do not have enough machines. What should they do?

4. Big Data Needs Cloud Computing, Cloud Computing Needs Big Data

At this point, you may recall cloud computing. When it comes to doing this work, many machines are needed to work together, truly achieving the ability to request resources whenever needed and have as much as required.

For example, if a big data analysis company needs to analyze its financial situation, it may do so weekly. If they had to keep one hundred or one thousand machines on standby, it would be a huge waste. So can they only deploy these one thousand machines when they need computing power, and let them do other tasks when they don’t need them?

Who can accomplish this? Only cloud computing can provide the resource-level flexibility needed for big data operations. Cloud computing can also deploy big data onto its PaaS platform as a very important general application. This is because big data platforms can enable multiple machines to work together, and these platforms are not something that can be easily developed or managed by just anyone; it generally requires a team of dozens or hundreds of people to operate effectively.

Therefore, like databases, it requires a team of professionals to manage it. Nowadays, most public clouds offer big data solutions, so when a small company needs a big data platform, they do not need to procure a thousand machines; they can simply click on the public cloud to have those thousand machines available, with the big data platform already deployed, ready for them to upload data and perform calculations.

Cloud computing needs big data, and big data needs cloud computing; thus, the two are combined.

IV. Artificial Intelligence Embraces Big Data

1. When Can Machines Understand Human Hearts?

Although big data is available, human desires are not fully met. While search engines exist within big data platforms, allowing users to find what they want easily, there are still cases where people cannot express what they want or cannot find what they are looking for.

For example, a music app may recommend a song that I have never heard of, and of course, I do not know the name and cannot search for it. However, if the app recommends it to me and I like it, that is something search cannot achieve. When people use such applications, they feel as if the machine knows what they want, rather than having to search for it when they want it. The machine seems to understand me like a friend, which is somewhat the essence of artificial intelligence.

People have been pondering this for a long time. Initially, they imagined a wall with a machine behind it; if I spoke to it, it would respond. If I could not tell whether it was a person or a machine, then it would truly be an artificial intelligence.

2. Teaching Machines to Reason

How can we achieve this? People thought: first, I need to teach the computer human reasoning abilities. What is important about humans? What distinguishes humans from animals is the ability to reason. If I could teach this reasoning ability to machines, allowing them to deduce appropriate answers based on your questions, that would be great!

Currently, people are gradually enabling machines to perform some reasoning tasks, such as proving mathematical formulas. This has been a surprising process, as machines can indeed prove mathematical formulas. However, it has also been discovered that this result is not as surprising as it seems. This is because mathematical formulas are very rigorous, and the reasoning process is also very strict, making it relatively easy to express mathematically and programmatically.

However, human language is not that simple. For instance, if you and your girlfriend have a date tonight, and she says: “If you come early and I am not there, you wait; if I come early and you are not there, you wait!” The machine would find this quite difficult to comprehend, but humans understand it.

3. Teaching Machines Knowledge

Therefore, simply teaching machines strict reasoning is not enough; they also need to learn some knowledge. However, teaching knowledge to machines is generally not something that ordinary people can do. Experts, such as language or finance specialists, might be able to do it.

Can knowledge in the fields of language and finance be expressed in a way that is slightly more rigorous like mathematical formulas? For example, language experts might summarize grammatical rules, such as subject-verb-object structures, stating that a subject must be followed by a verb, and a verb must be followed by an object. Summarizing these rules and expressing them rigorously should suffice, right?

However, it was discovered that this is not feasible; it is too difficult to summarize, as language expression can vary greatly. Take the subject-verb-object example; often, in spoken language, the verb is omitted. If someone asks, “Who are you?” and I answer, “I am Liu Chao,” you cannot require the machine to understand it in standard written language; that would not be intelligent.

This stage of artificial intelligence is called expert systems. Expert systems are difficult to succeed because, on one hand, knowledge is challenging to summarize, and on the other hand, the summarized knowledge is hard to teach to computers. If you are still confused and feel like there is a pattern but cannot articulate it, how can you program it to teach a machine?

4. Forget it, Let Machines Learn on Their Own

Thus, people thought: machines are entirely different beings from humans; let them learn on their own.

How do machines learn? Since machines have strong statistical capabilities, they can discover patterns from large amounts of data based on statistical learning.

In the entertainment industry, there is a good example of this:

A netizen compiled the lyrics of 117 songs from nine albums released by a well-known singer in mainland China. Each unique word in a song is counted only once, and the top ten adjectives, nouns, and verbs are as follows (the numbers indicate the frequency of occurrence):

If we randomly write a string of numbers and select a word from the adjective, noun, and verb lists according to the digits, what would happen?

For example, taking the digits of pi 3.1415926, the corresponding words are: strong, road, fly, freedom, rain, bury, confusion. If we connect and polish them a bit:

A strong child,

Still moving forward on the road,

Spreading wings to fly towards freedom,

Letting the rain bury his confusion.

Does that feel somewhat poetic? Of course, the actual statistical learning algorithms are much more complex than this simple statistic.

However, statistical learning can easily understand simple correlations: for example, if one word often appears with another, the two words should be related; yet it cannot express complex correlations. Moreover, the formulas used in statistical methods are often very complex, and to simplify calculations, various independence assumptions are often made, which reduce the computational difficulty of the formulas. However, in real life, events that are independent are relatively rare.

5. Simulating the Brain’s Functioning

Thus, humans began to reflect on how the human world works from the machine’s perspective.

The human brain does not store a large number of rules or record a vast amount of statistical data; rather, it operates through the activation of neurons. Each neuron receives inputs from other neurons, and when it receives input, it generates an output to stimulate other neurons. Thus, numerous neurons interact, ultimately producing various output results.

For example, when people see a beautiful woman, their pupils dilate; this is not because the brain judges based on body proportions or because it statistically analyzes all the beautiful women they have seen in life. Instead, it is the neurons triggered from the retina to the brain and back to the pupils. In this process, it is difficult to summarize which neurons contributed to the final result; they just work.

Thus, people began to use a mathematical unit to simulate neurons.

This neuron has inputs and outputs, and the relationship between inputs and outputs is expressed through a formula, where inputs influence outputs based on their importance (weights).

Thus, n neurons can be connected together to form a neural network. The number n can be very large, and all neurons can be arranged in many layers, with many neurons in each layer. Each neuron can have different weights for inputs, resulting in different formulas for each neuron. When people input something into this network, they hope to output a result that is correct to humans.

For example, if you input a picture with the number 2 written on it, the second number in the output list should be the largest. However, from the machine’s perspective, it does not know that the input image represents the number 2, nor does it understand the significance of the output numbers; it is fine as long as humans understand the significance. Just as neurons do not know that what they see is a beautiful woman or that dilating pupils are to see better; they just react accordingly.

For any neural network, no one can guarantee that if the input is 2, the output will definitely have the second number as the largest. To ensure this result, training and learning are required. After all, the dilation of pupils when seeing a beautiful woman is the result of many years of human evolution. The learning process involves inputting a large number of images; if the result is not as desired, adjustments are made.

How is the adjustment done? Each neuron’s weights are fine-tuned towards the target. Since there are many neurons and weights, the results produced by the entire network do not show binary outcomes; rather, they make gradual progress towards the target result.

Of course, the strategies for these adjustments are very skillful and require algorithm experts to fine-tune them. Just like when humans see a beautiful woman, the pupils do not initially dilate enough to see clearly, causing the beautiful woman to run away; the next learning result is to slightly dilate the pupils, not to dilate the nostrils.

6. No Reason, But It Works

It sounds somewhat unreasonable, but it indeed works; it is just that way!

The universality theorem of neural networks states that if someone gives you some complex and peculiar function, f(x):

No matter what this function looks like, there will always be a neural network that can produce an output for any possible input x that is equal to f(x) (or closely approximates it).

If the function represents a pattern, it means that no matter how wonderful or incomprehensible this pattern is, it can be expressed through many neurons and the adjustment of numerous weights.

7. Economic Explanation of Artificial Intelligence

This reminds me of economics, making it easier to understand.

We can consider each neuron as an individual engaged in economic activities in society. Thus, the neural network represents the entire economic society, where each neuron adjusts its output based on the input from society, such as wage increases, rising food prices, or falling stock prices, determining how to spend their money. Is there no pattern in this? Certainly, there is, but what exactly is the pattern? It is quite difficult to specify.

In contrast, expert-based economics resembles planned economy. The representation of economic laws does not rely on the independent decisions of each economic entity but rather on the high-level summaries of experts with foresight. However, experts can never know which city or street lacks a vendor selling sweet tofu.

As a result, experts may suggest how much steel or buns should be produced, but their recommendations often deviate significantly from the actual needs of people’s lives. Even if a plan is written in several hundred pages, it cannot express the hidden small patterns of people’s lives.

In contrast, statistical macro-control is much more reliable. Every year, the statistical bureau collects data on employment rates, inflation rates, GDP, etc. These indicators often represent many underlying patterns; while they may not be precisely expressed, they are relatively reliable.

However, summarizing and expressing statistical patterns is relatively coarse. For example, economists can analyze these statistical data to determine whether housing prices are rising or falling in the long term and whether stocks are likely to rise or fall. For instance, if the economy is generally improving, housing prices and stocks should both rise. Yet, based on statistical data, it is impossible to summarize the subtle fluctuations in stock prices and prices.

Microeconomics based on neural networks is the most accurate representation of all economic laws. Each individual adjusts their decisions based on the inputs from society, and these adjustments also feed back into society. Imagine the subtle fluctuation curves of the stock market; they are the results of the continuous transactions of independent individuals, without a unified pattern to follow.

When individuals make independent decisions based on societal inputs, certain factors can form macro-level statistical patterns through repeated training. This is what macroeconomics can observe. For example, whenever large amounts of currency are issued, housing prices tend to rise; after multiple training sessions, people learn this pattern.

8. Artificial Intelligence Needs Big Data

However, neural networks contain so many nodes, each with numerous parameters, resulting in an enormous amount of computation required. However, this is not a problem, as we have big data platforms that can leverage multiple machines to compute together, allowing us to obtain the desired results within a limited time.

Artificial intelligence can accomplish many tasks, such as identifying spam emails, recognizing violent or explicit text and images, etc. This has gone through three stages:

• The first stage relies on keyword blacklists and filtering techniques, which specify which words are considered explicit or violent. As online language evolves, the list of words constantly changes, making it difficult to keep the word bank updated.
• The second stage involves new algorithms, such as Bayesian filtering. You don’t need to worry about what Bayesian algorithms are, but you may have heard of them; they are based on probabilities.
• The third stage uses big data and artificial intelligence to perform more precise user profiling and text and image understanding.

Since artificial intelligence algorithms often rely on large amounts of data, these data typically need to accumulate over time in specific fields (e.g., e-commerce, email). Without data, even the best artificial intelligence algorithms are useless. Therefore, artificial intelligence programs are rarely installed for individual customers like IaaS and PaaS, allowing customers to use them. This is because when you install a set for a specific customer, if they lack relevant data for training, the results are often poor.

However, cloud computing vendors usually accumulate a large amount of data. Therefore, they install a set of services and expose an API. For example, if you want to determine whether a text involves explicit or violent content, you can directly use this online service. This form of service is referred to in cloud computing as Software as a Service, or SaaS.

Thus, artificial intelligence programs have entered cloud computing as SaaS platforms.

V. A Beautiful Life Based on the Relationship of the Three

Finally, the three brothers of cloud computing are gathered: IaaS, PaaS, and SaaS. Therefore, on a cloud computing platform, you can find cloud computing, big data, and artificial intelligence. A big data company that has accumulated a lot of data will use some artificial intelligence algorithms to provide certain services; an artificial intelligence company cannot be without a big data platform.

Therefore, when cloud computing, big data, and artificial intelligence are integrated, it completes the process of meeting, knowing, and understanding each other.

Editor: Yu Tengkai

Proofreader: Tan Jiayao