Overview

What is artificial intelligence? It was already a well-known field, but as systems such as image-generating AI have improved, more people seem to have become interested in studying it. Terms such as AI, deep learning, machine learning, data science, and big data are often used interchangeably. That can make it surprisingly difficult to explain what deep learning actually is and what the field studies.

The articles that follow refer to the blog for Stanford's CS231n course and are, for practical purposes, close to a translated set of notes. All rights belong to the original site. I wrote these posts for study and to share my own understanding with anyone who might find it helpful.

What Is AI?

This article is something of an orientation. The theoretical material begins in the next post, but I first wanted to establish the larger topic of the course. AI stands for artificial intelligence: intelligence created artificially.

I am not religious, and I do not mean to make a religious argument, but one of the seemingly impossible things for humans to create is another human—or an entity with comparable intelligence. Emotion and reason feel as boundless and difficult to fathom as the universe. Even so, I chose this path as an engineering student. While studying, I began to feel that working on artificial intelligence resembles astronomy: both involve stepping into an unknown domain and trying to uncover its secrets. Of course, these days I am usually too busy looking for the latest SOTA paper to think about it that grandly.

To return to the point, AI can be described as an entity that represents human intellect or intelligence. It may appear as a robot in hardware or as a program in software. If we translate the human process of receiving sensory information and interpreting it in the brain into a computing environment, we need interfaces that sense particular modalities and a processing module capable of combining and interpreting them.

AI is therefore the broadest boundary we commonly use, covering the ways artificially created intelligent entities are represented. Building such a system requires information received through interfaces and machine learning that gives a machine the ability to interpret that information. More recently, work on handling information has been further classified as data processing or data science. Data science tends to focus less on proposing methods by which a machine learns and more on refining information so that meaningful insights can be used by an algorithm.

Deep learning developed from one family of machine-learning methods: neural networks. Gradient-descent-based optimization, large datasets, and advances in hardware such as GPUs made neural-network training practical. Much of what people have recently encountered under the name AI is probably based on deep learning.

One major reason deep learning grew within AI, which had broadly advanced along the two axes of data and machine learning, is feature engineering. A computer needs data to be refined into useful information. Traditional machine-learning systems used various hand-designed algorithms to classify data or find the same object across different images. If no suitable algorithm could process the dataset's features, good performance was difficult to obtain.

Despite the name artificial intelligence, people had to provide not only labels for the desired outputs, but also algorithms that extracted meaningful features from the input modality. These manually designed algorithms often suffered another problem: their generalization did not continue to improve as the amount of data grew. What we wanted instead was a system that could discover useful features in its inputs and use those features to produce an answer. Deep learning, based on deep neural networks, became a field that surpassed many traditional machine-learning approaches in precisely this respect.

Here is a simple formulation. Let the input space be $X$, the feature space be $Y$, and let each input $x \sim X$ have a corresponding output $z \sim Z$. A space here is a set. We cannot have supervision for every causal relationship in the world, so suppose each space is open-ended and we consider only an observable subspace. A subspace can simply be understood as a subset.

In conventional machine learning, prediction from input to output can be written using fixed algorithms $F,~G$ as

$$F(x) = y,~G(y) = \hat{z} \approx z.$$

This resembles a composition of functions. These exact functions do not literally exist, of course; the goal is to model $F,~G$ so that output $Z$ can be predicted well from input $X$. In conventional machine learning, $F$ corresponds to a feature-extraction algorithm implemented through feature engineering. It does not update: additional data or training cannot improve the implemented algorithm itself. Function G then performs inference on the input using the engineered features. Because feature engineering had already reached a performance ceiling, further improvement in this stage also became difficult.

This leads to the familiar graph used to explain the difference between machine learning and deep learning. More data means more resources from which a system could learn, but traditional algorithms struggled to turn that growth into continued performance gains. Deep learning helped drive progress in AI by using the additional data to exceed the ceiling of those earlier approaches.

This has been a rather long explanation because I wanted to cover the broad picture, but the relationship is essentially the one shown above. Within the large concept of AI, machine learning provides methods for implementing intelligent performance in software or hardware. Deep learning advances neural networks, one family of algorithms within machine learning.

Closing Thoughts

I will end this post after this brief introduction to artificial intelligence. CS231n was one of the courses I took when I first began studying AI, and I still recommend it as a very standard introductory course. I am not entirely sure, however, that it is truly the right first introduction for everyone. One thing I keep learning is that this field has no single correct route. I seem to gain something only by putting in the work myself. This suddenly sounds like a diary entry, but that is enough for today.