Q&A on the topic of AI

There was an important and interesting discussion of AI models a few months back, when deepseek just came out.
And it looks like AI is quickly becoming more and more impactful on all our lives.
So I've decided to take some of my answers in that discussion, and create a more comprehensive Q&A, which I hope people will find interesting.

As I see it, the AI will revolutionize the world, as many other inventions did in the past:
The electricity, the car, the airplane, the computer, the printing press, etc.
But while other inventions initially only affected one or a few fields, it took decades or even centuries until their impact became global on many aspects of our lives.
With AI the progress is much faster, and all of us are either already feeling or will feel within a few years how it affects our lives, jobs, school, etc.

Questions & Answers:
Q: What is AI?
A: AI is Artificial Intelligence.
A computer run software, able to perform human tasks that require thinking, logic, creativity, etc. as well as or better than humans.

Q: How is AI built? What are Neural Nets / Neural Networks?
A: The basis of AI is a concept called "Neural Networks", which is taken from the way our brains work.
We can "teach" it by creating connections between the nodes.
For example, show a child enough pictures of a cat and say the word "cat" and they will start to associate the word with the image (a nueral link is created in the brain)
Similarly, if you show a neural network enough pictures of a cat and let it know it's called "cat", it will eventually will be able to "predict with a high probability" by looking at a picture, if there is a "cat" on it (a neural link will be created in the neural network).
The same principle can be used to train (teach) models not just on images, but also on audio, video, text, etc.
For more information see https://medium.com/@jereminuerofficial/neural-networks-for-dummies-bc1ed3f69027

Q: What is Machine Learning (ML)? What is Data Science (DS)?
A: Machines learning is the profession training various "automated computer models" or "machine learning models". We encompass these models by the general term of AI.
There are many many ways of building machine learning models.
The most common and well known is Neural Networks (explained above).
Data Science is the study subject of training machine learning models (among other things).
People who build/train ML models are usually called "Data Scientist" or "ML model engineer".

Q: How AI models create images/videos/songs/texts? What are GAN (Generative Adversarial Network)?
A: I explained above how Neural Networks are able to learn to "understand" visual, audio or othe input.
Generative Adversarial Network (GAN) is what enables Neural Networks to create output.
We basically pit 2 Neural Nets against each other.
For example, we want to create a Neural Net that can draw Picasso-like pictures.
We take a blank model that can draw, and tell it to draw pictures of random pixels.
We then take one model we trained on existing Picasso pictures, and can tell us a percentage (0% - 100%) how sure it is a picture it's seeing in Picasso.
Each time the first model draws a picture, the second model analyzes it, and gives it a percentage score of how close it is to Picasso. And gives it as feedback to the first model.
Each time the score becomes higher, the first model strengthens the neural link that was used for that image generation.
Each such iteration can take a split second. And as more and more iterations pass, the "correct" neural links are strengthened, and the first model is able to draw pictures that get higher scores from the second model (look more like Picasso).
After millions, or even billions of iterations, the first model will pass a benchmark defined by the ML engineer (for example: above 80% accuracy), which was determined to be "enough" for Picasso-like pictures.
If we run the first model until it reaches 100% accuracy - that would be bad. It's called "overfitting". It means every picture it will create will be a copy of an existing Picasso picture.
And that was not out goal.

Q: What are LLMs (Large Language Models)? Where do they come from?
A: LLMs (Large Language Models) were first created 2 years ago, and used to power the first iteration of ChatGPT.
While ML (Machine Learning) models have been with us for 10+ years. MMLs are relatively new.
The innovation with LLMs is - they showed us a product that's bigger than the sum of it's parts:
You start by taking an ML model that predicts your next word.
Think of writing a message on a cell phone, and it predicts your next word.
But it turns out that if you train it on a data set that is large enough, instead of being a system that is really really good at predicting your next word, but doesn't actually knows how to do anything beyond predicting your next word.
Instead you get a system that can actually write a book like it was Isaac Asimov.
Or discuss any topic you want with you.
Or write a computer program.
Or do the countless different things we use ChatGPT for today.
It's a living proof of the Infinite monkey theorem: a monkey hitting random keys for an infinite amount of time will eventually type out the complete works of William Shakespeare.
LLMs are what powers all of the LM models we call AI (Grok, Perplexity, Dall-E, ChatGPT, Deepseek, Llama, etc.)

Q: How are LLMs different from previous LM models?
A: The main difference of LLMs from previous ML models, is the fact that they are large.
Which is an understatement.
They are HUGE.
And each time they create a new model, they make it more and more HUGE. Like 10x 100x 1000x bigger every iteration.
Which caused several changes in the technological market:

1. Small companies can no longer compete and innovate in regards to AI.
   You cannot be the next Google or the next Meta in AI space.
   Because you need the resources of a behemoth like Google, Meta or Microsoft to create and train a modern LLM.
   Even Apple doesn't have it's own LLM yet. That's how difficult it has become to create/train one.

2. To train a single LLM model, you now need HUGE (HUMONGOUS) computing power.
   Literally thousands or tens of thousands powerful processors.
   And as it's more efficient to do on GPUs (Graphics Cards).
   (For more info see CPU vs. GPU for machine learning)
   Which is also the reason for the rise in the Nvidia stock in the last 2 years: Huge amounts of GPUs are needed to train the LLMs, and the bigger and more complex the LLMs, the bigger the number of GPUs needed for their training.
   And in addition to that, it takes HUGE amounts of electricity.
   To the point that the companies working on new LLM models, are now building their own nuclear power plants, to support their electricity needs, for training AI models.
   Which is why it costs Billions (with a B) of $$$ to develop new LLM models, and only huge & rich corporations can afford to do it.

Q: What is Deepseek? Why are they popular? How are they different from OpenAI?
A: Deepseek is a small start-up which (supposedly) was able to develop their own LLM models.
What was written above, one would presume that this would be impossible, as it would cost them Billions of $$$, which small start-ups don't have.
But they were able to train their model on a "minuscule" budget of only $6,000,000.
It seems they were able to do it, but using a lot of optimization, cutting corners (their model is slightly less accurate than the best ChatGPT model, but is infinitely cheaper) and by using existing models (ChatGPT, Grok, Llama) to participate in training the model.

Q: What is an "Open Source LLM"? How is it similar/different from an "Open Source Software"?
A: Open source software means getting access to the source code of the software.
For example, if software was a table.
Being "Open Source" means you get a full list of parts you would need to build it, and complete step-by-step instructions on how to build it yourself.
If you take the same components, and follow the instructions exactly - you end up with an identical table.
On the other hand, ML models are "black boxes" by definition.
Which means we can train one, but we don't really know or understand how it works inside.
We don't know why it makes one choice and not another - it's all based on the way the millions of neural nodes got linked inside it's brain.
It's like trying to predict what a person is thinking by looking at a CAT scan of their brain.
So "Open Source LLM" is like getting a smartphone with usage instructions.
You can take pictures with it, you can make calls, you can connect it to earphones.
But you can't actually build one yourself based on the instructions you received.
Providing "Open Source LLM" basically allows you to run it on your computer for free (instead of running it on the company's computer, and paying them for it).
But it doesn't provide any information on how it works, or how you can build one yourself.

Q: What are ChatGPT / Grok / Deepseek/ etc.? How are they built?
A: See answer to "What are LLMs (Large Language Models)?"

Feel free to ask additional questions, and I will add them to the Q&A.