CPU upgrade: AMD Ryzen on an AM4 socket

First question: any cpu that uses AM4 socket should be fine but you probably need to update bios for newer cpu, and you stuck with pcie gen3

2nd question: it uses the same socket should be compatible.

3rd question: if this is for gaming, I'd go with 5700x3d or 5800x3d

I did something similar a year or so ago. I know from prior experience that as a dead slot/socket standard gets older, its top-end CPUs get more expensive, not cheaper, so I upgrade to a Ryzen 9 5950X while it was still cheap enough to justify. That way, I know it'll keep me going for as long as possible before I need to upgrade my motherboard.

First question: Check your motherboard BIOS/UEFI against the current version. You might need to update it, if it's old enough that it doesn't support the new CPU you end out buying. As long as the manufacturer didn't drop support for the board prematurely, there should be an update that supports all AM4 CPUs (Asus is pretty good about that, so hopefully you're in good shape there).

Second question: It probably depends on your cooling setup. If yours is marginal, and only really suitable for managing heat on a 65W CPU, you might need to upgrade to a cooling setup that's suitable for the 105W TDP. I didn't have to upgrade mine, but when I first built this system, I put in a decent (not great) closed-loop liquid CPU cooler. If you're using AMD's OEM fan on the old 65W CPU, I'd be hesitant to use it as-is on a 105W CPU. If your computer is already shutting down when compressing movies, it sounds like your cooling is already inadequate. Nearly any upgrade (CPU or GPU) is going to make that problem worse unless you address your cooling problems.

Third question: You've already looked at benchmarks that will help answer this question. My old CPU and new CPU were both different from what you're looking at, but I can say that I saw a significant improvement when I upgraded from a 3rd to 5th generation CPU. I went from a 3900X to a 5950X. As I said in my intro paragraph, my personal feeling is that if you're going to upgrade CPU on a discontinued socket, it's worth getting the best CPU you can to maximize the life of that motherboard. It's a little more money, but if it gets you an extra few years before you need to upgrade your motherboard, it might be worth it, depending on your financial situation. Whatever you go with now, by the time you need to upgrade again, it won't make any sense to stick with AM4, so your choice now will determine how soon you need a new motherboard.

Second: are fans compatible between generations?

CPU coolers are mounted to the motherboard, and their specifics depend on the socket only. So technically you can keep it...

I can't compress movies without the computer shutting down due to overheating.

... however probably you shouldn't. Thermal shutdown is not normal, if the cooler is inadequate for your current CPU already, it certainly won't be enough for a more powerful one.

Oh, I type slow as it seems ¯\_(ツ)_/¯

First question: I'd say the first thing to look is the motherboard website. It will inform you of the CPU compatibility list (there shouldn't be any problem if it's the same AM4 socket but it never hurt to check) and you can determine which upgrades of BIOS (sometimes, you have to upgrade it several times) you should do for your new CPU.
Second: It should be compatible but if you say the new CPU has higher TDP, maybe you'll need a better fan or your PC will heat too fast.
Third: Too subjective, it depends of your own exigences and money you want to spend. CPU benchmarks are useful but you can also directly check some videos. It's easier for my shallow knowledge ;)

One last thing to check is your power supply
Well,I'm probably writing the same things as other... ^^

Edit: check if the offer for the AMD Ryzen 7 5800XT CPU come with a fan, it should be enough to do the job.

The higher the resolution you use the less important the CPU. I tested what I just told you myself with a 5700X and RTX 4090. In 1440p and above nearly always the GPU is the limiting factor.
I'd get a used mainboard with 5XX chipset as they are cheap and a more powerful GPU. Just now there are attractive used offers for e.g. AMD 6800 or Nvidia 3080 between 300-400 USD (or your currency equivalent) available. If this is over budget get a 3070 or 6750XT.

My cooling problems come that the PC has 5 years and I've never dusted off the insides. I bet I can lower it 20 celsius just by cleaning. I used to do it every year once, but somehow after I bought this one I simply let it go.

I think I might be better upgrading the GPU than the CPU. Gaming is my 1st usage ATM. So, a RTX 3080 is out of question, the prices here are so high. But I can get a refurbished 3070 for 300 to 400 bucks. I play at 4k whenever I can (Assetto Corsa, Forza Horizon 4, ETS2/ATS Truck Simulator, etc.).

However looking at comparisons makes me think that paying +300 bucks for a 35% improvement is again a waste: https://technical.city/en/video/GeForce-RTX-2060-Super-vs-GeForce-RTX-3070

A 3080 would be an improvement of +50% but it's 550 bucks (refurbished!) for me and that's too much: https://technical.city/en/video/GeForce-RTX-2060-Super-vs-GeForce-RTX-3080

You should see how fast the price increases for other brands, like the average is around one thousand and Ti models getting to €2,000.

So, right now I can't get better items, thus any upgrade has been aborted.

Thanks everyone for your help and suggestions.

For gaming, there isn't much of a reason to swap 3700x for anything else while staying on RTX 2060. Otherwise, 5700x3d / 5800x3d is way to go for a little bit of longevity, problem is, one is already overpriced and other unavailable. And that is, if you can get them cheaper than Ryzen 7700x (which outclasses them, but incompatible with AM4) and also plan to upgrade GPU in the perf. range of let's say up to RTX 5070 in the nearest future. In any other case, save up for a whole new platform at this point.

If your goal is to upgrade the PC for gaming purposes your best option is a 5800X3D CPU. Nothing else on AM4 can touch it when it comes to gaming performance. The higher core CPU's (like 5900X, 5950X, etc.) are actually slower for games due to splitting the cores across multiple CCD's and having less cache memory per-CCD, and should only be used if you have specific apps that are designed around high core counts.