I'm a sysadmin at a decently sized European university. While our large faculty server is stuck with ESXi for the near future (which I'm hoping to change), our smaller servers all run PVE. This includes our storage/backup/mass-deployment server, a cluster of several machines that deal with high compute power for AI and machine learning, a cluster that runs *several hundred* concurrent VMs for networking and white hat hacking exercises, and smaller roles too numerous to mention. I've never had a single "I wish PVE had x..." moment. The only thing that's a bit difficult to work with is the user and permission management, but it's not a deal-breaker. As for LXCs - I'd rather have them than not. My home server has very little memory, and having my services run inside Debian or Alpine LXCs that each consume ~20 megabytes of RAM is pretty baller.

regarding to the cpu types, if you choose "x86-64-v2-AES" you can live migrate VMs between Intel and AMD physical CPUs. :)

What proxmox devs are doing is little different, it is a place where all the opensource virtualization technology meets. Yes, it might not be general user intiuitive, but it provides well on core virtualization. And as a linux nerd I love it.

Life after VMware 🤣 perfect playlist for me

LXC containers are one of the main reasons I use Proxmox. They are extremely lightweight and fulfill all my requirements in my homelab. Compared to traditional VMs (Virtual Machines), their resource footprint is significantly lighter. I have only 16 threads available on my CPU, which means I need to be particularly cautious with thread reservation. LXC containers, on the other hand, offer great flexibility in terms of resource usage. I really love Proxmox and plan to continue using it in the future.

Re proxmox management from the console: The gui available via the web browser is only a think overlay over the command line tools, available from the command prompt. All VM and container related functionality is available from the CLI, and in some specialized cases, use of the CLI is still required to complete the task. The big advantage of the GUI is that it is much easier to use. The "short" version of the help text listing only available command line options for "qm" (the VM management tool) is several hundred lines long, spanning many pages. Super intimidating for a beginner, but a life-saver for an advanced user, who wants to automate routine tasks.

8:05 "all management needs to be done via web gui" completely incorrect. Proxmox makes virtually everything and more available via the command line. "pct" program and "pve" prefixed commands, tons of them. You can enter hosts via the "pct enter hostID" etc in /etc/pve/ all machines are available for config edit as well

I'm still on the fence as to which hypervisor to migrate to, so I appreciate the videos. However, I wanted to say that I appreciate the presentation style of these comparison videos. I've watched both, and they are fantastic examples of how to do an X vs Y comparison. Nice work!

I can understand the the immediate thought of Docker and kubernetes when it comes to containers, but remember that the container support in Proxmox actually started with support for OpenVZ containers, and moved to LXC even before docker even existed. Today I'm not sure that LXC makes that much sense anymore, but go back 10 years, and LXC provided significantly reduced overhead in contrast to a full VM. I've been using Proxmox since version 1.0, and with 3GB of ram on a server, shaving the memory usage by a few 100mb actually made a difference. As for the upgrade, it's actually not that hard to do. In fact I find it more unnerving to upgrade a vcenter, especially if NSX is involved.... Keeping my hands off the hypervisor and don't changing anything I've never had any issues upgrading a Proxmox host. That's not something I can say about upgrades of ESXi hosts...

I would love to see Nutanix

Good video, fair comparison, and good points, especially around the major version upgrades, such as going from 7 to 8.

Hi, it's important to emphasize that Proxmox leverages KVM for VM functionality. Great video, keep on home-labing!

thank for the video. As a linux nerd I say yes to a full featured gui with all options but a fallback to a shell for scripting helpfull tools is a great thing. Converting VM to proxmox works with simple scripts but every Vm should be testet after a transfer.

LXC have a lot of uses. It's basically for services that would best be run on bare metal, but you don't want to pollute host OS with it (samba server, nfs, ..)

Good video my dude. I would like to see a follow up video on clustering, vim networking, vm storage, virtual disks, migration, fault tolerence and "DRS".

Thank you for making these videos! My organization was just talking about this very thing, and this will help me explain options to my management team.

I haven´t made any experience on EXI because as a non professional user I don´t want to pay monthly for a Virtualization. I am running a mac mini but sometimes I have to use a Windows machine and I love Linux. Parallels for mac is so expensive and so a bought a mini PC for about 200 € and can now virtualize Windows and several Linux on that machine by using proxmox. Although I am 71 years old I was able to manage all this because there is a great community supporting proxmox users. And you will find big help on UKposts also. I think proxmox is a good software and free to use.

would love comparisons with Proxmox and XPG-ng vs Virtual Center. Our VMware bill went from $132k/year to 1.225 mil/year and we need to find some alternative.

As you just stated, the reason Proxmox CLI is minimal is the fact the CLI is debian with KVM enabled. I have Ubunutu servers with KVM installed making them TYPE1 hyperivosors and now I just manager them with their gui tools like Cockpit or Virt-Manager. yes I can get in to the hosts via ssh but I don't run additional services on the local host at this point and just manage VM's runing said services. KVM was designed to be open source and full customizable to your liking.

Should have really talked more about the ability to GPU and other pass through capabilities as well without paying an arm an a leg for Nvidia licensing. It's really nice for Plex or Jellyfin transcoding and running cheaper LLM/AI solutions. Also the Proxmox backup solution is truly amazing and probably needs to be talked about more. It doesn't just give you VM backups but it also does file backups as well. For me this was such a better and easier solution than going with Commvault or Veeam.

LXC is really nice to have, I wouldn't complain about it being there. If you need Docker or k8s, throw a VM with them in there .

I moved from Esx to HyperV then to Proxmox a few years ago. VM migration was a nightmare. However I’ve not regretted moving to Proxmox. It works well. Particularly good when you set up a Ceph cluster. There are easy ways to disable the nagging. I have a script that removes the nagging that executes on every boot. Not seen the nag in a very long time. The LXC containers don’t live migrate but they do migrate. They just shut down and start again on the destination host. They’re very useful for lightweight tasks such as running ddclient, letsencrypt, cloudflared etc. Generally, I’m glad I moved to Proxmox. Free HA, good for Linux and Windows VMs, an acceptable built-in VM backup system, Ceph for cluster storage. What’s not to like?

Not sure if this has been mentioned but you have multiple CLIs availabla: qm, pct, pvecm, etc, to launch, stop, enter, reboot, etc., vms, lcxs and managing the cluster. Also all the networking and virtual bridges (switches) can be managed from the linux terminal.

I started using proxmox last year and I have to say that it is clearly very good I agree concerning the anoying repository I must also add that to have the possibility to make snapshots of your vm you have to pay attention to the dis format of your VMs however up to now it is working like a charm

This was a great overview, I'll have to move my lab soon, I'd love to see some guides on moving (exporting) VM's and other config from esxi and importing into Proxmox where possible

I believe the reason Proxmox has the VMware features, the pvscsi, vmxnet3, etc is because if installed Vmware Player on a Linux host, it has to compile kernel modules. So they may have the modules precompiled and ready, not to mention there is Open-VM-Tools, which is an open source version of VMware Guest Additions for Linux. When the Open VM Tools became standard, VMware pretty much dropped support for their own proprietary guest additions for linux, especially for newer kernels. Maybe having these for guests is for compatibility should you wish to migrate from ESXi or any other VMware product.

I migrated ESXi VMs to Proxmox using the OVFtool from VMware. It was quite easy and where are guides out there. Having used VMware (ESX) since it was running on top of Linux (ya, I'm old) professionally and personally, I've been quite happy with the switch to Proxmox. I switched when ESXi 7 came out and made it harder to run on the same hardware as 6.5 did. Learn Linux. You'll be glad you did.

since it is debian based, couldn't you install any GUI of choice to interface with the host on the same machine?

I'm sure you will hear linux ops folks talking about how great the cmd line is for admin'ing... and I don't disagree... IF you're doing it frequently enough to remember how to do everything. For the rest of us who don't admin linux machines, we waste a LOT of time looking up commands and that goes away with a basic (even text based like with esxi) ui. I just started playing with Proxmox as a potential migration path from vsphere for my roughly 500 core cluster, the new vGPU non-prod release gives me hope that by the time I need to migrate there will be production gpu support which would make proxmox fully capable of replacing my vsphere e+ environment. Good overview but again I'd like a deeper dive in to advanced features like backup... upgrade workflow, software defined network and so on. oh and yea I DID assume containers meant app containers too.

The VM CPU type will be for clusters of hosts that have different CPUs. To maintain the ability to migrate you need the cpu features to be the same. in VMware this is the EVC settings.

I´m not exactly sure how user an group management is in esxi, but have you noticed that you can also add other realms in proxmox like from LDAP/AD or even OIDC?

Thank you for this and your other companions. How about a comparison with Hyper-V? My company is thinking of going with it as we have server licenses

Great video! I'd love to see a comparison of xcp-ng and proxmox now!

One thing I noticed in your XCP-ng comparison was the mention of the limitation of VM pausing during snapshot creation if memory is included in the snapshot. However I don't remember hearing about Proxmox snapshots in this video. Does it have the same VM pausing issue as XCP-ng for memory-inclusive snapshots? It looks like the snapshot capability is dependent on a number of factors such as the virtual disk format or the file system used to hold VM images (so you need to use QCow virtual disk files if your VM storage is networked using NFS or Samba), and that would have been good to mention.

I’ve been hearing about Scale Computing - do you know anything there? Would love to see a breakdown

I totally agree! If proxmox made the GUI a little easier, made the upgrade more seamless, and made it possible to migrate from vmware to proxmox smoothly, it would be a no brainer. But these points remains and a couple more you mentioned in the Video. Really hope to see these changes! Thanks for the video, good summary!

are you gonna make a video about harvester? it's a hypervisor from and meant to be integrated with suse's rancher platform

I just wiped my drives last night from Vmware ESXI 7u03 after using it since 2022. I ended up going with proxmox for now because due to having way more support in terms of DevOps tools such as Ansible and Terraform as well as installing nested hypervisors. I tested xcp-ng but the support base is too small esp for DevOps and API automation stuff at the moment. Also the RAM usage is slightly higher.

Since when can Proxmox automatically shift VMs to other hosts in the cluster to balance out workloads? It has HA so that in the event a host goes down, the VMs should power up on a host that's online.

Proxmox LXC system containers I find super useful in a homelab situation, for a production environment I use Incus (LXD fork by LXD devs).

Hey man, i have to say your X-CPNG and ProxMox vs ESXi has been great and super informative. I appreciate you taking the time to do this.. I run ESXi 6.7 in home lab and, given the state of the company going forward, i am looking to change that. Have you done any videos on running ProxMox in mini PC's? I know it is pretty compatible on desktops and would run fine on my R630's but looking to downsize the power usage footprint a bit. have you done video's on Proxmox Backup Server?? Wondering if that is a worthy addition to the PVE environment ... Either way, great video and i am looking forward to catching some more of your content - Liked & Sub'ed!! keep up the great work!

Great videos! Thank you. 😊

one of the critical feature linux based hypervisor (kvm) lack is storage offload API like VAAI

thanks for the comparison, i noticed that with proxmox unlike esxi, you can't install a promox host to a proxmox cluster if that host has vm's on it.. even if they are shutdown.. the promox most MUST be clean.

Vsphere is also available as a standard version, includes vsphere std hypervisor and vcenter std. Also essentials and essential plus are available. These will suite for SMB and will defenetily cost no more than before Broadcom. Only difference is that lowest support is now production support which will of course be more expensive than basic.

Great comparison. Thank you.

Wonder if the CPU type selection when creating a VM has to do with something similar to EVC on VMWare, where the VM is limited to a specific instruction set regardless of physical CPU, so you can move VM's around between hardware from different generations, so that if you set the CPU type to the lowest common denominator in a cluster of machines, it shouldn't have any issue moving a vm between those hosts.

I’m interested about similar comparison between Nutanix, Scale Computing, Oracle VM, Red Hat alternatives with RHEV and Openstack VM, Microsoft solutions for onprem VMs and containers.

For a Homelab i like having lxc. Easy to spin up a quick and dirty OS for testing something. I have started using more Ubuntu minimal cloud VMs which use very little resources as well.

And killed the VMWare nonprofit program. Great video, thanks for putting together

Yeah baby! The beard is back! That is good VM omen

The gui complaints doesn't really seem fair considering it's the full cluster stack compared to vcenter which is not free. Vcenter itself is very busy also and sometimes hard to find what you're looking for. And what a strange complaint that you can manage the host file directly, I mean I'd rather be able to than not. Native docker support is on the road map but don't discount the LXC support, they do have their place and can be very stable. Migration between hosts is literally seconds, they boot up incredibly fast and are very portable. Think application stack where you have 2 or more app servers that are load balanced.. you can kill an lxc container and reboot it in like 2 seconds, compared to having to reboot a vm that could take minutes.

Hi. It is possible to do a M.2 IOPS test on a virtual machine on proxmox? if you have a performant one? I used ISXi paid version and with a 980 pro i only got about 600k IOPS, and with hyper-v i got 990l IOPS; which is the very reason a switched from ESXi7 to hyper-v. I am wondering if proxmox allow full speed on m.2 ssds, i would love to give it a try, but i don't have any server available. Otherwise, from your video i see that proxmox is way better than ESXi considering that is free. I am enclined to believe that it might be a better solution than hyper-v also, since is a type1 hypervisor and not 2. Also, it is possible to have a folder sharred between multiple windows virtual servers? like to make them a D to share that folder? And a shared one for linux servers? I believe a folder shared between different OS types its impossible. I hope i get an answer from you. Thank you.

What about openstack / Ubuntu micro stack?

Any news on what they plan to do with VM Workstation Pro? I use Workstation Pro for testing and for keeping older OS's available so I can offer support for older OS's I have over 100 OS's from desktop to server to Appliances like pfsense Any word on those licenses or some sort of alternative?

Dayum, seems like I will have to move back to Proxmox due to ESXi not being free anymore.

I’d love to see a harvester and hyper-v videos

If you have not tested running multiple Proxmox nodes in a cluster with ZFS snapshot replication running between them, then you have not seen a huge part of the usefulness of Proxmox for small environments. Using that simple setup, you get the ability of almost instant HA or manual failover of VMs/LXCs between Proxmox nodes without needing any shared storage required. This is huge for small business or home users, to be able to have this VM migration capability without requiring any san or external storage.

I managed Esxi for years at my company (5 hosts, 35 VM) last summer we moved to Proxmox: 100 % satisfied, just a few small issues (solved). The only problem I still have is when I want to download and deploy a virtual appliance but it is available only to vmware. It works on proxmox after conversion but it's not ideal.

Is someone has some experience with proxmox in a cluster and storage? I need live migration a d snapshot and dont want ceph. Iscsi was a pain in the ass so far😅

The main question is, is PVE reliable, i mean rock solid so you would run your essential SAP Environment on it, 24/7 ?

Just spent the last year of my life migrating from nutanix ahv to vmware. Looks like im gonna be spending the next year moving elsewhere. Really wish i would have purchased 5 years of support to push this out

I'm using Proxmox for 3 years now. The switch from VMWare was extremely easy and I dindn't need to use the documentation ever. They are very similar. But i also manage a lot of headless Unix(FreeBSD and AIX) and Linux servers. So the command line is pretty easy for me.

You can use those repos, because they exist and are needed for pve updates. Instead of disabling, you need to change the apt sources with the community repo.

What about solusvm from webpros?

I’d be very interested in a comparison aimed at enterprise usage. Which doesn’t necessarily mean large deployments, but in terms of support, features and life expectancy - nobody wants to migrate to a platform that’s not financially backed and may stop being maintained at any time. Nutanix comes to mind, and RHE virtualisation for example.

Super well done video.

That was thorough

LXC containers are a lot more light weight than full vm's, they start up in ms and use less than 100 mb of ram. If your usecase does not require extreme security then they're great.

I completely hear the point on the benefit of a menu on the console. However, more and more system admins are so reliant on menus and GUIs, that they are not taking the time to learn the CLI. I have been in technology for 25 years and some of the best platforms that I have supported have been on Linux. Getting to "know" the OS and how to maneuver within it is a necessity that the next generation of administrators need to embrace. Just my 2 pennies... :)

At 8:03 you say all management meeds to be done frol the gui. I'm a novice proxmox user and my feeling generally is that sometimes in proxmox you have to use cli. That said I believe you _can_ do everything in cli that you can from the gui. And there's a lot of documentation around it. I agree a wizard for basic setup would make certain things easier though

I've been running a PVE cluster for a number of years now. I like it, and I'm not planning to move away from it, though I'm also playing with xcp-ng just a bit too. But I think a number of the comments here trying to defend PVE kind of miss the point: * Sure, a system admin should be able to configure a network interface from the shell (and I've had to do it on a variety of occasions). But a console menu like xcp-ng's would still be a welcome addition. And yes, you could get around this by just installing a full graphical desktop environment onto it, but that really isn't the way the product is intended to be used. * Yes, the subscription nag screen is obnoxious. It's easy enough to disable, but still obnoxious. I get it, devs gotta eat, but... * Agree on the enterprise repo being enabled by default. Like with the nag screen, it's easy enough to fix--easier now that repository management is included in the GUI--but it doesn't make sense as a default.

When I was reviewing Proxmox VE, I really wanted it to succeed, but I couldn't get promiscuous mode working for network packet capturing for security onion.

I’d love a video on harvester.

Does Proxmox have the stringent CPU requirements as ESXI does?

Great video. Btw, LXC live migration is coming. Docker comes from LXC. LXC (and the previous "original" container technology OpenVZ was around before Docker was a concept

Containers on linux has always meant LXC. LXC are terribly useful for use cases where you want containerized solution but need more access to hardware. Also as you mentioned it helps tamp down the ram required too. Don't disagree about full virtualized environment provides more security benefits, but for many use cases LXC is the perfect compromise. Plus most modern applications don't need the container itself to he HA to migration is not normally required. One last thing for LXC is with zfs copy on write file system you can spawn new containers off an existing one and not only does it use minimal RAM it uses minimal storage.

thank you for the video, for sure a straight migration path from VMware to Proxmox is something that has been needed for a while. also, LXC container is targeting the home lab I think. as some people have very lightweight workloads.

18:40 what about ESXi's EVC?

nope - no hell from me - i totally agree with you on the container thing...i kinda wish it had docker support out of the box so i wouldnt need to spool up a VM to run docker (or run on pve itself which aint right)

The repo nag is the price of free, no big deal.

Welcome to OpenShift Virt. Not a replacement, but a good alternative.

Proxmox FTW!!!

You know that you can do basically anything in pvr from the console via command line once you're logged in? If you want you can avoid the gui entirely and perform everything the gui can plus stuff the gui cannot do in the terminal? Yeah yeah you know it. But esxi loses. You can't do anything in esxi there even if you want. Let's say network fails, you're lost. You can't even check vm status... Sorry you're absolutely wrong

Well.. on the one hand you say you cannot expect everyone administering an Proxmox host to fix it using the command line, but on the other hand you say that editing the hosts file through the WebUI wasn't needed xD

As a big fan of infrastructure as code I must sy, I prefer XCP-NG the XenOrchestra Provider for terraform is just superior to the Proxmox one. At least last time I check the Proxmox one a couple years back it was pretty bare bones and was lacking a lot of features.

More ZFS gui controls... i have been simulating failure scenarios and a zfs mirror single drive failure should be easy to replace the drive and resilver thru the gui, but instead i had to learn zpool commands, proxmox boottool efs commands. Why? Ok if it a boot drive mirror ill give some slack but if its just a vm drive mirror why! Im in love with proxmox. I have successfully made many achievements. I installed prox backup environment as a vm but i passed thru a dedicated drive that i can select at boot instead of pve and restore the entire pve os from backup. Also passed thru a tape changer/drive thru to the pvb. Again that pvb can use if pve crashed. There are also a few other things that have made vmware obsolete, even if they were still "free". Proxmox it is!!!

tbh the major upgrade from 6to7 and 7to8 was so painless and easy. just a small check-script and the go for it. no much hassle

Great video!

I'm a little shocked you did not mention that it's possible to import vmdk files from vmware into proxmox and basically migrate your vm's as-is into proxmox with one command in the terminal.

Should create, "Exploring Proxmox and TrueNAS & UnRaid from a VMware User's Perspective" as they're the top three hypervisors (outside VMWare). I feel like TrueNAS is the baby of Unraid and ProxMox xD

The PVE VM CPU host section is just telling PVE how to simulate/emulate the chosen CPU family's instruction set. Choosing "host" is the most performant, becase it's basically just a CPU passthrough from the host to the VM. This could cause problems with HA, though, if you have a cluster of say an x86-64 node and an ARM64 node. Choosing anything else than "host" will lower your performance because PVE (or rather QEMU) either has to simulate or emulate the chosen instruction set, depending on the host CPU's virtualization capabilities for the chosen CPU. I think your example of choosing an AMD Epyc CPU while the host is an Intel one, this would probably result in emulation, so very slow performance. Choosing a desktop type AMD CPU while the host is a desktop type Intel CPU, this could probably become simulation, which would be faster but still slower than "host". I'm not on expert on this specific topic though, so take it with a grain of salt.

my research so far from a enterprise solution perspective: xcp-ng: good but 2TB drive limit is a deal breaker. proxmox: good but no real enterprise support is a deal breaker. nutanix: in progress

I get it that it is a review from a vmware perspective, but you got the console part plain wrong. qm commands allow you to manage your vms qm list shows your vms qm start, stop destroy ... allows you to manage them For the lxc containers you use pct pct list , start stop create destroy mount ... You can fully manage your proxmox(cluster) from the commandline, try that with vmware

Can you do Incus? It looks very promising.

Where I work will probably just pay the cost difference and stay on VMWare; but I’m watching these type videos just in case. One issue I have is we have a set of host with Nvidia L40 cards in them for AutoDeskand other graphic intensive software. Linux is notoriously horrible or horrendously complicated with PCIe pass through. What I’d like to see covered in one of the type of videos is how these alternatives deal with hardware like video cards.

I’m going to talk only about my experience with proxmox and I have tried to used several times. I do not like it I think that there are better options. My setup is a 3 node i5 13th gen with ceph and 32gb ram two nics one for ceph traffic and the other for all other traffic. I think it’s very slow, there is a problem, in my opinion, when stopping the vm’s when they get stuck or you made a mistake in some way with the vm. The templates can only be cloned from one node and they are attached to that node different from VMware of course you can migrate the templates. Installing a Linux vm in the traditional way takes a long time like several hours something like 4 hours or more. The ceph speed on ssd was around 13mb/s. I made a test by moving all my 10vm from 3 to only 2 nodes to test on the third node the speeds. Maybe it’s me and I’m not used to this kind of solution because I was a VCP on 5.5 and 6 I normally prefer fedora KVM because of cockpit but that doesn’t provide any way to cluster 2/3 machines. In sum I got tired of it and installed harvester hci and now a vm is installed in 5m or a bit more, longhorn gives speeds around 80mb/s. This is just my last experience and the previous ones. I hope this helps someone. Thank you.

21:31 again with just few commands you can install whole desktop environment in proxmox, I have been using proxmox installed on my primary Laptop and with Debian 12 Desktop environment Best of Both world, greater OS and Greatest Virtualization solution in on pack

Great run-through, thank you! Any replacements for VMWare Workstation?! (agree with you on the container thing!)

Remember, the linux command line is very user friendly. It's just picky about its friends... ;)