It would be interesting to know:
1. What kind of storage the people who say it's "fine" are running their system on Mechanical 5.4k/7.2k HDD, Fusion, SATA or NVMe SSD), and
2. What exactly does "fine" mean? For me, fine is like a 3-4 second load time on Microsoft Word... off of a 5,400 RPM Mechanical drive...
That's what I get on a Mid-2013 model AMD Laptop with a 5,400 RPM drive running Windows 10; so, not a high benchmark.
My Late 2013 iMac with 5,400 RPM drive boots about 4-5x slower than the older Windows Laptop. Loads applications 2-3x slower as well. Even iTunes loads much faster on that weaker Windows Machine, off of that slow HDD, than iTunes on my iMac.
The difference is stark.
It has to do with Mechanical HDD de-optimization. Performance degrades on HDD Macs, which is why Apple has been aggressive with SSDs and Hybrid Storage systems. This is not an issue with Windows systems, because:
1. Windows Disk Optimizer Exists
2. Windows Schedules Drives to Be Optimized, Automatically, Weekly
3. The OS will Automatically Defragment, Relocate (Optimize File Order), and Consolidate (Remove Gaps) when it runs this in the background (when the PC is idle)
So, HDDs in Windows machines (barring Hardware issues) continue to run at near optimal performance levels - but not so for Macs. This process turns application loading and booting into predominantly sequential read operations, since it will keep directory structures in largely contiguous, defragmented blocks on the HDD. This is optimal for mechanical HDDs, which perform best when they can read and write data sequentially.
macOS will perform much better on an SSD, because SSDs avoid the issues it has on mechanical drives with amazing Random R/W performance and very low latency... (cause no R/W head to move).
If the OP is running the Mac mini off of a HDD, then he should upgrade it to an SSD (if Possible). If it has 4GB RAM, definitely upgrade to 8. 4 is not enough, IMO. OSes like macOS and Windows 10 are designed for 8GB+ systems. The OS uses 2.5-3GB of RAM after a Fresh Boot, so you're at the mercy of the SWAP file if you have less - which means your performance is going to suffer on a HDD system due to the storage medium the SWAP file resides on being quite slow.
Never buy a Mac with a HDD of any type (Fusion or otherwise - avoid them).
Buy with a smaller SSD (i.e. 256GB) and just augment with external bulk storage, which is cheap. On smaller SSDs, you can save a lot of room by simply deleting apps like Pages, Numbers, Keynote, GarageBand, and iMovie - depending on what you use (someone with Office 2016 doesn't need any iWork apps, for example).
I don't think a Mac is worth it [at all], if you cannot afford to get it with All-SSD Storage, personally.