After lots of fiddling I came to the conclusion that the biggest problem with Audirvana running on a Mac Mini is I/O, not CPU/RAM.
My 2012 4-core Mac Mini has a horribly slow internal hard drive.
I switched to an external thunderbolt drive but it still “choked” now and then, especially when upsampling to DSD128.
I also noticed that the metadata, search and album art would take forever to load.
I solved it with a 30$ 256gb external USB SSD from Amazon. The difference is night and day.
I keep my OS and Audirvana database on the SSD, and the tracks are on an external drive in RAID0.