Комплексного готового решения не нашёл. Есть несколько утилит, которые могут ограничивать ЦПУ (приоритет, привязка к ядру) и память.
Process Lasso
Process Lasso is NOT yet another task manager. It is a process priority optimization and system automation utility. Priority optimization, affinity optimization, core optimization, automated rules, automated power profiles; you name it, and Process Lasso does it!
Process Tamer
Process Tamer is a tiny (140k) and super efficient utility for Microsoft Windows XP/2K/NT/Vista/Win7 that runs in your system tray and constantly monitors the cpu usage of other processes. When it sees a process that is overloading your cpu, it reduces the priority of that process temporarily, until its cpu usage returns to a reasonable level.
Minimem
Minimem now can reduce the memory footprint of any application. It runs in the background and can be set up to optimize as many proceses as you'd like, using a very intuitive graphical interface. Unlike other memory optimization products, it gives you full control over what applications you want to optimize, when and how often, as well as the possibility to only optimize memory when required and relevant.