>>Is there a way to change max memory in the Registry?
Changing memory settings using sp_configure does not require to reboot windows so I am assuming it is not stored in windows registry.
So answer is you **cannot**. That particular settings itself is probably stored in one of the system tables inside SQL server.
In this case a plain restart was not enough - SQL Server died because out of memory directly. I had to use -f to save the show.