>>Is there a way to change max memory in the Registry?
Changing memory settings using sp_configure does not require to reboot windows so I am assuming it is not stored in windows registry.
So answer is you **cannot**. That particular settings itself is probably stored in one of the system tables inside SQL server.
In this case a plain restart was not enough - SQL Server died because out of memory directly. I had to use -f to save the show.
In this case, we could add -f to SQL Server startup parameters which allows SQL Server to start with minimum configurations therefore uses least memory. After SQL Server starts, we could connect to SQL Server using SQL Server Management Studio and update the "max server memory" to an appropriate larger value.