Можно воспользоваться параметром CRF
ffmpeg -i 1.mp4 -c:v h264_nvenc -crf 30 -c:a libmp3lame -vf scale=-1:480 2.mp4
Больше число — ниже качество.
The range of the CRF scale is 0–51, where 0 is lossless, 23 is the default, and 51 is worst quality possible. A lower value generally leads to higher quality, and a subjectively sane range is 17–28. Consider 17 or 18 to be visually lossless or nearly so; it should look the same or nearly the same as the input but it isn't technically lossless.
Я лично использую
-c:v libx264
. Из-за того, что у вас другой энкодер, может не сработать.