Как донастроить Gitlab container registry с внешним nginx и https?

Question

[emoxam]

Всем привет!
Помогите избавится от ошибки

Error response from daemon: Get “https://cr.domain.com:5050/v2/”: Get “/jwt/auth?account=root&client_id=docker&offline_token=true&service=container_registry”: unsupported protocol scheme “”

Я поменял все домены на фейковые.
У меня есть роутер, где я пробрасываю порты 80, 443 и 5050 на nginx revers proxy (172.16.10.150).
Nginx reverse proxy (172.16.10.150) со сл. конфигами.

reg.domain.com.conf

server {
listen 80;
server_name reg.domain.com;
return 301 https://$server_name$request_uri;
}

server {
server_name reg.domain.com;

location / {
    proxy_set_header X-Forwarded-Host $host;
    proxy_set_header X-Forwarded-Server $host;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_connect_timeout 10000;
proxy_send_timeout 10000;
proxy_read_timeout 10000;
send_timeout 10000;
client_max_body_size 1000m;
    proxy_pass  http://172.16.10.157;
}


listen 443 ssl; # managed by Certbot
ssl_certificate /etc/letsencrypt/live/reg.domain.com/fullchain.pem; # managed by Certbot
ssl_certificate_key /etc/letsencrypt/live/reg.domain.com/privkey.pem; # managed by Certbot
 ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
}

cr.domain.com.conf

server {
server_name cr.domain.com;

location / {
    proxy_set_header X-Forwarded-Host $host;
    proxy_set_header X-Forwarded-Server $host;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_connect_timeout 10000;
proxy_send_timeout 10000;
proxy_read_timeout 10000;
send_timeout 10000;
client_max_body_size 1000m;
proxy_redirect          off;
proxy_set_header        X-Forwarded-Proto https;
proxy_set_header        Host              $http_host;
proxy_set_header        X-Real-IP         $remote_addr;
proxy_set_header        X-Forwarded-Ssl   on;
proxy_set_header        X-Frame-Options   SAMEORIGIN;
proxy_cache off;
proxy_buffering off;
proxy_request_buffering off;
proxy_http_version 1.1;
    proxy_pass  http://172.16.10.157:5050;
}


listen 5050 ssl; # managed by Certbot
ssl_certificate /etc/letsencrypt/live/cr.domain.com/fullchain.pem; # managed by Certbot
ssl_certificate_key /etc/letsencrypt/live/cr.domain.com/privkey.pem; # managed by Certbot
 ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
}

server {
listen 80;
server_name cr.domain.com;
return 301 https://$server_name$request_uri;
}

172.16.10.157 это VM с gitlab где я пытаюсь запустить container registry.

gitlab.rb

external_url = ‘https://reg.domain.com’
nginx[‘listen_port’] = 80
nginx[‘listen_https’] = false
nginx[‘proxy_protocol’] = true
nginx[‘real_ip_trusted_addresses’] = [ “127.0.0.0/8”, “172.16.10.150/32”]
nginx[‘proxy_set_headers’] = {
“X-Forwarded-Proto” => “http”,
“CUSTOM_HEADER” => “VALUE”
}

registry_external_url = ‘https://cr.domain.com’
gitlab_rails[‘registry_enabled’] = true
registry[‘enable’] = true
registry_nginx[‘enable’] = true
registry_nginx[‘proxy_set_headers’] = {
“Host” => “$http_host”,
“X-Real-IP” => “$remote_addr”,
“X-Forwarded-For” => “$proxy_add_x_forwarded_for”,
“X-Forwarded-Proto” => “https”,
“X-Forwarded-Ssl” => “on”
}
registry_nginx[‘listen_port’] = 5050
registry_nginx[‘listen_https’] = false

gitlab_rails[‘registry_enabled’] = true
gitlab_rails[‘registry_host’] = “cr.domain.com”
gitlab_rails[‘registry_port’] = “5050”
gitlab_rails[‘registry_path’] = “/var/opt/gitlab/gitlab-rails/shared/registry”
registry[‘registry_http_addr’] = “127.0.0.1:5000”

Как избавится от ошибки ?
Спасибо

Answer 1

[Александр Карабанов]

Добавь:

proxy_set_header X-Forwarded-Proto $scheme;

Comments

[emoxam]

Куда именно?

[Максим Федоров]

emoxam,

[emoxam]

Максим Федоров, Не помогло.
Я пытался это вписать и в /etc/gitlab/gitlab.rb
в виде "X-Forwarded-Proto" => "$scheme"
ошибка не меняется.
Я закомментировал большую часть конфигов - ошибка не меняется!

Вот та часть конфига nginx для хоста cr где происходят изменения

location / {
proxy_set_header X-Forwarded-Host $host;
proxy_set_header X-Forwarded-Server $host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_connect_timeout 10000;
proxy_send_timeout 10000;
proxy_read_timeout 10000;
send_timeout 10000;
client_max_body_size 1000m;
## proxy_redirect off;
# proxy_set_header X-Forwarded-Host $host;
# proxy_set_header X-Forwarded-Server $host;
# proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
## proxy_set_header X-Forwarded-Proto https;
## proxy_set_header Host $http_host;
## proxy_set_header X-Real-IP $remote_addr;
## proxy_set_header X-Forwarded-Ssl on;
# proxy_set_header X-Frame-Options SAMEORIGIN;
proxy_set_header X-Forwarded-Proto $scheme;
# proxy_headers_hash_max_size 512;
# proxy_headers_hash_bucket_size 128;
# proxy_cache off;
# proxy_buffering off;
# proxy_request_buffering off;
## proxy_http_version 1.1;
proxy_pass 172.16.10.157:5050;
}

/etc/gitlab/gitlab.rb сейчас выглядит вот так

external_url = 'https://reg.domain.com'
nginx['listen_port'] = 80
nginx['listen_https'] = false
#nginx['proxy_protocol'] = true
#nginx['real_ip_trusted_addresses'] = [ "127.0.0.0/8", "172.16.10.150/32"]
nginx['proxy_set_headers'] = {
# "X-Forwarded-Proto" => "$scheme",
# "X-Forwarded-Proto" => "http",
"CUSTOM_HEADER" => "VALUE"
}

registry_external_url = 'https://cr.domain.com:5050'
registry['enable'] = true
#registry_nginx['enable'] = true
#registry_nginx['proxy_set_headers'] = {
# "Host" => "$http_host",
# "X-Real-IP" => "$remote_addr",
# "X-Forwarded-For" => "$proxy_add_x_forwarded_for",
# "X-Forwarded-Proto" => "http",
# "X-Forwarded-Ssl" => "on",
# "X-Forwarded-Proto" => "$scheme"
# }
registry_nginx['listen_port'] = 5050
#registry_nginx['listen_https'] = false
### Settings used by GitLab application
gitlab_rails['registry_enabled'] = true
gitlab_rails['registry_host'] = "cr.domain.com"
gitlab_rails['registry_port'] = "5050"
gitlab_rails['registry_path'] = "/var/opt/gitlab/gitlab-rails/shared/registry"
#gitlab_rails['registry_api_url'] = "cr.domain.com:5000"
registry['registry_http_addr'] = "127.0.0.1:5000"

#nginx['redirect_http_to_https'] = true
#registry_nginx['redirect_http_to_https'] = true
#mattermost_nginx['redirect_http_to_https'] = true

Вот ошибка

Error response from daemon: Get https://cr.domain.com:5050/v2/: Get /jwt/auth?account=root&client_id=docker&offline_token=true&service=container_registry: unsupported protocol scheme ""

Вот такое в логах
tail -f /var/log/gitlab/nginx/gitlab_registry_access.log

172.16.10.150 - - [04/Sep/2022:07:16:07 +0000] "GET /v2/ HTTP/1.0" 401 87 "" "docker/20.10.7 go/go1.13.8 git-commit/20.10.7-0ubuntu5~18.04.3 kernel/4.15.0-112-generic os/linux arch/amd64 UpstreamClient(Docker-Client/20.10.7 \x5C(linux\x5C))" -

Не знаю полезен ли лог, может стоит повысить уровень логирования.
Проверяю уже снаружи, из германии, чтоб исключить влияние hairpin nat (или его отсутствия).
Спасибо

[emoxam]

Вероятно проблема была в этом.
В файле /var/opt/gitlab/registry/config.yml realm был указан как /jwt/auth
Эта строка формируется на основании конфига gitlab.rb
Судя по https://gitlab.com/gitlab-org/omnibus-gitlab/blob/...
На основании external_url должен заполнятся token_realm, о чем говорит строка
Gitlab['registry']['token_realm'] ||= Gitlab['external_url']
Но этого не происходит.
В дефолтном конфиге token_realm не встречается,
cat /etc/gitlab/gitlab.rb.backup | grep token_realm

Но он вскользь упоминается тут https://docs.gitlab.com/ee/administration/packages...
registry['token_realm']

Стало быть registry['token_realm'] = 'https://reg.domain.com'

И всё заработало.

Стало быть так решается ошибка unsupported protocol scheme “”.
Будем надеется я не поторопился.
P.S.
версия
gitlab-ee/now 15.3.2-ee.0 amd64 [installed,local]

[emoxam]

Итоговый конфиг выглядит так.

external_url = 'https://reg.domain.com'
nginx['listen_port'] = 80
nginx['listen_https'] = false
nginx['proxy_set_headers'] = {
  "Host" => "$http_host",
  "X-Real-IP" => "$remote_addr",
  "X-Forwarded-For" => "$proxy_add_x_forwarded_for",
  "X-Forwarded-Proto" => "https",
  "X-Forwarded-Ssl" => "on"
}

registry_external_url 'https://cr.domain.com'
gitlab_rails['registry_enabled'] = true
gitlab_rails['gitlab_host'] = 'reg.domain.com'
gitlab_rails['registry_host'] = 'cr.domain.com'
registry['token_realm'] = 'https://reg.domain.com'
#registry['registry_http_addr'] = "127.0.0.1:5000"
registry['enable'] = true
registry_nginx['enable'] = true
registry_nginx['proxy_set_headers'] = {
  "Host" => "$http_host",
  "X-Real-IP" => "$remote_addr",
  "X-Forwarded-For" => "$proxy_add_x_forwarded_for",
  "X-Forwarded-Proto" => "https",
  "X-Forwarded-Ssl" => "on"
}
registry_nginx['listen_port'] = 5050
registry_nginx['listen_https'] = false

[emoxam]

cr.domain.com.conf

server {
    server_name cr.domain.com;

    location / {
        proxy_set_header X-Forwarded-Host $host;
        proxy_set_header X-Forwarded-Server $host;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    proxy_connect_timeout 10000;
    proxy_send_timeout 10000;
    proxy_read_timeout 10000;
    send_timeout 10000;
    client_max_body_size 1000m;

        proxy_pass  http://172.16.10.157:5050;
    }


    listen 443 ssl; # managed by Certbot
    ssl_certificate /etc/letsencrypt/live/cr.domain.com/fullchain.pem; # managed by Certbot
    ssl_certificate_key /etc/letsencrypt/live/cr.domain.com/privkey.pem; # managed by Certbot
    ssl_protocols TLSv1 TLSv1.1 TLSv1.2;

}

server {
    listen 80;
    server_name cr.domain.com;
    return  301 https://$server_name$request_uri;
}

reg.domain.com.conf

server {
    listen 80;
    server_name reg.domain.com;
    return  301 https://$server_name$request_uri;
}

server {
    server_name reg.domain.com;

    location / {
        proxy_set_header X-Forwarded-Host $host;
        proxy_set_header X-Forwarded-Server $host;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_connect_timeout 10000;
    proxy_send_timeout 10000;
    proxy_read_timeout 10000;
    send_timeout 10000;
    client_max_body_size 1000m;
        proxy_pass  http://172.16.10.157;
    }


    listen 443 ssl; # managed by Certbot
    ssl_certificate /etc/letsencrypt/live/reg.domain.com/fullchain.pem; # managed by Certbot
    ssl_certificate_key /etc/letsencrypt/live/reg.domain.com/privkey.pem; # managed by Certbot
     ssl_protocols TLSv1 TLSv1.1 TLSv1.2;

}