Какую кодировку строк использовать в linux?

Question

[arseniylebedev]

Делаю авторизацию по логину и паролю в openvpn сервере, есть такой скрипт на проверку логина и пароля:

verify.py

#!/usr/bin/python3
# coding=utf-8

import sys
import os
import requests
import config

if config.show_time:
	import time
	
	start_time = time.perf_counter()
	

SUCCESS_AUTH = 0
ERROR_AUTH = 1

username = os.environ.get("username")
password = os.environ.get("password")
ip = os.environ.get("untrusted_ip")
port = os.environ.get("untrusted_port")

print(config.verify_log.format("Authorization attempt for username: " + username + " ip: " + ip + " port: " + port))

data = {
	"username": username,
	"password": password,
	"ip": ip,
	"port": port,
}

try:
	r = requests.post(url=config.verify_url, data=data)
except Exception as e:
	print(config.verify_error_log.format("Failed to complete server request: {0}".format(e)))
	sys.exit(ERROR_AUTH)


if r.status_code != 200:
	print(config.verify_error_log.format("Error status server code " + str(r.status_code) + " and should be 200"))
	sys.exit(ERROR_AUTH)
	
	
if config.show_time:
	stop_time = time.perf_counter()
	diff_time = str(config.toFixed((stop_time - start_time), 2))
	print(config.verify_log.format("Request time: " + diff_time + " seconds"))
	

try:
	r_json = r.json()
except Exception:
	print(config.verify_error_log.format("Server error returned not json"))
	sys.exit(ERROR_AUTH)


if r_json["ok"] == False:
	print(config.verify_log.format("Authorisation Error. Server message: " + r_json["message"]))
	sys.exit(ERROR_AUTH)
else:
	print(config.verify_log.format("Successful authorization username: " + username + " ip: " + ip + " port: " + port))
	sys.exit(SUCCESS_AUTH)

Если в параметрах логина или пароля передать кириллицу, то запрос на сервер не отправится, будет ошибка в этой строке r = requests.post(url=config.verify_url, data=data).
Ошибка "'utf-8' codec can't encode characters in position 30-31: surrogates not allowed"
В какую кодировку енкодить строку перед отправкой на сервер?

traceback

Dec 12 14:37:02 shadowvpn openvpn[4349]: Traceback (most recent call last):
Dec 12 14:37:02 shadowvpn openvpn[4349]:   File "/etc/openvpn/scripts/verify.py", line 34, in <module>
Dec 12 14:37:02 shadowvpn openvpn[4349]:     r = requests.post(url=config.verify_url, data=data)
Dec 12 14:37:02 shadowvpn openvpn[4349]:   File "/usr/lib/python3/dist-packages/requests/api.py", line 112, in post
Dec 12 14:37:02 shadowvpn openvpn[4349]:     return request('post', url, data=data, json=json, **kwargs)
Dec 12 14:37:02 shadowvpn openvpn[4349]:   File "/usr/lib/python3/dist-packages/requests/api.py", line 58, in request
Dec 12 14:37:02 shadowvpn openvpn[4349]:     return session.request(method=method, url=url, **kwargs)
Dec 12 14:37:02 shadowvpn openvpn[4349]:   File "/usr/lib/python3/dist-packages/requests/sessions.py", line 506, in request
Dec 12 14:37:02 shadowvpn openvpn[4349]:     prep = self.prepare_request(req)
Dec 12 14:37:02 shadowvpn openvpn[4349]:   File "/usr/lib/python3/dist-packages/requests/sessions.py", line 449, in prepare_request
Dec 12 14:37:02 shadowvpn openvpn[4349]:     hooks=merge_hooks(request.hooks, self.hooks),
Dec 12 14:37:02 shadowvpn openvpn[4349]:   File "/usr/lib/python3/dist-packages/requests/models.py", line 308, in prepare
Dec 12 14:37:02 shadowvpn openvpn[4349]:     self.prepare_body(data, files, json)
Dec 12 14:37:02 shadowvpn openvpn[4349]:   File "/usr/lib/python3/dist-packages/requests/models.py", line 499, in prepare_body
Dec 12 14:37:02 shadowvpn openvpn[4349]:     body = self._encode_params(data)
Dec 12 14:37:02 shadowvpn openvpn[4349]:   File "/usr/lib/python3/dist-packages/requests/models.py", line 104, in _encode_params
Dec 12 14:37:02 shadowvpn openvpn[4349]:     v.encode('utf-8') if isinstance(v, str) else v))
Dec 12 14:37:02 shadowvpn openvpn[4349]: UnicodeEncodeError: 'utf-8' codec can't encode characters in position 30-31: surrogates not allowed

Comments

[arseniylebedev]

Pavel Denisov,

traceback

Dec 12 14:37:02 shadowvpn openvpn[4349]: Traceback (most recent call last):
Dec 12 14:37:02 shadowvpn openvpn[4349]:   File "/etc/openvpn/scripts/verify.py", line 34, in <module>
Dec 12 14:37:02 shadowvpn openvpn[4349]:     r = requests.post(url=config.verify_url, data=data)
Dec 12 14:37:02 shadowvpn openvpn[4349]:   File "/usr/lib/python3/dist-packages/requests/api.py", line 112, in post
Dec 12 14:37:02 shadowvpn openvpn[4349]:     return request('post', url, data=data, json=json, **kwargs)
Dec 12 14:37:02 shadowvpn openvpn[4349]:   File "/usr/lib/python3/dist-packages/requests/api.py", line 58, in request
Dec 12 14:37:02 shadowvpn openvpn[4349]:     return session.request(method=method, url=url, **kwargs)
Dec 12 14:37:02 shadowvpn openvpn[4349]:   File "/usr/lib/python3/dist-packages/requests/sessions.py", line 506, in request
Dec 12 14:37:02 shadowvpn openvpn[4349]:     prep = self.prepare_request(req)
Dec 12 14:37:02 shadowvpn openvpn[4349]:   File "/usr/lib/python3/dist-packages/requests/sessions.py", line 449, in prepare_request
Dec 12 14:37:02 shadowvpn openvpn[4349]:     hooks=merge_hooks(request.hooks, self.hooks),
Dec 12 14:37:02 shadowvpn openvpn[4349]:   File "/usr/lib/python3/dist-packages/requests/models.py", line 308, in prepare
Dec 12 14:37:02 shadowvpn openvpn[4349]:     self.prepare_body(data, files, json)
Dec 12 14:37:02 shadowvpn openvpn[4349]:   File "/usr/lib/python3/dist-packages/requests/models.py", line 499, in prepare_body
Dec 12 14:37:02 shadowvpn openvpn[4349]:     body = self._encode_params(data)
Dec 12 14:37:02 shadowvpn openvpn[4349]:   File "/usr/lib/python3/dist-packages/requests/models.py", line 104, in _encode_params
Dec 12 14:37:02 shadowvpn openvpn[4349]:     v.encode('utf-8') if isinstance(v, str) else v))
Dec 12 14:37:02 shadowvpn openvpn[4349]: UnicodeEncodeError: 'utf-8' codec can't encode characters in position 30-31: surrogates not allowed

[arseniylebedev]

Pavel Denisov,

{'username': '2WLjulhUGPAhNJ6bANmr1EXp3jjmQD\udcd0\udcb0', 'password': 'IMAn5BifVgN2k1gyVwmr7fGhqG94li', 'ip': '10.0.2.2', 'port': '18375'}

В конец username добавил русский символ а.
Вот так выглядит исходный username 2WLjulhUGPAhNJ6bANmr1EXp3jjmQDа

[arseniylebedev]

Pavel Denisov,

root@shadowvpn:/etc/openvpn/scripts# echo '2WLjulhUGPAhNJ6bANmr1EXp3jjmQDа'
2WLjulhUGPAhNJ6bANmr1EXp3jjmQDа

А где можно системную кодировку посмотреть?

[arseniylebedev]

Pavel Denisov,
print(username) сделал

Dec 12 15:11:35 shadowvpn openvpn[4349]: verify.py -- Authorization attempt for username: 2WLjulhUGPAhNJ6bANmr1EXp3jjmQDа ip: 10.0.2.2 port: 39540
Dec 12 15:11:35 shadowvpn openvpn[4349]: 2WLjulhUGPAhNJ6bANmr1EXp3jjmQDа
Dec 12 15:11:35 shadowvpn openvpn[4349]: {'username': '2WLjulhUGPAhNJ6bANmr1EXp3jjmQD\udcd0\udcb0', 'password': 'IMAn5BifVgN2k1gyVwmr7fGhqG94li', 'ip': '10.0.2.2', 'port': '39540'}

[arseniylebedev]

Pavel Denisov,
Нет, вот код который ее выводит:

data = {
	"username": username,
	"password": password,
	"ip": ip,
	"port": port,
}

print(username)
print(data)

[arseniylebedev]

Pavel Denisov, проверил

Dec 12 15:37:54 shadowvpn openvpn[4349]: 3.6.7 (default, Oct 22 2018, 11:32:17)

Answer 1

[Kastuś]

любую, т.к. сайты разные бывают, а встречал и с 2мя кодировками одновременно
с encode/decode поиграйся, надо явно указать кодировку в request, точнее не скажу, давно было...