Record curl submit data encoding and javascript transform unicode problem

Posted by boiy on Sun, 10 May 2020 16:19:42 +0200

1. Data coding problem of curl submission

When using curl to send a POST request under windows, there is a problem. The parameters transmitted are not UTF-8 encoded.

  • 1. In the GIT bash test, locale is UTF-8, no effect, the server received or GBK encoding.
  • 2. Tested under cmd, chcp 65001 has no effect. The server receives GBK code.
  • 3. Write script file (sh/bat). In both environments, it is still GBK.

Finally, only echo can be used to output data, and then curl can read from stdin.

echo  '{"url":"Here are Chinese and zimu","time":1526129881}' | \
curl 'http://192.168.17.11:8010/testsvr/post' \
      -H 'User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:76.0) Gecko/20100101 Firefox/76.0' \
      -H 'Accept: application/json, text/plain, */*' \
      -H 'Accept-Language: zh-CN,zh;q=0.8,zh-TW;q=0.7,zh-HK;q=0.5,en-US;q=0.3,en;q=0.2' \
      -H 'Content-Type: application/json;charset=utf-8' \
      -H 'Origin: http://192.168.17.11:8010' \
      -H 'Connection: keep-alive' \
      -H 'Referer: http://192.168.17.11:8010/' \
      -H 'Cookie: id=cdfb03dfd29351d7b3aeb96b24218b86' \
      -X POST-d @-

2. The problem of translating UniCode into javascript

Many of the codes on the Internet are as follows:

function encodeUnicode(str) {
    var res = [];
    for ( var i=0; i<str.length; i++ ) {
	res[i] = ( "00" + str.charCodeAt(i).toString(16) ).slice(-4);
    }
    return "\\u" + res.join("\\u");
}

There is a problem, that is, ascii characters will also be encoded, but most of the time, we only need to encode non ascii characters, so it is modified as follows

function encodeUnicode(str) {
    var value = '';
    for (var i = 0; i < str.length; i++) {
        ch = str.charCodeAt(i);
        if (ch > 127) {
            value += '\\u' + str.charCodeAt(i).toString(16);
        } else {
            value += str[i];
        }
    }
    return value;
}

The test results are as follows:

console.log(encodeUnicode('{"url":"Here are Chinese and zimu","time":1526129881}'));
# First result
> "\u007b\u0022\u0075\u0072\u006c\u0022\u003a\u0022\u8fd9\u91cc\u6709\u4e2d\u6587\u548c\u007a\u0069\u006d\u0075\u0022\u002c\u0022\u0074\u0069\u006d\u0065\u0022\u003a\u0031\u0035\u0032\u0036\u0031\u0032\u0039\u0038\u0038\u0031\u007d"
# Second result
> "{"url":"\u8fd9\u91cc\u6709\u4e2d\u6587\u548czimu","time":1526129881}"

Topics: curl Windows JSON ascii