statement

All contents in this article are for learning and communication only. The packet capturing content, sensitive website and data interface have been desensitized. It is strictly prohibited to use them for commercial and illegal purposes, otherwise all the consequences have nothing to do with the author. If there is infringement, please contact me and delete them immediately!

Reverse target

• Objective: to encrypt the PEDATA MAX information of SAAS system in an investment field
• Home page: aHR0cHM6Ly9tYXgucGVkYXRhLmNuL2NsaWVudC9uZXdzL25ld3NmbGFzaA==
• Interface: aHR0cHM6Ly9tYXgucGVkYXRhLmNuL2FwaS9xNHgvbmV3c2ZsYXNoL2xpc3Q=
• Reverse parameter: the encryption result returned by the request, data: "L+o+YmIyNDE..."

Packet capture analysis

On the home page, click to view all 24-hour information and pull down. The information is loaded in the form of Ajax. We select the developer tool XHR for filtering. It is easy to find a list request. The return value data is a string of encrypted strings. exor doesn't know what it is, but it may be useful later. ts is the timestamp, as shown in the following figure:

There is nothing special about the parameters in the Payload, but some page turning information. Let's look at the request header. Here, note the Cookie and HTTP-X-TOKEN parameters. You need to log in to the page. Generally speaking, the Cookie is used to identify different users, but after brother K's test, it is found that in this case, the HTTP-X-TOKEN parameter is used to identify users, So you don't need cookies. Just mention it. We often see HM in cookies_ lvt_ XXX and Hm_lpvt_xxx is used for Baidu alliance advertising data statistics, which has nothing to do with crawlers.

Encryption reverse

We noticed that a dictionary is returned, and there must be a value taking process after obtaining the encrypted data, so we directly search the key, and there is only one search exor result:

Here e.data is the returned dictionary, e.data data,e.data.exor takes the encrypted value and exor in turn. Here we can guess that the encrypted value is taken out for decryption. We also make a breakpoint at the end of this function to see whether the data value becomes clear text after the execution of this Code:

As expected, the code of Object(p["y"] (e.data.data, e.data. EXOR) is the decryption function. Object(p["y"]) actually calls the M method. Follow to see:

The passed in t and n are the encrypted value and exor respectively, and the last returned JSON Parse (c) is the decryption result:

Key codes:

function M(t, n) {
    var a = L(Object(s["a"])(), n)
    , r = Y(B(t), a)
    , c = o.a.gunzipSync(e.from(r)).toString("utf-8");
    return JSON.parse(c)
}
Copy code

Deduct the functions one by one. I won't talk about it simply. Selecting Object(s["a"]) actually calls the c method, follows up the c method, and actually takes the loginToken. This loginToken is the HTTP-X-TOKEN in the request header we analyzed earlier, which contains your login information.

Expand knowledge: window The localStorage attribute is used to store data in the form of key value pairs in the browser. localStorage is similar to sessionStorage. The difference is that the data in localStorage can be retained for a long time without expiration time until it is manually deleted. The data of sessionStorage is only saved in the current session and will be deleted after closing the window or tab.

Look down again, as like as two peas, we'll have a o.a.gunzipSync(), first let's put it first, then we'll look at the parameter e.from(r) first, then we can see what we can't see through the follow-up. We will find that Uint8Array and r are the same as the r and e.from.

Let's look at o.a.gunzipSync(), which actually calls chunk vendors Anonymous function in JS. It doesn't matter if you don't know this JS. We noticed chunk vendors There are more than 140000 lines of code in JS. With this strange name and what module supplier, it is not difficult to think that this is a JS generated by a system or a third party. In fact, it is a file created during the construction of vue application. For our crawler engineers, it is roughly understood as similar to jQuery JS is OK. We usually don't deduct jQuery JS, the same chunk vendors JS can't buckle foolishly.

Let's focus on the function name, gunzipSync. If you don't know others, but know zip, you can think that it should be related to compression. It doesn't matter if you don't understand it. Use Baidu Dafa directly:

This directly gives the implementation method in nodejs. The zlib module is used. Find an example to see the usage:

var zlib = require('zlib');
var input = "Nidhi";
var gzi = zlib.gzipSync(input);
var decom = zlib.gunzipSync(new Buffer.from(gzi)).toString();

console.log(decom);
Copy code

Further study, we can know zlib The gunzipsync () method is the built-in application programming interface of the zlib module, which is used to decompress data blocks using Gunzip. The incoming data can be of Buffer, TypedArray, DataView, ArrayBuffer and string types. In the official document, we can see the update history in V8 After 0.0, the incoming data supports Uint8Array:

Combined with our previous analysis of r value, in nodejs, the r value is directly passed into zlib Just use the gunzipsync () method. Deduct the L, V and B methods used, and then rewrite them with the zlib library to get the decompressed data:

function getDecryptedData(encryptedData, exor, loginToken) {
    var a = L(loginToken, exor);
    var r = Y(B(encryptedData), a)
    var decryptedData = zlib.gunzipSync(r).toString();
    return decryptedData
}
Copy code

Complete code

github.com/kgepachong/

The following only demonstrates part of the key code and cannot be run directly! Full code warehouse address: github.com/kgepachong/...

JavaScript encryption code

/* ==================================
# @Time    : 
# @Author  : 
# @FileName: main.js
# @Software: PyCharm
# ================================== */

var zlib = require('zlib');

function L(e, t) {
    if ("1" == t)
        return [7, 65, 75, 31, 71, 101, 57, 0];
    for (var n = [], a = 0, r = t.length; a < r; a += 2)
        n.push(e.substr(1 * t.substr(a, 2), 1).charCodeAt());
    return n
}

function Y(e, t) {
    for (var n, a = new Uint8Array(e.length), r = 0, c = e.length; r < c; r++)
        n = t[r % t.length],
            a[r] = e[r].charCodeAt() ^ n;
    return a
}

function B(e) {
    var t, n, a, r, c, u, i, o = "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/=", s = "", f = 0;
    e = e.replace(/[^A-Za-z0-9\+\/\=]/g, "");
    while (f < e.length)
        r = o.indexOf(e.charAt(f++)),
            c = o.indexOf(e.charAt(f++)),
            u = o.indexOf(e.charAt(f++)),
            i = o.indexOf(e.charAt(f++)),
            t = r << 2 | c >> 4,
            n = (15 & c) << 4 | u >> 2,
            a = (3 & u) << 6 | i,
            s += String.fromCharCode(t),
        64 != u && (s += String.fromCharCode(n)),
        64 != i && (s += String.fromCharCode(a));
    return s
}

function getDecryptedData(encryptedData, exor, loginToken) {
    var a = L(loginToken, exor);
    var r = Y(B(encryptedData), a)
    var decryptedData = zlib.gunzipSync(r).toString();
    return decryptedData
}
Copy code

Python sample code

# ==================================
# --*-- coding: utf-8 --*--
# @Time    : 
# @Author  : 
# @FileName: main.py
# @Software: PyCharm
# ==================================


import execjs
import requests

news_est_url = "Desensitization, complete code attention GitHub: https://github.com/kgepachong/crawler"
login_token = "token Change it to your own!"
headers = {
    "Accept": "application/json, text/plain, */*",
    "Content-Type": "application/json",
    "Host": "Desensitization, complete code attention GitHub: https://github.com/kgepachong/crawler",
    "HTTP-X-TOKEN": login_token,
    "Origin": "Desensitization, complete code attention GitHub: https://github.com/kgepachong/crawler",
    "Referer": "Desensitization, complete code attention GitHub: https://github.com/kgepachong/crawler",
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36"
}


def get_decrypted_data(encrypted_data, exor):
    with open('pedata_decrypt.js', 'r', encoding='utf-8') as f:
        pedata_js = f.read()
    decrypted_data = execjs.compile(pedata_js).call('getDecryptedData', encrypted_data, exor, login_token)
    return decrypted_data


def get_encrypted_data():
    data = {
        "type": "",
        "module": "LP",
        "page":
            {
                "currentPage": 1,
                "pageSize": 10
            }
    }
    response = requests.post(url=news_est_url, headers=headers, json=data).json()
    encrypted_data, exor = response["data"], response["exor"]
    return encrypted_data, exor


def main():
    encrypted_data, exor = get_encrypted_data()
    decrypted_data = get_decrypted_data(encrypted_data, exor)
    print(decrypted_data)


if __name__ == '__main__':
    main()
Copy code