py2exe packaging oss2 and pycryptodome failure problem investigation

Posted by gurhy on Sat, 31 Aug 2019 21:26:37 +0200

Background:
Recently, a program was developed with Python that requires running on a computer without Python installed.After comparing python's packaged exe tools, I chose py2exe(py2exe is officially not updated and maintained by third-party developers).

During use, some file libraries were found missing after py2exe was packaged, so they were deliberately recorded.

Sketch:
py2exe is a Python Distutils extension that converts Python scripts into executable Windows programs and runs without installing Python.

This is mainly to introduce the troubleshooting ideas for packaging errors with py2exe. The use of py2exe can be found on the py2exe website: http://www.py2exe.org/

Test environment:

System: win10 x64
python3.6.3

Lazy:

oss2==2.8.0
py2exe==0.9.3.2  # Download address: https://github.com/albertosottile/py2exe/releases
# pycryptodome==3.8.2 Installing oss2 automatically downloads and installs oss2. The installed directory name is Crypto

Test code app.py

import oss2


class OSSHandler:
    def __init__(self):
        self.endpoint = "endpoint"
        self.auth = oss2.Auth("access_key_id", "access_key_secret")
        self.bucket = oss2.Bucket(self.auth, self.endpoint, "bucket_name")

    def iterator(self):
        """
        //Traversing bucket files
        :return:
        """
        for object_info in oss2.ObjectIterator(bucket=self.bucket):
            print(object_info.key)


def main():
    OSSHandler().iterator()


if __name__ == "__main__":
    main()

Packaging code setup.py, note: console will run using terminal, otherwise it will flash by

import py2exe
from distutils.core import setup

py2exe_options = {
    "dist_dir": "dist",
    "compressed": 1,
    "optimize": 2,
    "ascii": 0,
}

setup(
    name='oss',
    version='0.1.0',
    description="win tool",
    options={'py2exe': py2exe_options},
    # Note: console will run using terminal, otherwise it will flash by
    console=[{
        # windows = [{
        "script": "app.py",
    }],
    zipfile="lib/shared.lib",
    data_files=[],
)

tar

python setup.py py2exe

Report errors
Error running dist folder app.exe, found module not found

Traceback (most recent call last):
  File "app.py", line 1, in <module>
    import oss2
  File "<frozen importlib._bootstrap>", line 971, in _find_and_load
  File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 656, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 626, in _load_backward_compatible
  File "oss2\__init__.pyc", line 3, in <module>
  File "<frozen importlib._bootstrap>", line 971, in _find_and_load
  File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 656, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 626, in _load_backward_compatible
  File "oss2\models.pyc", line 10, in <module>
  File "<frozen importlib._bootstrap>", line 971, in _find_and_load
  File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 656, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 626, in _load_backward_compatible
  File "oss2\utils.pyc", line 30, in <module>
  File "<frozen importlib._bootstrap>", line 971, in _find_and_load
  File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 656, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 626, in _load_backward_compatible
  File "Crypto\Cipher\__init__.pyc", line 27, in <module>
  File "<frozen importlib._bootstrap>", line 971, in _find_and_load
  File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 656, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 626, in _load_backward_compatible
  File "Crypto\Cipher\_mode_ecb.pyc", line 47, in <module>
  File "Crypto\Util\_raw_api.pyc", line 300, in load_pycryptodome_raw_lib
OSError: Cannot load native module 'Crypto.Cipher._raw_ecb': Trying '_raw_ecb.cp36-win_amd64.pyd': [WinError 126] The specified module could not be found., Trying '_raw_ecb.pyd': [WinError 126] The specified module could not be found.

Looking at the distlibshared.lib file, decompressing the view with the decompression tool revealed that the py2exe tool did not package the *.pyd file, causing the module search to fail

modify
From the error information above, you can see that the module call file location is "CryptoUtil_raw_api.pyc", line 300

try:
    filename = basename + ext
    # Debugging found that the pycryptodome_filename function could not find *.pyd caused an error
    return load_lib(pycryptodome_filename(dir_comps, filename),
                    cdecl)
except OSError as exp:
    attempts.append("Trying '%s': %s" % (filename, str(exp)))

View the pycryptodome_filename function CryptoUtil_file_system.py

util_lib, _ = os.path.split(os.path.abspath(__file__))
root_lib = os.path.join(util_lib, "..")
# Add print information to see where the module is loaded
print("root_lib: ", root_lib)

return os.path.join(root_lib, *dir_comps)

Add the print information, look at the module load location, and find that the directory is under shared.lib, but.pyd is not packaged in, so just let the program load into.pyd.

root_lib:  D:\code\dist\lib\shared.lib\Crypto\Util\..

Modify the *.pyd load directory so that root_lib loads the directory distlib, note that this method can affect the normal use of unpacked oss and pycryptodome

# util_lib, _ = os.path.split(os.path.abspath(__file__))
# root_lib = os.path.join(util_lib, "..")
util_lib = os.path.dirname(os.path.dirname(os.path.dirname(os.path.dirname(os.path.abspath(__file__)))))
root_lib = os.path.join(util_lib, "Crypto")
return os.path.join(root_lib, *dir_comps)

Repack, copy Crypto folder to distlib, Libsite-packagesCrypto folder only needs to keep *.pyd file, others can be deleted

Found that the report module could not be loaded, instead the report*.json file could not be loaded

Refer to the above method to modify the Libsite-packagesaliyunsdkcoreutils_init_.py file

# base_dir = os.path.dirname(os.path.abspath(aliyunsdkcore.__file__))
base_dir = os.path.dirname(os.path.dirname(os.path.dirname(os.path.abspath(aliyunsdkcore.__file__))))

Copy Libsite-packagesaliyunsdkcoredata to distlib, final directory

Function
This program may be working properly by now

With the above ideas, when packaging py2exe encounters a file that cannot be loaded, console mode can be used to find the directory where the file is loaded, and then make the appropriate modifications.

Topics: Python Windows github ascii