linux, source replacement, common Python operations, GPU server configuration, python virtual environment migration, offline deployment

Posted by superhaggis on Mon, 08 Nov 2021 20:20:09 +0100

linux configuration

Replace source

sudo chmod 777 /etc/apt/sources.list

vim /etc/apt/sources.list

Enter 10000 first, and then press d repeatedly to delete the existing

# The source image is annotated by default to improve apt update speed. You can cancel the annotation if necessary
deb https://mirrors.tuna.tsinghua.edu.cn/ubuntu/ focal main restricted universe multiverse
# deb-src https://mirrors.tuna.tsinghua.edu.cn/ubuntu/ focal main restricted universe multiverse
deb https://mirrors.tuna.tsinghua.edu.cn/ubuntu/ focal-updates main restricted universe multiverse
# deb-src https://mirrors.tuna.tsinghua.edu.cn/ubuntu/ focal-updates main restricted universe multiverse
deb https://mirrors.tuna.tsinghua.edu.cn/ubuntu/ focal-backports main restricted universe multiverse
# deb-src https://mirrors.tuna.tsinghua.edu.cn/ubuntu/ focal-backports main restricted universe multiverse
deb https://mirrors.tuna.tsinghua.edu.cn/ubuntu/ focal-security main restricted universe multiverse
# deb-src https://mirrors.tuna.tsinghua.edu.cn/ubuntu/ focal-security main restricted universe multiverse

# Pre release software source, not recommended
# deb https://mirrors.tuna.tsinghua.edu.cn/ubuntu/ focal-proposed main restricted universe multiverse
# deb-src https://mirrors.tuna.tsinghua.edu.cn/ubuntu/ focal-proposed main restricted universe multiverse

sudo apt-get install net-tools

anaconda

wget -P /tmp https://repo.anaconda.com/archive/Anaconda3-2020.02-Linux-x86_64.sh
bash /tmp/Anaconda3-2020.02-Linux-x86_64.sh
source ~/.bashrc  # Activate environment variable
conda update --all # Upgrade anaconda

Create virtual environment conda create -n name python==3.7

anaconda replacement source

conda config --add channels https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/free/`
conda config --add channels https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main/`
conda config --add channels https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/pytorch/`
conda config --add channels httlsps://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/conda-forge/`
conda config --set show_channel_urls yes

Remove source
conda config --remove channels https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/conda-forge/

pip source

pip config set global.index-url https://pypi.tuna.tsinghua.edu.cn/simple

Installation of pycocotools under linux

https://blog.csdn.net/wlsccc/article/details/112728239

Decompression and compression

tar -xvf unzip tar.gz

tar -vczf test.tar.gz test compression

https://www.cnblogs.com/jyaray/archive/2011/04/30/2033362.html

Delete environment

conda remove -n xxx --all

conda remove package
conda remove -n learn --all / / delete the learn environment and all subordinate packages
pip uninstall package

Common python operations

1. Time string

now_time = datetime.datetime.strptime(now_time, "%Y-%m-%d %H:%M:%S") #To datetime
time.strftime("%Y-%m-%d %H:%M:%S")  #Convert to string

2. pandas

concat

pd.concat(objs=[df1,df2],axis=0)

Reference link

3. Use of jupyter notebook virtual environment

conda install nb_conda

Add and remove virtual environments
Reference link 1: Zhihu
Reference link 2: CSDN

4. Tensorflow 1.9 and python 3.6 configurations

pip installation

pip install --user tensorflow==1.9.0 -i https://pypi.doubanio.com/simple/

https://pypi.tuna.tsinghua.edu.cn/simple/

Error reporting numpy

FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_qint8 = np.dtype([("qint8", np.int8, 1)])

resolvent

pip uninstall numpy

pip install numpy==1.16.4

5. Data set download

print("Downloading " + file_name + " ... ")  # Start downloading
urllib.request.urlretrieve(url_base + file_name, file_path)
print("Done")

Conversion of data sets

print("Converting " + file_name + " to NumPy Array ...")
with gzip.open(file_path, 'rb') as f:   # Open to read binary data or text data
    data = np.frombuffer(f.read(), np.uint8, offset=16)  # np.frombuffer reads data from the stream,
data = data.reshape(-1, img_size)  # One line is a picture
print("Done")

np.uint8  0~255

>>> s = b'hello world'   # b stands for bytes, and u and r
>>> np.frombuffer(s, dtype='S1', count=5, offset=6)  #The offset is 6
array([b'w', b'o', b'r', b'l', b'd'], dtype='|S1')

Encapsulation and opening of data sets

print("Creating pickle file ...")  #Here is a dictionary stored
with open(save_file, 'wb') as f:   #f is the name of the saved file
    pickle.dump(dataset, f, -1)    #-1 specifies the serialization method of Python 3
print("Done!")


# Open the pickle file.

with open(save_file, 'rb') as f:
    dataset = pickle.load(f)

Common errors in opening a dataset pickle

When picking. Load (f), you may encounter coding errors, so you need to change it to pick. Load (F, encoding = 'bytes')

6. python program encryption

Package as exe

https://zhuanlan.zhihu.com/p/109266820

https://zhuanlan.zhihu.com/p/59442292

Python generates pyd files

You need to install a vsstudio c + +

Go to the official website to download the wheel file (the direct download is wrong)

During installation,

Create a setup file

from distutils.core import setup
from Cython.Build import cythonize

setup(name='api_sign', ext_modules=cythonize(['in_main.py','main_use.py','data_load.py','subprogram.py','unique.py']))

# python setup.py build_ext --inplace

Multiple files can be placed here

Execute the command to generate a pyd file:

python setup.py build_ext --inplace

Error reporting processing of generated pyd

https://blog.csdn.net/Gavinmiaoc/article/details/84340736?utm_medium=distribute.pc_relevant.none-task-blog-2%7Edefault%7EBlogCommendFromMachineLearnPai2%7Edefault-13.control&depth_1-utm_source=distribute.pc_relevant.none-task-blog-2%7Edefault%7EBlogCommendFromMachineLearnPai2%7Edefault-13.control

easycython generates pyd

http://www.xoxxoo.com/index/index/article/id/772.html

GPU server configuration

Linux system directory structure: https://www.runoob.com/linux/linux-system-contents.html

Use windows as the ssh server and linux as the ssh server to connect.

1. Use windows to generate public and private keys

ssh-keygen -t rsa

Then select the location

Generally stored under. ssh /

Generate two files [Ubuntu 20.04 LTS.lnk](Ubuntu 20.04 LTS.lnk)

id_rsa， id_rsa.pub

2. Put the id_rsa.pub is transferred to the linux server

cat id_rsa.pub >> ~/.ssh/authorized_key

Add this new public key

3. windows config configuration

Create a new config file under. ssh / without suffix

Host my(Simple common name)
	HostName ip
	User myname
	Port 22
	IdentityFile (Which private key is used)
Host my(Simple common name)
	HostName ip
	User myname
	IdentityFile (Which private key is used)

https://www.xylnk.com/2020/206.html

The config of the server is generally port and permission configuration

4. File transfer

scp local file my: server path

scp -r local folder my: server path

rsync -avP local file my: server path (very fast for large folders)

https://www.bilibili.com/video/BV1nk4y1k742

python virtual environment migration

Installing miniconda

curl -k -o Miniconda2-latest-Linux-x86_64.sh https://repo.anaconda.com/miniconda/Miniconda2-latest-Linux-x86_64.sh

https://docs.conda.io/en/latest/miniconda.html

-k is to skip ssl certificate detection

-o yes, save as target file

https://www.ruanyifeng.com/blog/2019/09/curl-reference.html This is curl's reference website

Unreliable method

pip installed packages

pip freeze > requirements.txt

pip install -r requirements.txt

Offline on the same platform

pip download -d ~/my_env_pkgs/ -r requirements.txt

pip install -r requirements.txt --no-index --find-links=file:///home/user_name/my_env_pkgs

Cross platform offline

pip download --no-binary=:all: -d ~/my_env_pkgs/ -r requirements.txt

pip install -r requirements.txt --no-index --find-links=file:///home/user_name/my_env_pkgs

Package installed by conda

conda env export -f environment.yml

conda env create -f environment.yml

Manually delete the pip part and install only the conda part

Offline deployment

Installation (both the test machine and the machine to be deployed need to be installed, which is recommended to be installed in the base environment)

conda install -c conda-forge conda-pack

Export environment

conda pack -n my_env -o out_name.tar.gz,

Create a new directory for the target environment, usually under. conda/envs /

mkdir -p my_env

Extract it

tar -xzf out_name.tar.gz -C my_env

When the target machine activates the environment, it will_ Env / bin is added to the path environment variable before conda activate is supported

source my_env/bin/activate

Replace prefix on deployment machine

conda-unpack

At this time, if you encounter the problem of "Files managed by conda were found to have been deleted/overwritten...", you should use pip to uninstall the packages installed with conda. At this time, you need to uninstall these packages correctly with conda first, and then perform the packaging operation

Cross platform deployment

Delete the build field from the yml file

be careful

conda --clone and – offline are not recommended

If it is an offline host, you can directly use the networked virtual machine for conda pack.

If it is an online host, use txt and yml files, such as online download.