linux, source replacement, common Python operations, GPU server configuration, python virtual environment migration, offline deployment

Posted by superhaggis on Mon, 08 Nov 2021 20:20:09 +0100

linux configuration

Replace source

sudo chmod 777 /etc/apt/sources.list

vim /etc/apt/sources.list

Enter 10000 first, and then press d repeatedly to delete the existing

# The source image is annotated by default to improve apt update speed. You can cancel the annotation if necessary
deb focal main restricted universe multiverse
# deb-src focal main restricted universe multiverse
deb focal-updates main restricted universe multiverse
# deb-src focal-updates main restricted universe multiverse
deb focal-backports main restricted universe multiverse
# deb-src focal-backports main restricted universe multiverse
deb focal-security main restricted universe multiverse
# deb-src focal-security main restricted universe multiverse

# Pre release software source, not recommended
# deb focal-proposed main restricted universe multiverse
# deb-src focal-proposed main restricted universe multiverse

sudo apt-get install net-tools


wget -P /tmp
bash /tmp/
source ~/.bashrc  # Activate environment variable
conda update --all # Upgrade anaconda

Create virtual environment conda create -n name python==3.7

anaconda replacement source

conda config --add channels`
conda config --add channels`
conda config --add channels`
conda config --add channels httlsps://`
conda config --set show_channel_urls yes

Remove source
conda config --remove channels

pip source

pip config set global.index-url

Installation of pycocotools under linux

Decompression and compression

tar -xvf unzip tar.gz

tar -vczf test.tar.gz test compression

Delete environment

conda remove -n xxx --all

conda remove package
conda remove -n learn --all / / delete the learn environment and all subordinate packages
pip uninstall package

Common python operations

1. Time string

now_time = datetime.datetime.strptime(now_time, "%Y-%m-%d %H:%M:%S") #To datetime
time.strftime("%Y-%m-%d %H:%M:%S")  #Convert to string

2. pandas

  • concat

Reference link

3. Use of jupyter notebook virtual environment

conda install nb_conda

Add and remove virtual environments
Reference link 1: Zhihu
Reference link 2: CSDN

4. Tensorflow 1.9 and python 3.6 configurations

pip installation

pip install --user tensorflow==1.9.0 -i

Error reporting numpy

FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_qint8 = np.dtype([("qint8", np.int8, 1)])


pip uninstall numpy

pip install numpy==1.16.4

5. Data set download

print("Downloading " + file_name + " ... ")  # Start downloading
urllib.request.urlretrieve(url_base + file_name, file_path)

Conversion of data sets

print("Converting " + file_name + " to NumPy Array ...")
with, 'rb') as f:   # Open to read binary data or text data
    data = np.frombuffer(, np.uint8, offset=16)  # np.frombuffer reads data from the stream,
data = data.reshape(-1, img_size)  # One line is a picture

np.uint8  0~255

>>> s = b'hello world'   # b stands for bytes, and u and r
>>> np.frombuffer(s, dtype='S1', count=5, offset=6)  #The offset is 6
array([b'w', b'o', b'r', b'l', b'd'], dtype='|S1')

Encapsulation and opening of data sets

print("Creating pickle file ...")  #Here is a dictionary stored
with open(save_file, 'wb') as f:   #f is the name of the saved file
    pickle.dump(dataset, f, -1)    #-1 specifies the serialization method of Python 3

# Open the pickle file.

with open(save_file, 'rb') as f:
    dataset = pickle.load(f)

Common errors in opening a dataset pickle

When picking. Load (f), you may encounter coding errors, so you need to change it to pick. Load (F, encoding = 'bytes')

6. python program encryption

Package as exe

Python generates pyd files

You need to install a vsstudio c + +

Go to the official website to download the wheel file (the direct download is wrong)

During installation,

Create a setup file

from distutils.core import setup
from Cython.Build import cythonize

setup(name='api_sign', ext_modules=cythonize(['','','','','']))

# python build_ext --inplace

Multiple files can be placed here

Execute the command to generate a pyd file:

python build_ext --inplace

Error reporting processing of generated pyd

easycython generates pyd

GPU server configuration

Linux system directory structure:

Use windows as the ssh server and linux as the ssh server to connect.

1. Use windows to generate public and private keys

ssh-keygen -t rsa

Then select the location

Generally stored under. ssh /

Generate two files [Ubuntu 20.04 LTS.lnk](Ubuntu 20.04 LTS.lnk)


2. Put the is transferred to the linux server

cat >> ~/.ssh/authorized_key

Add this new public key

3. windows config configuration

Create a new config file under. ssh / without suffix

Host my(Simple common name)
	HostName ip
	User myname
	Port 22
	IdentityFile (Which private key is used)
Host my(Simple common name)
	HostName ip
	User myname
	IdentityFile (Which private key is used)

The config of the server is generally port and permission configuration

4. File transfer

scp local file my: server path

scp -r local folder my: server path

rsync -avP local file my: server path (very fast for large folders)

python virtual environment migration

Installing miniconda

curl -k -o

-k is to skip ssl certificate detection

-o yes, save as target file This is curl's reference website

Unreliable method

pip installed packages

pip freeze > requirements.txt

pip install -r requirements.txt

Offline on the same platform

pip download -d ~/my_env_pkgs/ -r requirements.txt

pip install -r requirements.txt --no-index --find-links=file:///home/user_name/my_env_pkgs

Cross platform offline

pip download --no-binary=:all: -d ~/my_env_pkgs/ -r requirements.txt

pip install -r requirements.txt --no-index --find-links=file:///home/user_name/my_env_pkgs

Package installed by conda

conda env export -f environment.yml

conda env create -f environment.yml

Manually delete the pip part and install only the conda part

Offline deployment

Installation (both the test machine and the machine to be deployed need to be installed, which is recommended to be installed in the base environment)

conda install -c conda-forge conda-pack

Export environment

conda pack -n my_env -o out_name.tar.gz,

Create a new directory for the target environment, usually under. conda/envs /

mkdir -p my_env

Extract it

tar -xzf out_name.tar.gz -C my_env

When the target machine activates the environment, it will_ Env / bin is added to the path environment variable before conda activate is supported

source my_env/bin/activate

Replace prefix on deployment machine


At this time, if you encounter the problem of "Files managed by conda were found to have been deleted/overwritten...", you should use pip to uninstall the packages installed with conda. At this time, you need to uninstall these packages correctly with conda first, and then perform the packaging operation

Cross platform deployment

Delete the build field from the yml file


be careful

conda --clone and – offline are not recommended

If it is an offline host, you can directly use the networked virtual machine for conda pack.

If it is an online host, use txt and yml files, such as online download.

Reference link

The project is deployed offline and the virtual environment is started for operation

plt.rcParams.update({'figure.figsize':(9,7), 'figure.dpi':120})

[external chain picture transfer failed. The source station may have anti-theft chain mechanism. It is recommended to save the picture and upload it directly (img-gjolmpi0-163662171870) (Python% E5% B8% B8% E7% 94% A8% E6% 93% 8D% E4% BD% 9C. Assets / image-20210815115527645. PNG)]

p = d = q = range(0, 2)
pdq = list(itertools.product(p, d, q))
# Ignore warning
with warnings.catch_warnings():
    error = walk_forward_validation(data, n_test, cfg)
key = str(cfg) #Parameter configuration is tuple
print(f'> Model{key} {error:.3f}')

Topics: Python Linux server computer