Building a deep and intensive learning environment

Posted by tooNight on Sat, 29 Jan 2022 14:47:28 +0100

Conda environment construction

Download and install

Download from official website
Tsinghua source image download

New environment

conda create -n rl python=3.8

Activate environment

conda activate rl

Install software

The IDE recommends installing Jupyter Lab or pycham

Execute the following command to install and run jupyter lab in rl environment:

# Activate environment
conda activate robocup
# install
conda install -c conda-forge jupyterlab
# function
jupyter-lab

Construction of deep learning environment

Windows

View supported CUDA versions

Pay attention to CUDA Version

nvidia-smi

# Output:
Mon Nov  8 18:37:26 2021
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 496.49       Driver Version: 496.49       CUDA Version: 11.5     |
|-------------------------------+----------------------+----------------------+
| GPU  Name            TCC/WDDM | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  NVIDIA GeForce ... WDDM  | 00000000:01:00.0 Off |                  N/A |
| N/A    0C    P8    12W /  N/A |    162MiB /  6144MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+

Install drive and CUDA

Video card driver download:

Official GeForce driver | NVIDIA

Remember to open and update the graphics card driver after the program is installed

CUDA toolkit Download:
CUDA Toolkit 11.5 Downloads | NVIDIA Developer

If the download speed of the official website is too slow, you can download the driver from Baidu online disk:

Link: https://pan.baidu.com/s/1qy7hJKPAktqoNy8xwFxM3Q
Extraction code: d7z3

Is the installation successful

nvcc -V

# Output of successful installation:
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2021 NVIDIA Corporation
Built on Mon_Sep_13_20:11:50_Pacific_Daylight_Time_2021
Cuda compilation tools, release 11.5, V11.5.50
Build cuda_11.5.r11.5/compiler.30411180_0

Ubuntu

Check dependency

Verify your GPU Is there a GPU that can support CUDA

lspci | grep -i nvidia

Verify that your Linux version supports CUDA:

uname -m && cat /etc/*release

Verify whether gcc is installed in the system. Enter in the terminal:

gcc –v

Verify whether kernel header and package development are installed in the system

sudo apt-get install linux-headers-$(uname -r)

Install the drive

Check your NVIDIA graphics card model and recommended driver installation model.

Enter command:

ubuntu-drivers devices

The recommended driver is nvidia-390.
Install driver command:

sudo ubuntu-drivers autoinstall

The command is to automatically install the appropriate graphics card driver. You can also select the required driver for installation. The command is as follows:

sudo apt-get install nvidia-driver-390

Then wait for the installation to complete and restart the system.

sudo reboot

Finally, check whether the installation is successful:

Enter the following command:

nvidia-smi

Install CUDA

CUDA toolkit Download:
CUDA Toolkit 11.5 Downloads | NVIDIA Developer

Check whether the installation is successful:

ls /dev/nvidia*

Set environment variables:

sudo vim ~/.bashrc

Add the following:

export PATH=$PATH:/usr/local/cuda/bin
export LD_LIBRARY_PATH=/usr/local/cuda-10.0/bin/lib64:$LD_LIBRARY_PATH

Update profile:

source ~/.bashrc

Install cuDNN

Download the corresponding version of cuda from the official website cudnn

After downloading, unzip it into Cuda's directory and execute the following commands in sequence:

tar -xzvf cudnn-8.0-linux-x64-v6.0.tgz
sudo cp cuda/include/cudnn.h /usr/local/cuda/include
sudo cp cuda/lib64/libcudnn* /usr/local/cuda/lib64
sudo chmod a+r /usr/local/cuda/include/cudnn.h /usr/local/cuda/lib64/libcudnn*

Is the installation successful

Build an example:

cd NVIDIA_CUDA-10.0_Samples/
make
cd NVIDIA_CUDA-10.0_Samples/bin/x86_64/linux/release
./deviceQuery

Torch

Install Torch

Torch official website: Start Locally | PyTorch
I installed PyTorch with the following command (combined with the CUDA version installed above):

pip install torch==1.10.0+cu113 torchvision==0.11.1+cu113 torchaudio===0.10.0+cu113 -f https://download.pytorch.org/whl/cu113/torch_stable.html
 
# GPU version:
conda install pytorch torchvision torchaudio cudatoolkit=11.3 -c pytorch
# CPU version:
conda install pytorch torchvision torchaudio cpuonly -c pytorch

Verify GPU

Open jupyter lab input and run:

import torch
torch.cuda.is_available()

# If True is output, the CUDA version of Torch is successfully installed

TensorFlow

Install TensorFlow

conda install tensorflow # Installing TensorFlow for cpu using conda
conda install tensorflow-gpu # Install TensorFlow of gpu version with conda (CUDA and cudnn will be installed automatically)
pip install --ignore-installed --upgrade tensorflow-gpu # Installing using pip
pip install --ignore-installed --upgrade tensorflow_gpu==1.8.0 # Specified version

Verify GPU

import tensorflow as tf
# Creates a graph.
tf.compat.v1.disable_eager_execution()
a = tf.constant([1.0, 2.0, 3.0, 4.0, 5.0, 6.0], shape=[2, 3], name='a')
b = tf.constant([1.0, 2.0, 3.0, 4.0, 5.0, 6.0], shape=[3, 2], name='b')
c = tf.matmul(a, b)
# Creates a session with log_device_placement set to True.
sess = tf.compat.v1.Session(config=tf.compat.v1.ConfigProto(log_device_placement=True))
# Runs the op.
print(sess.run(c))

Strengthen the construction of learning environment

Installation dependent environment

TensorFlow may need to be installed first

pip install scipy numpy mkl matplotlib opencv-contrib-python pandas ray networkx -i https://pypi.tuna.tsinghua.edu.cn/simple

Install Gym

Execute the third order:

pip install gym # Basic installation
pip install gym[all] # Full installation (mujoco needs to be installed first)
pip install gym[all] --no-deps mujoco_py # Exclude mujoco_py dependent installation

Is the installation successful

Open jupyter lab input and run:

import gym
env = gym.make('CartPole-v0')
env.reset()
for _ in range(1000):
    env.render()
    env.step(env.action_space.sample()) # take a random action
env.close()

other

Essential software

sudo apt-get install build-essential cmake git vim

Configure pip source

Execute the following command to upgrade pip to the latest version (> = 10.0.0) and configure it:

pip install -i https://pypi.tuna.tsinghua.edu.cn/simple pip -U
pip config set global.index-url https://pypi.tuna.tsinghua.edu.cn/simple

Temporary use:

pip install -i https://pypi.tuna.tsinghua.edu.cn/simple
# Followed by the package to install

Note that simple cannot be less. It is https instead of http

Update pip:

python -m pip install --upgrade pip

Reference: help for using Tsinghua source pypi image

Configure Ubuntu 18 04 source

Back up the configuration file and execute the following commands:

sudo cp /etc/apt/sources.list /etc/apt/sources.list.bak
sudo vim /etc/apt/sources.list

Replacement:

# The source image is annotated by default to improve apt update speed. You can cancel the annotation if necessary
deb http://mirrors.tuna.tsinghua.edu.cn/ubuntu/ bionic main restricted universe multiverse
# deb-src https://mirrors.tuna.tsinghua.edu.cn/ubuntu/ bionic main restricted universe multiverse
deb http://mirrors.tuna.tsinghua.edu.cn/ubuntu/ bionic-updates main restricted universe multiverse
# deb-src https://mirrors.tuna.tsinghua.edu.cn/ubuntu/ bionic-updates main restricted universe multiverse
deb http://mirrors.tuna.tsinghua.edu.cn/ubuntu/ bionic-backports main restricted universe multiverse
# deb-src https://mirrors.tuna.tsinghua.edu.cn/ubuntu/ bionic-backports main restricted universe multiverse
deb http://mirrors.tuna.tsinghua.edu.cn/ubuntu/ bionic-security main restricted universe multiverse
# deb-src https://mirrors.tuna.tsinghua.edu.cn/ubuntu/ bionic-security main restricted universe multiverse

# Pre release software source, not recommended
# deb https://mirrors.tuna.tsinghua.edu.cn/ubuntu/ bionic-proposed main restricted universe multiverse
# deb-src https://mirrors.tuna.tsinghua.edu.cn/ubuntu/ bionic-proposed main restricted universe multiverse

Run the command to update:

sudo apt-get update
sudo apt-get upgrade
sudo apt-get install -f

Reference: Tsinghua source Ubuntu image help

Configure conda source

# Execute the following command to generate condarc file
conda config --set show_channel_urls yes

# Edit condarc file
vim ~/.condarc

Modified contents of. condarc file:

channels:
  - defaults
show_channel_urls: true
default_channels:
  - http://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
  - http://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/r
  - http://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/msys2
custom_channels:
  conda-forge: http://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud
  msys2: http://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud
  bioconda: http://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud
  menpo: http://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud
  pytorch: http://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud
  simpleitk: http://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud

After modification, run the following command to clear the index cache

conda clean -i 

# Test it
# Create a new environment
conda create -n myenv numpy
# Activate environment
conda activate myenv
# Delete environment
conda remove -n myenv --all

Reference: Tsinghua source Anaconda image help

Installing Typora for Ubuntu

# or see https://support.typora.com.cn/Typora-on-Linux/
sudo snap install typora

Reference: Typora Chinese station

GitHub image access

Here are the two most commonly used image addresses:

github.com.cnpmjs.org
hub.fastgit.org

In other words, the above image is a cloned version of GitHub. You can visit the above image website. The content of the website is a complete and synchronous image with GitHub, and then download and clone in this website. The use method is simple website replacement, such as https://github.com/openai/gym.git Replace with https://github.com.cnpmjs.org/openai/gym.git .

Ubuntu18.04 install Sogou input method

Download installation package

Installation tutorial

The following are concise installation commands:

# Update source
sudo apt update
# Install the fcitx input method framework (some operations need to be done in subsequent settings)
sudo apt install fcitx
# Set fcitx startup self startup
sudo cp /usr/share/applications/fcitx.desktop /etc/xdg/autostart/
# Uninstall the ibus input method framework of the system
sudo apt purge ibus
# Install Sogou input method
sudo dpkg -i Installation package name
# Install input method dependency
sudo apt install libqt5qml5 libqt5quick5 libqt5quickwidgets5 qml-module-qtquick2
sudo apt install libgsettings-qt1
# Restart the computer (you may need to add Sogou input method before restarting)
# super + space switching language
# ctrl + space switch input method

Windows Terminal configuration Anaconda

Windows Terminal is built-in in Win 11, and Win10 can be downloaded in the app store
Try running as administrator when permission error occurs

Name: Anaconda
Command line: CMD exe /K C:\Users\shang\anaconda3\Scripts\activate. bat
Startup directory:% USERPROFILE%
Do not use the parent process directory (otherwise it may enter the Windows directory of drive C by default)
Icon: C: \ users \ Shang \ anaconda3 \ menu \ Anaconda navigator ico
Tab Title: Anaconda

Topics: Machine Learning

Programmer Think

Building a deep and intensive learning environment

Conda environment construction

Download and install

New environment

Activate environment

Install software

Construction of deep learning environment

Windows

View supported CUDA versions

Install drive and CUDA

Is the installation successful

Ubuntu

Check dependency

Install the drive

Install CUDA

Install cuDNN

Is the installation successful

Torch

Install Torch

Verify GPU

TensorFlow

Install TensorFlow

Verify GPU

Strengthen the construction of learning environment

Installation dependent environment

Install Gym

Is the installation successful

other

Essential software

Configure pip source

Configure Ubuntu 18 04 source

Configure conda source

Installing Typora for Ubuntu

GitHub image access

Ubuntu18.04 install Sogou input method

Windows Terminal configuration Anaconda

Hot Topics