Reading yaml file with linux shell

Posted by techcone on Thu, 16 Jan 2020 06:30:04 +0100

1, Installing python3.7.0 on Linux

centos 6 / 7, the default installation of centos system is python2. X (in fact, I'm centos 6.5, and centos default installation of python2.6.6). Version x is different according to different versions of the system. You can view the python version of the system through python --V or python --version. Some system commands require python2, which cannot be uninstalled

1. Install dependency package

1) Install the gcc compiler first. Some system versions of GCC have been installed by default. Check through gcc --version. For those not installed, install GCC first, yum -y install gcc

2) Install other dependent packages. (Note: do not lack them, otherwise there may be an error in installing python. For versions under 3.7.0, libffi devel is not required.)

yum -y install zlib-devel bzip2-devel openssl-devel ncurses-devel sqlite-devel readline-devel tk-devel gdbm-devel db4-devel libpcap-devel xz-devel libffi-devel

2. Download the source code of python3.7.0 as required

1) Choose the python source package you need from https://www.python.org/ftp/python/. I downloaded Python 3.7.0

(you can also go https://www.python.org Download from the homepage of the official website, find downloads, and you can choose the version you want to download.)

2) Download

wget https://www.python.org/ftp/python/3.7.0/Python-3.7.0.tgz

3. Unzip Python-3.7.0.tgz

cd /usr/local
tar -zxvf /opt/software/Python-3.7.0.tgz

4. Create an empty folder to store the python 3 program

mkdir /usr/local/python3 

5. Execute configuration file, compile, compile and install

cd Python-3.7.0
./configure --prefix=/usr/local/python3
make && make install

The installation was completed without any error

6. Establish soft connection

ln -s /usr/local/python3/bin/python3.7 /usr/bin/python3
ln -s /usr/local/python3/bin/pip3.7 /usr/bin/pip3

7. Test whether Python 3 can be used

[root@mini Python-3.7.0]# python3
Python 3.7.0 (default, Jul 28 2018, 22:47:29) 
[GCC 4.8.5 20150623 (Red Hat 4.8.5-28)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> print("hello world!")
hello world!
>>> exit()
[root@mini Python-3.7.0]# pip3 --version
pip 10.0.1 from /usr/local/python3/lib/python3.7/site-packages/pip (python 3.7)

You can see that Python 3.7.0 works normally

2, bug: using pip command failed

1. Error message

1.1 information

pip is configured with locations that require TLS/SSL, however the ssl module in Python is not available.
Collecting virtualenv
Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError("Can't connect to HTTPS URL because the SSL module is not available.")': /simple/virtualenv/
Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError("Can't connect to HTTPS URL because the SSL module is not available.")': /simple/virtualenv/
Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError("Can't connect to HTTPS URL because the SSL module is not available.")': /simple/virtualenv/
Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError("Can't connect to HTTPS URL because the SSL module is not available.")': /simple/virtualenv/
Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError("Can't connect to HTTPS URL because the SSL module is not available.")': /simple/virtualenv/
Could not fetch URL https://pypi.org/simple/virtualenv/: There was a problem confirming the ssl certificate: HTTPSConnectionPool(host='pypi.org', port=443): Max retries exceeded with url: /simple/virtualenv/ (Caused by SSLError("Can't connect to HTTPS URL because the SSL module is not available.")) - skipping
Could not find a version that satisfies the requirement virtualenv (from versions: )
No matching distribution found for virtualenv
pip is configured with locations that require TLS/SSL, however the ssl module in Python is not available.
Could not fetch URL https://pypi.org/simple/pip/: There was a problem confirming the ssl certificate: HTTPSConnectionPool(host='pypi.org', port=443): Max retries exceeded with url: /simple/pip/ (Caused by SSLError("Can't connect to HTTPS URL because the SSL module is not available.")) - skipping

1.2 error reason

The system version is CentOS 6.5, in which the version of openssl is OpenSSL 1.0.1e-fips 11 Feb 2013, while the version of openssl required by Python 3.7 is 1.0.2 or 1.1.x. it needs to upgrade openssl and recompile Python 3.7.0. The version of openssl installed by yum is relatively low.

2. Solution - upgrade openssl

2.1 download openssl

wget https://www.openssl.org/source/openssl-1.1.1a.tar.gz
cd /usr/local
tar -zxvf openssl-1.1.1a.tar.gz
cd openssl-1.1.1amv openssl-1.1.1a openssl

2.2 compilation and installation

./config --prefix=/usr/local/openssl no-zlib #zlib is not required
make
make install

2.3 back up the original configuration

mv /usr/bin/openssl /usr/bin/openssl.bak
mv /usr/include/openssl/ /usr/include/openssl.bak
//New configuration
ln -s /usr/local/openssl/include/openssl /usr/include/openssl
ln -s /usr/local/openssl/lib/libssl.so.1.1 /usr/local/lib64/libssl.so
ln -s /usr/local/openssl/bin/openssl /usr/bin/openssl
//Modify system configuration
\## Search path to write openssl library file
echo "/usr/local/openssl/lib" >> /etc/ld.so.conf
\## Make the modified / etc/ld.so.conf effective
ldconfig -v

2.4 view openssl version

openssl version
openssl version Tips:
 /usr/local/openssl/bin/openssl: error while loading shared libraries: libssl.so.1.1: cannot open shared object file: No such file or directory

//If your libssl.so.1.1 file is under / usr/local/openssl/lib /, you can do this
ln -s /usr/local/openssl/lib/libssl.so.1.1 /usr/lib64/libssl.so.1.1
ln -s /usr/local/openssl/lib/libcrypto.so.1.1 /usr/lib64/libcrypto.so.1.1

2.5 reinstall python

./configure --prefix=/usr/local/python3 --with-openssl=/usr/local/openssl
make
make install

3, shell reads shyaml of yaml

1. Install shyaml

pip3 install shyaml

2. Case analysis

2.1 file.yaml file content

create a file
cd /usr/local
vim file.yaml
---
idc_group:
    name: bx
    bx:
        news_bx: news_bx
        web3_bx: web3_php-fpm_bx
        task: [1,2,3,4,5,6]

2.2 shell operation

(testenv3.7) localhost:testenv3.7 macname$ cat file.yaml | shyaml get-value idc_group.name
bx
(testenv3.7) localhost:testenv3.7 macname$ 
(testenv3.7) localhost:testenv3.7 macname$ cat file.yaml | shyaml key-values idc_group.bx
news_bx
news_bx
web3_bx
web3_php-fpm_bx
(testenv3.7) localhost:testenv3.7 macname$ 


get-value: Get value
(testenv3.7) localhost:testenv3.7 macname$ cat file.yaml | shyaml get-value idc_group.bx.news_bx
news_bx
(testenv3.7) localhost:testenv3.7 macname$ 


get-type: Get the corresponding type
(testenv3.7) localhost:testenv3.7 macname$ cat file.yaml | shyaml get-type idc_group.bx.news_bx
str
(testenv3.7) localhost:testenv3.7 macname$ 


get-values{,-0}: For sequence types, get a list of values
(testenv3.7) localhost:testenv3.7 macname$ cat file.yaml | shyaml get-values idc_group.bx.task
1
2
3
4
5
6
(testenv3.7) localhost:testenv3.7 macname$ 


keys{,-0}: Back to key list
(testenv3.7) localhost:testenv3.7 macname$ cat file.yaml | shyaml keys idc_group
name
bx
(testenv3.7) localhost:testenv3.7 macname$ cat file.yaml | shyaml keys idc_group.bx
news_bx
web3_bx
task
(testenv3.7) localhost:testenv3.7 macname$ 


values{,-0}: Return value list
(testenv3.7) localhost:testenv3.7 macname$ cat file.yaml | shyaml values idc_group.bx
news_bx
web3_php-fpm_bx
- 1
- 2
- 3
- 4
- 5
- 6
(testenv3.7) localhost:testenv3.7 macname$ cat file.yaml | shyaml values idc_group
bx
news_bx: news_bx
web3_bx: web3_php-fpm_bx
task:
- 1
- 2
- 3
- 4
- 5
- 6
(testenv3.7) localhost:testenv3.7 macname$ 


key-values,{,-0}: Return key value pair
(testenv3.7) localhost:testenv3.7 macname$ cat file.yaml | shyaml key-values idc_group.bx
news_bx
news_bx
web3_bx
web3_php-fpm_bx
task
- 1
- 2
- 3
- 4
- 5
- 6
(testenv3.7) localhost:testenv3.7 macname$ cat file.yaml | shyaml key-values idc_group
name
bx
bx
news_bx: news_bx
web3_bx: web3_php-fpm_bx
task:
- 1
- 2
- 3
- 4
- 5
- 6
(testenv3.7) localhost:testenv3.7 macname$ 
Published 1 original article · praised 0 · visited 2
Private letter follow

Topics: OpenSSL Python SSL pip