1, jq

zcat client_log_20211226/*.gz | head -n 10000 | jq .event | sort | uniq -c

jq is a lightweight json processing command. json data can be sliced, filtered, mapped and transformed
jq . Format and output json data

Common options

• -c compact output json data
• -s reads all inputs into an array
• -r outputs the original string instead of a JSON format
• -S sort object

1. Compact output json data

jq -c . test.json
[{"lon":113.30765,"name":"Guangzhou City","code":"4401","lat":23.422825},{"lon":113.59446,"name":"Shaoguan City","code":"4402","lat":24.80296}]

2. Read all the output to an array (that is, the outermost set of json data)

echo '{"safd":"fsafd"}'
{"safd":"fsafd"}

echo '{"safd":"fsafd"}' | jq -r .
{
  "safd": "fsafd"
}

echo '{"safd":"fsafd"}' | jq -s .
[
  {
    "safd": "fsafd"
  }
]

3. Output the original string instead of a JSON format

echo '{"safd":"fsafd"}' | jq  .[]
"fsafd"

echo '{"safd":"fsafd"}' | jq -r .[]
fsafd

4. Sort object

jq . test.json 
[
  {
    "lon": 113.30765,
    "name": "Guangzhou City",
    "code": "4401",
    "lat": 23.422825
  },
  {
    "lon": 113.59446,
    "name": "Shaoguan City",
    "code": "4402",
    "lat": 24.80296
  }
]

jq -S . test.json 
[
  {
    "code": "4401",
    "lat": 23.422825,
    "lon": 113.30765,
    "name": "Guangzhou City"
  },
  {
    "code": "4402",
    "lat": 24.80296,
    "lon": 113.59446,
    "name": "Shaoguan City"
  }
]

5. Get the name value in the above geographic json data

jq '.[]|{name}' test.json 
{
  "name": "Guangzhou City"
}
{
  "name": "Shaoguan City"
}

6. Get the first name value

jq '.[0]|{name}' test.json 
{
  "name": "Guangzhou City"
}

7. Print out only the value of the first map:

jq '.[0]|.[]' test.json 
113.30765
"Guangzhou City"
"4401"
23.422825

8. Print out the name value of a map

jq '.[0]|.name' test.json 
"Guangzhou City"

9. Print out the name value of a map and display it in a normal string

jq -r '.[0]|.name' test.json 
Guangzhou City

case

Test json data as follows:

{
    "name": "xueyuan",
    "age": 21,
    "birthday": "10th August",
    "email": "im.hexueyuan@outlook.com",
    "skills" : [
        "C/C++",
        "Python",
        "Golang",
        "Node.js"
    ]
}

Use python's json library to process it into a string, as follows:

{"skills": ["C/C++", "Python", "Golang", "Node.js"], "age": 21, "birthday": "10th August", "name": "xueyuan", "email": "im.hexueyuan@outlook.com"}

This format is often seen in actual production. For example, we use curl to request an interface and return a json. For example, we test and output a json data in our own project. This format is often not readable. We need to process it before viewing it.
To convert to the readable form before python processing is very simple, execute:

cat test.json | jq '.'
#You can also write cat json2 data | jq .
#No quotation marks, but it's better to write it for the sake of standard, because sometimes it will be a problem

jq converts the data into a readable format and adds color highlighting, where key and value use different colors.

{
  "skills": [
    "C/C++",
    "Python",
    "Golang",
    "Node.js"
  ],
  "age": 21,
  "birthday": "10th August",
  "name": "xueyuan",
  "email": "im.hexueyuan@outlook.com"
}

If the json data is large and we only want to see one of the field data, use the following syntax:

#key is the field name
jq '.<key>'

cat test.json | jq .skills
[
  "C/C++",
  "Python",
  "Golang",
  "Node.js"
]

cat test.json | jq .name
"xueyuan"

When a field is a list, jq you can slice it

jq '.<list-key>[s:e]'

cat test.json | jq '.skills[0]'
"C/C++"

cat test.json | jq '.skills[0:3]'
[
  "C/C++",
  "Python",
  "Golang"
]

cat test.json | jq '.skills[:3]'
[
  "C/C++",
  "Python",
  "Golang"
]

cat test.json | jq '.skills[:]'
jq: error: syntax error, unexpected ']' (Unix shell quoting issues?) at <top-level>, line 1:
.skills[:]
jq: 1 compile error

cat test.json | jq '.skills[2:]'
[
  "Golang",
  "Node.js"
]

cat test.json | jq '.skills[]'
"C/C++"
"Python"
"Golang"
"Node.js"

Note that it is different from python list slicing. Make a difference 'skills' and ' Skills [] 'two types. You can see that the output of the former is a list, and the latter is a list member in non json format.

would rather

jq obtains text input from linux pipes or files. If the text does not meet the json format, jq will report an error. You can use this method to check whether a text meets the json check:

jq '.' json_file > /dev/null

jq uses a filter to process json text and output the content processed by the filter to the standard output. The filter is used to filter the fields that meet the requirements, such as the simplest filter ', This indicates that there is no filtering policy, so all json text will be output.

key filter

'.<key>'

Filter the field names that meet the key and output the value of the key.

key-value filter

'<key>'

Output key and value. The difference between key filter is as follows

cat test.json | jq '{age}'
{
  "age": 21
}

cat test.json | jq '.age'
21

Because the key value must belong to an object, the outer layer {} is added.

index filter

'.<list-key>[index]'
'.<list-key>[index1, index2]'
'.<list-key>[s:e]'
'.<list-key>[:e]'
'.<list-key>[s:]'
'.<list-key>[]'

Array indexes and slices are used to manipulate list elements.

Nested level filter

'.key1.key2.key3'

json data filter for nesting.

Multiple filter s

'.key1, .key2, .key3'

Separate the filter with English single byte comma, which is used to filter multiple fields in one filter.

filter pipe

'filter1 | filter2 | filter3'

jq '.contact | .phone | .home' people.json

Since the output of most filters is still json data, you can connect the filters through pipes.

Data output after reorganizing the filter
Sometimes we need to reconstruct the structure of json, such as removing a certain layer of nesting, such as taking several fields to form a new json. At this time, we need to change the structure of json again. We can use [] and {} to reorganize json.

#Organize the output into a list
jq '[filter1, filter2, filter3]' data.json

cat test.json | jq '[.age,.name]'
[
  21,
  "xueyuan"
]

#Organize the output into new json objects
jq '{filter1, filter2, filter3}' data.json

cat test.json | jq '{age,name,email,skills: .skills[2:]}'
{
  "age": 21,
  "name": "xueyuan",
  "email": "im.hexueyuan@outlook.com",
  "skills": [
    "Golang",
    "Node.js"
  ]
}

Recursively expand json structure
Sometimes we need to find a field in a json, but we really don't know which nesting this field is in. If the nesting is very deep, it's difficult to find it. jq you can expand the nesting and flatten it before finding it.

#Expand nesting
jq '..' data.json

cat test.json | jq '..'
{
  "skills": [
    "C/C++",
    "Python",
    "Golang",
    "Node.js"
  ],
  "age": 21,
  "birthday": "10th August",
  "name": "xueyuan",
  "email": "im.hexueyuan@outlook.com"
}
[
  "C/C++",
  "Python",
  "Golang",
  "Node.js"
]
"C/C++"
"Python"
"Golang"
"Node.js"
21
"10th August"
"xueyuan"
"im.hexueyuan@outlook.com"

After expansion, you can find the key by combining the pipeline with the filter again.

length filter
Calculate the element length. For an object, length represents the number of elements in the object, for a string, length represents the number of string characters, and for a list, it represents the number of list elements.

cat test.json | jq '. | length'
5

cat test.json | jq '.age | length'
21

cat test.json | jq '.name | length'
7

cat test.json | jq '.skills | length'
4

keys filter
Output all key s in list form

cat test.json | jq '. | keys'
[
  "age",
  "birthday",
  "email",
  "name",
  "skills"
]

Check whether a key exists
If the input is an object, the element of the object exists in the form of "key value". Use

jq 'has("key-name")' data.json

Check whether the json object contains a key.

List traversal
jq supports the use of map() or map_values() iterates through the list, or the value of the object.

echo '{"a":1,"b":2,"c":3}' | jq 'map_values(1 + .)'
{
  "a": 2,
  "b": 3,
  "c": 4
}

echo '[1,2,3]' | jq 'map(1 + .)'
[
  2,
  3,
  4
]

Delete a key

jq 'del(filter)' json.data

Delete the key value described by the filter using del().

jq supports pipelines, which are like pipelines in linux commands -- Taking the output of the previous command as the input of the following command.

2, vim

Related commands in command line mode

1. Move the cursor

• h: ← shift left
• l: → shift right
• j: ↓ move down
• k: ↑ move up
• gg: move the cursor to the beginning of the file
• G: Move the cursor to the end of the file
• 0: the cursor moves to the beginning of the line
• $: cursor moves to end of line
• 123G: jump to line 123

2. Delete characters

• x: Delete one character after the cursor, which is equivalent to Del
• 10: Delete the character before the cursor, which is equivalent to Backspace
• dw: delete the word at the beginning of the cursor, including the character where the cursor is located
• The cursor must move over the first character of the deleted word
• d0: delete all contents of the line before the cursor, excluding the character of the cursor
• D(d $): all contents of the line after deleting the cursor, including the character where the cursor is located
• dd: delete the line where the cursor is located
• n dd deletes the specified number of rows
• It's not really deleted, it's actually cut

3. Undo operation

• u: Undo step by step
• Ctr-r: cancel revocation

4. Copy and paste

• yy: copy the current row, n yy copy n rows
• p: Open a new line down at the cursor position and paste
• P: Start pasting from the line where the cursor is located

5. Visual mode

• v: Move by word
• Used with h, j, k and l
• Use y to copy the selection

6. Find operation

• / hello - > find hello backwards from the cursor position
• n: Next
• N: Previous
• ? hello - > find hello from the cursor position
• n: Previous
• N: Next
• Use # to find the word to query

7. Replacement operation

• r: replace current character

8. Text line movement

• >>: shift text line right
• < <: move text line left

9.Man Page

• Move the cursor over the function and Shift-k cursor over the function
• 3Shift-k, see ManPage in Chapter 3

Related commands in text mode

1. Enter the input mode

• i: insert one character before the cursor
• 1: I nsert line beginning
• a: Insert one character after cursor
• A: Insert row not
• o: Open a new line down and insert the beginning of the line
• O: Open a new line up and insert the beginning of the line
• s: Delete the character of the cursor
• S: Delete current row

Related commands in last line mode

1. Line jump

• : 123 - > jump to line 123

2. Replacement

(1) Replace one line

• : s/abc/123 ​
• ->Replace the first abc in the current line with 123
• : s/abc/123/g ​
• ->Replace all abc in the current line with 123

(2) Replace all

• ​ :%s/abc/123 ​
• ->Replace the first abc in all rows with 123
• :%s/abc/123/g ​
• ->Replace abc in all lines with 123

(3) Replace specified row

• ​ :10,30s/abc/123/g ​
• ->Replace abc in lines 10-30 with 123

3. Execute shell command

• Input in the last line mode!, Followed by command

4. Split screen operation

(1) Enter split screen mode

• Command: sp divides the screen into two parts -- > horizontal
• Command: vsp divides the screen into two parts -- > vertical
• Command: sp(vsp) + file name horizontal or vertical split window displays two different files

(2) Exit split screen mode

• Command: wqall save and exit all screens
• Command: wq save and exit the screen where the cursor is located
• Ctrl+ww switch between two screens

reference resources: https://blog.csdn.net/Cheat1173010256/article/details/118230562