python data analysis tool
The data analysis function of python itself is not strong, so we need to install some third-party extension libraries to enhance its corresponding functions.
Extension library related to python data analysis and mining;
| expanded memory bank | brief introduction |
|---|---|
| NumPy | Provide array support and corresponding efficient processing functions |
| SciPy | Provide matrix support and matrix related numerical calculation modules |
| Matplotlib | Powerful data visualization tools and drawing library |
| pandas | Powerful and flexible data analysis and exploration tools |
| StatsModels | Statistical modeling and econometrics, including descriptive statistics, statistical model estimation and inference |
| scikit-learn | Support powerful machine learning libraries such as regression, classification and clustering |
| Keras | Deep learning library is used to establish neural network and deep learning model |
| Gensim | It is used as a text topic model library, which may be used in text mining |
Of course, there are other libraries. For example, the pilot library can be used for image processing, OpenCV can be used for video processing, GMPY2 can be used for high-precision computing, etc. Of course, when dealing with problems, we can search relevant information on the Internet.
If Anaconda distribution is used, many libraries already come with them, such as NumPy, SciPy, Matplotlib, pandas and scikit learn.
Of course, if you use other compilers, you need to install the relevant library files yourself.
NumPy
- python does not provide the array function. Although the list can complete the basic array function, when the amount of data is large, the speed of using the list will be very slow;
- NumPy provides real array functions and functions for fast data processing;
- NumPy is a dependent Library of many advanced libraries;
Using NumPy to manipulate arrays
import numpy as np a = np.array([2, 0, 1, 5]) #Create array print(a) print(a[:3]) print(a.min()) a.sort() print(a) b = np.array(([[1,2,3], [4,5,6]])) print(b*b)
Output result: [2 0 1 5] [2 0 1] 0 [0 1 2 5] [[ 1 4 9] [16 25 36]]
Reference links:
Scipy
The functions of SciPy include optimization, linear algebra, integration, interpolation, fitting, special functions, fast Fourier transform, signal processing and image processing, ordinary differential equation solving and other calculations commonly used in science and engineering.
from scipy.optimize import fsolve #Import functions for solving equations
def f(x):
x1 = x[0]
x2 = x[1]
return [2*x1 - x2**2 - 1, x1**2 - x2 - 2]
result = fsolve(f, [1, 1])
print(result)
from scipy import integrate #Import integral function
def g(x): #Integral function definition
return (1 - x**2)**0.5
pi_2, err = integrate.quad(g, -1, 1) # Integration results and errors
print(pi_2 * 2)
print(err)
Output result: [1.91963957 1.68501606] 3.1415926535897967 1.0002354500215915e-09
Reference link:
Matplotlib
Matplotlib is the most famous drawing library, which is mainly used for two-dimensional drawing. Of course, it can also be used for simple three-dimensional drawing.
import numpy as np
import matplotlib.pyplot as plt
x = np.linspace(0, 10, 1000) #Independent variable of drawing
y = np.sin(x) + 1 #Dependent variable y
z = np.cos(x**2) + 1 #Dependent variable z
plt.figure(figsize=(8,4)) #Set image size
plt.plot(x, y, label = '$\sin x+1$', color = 'red', linewidth = 2) #Drawing, setting label, line color and line size
plt.plot(x, z, 'b--', label = '$\cos x^2+1$') #Drawing, setting label and line type
plt.xlabel('Time(s) ') #x-axis name
plt.ylabel('Volt') #y-axis name
plt.title('A Simple Example')
plt.ylim(0,2.2)# y-axis range
plt.legend()#Show Legend
plt.show()
Chinese font needs to be manually specified. The default font is Chinese font.
reference material:
pandas
pandas is the most powerful data analysis and exploration tool in python. pandas is very powerful;
- Support data addition, deletion, query and modification similar to SQL, and have rich data processing functions;
- Support time series analysis function; Support flexible data processing;
The basic data structures of pandas are Series and DataFrame.
import numpy as np
import pandas as pd
s = pd.Series([1, 2, 3], index=list('abc'))
d = pd.DataFrame([[1,2,3],[4,5,6]], columns=['a','b','c'])
d2 = pd.DataFrame(s)
print(d.head())
print(d.describe())
Output result:
a b c
0 1 2 3
1 4 5 6
a b c
count 2.00000 2.00000 2.00000
mean 2.50000 3.50000 4.50000
std 2.12132 2.12132 2.12132
min 1.00000 2.00000 3.00000
25% 1.75000 2.75000 3.75000
50% 2.50000 3.50000 4.50000
75% 3.25000 4.25000 5.25000
max 4.00000 5.00000 6.00000
#When reading the file, note that the storage path of the file cannot contain Chinese, otherwise the reading may be wrong;
pd.read_excel('data.xls') #Read Excel file and create DataFrame;
pd.read_csv('data.csv', encoding='utf-8') #When reading data in text format, encoding is generally used to specify the encoding;
Reference documents;
StatsModels
Pandas focuses on data reading, processing and exploration, while StatsModels pays more attention to data statistical modeling and analysis, which makes Python have the flavor of R language. StatsModels supports data interaction with pandas, so it combines with pandas to become a powerful data mining combination under python.
from statsmodels.tsa.stattools import adfuller as ADF #Import ADF inspection import pandas as pd import numpy as np print(ADF(np.random.rand(100))) #The returned results include ADF value, p value, etc
Output:
(-8.103123291388002, 1.2838791095546032e-12, 1, 98, {'1%': -3.4989097606014496, '5%': -2.891516256916761, '10%': -2.5827604414827157}, 30.91636795599902)
Reference link:
scikit-learn
This is a library related to machine learning. Scikit learn is a powerful machine learning toolkit under python. It provides a perfect machine learning toolkit, including data preprocessing, classification, regression, clustering, prediction, model analysis and so on.
Scikit learn relies on NumPy, SciPy and Matplotlib.
from sklearn.linear_model import LinearRegression #Import linear regression model model = LinearRegression() #Establish linear regression model print(model)
1) The interfaces provided by all models are: model for training model Fit() is fit(X,y) for supervised models and fit(X) for unsupervised models
2) The supervision model provides the following interfaces:
- model.predict(X_new): predict new samples;
- model.predict_proba(X_new): prediction probability, which is only useful for some models (such as LR);
- model.score(): the higher the score, the better the fit;
3) The unsupervised model provides the following interfaces:
- model.transform(): learn new base space from data;
- model.fit_transform(): learn a new base from the data and convert the data according to this set of bases;
Scikit learn itself provides some example data for us to learn. The more common ones are Anderson iris flower data set, handwritten graphic data set and so on.
from sklearn import datasets # Import dataset iris = datasets.load_iris() # Load dataset print(iris.data.shape) from sklearn import svm #Import SVM model clf = svm.LinearSVC() #Establish SVM classifier clf.fit(iris.data, iris.target) #Training model with data clf.predict([[5.0, 3.6, 1.3, 0.25]]) #After inputting the trained model, input new data for prediction print(clf.coef_) #View the parameters of the trained model
Output result: (150, 4) [[ 0.18423149 0.45122757 -0.8079383 -0.45071932] [ 0.05554602 -0.9001544 0.40811885 -0.96012405] [-0.85077276 -0.98663003 1.3810384 1.86530666]]
Reference link:
Keras
Artificial intelligence neural network is a powerful but simple model. It plays an important role in the fields of language processing, image recognition and so on. Keras library can be used to build neural networks. In fact, keras is not a simple neural network library, but a powerful deep learning library based on Theano. It can be used to build not only ordinary neural networks, but also various deep learning models, such as self encoder, cyclic neural networks, recursive neural networks, convolutional neural networks, etc.
To use Keras, you need to install TensorFlow package.
from keras.models import Sequential
from keras.layers.core import Dense, Dropout, Activation
from keras.optimizers import SGD
model = Sequential()#Model initialization
model.add(Dense(20,64))#Add the connection of the input layer (20 nodes) and the first hidden layer (64 nodes)
model.add(Activation('tanh'))#The first hidden layer uses tanh as the activation function
model.add(Dropout(0.5))#Use Dropout to prevent overfitting
model.add(Dense(64, 64))#Add the connection of the first hidden layer (64 nodes) and the second hidden layer (64 nodes)
model.add(Activation('tanh'))
model.add(Dropout(0.5))
model.add(Dense(64,1))#Add the connection of the second hidden layer (64 nodes) and the output layer (1 node)
model.add(Activation('sigmoid'))#The output layer uses sigmoid as the activation function
sgd = SGD(lr=0.1, decay=1e-6,momentum=0.9,nesterov=True)#Define solution algorithm
model.compile(loss='mean_squraed_error', optimizer=sgd)#Compile and generate the model, and the loss function is the sum of squares of the average error
model.fit(X_train, y_train, nb_epoch=20,batch_size=16)#Training model
score = model.evaluate(X_test, y_test, batch_size=16)#test model
reference material;
Gensim
Gensim is used to deal with language tasks, such as text similarity calculation, LDA, Word2Vec, etc. tasks in these fields often need more background knowledge.
import gensim, logging logging.basicConfig(format='%(asctime)s: %(levelname)s : %(message)s', level=logging.INFO) #logging is used to output training logs #Divide the sentences into words, and input each sentence in the form of word list sentences=[['first', 'sentence'],['secend','sentence']] #The above sentence is trained with vector model model = gensim.models.Word2Vec(sentences, min_count=1) print(model['sentence'])#Output the word vector of the word sentence
reference material: