pandas uses the floating-point value NaN to represent missing data in floating-point and non floating-point arrays:

In [14]: string_data = Series(['aardvark','artichoke',np.nan,'avocado'])
In [15]: string_data
Out[15]:
0     aardvark
1    artichoke
2          NaN
3      avocado
dtype: object
In [16]: string_data.isnull()
Out[16]:
0    False
1    False
2     True
3    False
dtype: bool

 

python's built-in None value is also treated as NA:

In [17]: string_data[0] = None
In [18]: string_data.isnull()
Out[18]:
0     True
1    False
2     True
3    False
dtype: bool

 

1. Filter out missing data

dropna is used to return a Series with only non empty data and index values:

In [20]: data = Series([1,NA,3.5,NA,7])
In [21]: data.dropna()
Out[21]:
0    1.0
2    3.5
4    7.0
dtype: float64

 

Through Boolean index, you can also:

In [22]: data[data.notnull()]
Out[22]:
0    1.0
2    3.5
4    7.0
dtype: float64

 

For DataFrame, dropna discards rows with missing values by default

In [23]: data = DataFrame([[1.,6.5,3.],[1.,NA,NA],[NA,NA,NA],[NA,6.5,3.]])
In [24]: cleaned = data.dropna()
In [25]: data
Out[25]:
     0    1    2
0  1.0  6.5  3.0
1  1.0  NaN  NaN
2  NaN  NaN  NaN
3  NaN  6.5  3.0
In [26]: cleaned
Out[26]:
     0    1    2
0  1.0  6.5  3.0

 

Incoming how="all" discard all NaN rows:

In [27]: data.dropna(how="all")
Out[27]:
     0    1    2
0  1.0  6.5  3.0
1  1.0  NaN  NaN
3  NaN  6.5  3.0

 

Discard columns with NaN:

In [28]: data[4] = NA
In [29]: data
Out[29]:
     0    1    2   4
0  1.0  6.5  3.0 NaN
1  1.0  NaN  NaN NaN
2  NaN  NaN  NaN NaN
3  NaN  6.5  3.0 NaN
In [30]: data.dropna(axis=1,how="all")
Out[30]:
     0    1    2
0  1.0  6.5  3.0
1  1.0  NaN  NaN
2  NaN  NaN  NaN
3  NaN  6.5  3.0

 

thresh can choose how many useful rows to filter

In [41]: df
Out[41]:
          0         1         2
0 -0.184676       NaN       NaN
1  0.565214       NaN       NaN
2  0.440203       NaN       NaN
3  0.188283       NaN  0.146847
4  1.696903       NaN  0.554640
5 -1.287915  0.139527 -0.494558
6  0.854922  0.299511  0.773247
In [42]: df.dropna(thresh=2)  # thresh=2 Indicates that there are at least two non empty data
Out[42]:
          0         1         2
3  0.188283       NaN  0.146847
4  1.696903       NaN  0.554640
5 -1.287915  0.139527 -0.494558
6  0.854922  0.299511  0.773247
In [43]: df.dropna(thresh=1)
Out[43]:
          0         1         2
0 -0.184676       NaN       NaN
1  0.565214       NaN       NaN
2  0.440203       NaN       NaN
3  0.188283       NaN  0.146847
4  1.696903       NaN  0.554640
5 -1.287915  0.139527 -0.494558
6  0.854922  0.299511  0.773247

 

2. Fill in missing data

fillna is the main method to fill in the default data:

In [9]: df.fillna(0)
Out[9]:
          0         1         2
0  0.863556  0.000000  0.000000
1 -0.099558  0.000000  0.000000
2 -0.605804  0.000000  0.000000
3 -0.934688  0.000000 -1.198976
4  0.741383  0.000000  0.229845
5 -1.415495  0.511485 -0.086808
6 -0.748325  0.437964 -2.458319

 

Call fillna through the dictionary to fill in different values for different columns:

In [11]: df.fillna({1:0.5,2:-1})
Out[11]:
          0         1         2
0  0.863556  0.500000 -1.000000
1 -0.099558  0.500000 -1.000000
2 -0.605804  0.500000 -1.000000
3 -0.934688  0.500000 -1.198976
4  0.741383  0.500000  0.229845
5 -1.415495  0.511485 -0.086808
6 -0.748325  0.437964 -2.458319