Python Pandas Tutorial: Series Methods

The Series is one of the most common Pandas data structures. It is similar to a Python list and is used to represent a column of data. After looking into the basics of creating and initializing a pandas Series object, we now delve into some common usage patterns and methods.

Series Information

After a Series is created, it is most important to look into various details of its structure. These include the size of the series, whether there are NaNs in it, etc. Here are some commonly used methods which help clarify the situation.

Size of the Series

There are several methods to determine how big the Series is.

The first is the attribute shape, which returns a tuple.

a = pd.Series(random.sample(xrange(100), 6))
print a.shape

# prints
(6,)

We also have the count() method which returns the size of the Series as an integer.

print a.count()

# prints
6

However, note that count() only reports the number of non-NaN elements, while shape  reports both.

Another attribute for getting the count of elements is size. It reports the count as an integer and includes NaN elements if any.

a = pd.Series(random.sample(xrange(100), 6))
print 'count of a =>', a.count(), '\n'
b = a.append(pd.Series(np.nan, index=list('abcd')), ignore_index=True)
print 'b => ', b, '\n'
print 'count of b =>', b.count(), '\n'
print 'shape of b =>', b.shape, '\n'
print 'size of b =>', b.size

# prints
count of a => 6 

b =>  0    76.0
1    92.0
2    75.0
3    60.0
4    42.0
5    44.0
6     NaN
7     NaN
8     NaN
9     NaN
dtype: float64 

count of b => 6 

shape of b => (10,)

size of b => 10

Series Details

Get some detailed stats on the Series using describe(). This method returns a Series object with the index (or labels) as shown.

x = pd.Series(random.sample(xrange(100), 6))
x.describe()

# prints
count     6.000000
mean     60.500000
std      30.742479
min      20.000000
25%      44.000000
50%      53.500000
75%      86.250000
max      98.000000
dtype: float64

Head and Tail

Show the first 5 or last 5 rows of the Series using head() or tail().

x = pd.Series(random.sample(xrange(100), 10))
print x, '\n'
print x.head(), '\n'
print x.tail(), '\n'

# prints
0    24
1    39
2    56
3    77
4    81
5    26
6     8
7    87
8    34
9    68
dtype: int64 

0    24
1    39
2    56
3    77
4    81
dtype: int64 

5    26
6     8
7    87
8    34
9    68
dtype: int64

Add Elements to Series

Adding elements to a Series is accomplished by using append(). The argument must be a single Series object, or a list (or tuple) of Series objects.

x  = pd.Series(random.sample(xrange(100), 6))
print x, '\n'
print 'appended =>\n', x.append([pd.Series(2), pd.Series([3, 4, 5])])

# prints
0    62
1    29
2    20
3    69
4    53
5    22
dtype: int64 

appended =>
0    62
1    29
2    20
3    69
4    53
5    22
0     2
0     3
1     4
2     5
dtype: int64

You might notice the oddball labels after appending. Each Series is appended with a default index starting from 0, regardless of whether this creates duplicate labels. One way to fix this is to specify ignore_index=True to ensure re-labeling.

print 'appended =>\n', x.append([pd.Series(2), pd.Series([3, 4, 5])], ignore_index=True)

# prints
appended =>
0    62
1    29
2    20
3    69
4    53
5    22
6     2
7     3
8     4
9     5
dtype: int64

What if you don’t want to re-label but ensure that append() succeeds only if the labels are unique? Keep your precious labels intact and unique by specifying verify_integrity=True.

print 'appended =>\n', x.append([pd.Series(2), pd.Series([3, 4, 5])], verify_integrity=True)

# throws exception
ValueError: Indexes have overlapping values: [0, 1, 2]

Delete Elements

You can delete elements from a Series using the following methods.

By Label

Use drop() and specify a single label or a list of labels to drop.

x = pd.Series(random.sample(xrange(100), 6), index=list('ABCDEF'))
print x, '\n'
print 'drop one =>\n', x.drop('C'), '\n'
print 'drop many =>\n', x.drop(['C', 'D'])

# prints
A    67
B    18
C     1
D    54
E    38
F     3
dtype: int64 

drop one =>
A    67
B    18
D    54
E    38
F     3
dtype: int64 

drop many =>
A    67
B    18
E    38
F     3
dtype: int64

Duplicate Elements

Get rid of duplicate elements by invoking drop_duplicates().

x = pd.Series([1, 2, 2, 4, 5, 7, 3, 4])
print x, '\n'
print 'drop duplicates =>\n', x.drop_duplicates(), '\n'

# prints
0    1
1    2
2    2
3    4
4    5
5    7
6    3
7    4
dtype: int64 

drop duplicates =>
0    1
1    2
3    4
4    5
5    7
6    3
dtype: int64

By default, the method retains the first repeated value. Get rid of all duplicates (including the first) by specifying keep=False.

drop all duplicates =>
0    1
4    5
5    7
6    3
dtype: int64

NaN Elements

Use the dropna() to drop elements without a value (NaN).

x = pd.Series([1, 2, 3, 4, np.nan, 5, 6])
print x, '\n'
print 'drop na =>\n', x.dropna()

# prints
0    1.0
1    2.0
2    3.0
3    4.0
4    NaN
5    5.0
6    6.0
dtype: float64 

drop na =>
0    1.0
1    2.0
2    3.0
3    4.0
5    5.0
6    6.0
dtype: float64

Replace NaN Elements

When you want to replace NaN elements in a Series, use fillna().

x = pd.Series([1, 2, 3, 4, np.nan, 5, 6])
print x, '\n'
print 'fillna w/0 =>\n', x.fillna(0)

# prints
0    1.0
1    2.0
2    3.0
3    4.0
4    NaN
5    5.0
6    6.0
dtype: float64 

fillna w/0 =>
0    1.0
1    2.0
2    3.0
3    4.0
4    0.0
5    5.0
6    6.0
dtype: float64

Select Elements

Select elements from a Series based on various conditions as follows.

In a Range

Use the between() method, which returns a Series of boolean values indicating whether the element lies within the range.

a = pd.Series(random.sample(xrange(100), 10))
print a
print a.between(30, 50)

# prints
0    85
1    42
2    63
3    69
4    81
5    45
6    50
7    72
8    66
9    34
dtype: int64
0    False
1     True
2    False
3    False
4    False
5     True
6     True
7    False
8    False
9     True
dtype: bool

You can use this the returned boolean Series as a predicate into the original Series.

print a[a.between(30, 50)]

# prints
1    42
5    45
6    50
9    34
dtype: int64

Using a Function

Select elements using a predicate function as the argument to select().

x = pd.Series(random.sample(xrange(100), 6))
print x, '\n'
print 'select func =>\n', x.select(lambda a: x.iloc[a] > 20)

# prints
0    83
1    96
2    29
3    15
4    28
5    12
dtype: int64 

select func =>
0    83
1    96
2    29
4    28
dtype: int64

By List of Labels

Use filter(items=[..]) with the labels to be selected in a list.

x = pd.Series([1, 2, 3, 4, np.nan, 5, 6])
print x, '\n'
print 'filtered =>\n', x.filter(items=[1, 2, 6])

# prints
0    1.0
1    2.0
2    3.0
3    4.0
4    NaN
5    5.0
6    6.0
dtype: float64 

filtered =>
1    2.0
2    3.0
6    6.0
dtype: float64

Regex Match on Label

Select labels to filter using a regular expression match with filter(regex=’..’).

x = pd.Series({'apple': 1.99,
              'orange': 2.49,
              'banana': 0.99,
              'grapes': 1.49,
              'melon': 3.99})
print x, '\n'
print 'regex filter =>\n', x.filter(regex='a$')

# prints
apple     1.99
banana    0.99
grapes    1.49
melon     3.99
orange    2.49
dtype: float64 

regex filter =>
banana    0.99
dtype: float64

Substring Match on Label

Use the filter(like=’..’) version to perform a substring match on the labels to be selected.

print 'like filter =>\n', x.filter(like='an')

# prints
like filter =>
banana    0.99
orange    2.49
dtype: float64

Sorting

Ah! Sorting. The all important functionality when playing with data.

Here is how you can sort a Series by labels or by value.

By Index (or Labels)

Use sort_index().

x = pd.Series(random.sample(xrange(100), 6), index=random.sample(map(chr, xrange(ord('a'), ord('z'))), 6))
print x, '\n'
print 'sort by index: =>\n', x.sort_index(), '\n'

# prints
p    37
e    44
b    93
l    75
n     4
s    83
dtype: int64 

sort by index: =>
b    93
e    44
l    75
n     4
p    37
s    83
dtype: int64

By Values

Use sort_values() to sort by the values.

print 'sort by value: =>\n', x.sort_values()

# prints
sort by value: =>
n     4
p    37
e    44
l    75
s    83
b    93
dtype: int64

Summary

This article covers some commonly used methods to deal with a Series object, including methods to retrieve general information about a Series, modifying a Series, selection, and sorting.