Foundations of Data Science unit IV

 

UNIT-IV
pandas Data Structures

Pandas – Series, DataFrame and Essential Functionality

1. Explain the Series, Data Frame?

Series:

A Series is a one-dimensional array-like object containing a sequence of values and an associated array of data labels, called its index. The simplest Series is formed from only an array of data.

>>> import numpy as np
>>> import pandas as pd

>>> obj = pd.Series([4, 7, -5, 3])
>>> obj
0    4
1    7
2   -5
3    3
dtype: int64

We did not specify an index for the data, so a default index consisting of the integers 0 through N - 1 is created.

You can get the array representation and index object of the Series via its values and index attributes, respectively.

>>> obj.values
array([ 4,  7, -5,  3])

>>> obj.index
RangeIndex(start=0, stop=4, step=1)

Create a Series with an index for the data points with a label:

>>> obj2 = pd.Series([4, 7, -5, 3], index=['d', 'b', 'a', 'c'])

>>> obj2
d    4
b    7
a   -5
c    3
dtype: int64

>>> obj2.index
Index(['d', 'b', 'a', 'c'], dtype='object')

Compared with NumPy arrays, you can use labels in the index when selecting single values or a set of values:

>>> obj2['a']
-5

>>> obj2['d'] = 6

>>> obj2[['c', 'a', 'd']]
c    3
a   -5
d    6
dtype: int64

---

DataFrame:

A DataFrame represents a rectangular table of data and contains an ordered collection of columns, each of which can be a different value type (numeric, string, boolean, etc.).

The DataFrame has both a row and column index; it can be thought of as a dict of Series all sharing the same index.

There are many ways to construct a DataFrame, though one of the most common is from a dict of equal-length lists or NumPy arrays.

>>> data = {
...     'state': ['Ohio', 'Ohio', 'Ohio', 'Nevada', 'Nevada', 'Nevada'],
...     'year': [2000, 2001, 2002, 2001, 2002, 2003],
...     'pop': [1.5, 1.7, 3.6, 2.4, 2.9, 3.2]
... }

>>> frame = pd.DataFrame(data)

The resulting DataFrame will have its index assigned automatically as with Series, and the columns are placed in sorted order.

     pop  state  year
0    1.5  Ohio   2000
1    1.7  Ohio   2001
2    3.6  Ohio   2002
3    2.4  Nevada 2001
4    2.9  Nevada 2002
5    3.2  Nevada 2003

For large DataFrames, the head method selects only the first five rows:

>>> frame.head()

If you specify a sequence of columns, the DataFrame's columns will be arranged in that order:

>>> pd.DataFrame(
...     data,
...     columns=['year', 'state', 'pop']
... )

2. Explain the Dropping Entries ?

Dropping one or more entries from an axis is easy if you already have an index array or list without those entries.

As that can require a bit of munging and set logic, the drop method will return a new object with the indicated value or values deleted from an axis.

>>> obj = pd.Series(np.arange(5.), index=['a', 'b', 'c', 'd', 'e'])
>>> obj

a    0.0
b    1.0
c    2.0
d    3.0
e    4.0
dtype: float64

>>> new_obj = obj.drop('c')
>>> new_obj

a    0.0
b    1.0
d    3.0
e    4.0
dtype: float64

>>> obj.drop(['d', 'c'])

a    0.0
b    1.0
e    4.0
dtype: float64

Dropping Entries from DataFrame

With DataFrame, index values can be deleted from either axis.

>>> data = pd.DataFrame(
...     np.arange(16).reshape((4, 4)),
...     index=['Ohio', 'Colorado', 'Utah', 'New York'],
...     columns=['one', 'two', 'three', 'four']
... )

>>> data

          one  two  three  four
Ohio        0    1      2     3
Colorado    4    5      6     7
Utah        8    9     10    11
New York   12   13     14    15

Calling drop with a sequence of labels will drop values from the row labels (axis 0):

>>> data.drop(['Colorado', 'Ohio'])

          one  two  three  four
Utah        8    9     10    11
New York   12   13     14    15

You can drop values from the columns by passing axis=1:

>>> data.drop('two', axis=1)

          one  three  four
Ohio        0      2     3
Colorado    4      6     7
Utah        8     10    11
New York   12     14    15

3. Explain the Indexing, Selection, and Filtering?

Series indexing (obj[...]) works analogously to NumPy array indexing, except you can use the Series's index values instead of only integers.

>>> import numpy as np
>>> import pandas as pd

>>> obj = pd.Series(np.arange(4.), index=['a', 'b', 'c', 'd'])
>>> obj
a    0.0
b    1.0
c    2.0
d    3.0
dtype: float64

>>> obj[1]
1.0

>>> obj[2:4]
c    2.0
d    3.0
dtype: float64

>>> obj[['b', 'a', 'd']]
b    1.0
a    0.0
d    3.0
dtype: float64

>>> obj[[1, 3]]
b    1.0
d    3.0
dtype: float64

>>> obj[obj < 2]
a    0.0
b    1.0
dtype: float64

Slicing with Labels:

Slicing with labels behaves differently than normal Python slicing in that the end point is inclusive:

>>> obj['b':'c']

b    1.0
c    2.0
dtype: float64

Setting using these methods modifies the corresponding section of the Series:

>>> obj['b':'c'] = 5
>>> obj

a    0.0
b    5.0
c    5.0
d    3.0
dtype: float64

---

Indexing into a DataFrame: 4M

Indexing into a DataFrame is for retrieving one or more columns either with a single value or sequence.

>>> data = pd.DataFrame(
...     np.arange(16).reshape((4, 4)),
...     index=['Ohio', 'Colorado', 'Utah', 'New York'],
...     columns=['one', 'two', 'three', 'four']
... )

>>> data

          one  two  three  four
Ohio        0    1      2     3
Colorado    4    5      6     7
Utah        8    9     10    11
New York   12   13     14    15

>>> data['two']

Ohio          1
Colorado      5
Utah          9
New York     13
Name: two, dtype: int64

>>> data[['three', 'one']]

          three  one
Ohio           2    0
Colorado       6    4
Utah          10    8
New York      14   12

Boolean Array Selection

Slicing or selecting data with a Boolean array:

>>> data[:2]

          one  two  three  four
Ohio        0    1      2     3
Colorado    4    5      6     7

>>> data[data['three'] > 5]

          one  two  three  four
Colorado    4    5      6     7
Utah        8    9     10    11
New York   12   13     14    15

Another use case is in indexing with a Boolean DataFrame, such as produced by a scalar comparison:

>>> data < 5

             one    two  three   four
Ohio        True   True   True   True
Colorado    True  False  False  False
Utah       False  False  False  False
New York   False  False  False  False

>>> data[data < 5]

          one  two  three  four
Ohio        0    0      0     0
Colorado    4    5      6     7
Utah        8    9     10    11
New York   12   13     14    15

---

Selection with loc and iloc:  4M

The special indexing operators loc (location) and iloc (integer location) enable you to select a subset of the rows and columns from a DataFrame with NumPy-like notation using either axis labels (loc) or integers (iloc).

Select a single row and multiple columns by label:

>>> data.loc['Colorado', ['two', 'three']]

two      5
three    6
Name: Colorado, dtype: int64

Selections with integers using iloc:

>>> data.iloc[2, [3, 0, 1]]

four     11
one       8
two       9
Name: Utah, dtype: int64

>>> data.iloc[2]

one       8
two       9
three    10
four     11
Name: Utah, dtype: int64

>>> data.iloc[[1, 2], [3, 0, 1]]

          four  one  two
Colorado     7    4    5
Utah        11    8    9

Both indexing functions work with slices in addition to single labels or lists of labels:

>>> data.loc[:, ['two', 'three']]

          two  three
Ohio        1      2
Colorado    5      6
Utah        9     10
New York   13     14

>>> data.iloc[:, :3]

          one  two  three
Ohio        0    1      2
Colorado    4    5      6
Utah        8    9     10
New York   12   13     14

---

Indexing Options with DataFrame:

Indexing Option	Description
df[val]	Select single column or sequence of columns from the DataFrame; special cases: Boolean array or DataFrame filtering
df.loc[val]	Select a single row or subset of rows from the DataFrame by label
df.loc[:, val]	Select a single column or subset of columns by label
df.loc[val1, val2]	Select both rows and columns by label
df.iloc[where]	Select a single row or subset of rows from the DataFrame by integer position
df.iloc[:, where]	Select a single column or subset of columns by integer position
df.iloc[where_i, where_j]	Select both rows and columns by integer position
df.at[label_i, label_j]	Select a single scalar value by row and column label
df.iat[i, j]	Select a single scalar value by row and column position
reindex method	Select either rows or columns by labels
get_value, set_value methods	Select single value by row and column label

---

4. Explain the Function Application and Mapping ?

NumPy ufuncs (universal functions) also work with pandas objects.

>>> import numpy as np
>>> import pandas as pd

>>> frame = pd.DataFrame(
...     np.random.randn(4, 3),
...     columns=['b', 'd', 'e'],
...     index=['Utah', 'Ohio', 'Texas', 'Oregon']
... )

>>> frame

               b         d         e
Utah      -0.542787  2.882672  0.850555
Ohio      -0.105346  0.500762  0.884608
Texas     -0.337065 -2.184005  0.264175
Oregon     0.715226  0.462233  0.400807

>>> np.abs(frame)

               b         d         e
Utah       0.542787  2.882672  0.850555
Ohio       0.105346  0.500762  0.884608
Texas      0.337065  2.184005  0.264175
Oregon     0.715226  0.462233  0.400807

The DataFrame's apply method does exactly this:

>>> f = lambda x: x.max() - x.min()

>>> frame.apply(f)

b    1.258012
d    5.066677
e    0.620433
dtype: float64

If you pass axis='columns' to apply, the function will be invoked once per row instead:

>>> frame.apply(f, axis='columns')

Utah      3.425458
Ohio      0.989954
Texas     2.448181
Oregon    1.177458
dtype: float64

The function passed to apply need not return a scalar value; it can also return a Series with multiple values:

>>> def f(x):
...     return pd.Series([x.min(), x.max()], index=['min', 'max'])

>>> frame.apply(f)

            b         d         e
min   -0.542787 -2.184005  0.264175
max    0.715226  2.882672  0.884608

To compute a formatting function from each floating-point value in frame, you can do this with applymap:

>>> format = lambda x: '%.2f' % x

>>> frame.applymap(format)

           b      d     e
Utah      -0.54   2.88  0.85
Ohio      -0.11   0.50  0.88
Texas     -0.34  -2.18  0.26
Oregon     0.72   0.46  0.40

The reason for the name applymap is that Series has a map method for applying an element-wise function:

>>> frame['e'].map(format)

Utah      0.85
Ohio      0.88
Texas     0.26
Oregon    0.40
Name: e, dtype: object

---

5. Explain the Sorting and Ranking ? 10M

Sorting:

Sorting a dataset by some criterion is another important built-in operation. To sort lexicographically by row or column index, use the sort_index method, which returns a new, sorted object.

>>> obj = pd.Series(range(4), index=['c', 'a', 'b', 'd'])

>>> obj
c    0
a    1
b    2
d    3
dtype: int64

>>> obj.sort_index()

a    1
b    2
c    0
d    3
dtype: int64

With a DataFrame, you can sort by index on either axis:

>>> frame = pd.DataFrame(
...     np.arange(8).reshape((2, 4)),
...     index=['three', 'one'],
...     columns=['d', 'a', 'b', 'c']
... )

>>> frame

       d  a  b  c
three  0  1  2  3
one    4  5  6  7

>>> frame.sort_index()

       d  a  b  c
one    4  5  6  7
three  0  1  2  3

>>> frame.sort_index(axis=1)

       a  b  c  d
three  1  2  3  0
one    5  6  7  4

The data is sorted in ascending order by default, but can be sorted in descending order, too:

>>> frame.sort_index(axis=1, ascending=False)

       d  c  b  a
three  0  3  2  1
one    4  7  6  5

To sort a Series by its values, use its sort_values method:

>>> obj = pd.Series([4, 7, -3, 2])

>>> obj.sort_values()

2   -3
3    2
0    4
1    7
dtype: int64

---

Ranking:

Ranking assigns ranks from one through the number of valid data points in an array.

The rank methods for Series and DataFrame are the place to look; by default rank breaks ties by assigning each group the mean rank.

>>> obj = pd.Series([7, -5, 7, 4, 2, 0, 4])

>>> obj.rank()

0    6.5
1    1.0
2    6.5
3    4.5
4    3.0
5    2.0

6    4.5
dtype: float64
6. Write Summarizing and Computing some Descriptive Statistics? 10M
Pandas objects are equipped with a set of common mathematical and statistical methods. 
Most of these fall into the category of reductions or summary statistics, methods that 
extract a single value (like sum() or mean()) from a Series or a Series of values from
the rows or columns of a DataFrame. Pandas objects also have built-in handling for 
missing data.
Consider a small DataFrame:
>>> import numpy as np
>>> import pandas as pd

>>> df = pd.DataFrame([[1.4, np.nan], [7.1, -4.5],
...                    [np.nan, np.nan], [0.75, -1.3]],
...                   index=['a', 'b', 'c', 'd'],
...                   columns=['one', 'two'])

>>> df
    one   two
a   1.40  NaN
b   7.10 -4.5
c   NaN   NaN
d   0.75 -1.3
Calling DataFrame's sum method returns a Series containing column sums:
>>> df.sum()
one    9.25
two   -5.80
dtype: float64
Passing axis='columns' or axis=1 sums across the columns instead:
>>> df.sum(axis='columns')
a    1.40
b    2.60
c    0.00
d   -0.55
dtype: float64
NA values are excluded unless the entire slice (row or column in this case) is NA.
This can be disabled with the skipna option:
>>> df.sum(axis='columns', skipna=False)
a      NaN
b     1.300
c      NaN
d    -0.275
dtype: float64
Options for Reduction Methods: 
Method
Description:
axis
Axis to reduce over; 0 for DataFrame's rows and 1 for columns
skipna
Exclude missing values; True by default
level
Reduce grouped by level if the axis is hierarchically indexed
Some methods, like idxmin and idxmax, return indirect statistics like the index
value where the minimum or maximum values are attained:
>>> df.idxmax()
one    b
two    d
dtype: object
Other Methods and Accumulations
>>> df.cumsum()
     one  two
a    1.40  NaN
b    8.50 -4.5
c     NaN  NaN
d    9.25 -5.8
7. Explain the Unique Values, Value Counts, 
    and Membership? 10M
Another class of related methods extracts information about the values 
contained in a one-dimensional Series.
Consider this example:
>>> import numpy as np
>>> import pandas as pd

>>> obj = pd.Series(['c', 'a', 'd', 'a', 'a', 'b', 'b', 'c', 'c'])
The first function is unique, which gives you an array of the unique 
values in a Series:
>>> uniques = obj.unique()
>>> uniques
array(['c', 'a', 'd', 'b'], dtype=object)
value_counts computes a Series containing value frequencies:
>>> obj.value_counts()
c    3
a    3
b    2
d    1
Name: count, dtype: int64
isin performs a vectorized set membership check and can be useful 
in filtering a dataset down to a subset of values in a Series or 
columns in a DataFrame.
>>> obj
0    c
1    a
2    d
3    a
4    a
5    b
6    b
7    c
8    c
dtype: object
>>> mask = obj.isin(['b', 'c'])
>>> mask
0    True
1    False
2    False
3    False
4    False
5    True
6    True
7    True
8    True
dtype: bool
>>> obj[mask]

5    b
6    b
7    c
8    c
dtype: object
Related to isin is the Index.get_indexer method, which gives you 
an index array from an array of possibly non-distinct values into
 another array of distinct values:
>>> to_match = pd.Series(['c', 'a', 'b', 'b', 'c', 'w'])
>>> unique_vals = pd.Series(['c', 'b', 'a'])

>>> pd.Index(unique_vals).get_indexer(to_match)
array([0, 2, 1, 1, 0, 2])
Example with Multiple DataFrame Columns:
You may want to compute a histogram on multiple related columns in a DataFrame:
>>> data = pd.DataFrame({
...     'Qu1': [1, 3, 4, 3, 4],
...     'Qu2': [2, 3, 1, 2, 3],
...     'Qu3': [1, 5, 2, 4, 4]
... })

>>> data

   Qu1  Qu2  Qu3
0    1    2    1
1    3    3    5
2    4    1    2
3    3    2    4
4    4    3    4
>>> result = data.apply(pd.value_counts).fillna(0)
>>> result
8. How to Reading and Writing Data in Text Format? 
                                               10M IMP
Pandas features a number of functions for reading tabular data as a 
DataFrame object.
Parsing Functions in Pandas: 4M
Optional Arguments:
The optional arguments for these functions may fall into a few categories:
Indexing:

It can be one or more columns as the returned DataFrame, and whether to
get columnnames from the file, the user, or not at all.
Type inference and data conversion:

This includes the user-defined value conversions and custom list of 
missing value markers.
Date/time parsing:

It includes combining capability, including combining date and time 
information spread over multiple columns into a single column in the result.
Iterating:

It supports for iterating over chunks of very large files.
Uneeded data issues:

It includes skipping rows or a footer, comments, or other minor things like 
numeric data with thousands separated by commas.
Reading Text Files in Pieces:
We can use read_csv to read it into a DataFrame:
>>> import numpy as np
>>> import pandas as pd

>>> df = pd.read_csv('examples/ex1.csv')

>>> df
   a  b  c  d  message
0  0  1  2  3  hello
1  5  6  7  8  world
2  9 10 11 12  foo
Using the index_col argument:
>>> pd.read_csv('examples/ex1.csv', index_col='message')
         a  b  c  d
message
hello    0  1  2  3
world    5  6  7  8
foo      9 10 11 12
Writing Data to Text Format:
Other delimiters can be used, of course, including writing to sys.stdout
so it prints the text result to the console:
>>> import sys

>>> data.to_csv(sys.stdout, sep='|')
|a|b|c|d|message
0|1|2|3|hello
1|5|6|7|world
2|9|10|11|12|foo
Series also has a to_csv method:
>>> dates = pd.date_range('1/1/2000', periods=7)

>>> s = pd.Series(np.arange(7), index=dates)

>>> s.to_csv('examples/ex1.csv')
Note: The ex1.csv has some data that can be changed automatically 
below figure shows.

How to Working with Delimited Formats Explain? 4M
For any file with a single-character delimiter, you can use Python's
built-in csv module.
>>> import csv

>>> f = open('examples/ex1.csv')

>>> reader = csv.reader(f)

>>> for line in reader:
...     print(line)
Output:
['01-01-2000', '0']
['02-01-2000', '1']
['03-01-2000', '2']
['04-01-2000', '3']
['05-01-2000', '4']
['06-01-2000', '5']
['07-01-2000', '6']


Explain JSON Data ? 4M
JSON (short for JavaScript Object Notation) has become one 
of the standard formats for sending data by HTTP request between 
web browsers and other applications. It is a much more free-form
 data format than a tabular text format like CSV.
Example:
>>> obj = """
... {"name": "Wes",
...  "places_lived": ["United States", "Spain", "Germany"],
...  "pet": null,
...  "siblings": [
...      {"name": "Scott", "age": 30,
...       "pets": ["Zeus", "Zuko"]},
...      {"name": "Katie", "age": 38,
...       "pets": ["Sixes", "Stache", "Cisco"]}
...  ]
... }
... """
>>> result = json.loads(obj)
>>> result
{'name': 'Wes',
 'places_lived': ['United States', 'Spain', 'Germany'],
 'pet': None,
 'siblings': [
     {'name': 'Scott', 'age': 30,
      'pets': ['Zeus', 'Zuko']},
     {'name': 'Katie', 'age': 38,
      'pets': ['Sixes', 'Stache', 'Cisco']}
 ]}
Conveniently, you can pass a list of dicts (which were previously JSON objects)
 to the DataFrame constructor and select a subset of the data fields:
>>> siblings = pd.DataFrame(
...     result['siblings'],
...     columns=['name', 'age']
... )

>>> siblings
    name   age
0   Scott   30
1   Katie   38