WIXLIB

CHAPTERTHREEGETTING STARTED{{ header }}3.1 Package overviewpandas is a Python package providing fast, flexible, and expressive data structures designed to make working withrelational or labeled data both easy and intuitive. It aims to be the fundamental high-level building block for doingpractical, real world data analysis in Python. Additionally, it has the broader goal of becoming the most powerfuland flexible open source data analysis / manipulation tool available in any language. It is already well on its waytoward this goal.pandas is well suited for many different kinds of data: Tabular data with heterogeneously-typed columns, as in an SQL table or Excel spreadsheet Ordered and unordered (not necessarily fixed-frequency) time series data. Arbitrary matrix data (homogeneously typed or heterogeneous) with row and column labels Any other form of observational / statistical data sets. The data actually need not be labeled at all to be placedinto a pandas data structureThe two primary data structures of pandas, Series (1-dimensional) and DataFrame (2-dimensional), handle thevast majority of typical use cases in finance, statistics, social science, and many areas of engineering. For R users,DataFrame provides everything that Rs data.frame provides and much more. pandas is built on top of NumPyand is intended to integrate well within a scientific computing environment with many other 3rd party libraries.Here are just a few of the things that pandas does well: Easy handling of missing data (represented as NaN) in floating point as well as non-floating point data Size mutability: columns can be inserted and deleted from DataFrame and higher dimensional objects Automatic and explicit data alignment: objects can be explicitly aligned to a set of labels, or the user cansimply ignore the labels and let Series, DataFrame, etc. automatically align the data for you in computations Powerful, flexible group by functionality to perform split-apply-combine operations on data sets, for both aggregating and transforming data Make it easy to convert ragged, differently-indexed data in other Python and NumPy data structures intoDataFrame objects Intelligent label-based slicing, fancy indexing, and subsetting of large data sets Intuitive merging and joining data sets Flexible reshaping and pivoting of data sets11

pandas: powerful Python data analysis toolkit, Release 0.25.3 Hierarchical labeling of axes (possible to have multiple labels per tick) Robust IO tools for loading data from flat files (CSV and delimited), Excel files, databases, and saving / loadingdata from the ultrafast HDF5 format Time series-specific functionality: date range generation and frequency conversion, moving window statistics,moving window linear regressions, date shifting and lagging, etc.Many of these principles are here to address the shortcomings frequently experienced using other languages / scientificresearch environments. For data scientists, working with data is typically divided into multiple stages: munging andcleaning data, analyzing / modeling it, then organizing the results of the analysis into a form suitable for plotting ortabular display. pandas is the ideal tool for all of these tasks.Some other notes pandas is fast. Many of the low-level algorithmic bits have been extensively tweaked in Cython code. However,as with anything else generalization usually sacrifices performance. So if you focus on one feature for yourapplication you may be able to create a faster specialized tool. pandas is a dependency of statsmodels, making it an important part of the statistical computing ecosystem inPython. pandas has been used extensively in production in financial applications.3.1.1 Data on1D labeled homogeneously-typed arrayGeneral 2D labeled, size-mutable tabular structure with potentiallyheterogeneously-typed columnWhy more than one data structure?The best way to think about the pandas data structures is as flexible containers for lower dimensional data. Forexample, DataFrame is a container for Series, and Series is a container for scalars. We would like to be able to insertand remove objects from these containers in a dictionary-like fashion.Also, we would like sensible default behaviors for the common API functions which take into account the typicalorientation of time series and cross-sectional data sets. When using ndarrays to store 2- and 3-dimensional data, aburden is placed on the user to consider the orientation of the data set when writing functions; axes are consideredmore or less equivalent (except when C- or Fortran-contiguousness matters for performance). In pandas, the axes areintended to lend more semantic meaning to the data; i.e., for a particular data set there is likely to be a right way toorient the data. The goal, then, is to reduce the amount of mental effort required to code up data transformations indownstream functions.For example, with tabular data (DataFrame) it is more semantically helpful to think of the index (the rows) and thecolumns rather than axis 0 and axis 1. Iterating through the columns of the DataFrame thus results in more readablecode:for col in df.columns:series df[col]# do something with series12Chapter 3. Getting started

pandas: powerful Python data analysis toolkit, Release 0.25.33.1.2 Mutability and copying of dataAll pandas data structures are value-mutable (the values they contain can be altered) but not always size-mutable. Thelength of a Series cannot be changed, but, for example, columns can be inserted into a DataFrame. However, the vastmajority of methods produce new objects and leave the input data untouched. In general we like to favor immutabilitywhere sensible.3.1.3 Getting supportThe first stop for pandas issues and ideas is the Github Issue Tracker. If you have a general question, pandas communityexperts can answer through Stack Overflow.3.1.4 Communitypandas is actively supported today by a community of like-minded individuals around the world who contribute theirvaluable time and energy to help make open source pandas possible. Thanks to all of our contributors.If youre interested in contributing, please visit the contributing guide.pandas is a NumFOCUS sponsored project. This will help ensure the success of development of pandas as a worldclass open-source project, and makes it possible to donate to the project.3.1.5 Project governanceThe governance process that pandas project has used informally since its inception in 2008 is formalized in ProjectGovernance documents. The documents clarify how decisions are made and how the various elements of our community interact, including the relationship between open source collaborative development and work that may be fundedby for-profit or non-profit entities.Wes McKinney is the Benevolent Dictator for Life (BDFL).3.1.6 Development teamThe list of the Core Team members and more detailed information can be found on the peoples page of the governancerepo.3.1.7 Institutional partnersThe information about current institutional partners can be found on pandas website page.3.1.8 LicenseBSD 3-Clause LicenseCopyright (c) 2008-2012, AQR Capital Management, LLC, Lambda Foundry, Inc. and PyData, Development TeamAll rights reserved.Redistribution and use in source and binary forms, with or withoutmodification, are permitted provided that the following conditions are met:(continues on next page)3.1. Package overview13

pandas: powerful Python data analysis toolkit, Release 0.25.3(continued from previous page)* Redistributions of source code must retain the above copyright notice, thislist of conditions and the following disclaimer.* Redistributions in binary form must reproduce the above copyright notice,this list of conditions and the following disclaimer in the documentationand/or other materials provided with the distribution.* Neither the name of the copyright holder nor the names of itscontributors may be used to endorse or promote products derived fromthis software without specific prior written permission.THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THEIMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE AREDISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLEFOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIALDAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS ORSERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVERCAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USEOF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.{{ header }}3.2 10 minutes to pandasThis is a short introduction to pandas, geared mainly for new users. You can see more complex recipes in the Cookbook.Customarily, we import as follows:In [1]: import numpy as npIn [2]: import pandas as pd3.2.1 Object creationSee the Data Structure Intro section.Creating a Series by passing a list of values, letting pandas create a default integer index:In [3]: s pd.Series([1, 3, 5, np.nan, 6, 8])In [4]: sOut[4]:01.013.025.03NaN46.058.0dtype: float64Creating a DataFrame by passing a NumPy array, with a datetime index and labeled columns:14Chapter 3. Getting started

pandas: powerful Python data analysis toolkit, Release 0.25.3In [5]: dates pd.date range('20130101', periods 6)In [6]: datesOut[6]:DatetimeIndex(['2013-01-01', '2013-01-02', '2013-01-03', '2013-01-04','2013-01-05', '2013-01-06'],dtype 'datetime64[ns]', freq 'D')In [7]: df pd.DataFrame(np.random.randn(6, 4), index dates, columns list('ABCD'))In [8]: dfOut[8]:AB2013-01-01 1.832747 1.5153862013-01-02 -0.913436 0.0351412013-01-03 -1.323650 0.4273552013-01-04 0.509859 -2.7695862013-01-05 0.139488 -0.2593282013-01-06 -0.130327 2452-0.424347Creating a DataFrame by passing a dict of objects that can be converted to series-like.In [9]: df2 pd.DataFrame({'A': 1.,.:'B': pd.Timestamp('20130102'),.:'C': pd.Series(1, index list(range(4)), dtype 'float32'),.:'D': np.array([3] * 4, dtype 'int32'),.:'E': pd.Categorical(["test", "train", "test", "train"]),.:'F': 'foo'}).:In [10]: df2Out[10]:AB0 1.0 2013-01-021 1.0 2013-01-022 1.0 2013-01-023 1.0 ofoofoofooThe columns of the resulting DataFrame have different dtypes.In [11]: int32EcategoryFobjectdtype: objectIf youre using IPython, tab completion for column names (as well as public attributes) is automatically enabled. Heresa subset of the attributes that will be completed:In [12]: df2. TAB df2.Adf2.absdf2.add# noqa: E225, E999df2.booldf2.boxplotdf2.C(continues on next page)3.2. 10 minutes to pandas15

pandas: powerful Python data analysis toolkit, Release 0.25.3(continued from previous page)df2.add prefixdf2.add 2.applymapdf2.Ddf2.clipdf2.clip lowerdf2.clip upperdf2.columnsdf2.combinedf2.combine firstdf2.compounddf2.consolidateAs you can see, the columns A, B, C, and D are automatically tab completed. E is there as well; the rest of the attributeshave been truncated for brevity.3.2.2 Viewing dataSee the Basics section.Here is how to view the top and bottom rows of the frame:In [13]: df.head()Out[13]:AB2013-01-01 1.832747 1.5153862013-01-02 -0.913436 0.0351412013-01-03 -1.323650 0.4273552013-01-04 0.509859 -2.7695862013-01-05 0.139488 D-0.360634-1.106914-0.000698-0.865748-0.902452In [14]: df.tail(3)Out[14]:AB2013-01-04 0.509859 -2.7695862013-01-05 0.139488 -0.2593282013-01-06 -0.130327 -0.372906CD1.000521 -0.8657481.082034 -0.9024521.072236 -0.424347Display the index, columns:In [15]: df.indexOut[15]:DatetimeIndex(['2013-01-01', '2013-01-02', '2013-01-03', '2013-01-04','2013-01-05', '2013-01-06'],dtype 'datetime64[ns]', freq 'D')In [16]: df.columnsOut[16]: Index(['A', 'B', 'C', 'D'], dtype 'object')DataFrame.to numpy() gives a NumPy representation of the underlying data. Note that this can be an expensiveoperation when your DataFrame has columns with different data types, which comes down to a fundamental difference between pandas and NumPy: NumPy arrays have one dtype for the entire array, while pandas DataFrameshave one dtype per column. When you call DataFrame.to numpy(), pandas will find the NumPy dtype thatcan hold all of the dtypes in the DataFrame. This may end up being object, which requires casting every value to aPython object.For df, our DataFrame of all floating-point values, DataFrame.to numpy() is fast and doesnt require copyingdata.16Chapter 3. Getting started

pandas: powerful Python data analysis toolkit, Release 0.25.3In [17]: df.to numpy()Out[17]:array([[ 1.83274697e 00, 1.51538609e 00,-3.60634458e-01],[-9.13435768e-01, 3.51414290e-02,-1.10691447e 00],[-1.32365020e 00, 4.27354647e-01,-6.97782589e-04],[ 5.09859417e-01, -2.76958615e 00,-8.65747849e-01],[ 1.39488428e-01, -2.59327906e-01,-9.02451725e-01],[-1.30327388e-01, -3.72906082e-01,-4.24346700e-01]])1.79354724e 00,3.43748191e 00,8.35343007e-01,1.00052083e 00,1.08203428e 00,1.07223611e 00,For df2, the DataFrame with multiple dtypes, DataFrame.to numpy() is relatively expensive.In [18]: df2.to numpy()Out[18]:array([[1.0, Timestamp('2013-01-02[1.0, Timestamp('2013-01-02[1.0, Timestamp('2013-01-02[1.0, Timestamp('2013-01-02dtype ,1.0,1.0,1.0,1.0,3,3,3,3,'test', 'foo'],'train', 'foo'],'test', 'foo'],'train', 'foo']],Note: DataFrame.to numpy() does not include the index or column labels in the output.describe() shows a quick statistic summary of your data:In [19]: df.describe()Out[19]:ABcount 6.000000 6.000000mean0.019114 -0.237323std1.117102 1.415574min-1.323650 -2.76958625%-0.717659 -0.34451250%0.004581 -0.11209375%0.417267 0.329301max1.832747 osing your data:In [20]: 0327-0.3729061.072236-0.424347Sorting by an axis:In [21]: df.sort index(axis 1, ascending False)Out[21]:(continues on next page)3.2. 10 minutes to pandas17

pandas: powerful Python data analysis toolkit, Release 0.25.3(continued from previous 48-0.902452-0.424347CBA1.793547 1.515386 1.8327473.437482 0.035141 -0.9134360.835343 0.427355 -1.3236501.000521 -2.769586 0.5098591.082034 -0.259328 0.1394881.072236 -0.372906 -0.130327Sorting by values:In [22]: df.sort values(by 'B')Out[22]:AB2013-01-04 0.509859 -2.7695862013-01-06 -0.130327 -0.3729062013-01-05 0.139488 -0.2593282013-01-02 -0.913436 0.0351412013-01-03 -1.323650 0.4273552013-01-01 1.832747 698-0.3606343.2.3 SelectionNote: While standard Python / Numpy expressions for selecting and setting are intuitive and come in handy forinteractive work, for production code, we recommend the optimized pandas data access methods, .at, .iat, .locand .iloc.See the indexing documentation Indexing and Selecting Data and MultiIndex / Advanced Indexing.GettingSelecting a single column, which yields a Series, equivalent to df.A:In [23]: .1394882013-01-06-0.130327Freq: D, Name: A, dtype: float64Selecting via [], which slices the rows.In [24]: df[0:3]Out[24]:A2013-01-01 1.8327472013-01-02 -0.9134362013-01-03 -1.323650B1.5153860.0351410.427355CD1.793547 -0.3606343.437482 -1.1069140.835343 -0.000698In [25]: df['20130102':'20130104']18Chapter 3. Getting started

pandas: powerful Python data analysis toolkit, Release 0.25.3Out[25]:AB2013-01-02 -0.913436 0.0351412013-01-03 -1.323650 0.4273552013-01-04 0.509859 -2.769586CD3.437482 -1.1069140.835343 -0.0006981.000521 -0.865748Selection by labelSee more in Selection by Label.For getting a cross section using a label:In [26]: 7D-0.360634Name: 2013-01-01 00:00:00, dtype: float64Selecting on a multi-axis by label:In [27]: df.loc[:, ['A', 'B']]Out[27]:AB2013-01-01 1.832747 1.5153862013-01-02 -0.913436 0.0351412013-01-03 -1.323650 0.4273552013-01-04 0.509859 -2.7695862013-01-05 0.139488 -0.2593282013-01-06 -0.130327 -0.372906Showing label slicing, both endpoints are included:In [28]: df.loc['20130102':'20130104', ['A', 'B']]Out[28]:AB2013-01-02 -0.913436 0.0351412013-01-03 -1.323650 0.4273552013-01-04 0.509859 -2.769586Reduction in the dimensions of the returned object:In [29]: df.loc['20130102', ['A', 'B']]Out[29]:A-0.913436B0.035141Name: 2013-01-02 00:00:00, dtype: float64For getting a scalar value:In [30]: df.loc[dates[0], 'A']Out[30]: 1.8327469709663295For getting fast access to a scalar (equivalent to the prior method):3.2. 10 minutes to pandas19

pandas: powerful Python data analysis toolkit, Release 0.25.3In [31]: df.at[dates[0], 'A']Out[31]: 1.8327469709663295Selection by positionSee more in Selection by Position.Select via the position of the passed integers:In [32]: 865748Name: 2013-01-04 00:00:00, dtype: float64By integer slices, acting similar to numpy/python:In [33]: df.iloc[3:5, 0:2]Out[33]:AB2013-01-04 0.509859 -2.7695862013-01-05 0.139488 -0.259328By lists of integer position locations, similar to the numpy/python style:In [34]: df.iloc[[1, 2, 4], [0, 2]]Out[34]:AC2013-01-02 -0.913436 3.4374822013-01-03 -1.323650 0.8353432013-01-05 0.139488 1.082034For slicing rows explicitly:In [35]: df.iloc[1:3, :]Out[35]:AB2013-01-02 -0.913436 0.0351412013-01-03 -1.323650 0.427355CD3.437482 -1.1069140.835343 -0.000698For slicing columns explicitly:In [36]: df.iloc[:, 1:3]Out[36]:BC2013-01-01 1.515386 1.7935472013-01-02 0.035141 3.4374822013-01-03 0.427355 0.8353432013-01-04 -2.769586 1.0005212013-01-05 -0.259328 1.0820342013-01-06 -0.372906 1.072236For getting a value explicitly:20Chapter 3. Getting started

pandas: powerful Python data analysis toolkit, Release 0.25.3In [37]: df.iloc[1, 1]Out[37]: 0.03514142900432859For getting fast access to a scalar (equivalent t

pandas: powerful Python data analysis toolkit, Release 0.25.3 Date: Nov 02, 2019 Version: 0.25.3 Download