WIXLIB

Published by :http://www.ijert.orgInternational Journal of Engineering Research & Technology (IJERT)ISSN: 2278-0181Vol. 10 Issue 07, July-2021Data Analysis using PythonKiranbala NongthombamDeepika SharmaUniversity Institute of Sciences(Mathematics Department) Chandigarh University,Punjab, IndiaUniversity Institute of Sciences(Mathematics Department)Chandigarh University, Punjab, IndiaAbstract- In this paper, the analysis of data using PythonProgramming Language is studied. The very basic processes ofdata analysis like cleaning, transforming, modeling of data isbriefly explained in this paper and focus more on exploratorydata analysis of an already existing dataset and finding theinsights. Some graphical analysis of the data from the dataset willbe shown using different libraries and functions of Python. Here,a dataset named “World Happiness report 2021” is used toanalyze and extract various information in both numerical andpictorial form.Keywords:- Data analysis; python; data visualization; pandas;seaborn; exploratory data analysisI.INTRODUCTIONData are those raw facts and figures with no properinformation hence need to be processed to get the desiredinformation. While information is those results which we getafter processing the raw data in different levels or extractedconclusions from a given dataset through a process called dataanalysis.Data Analysis is simply the analysis of various data meanscleaning the data, transforming it into understandable form, andthen modeling data to extract some useful information forbusiness use or an organizational use. It is mainly used in takingbusiness decisions. Many libraries are available for doing theanalysis. For example, NumPy, Pandas, Seaborn, Matplotlib,Sklearn, etc. [7]. NumPy: NumPy is a library written in Python, usedfor numerical analysis in Python. It stores the data inthe form of nd-arrays (n-dimensional arrays). Pandas: Pandas is mainly used for converting data intotabular form and hence, makes the data morestructured and easily to read. Matplotlib: Matplotlib is a data visualisation andgraphical plotting package for Python and itsnumerical extension NumPy that runs on all platforms. Seaborn: Seaborn is a Python data visualisationpackage based on matplotlib that is tightly connectedwith pandas data structures. The core component ofSeaborn is visualisation, which aids in dataexploration and comprehension. Sklearn: Scikit-learn is the most useful library formachine learning in Python. It includes numeroususeful tools for classification, regression, clustering,and dimensionality reduction.Data visualization will help the data analysis to make it moreunderstandable and interactive by plotting or displaying thedata in pictorial form. Pandas, a Python open-source packagethat deals with three different data structures: series, dataIJERTV10IS070241frames, and panels, solves that need of analyzing andvisualization of data [2].Data analysis using Python makes task easier since PythonProgramming language has many advantages over any otherprogramming language. It has prominent features like being ahigh-level programming language (the codes are in humanreadable form) it is easy to understand and use by anyprogrammer or user. Many libraries and functions for statistical,numerical analysis are available in Python. Moreover, thesource code is freely available to anyone (free and open source).This paper includes all the basic terms and functions which aremuch needed by a beginner to know what data analysis is. Thepaper is divided broadly into 4 sections. In section II, the mainsteps in data analysis will be discussed. In section III, dataanalysis using python will be studied with all the basic needs ofpython in doing data analysis and data visualization will aid theanalysis by representing them in picture format. In section IV,conclusion of the paper is given.II.MAIN PHASES IN DATA ANALYSISA. Data requirementsData are the most important unit in any study. Data mustbe provided as inputs to the analysis based on the analysis’requirements. The term “experimental unit” refers to the typeof organization that would be used to gather data (e.g., aperson or population of people). It is possible to identify andobtain specific population variables (such as height, weight,age, and salary). It doesn’t matter whether the data isnumerical or categorical.B. Data Collecting:The collecting of data is simply known as Data Collecting.Data is gathered from a variety of sources, including relationaldatabases, cloud databases, and other sources, depending onthe study’ needs. Field sensors, such as traffic cameras,satellites, monitoring systems, and so on, can also be used asdata sources.C. Data processingData that are collected must be processed or organized foranalysis. For instance, these may involve arranging data intorows and columns in a table format (known as structured data)for further analysis, often through the use of spreadsheet orstatistical software.D. Data cleaning:The method of cleaning data after it has been processedand organized is known as data cleaning. It scans for datawww.ijert.org(This work is licensed under a Creative Commons Attribution 4.0 International License.)463

Published by :http://www.ijert.orgInternational Journal of Engineering Research & Technology (IJERT)ISSN: 2278-0181Vol. 10 Issue 07, July-2021inconsistencies, duplicates, and errors, and then removesthem. The data cleaning process includes tasks such as recordmatching, identifying data inaccuracy, data sort, outlier dataidentification, textual data spell checker, and data qualitymaintenance. As a consequence, it keeps us from havingunexpected outcomes and assists us in delivering high-qualitydata, which is essential for a successful outcome.C. Platform used: Anaconda (Jupyter Notebook)D. Dataset used: World Happiness record 2021E. Exploratory data analysis:Once the datasets are cleaned and free of error, it can thenbe analyzed. A variety of techniques can be applied such asexploratory data analysis- understanding the messagescontained within the obtained data and descriptive statisticsfinding average, median, etc. Data visualization is also atechnique used, in which the data is represented in a graphicalformat in order to obtain additional insights, regarding theinformation within the data [4].F. Modeling and algorithms:Mathematical formulas or models (known as algorithms),may be applied to the data in order to identify relationshipsamong the variables; for example, using correlation or causation.G. Data productA data product, is a computer application that takes datainputs and generates outputs, feeding them back into theenvironment. It may be based on a model or algorithm.III.DATA ANALYSIS USING PYTHONIn this section, data analysis using python will be studied. Themost basic things like why using python for data analysis willbe understood. Moreover, how anyone can start using pythonwill be shown. The important libraries, the platforms, thedataset to carry out the analysis will be introduced. Usage ofvarious python functions for numerical analysis are givenalong with various methods of plotting graphs or charts arediscussed.A. Why using Python?Python is a high-level, interpreted, multi-purposeprogramming language. Many programming paradigms rogramming is supported in python. It can be used for manyapplications, that includes statistical computing with variouspackages and functions. Moreover, it is easy to learn. It canbe picked up by anyone including those who has lessprogramming skills [9].Some features of Python are as listed below: Open source and free Interpreted language Dynamic typesetting Portable Numerous IDEFig. 1. A view of the dataset (World Happiness record 2021)E. Working with dataset Importing libraries:Libraries that would be used in the process of analysis are to beimported first. Here are the codes to import the libraries.import pandas as pdimport numpy as npimport matplotlib.pyplot as pltimport seaborn as snsFig. 2. Importing libraries Importing datasetHere, the dataset (World Happiness report 2021) is imported inthe jupyter notebook.mydata pd.read csv(“World Happiness report 2021.csv”)mydataB. Packages used: NumpyPandasSeabornMatplotlibIJERTV10IS070241Fig. 3. Importing datasetwww.ijert.org(This work is licensed under a Creative Commons Attribution 4.0 International License.)464

Published by :http://www.ijert.orgInternational Journal of Engineering Research & Technology (IJERT)ISSN: 2278-0181Vol. 10 Issue 07, July-2021 Cleaning DataRemoving unwanted data or null values are done in the processof data cleaning. So, first we need to check the dataset whetherit contains any null value or empty cells [6].# isnull() returns true in the entry where there is no value or NAvalue. And sum() is used together with isnull() to find the totalnumber of null values in every columns.mydata.isnull().sum()F. Exploratory Data AnalysisIn statistics, exploratory data analysis is an approach ofanalyzing data sets to summarize their main characteristics,often using statistical graphics and other data visualizationmethods. A statistical model can be used or not, but primarilyEDA is for seeing what the data can tell us beyond the formalmodeling or hypothesis testing task. Exploratory data analysiswas promoted by John Tukey to encourage statisticians toexplore the data, and possibly formulate hypotheses that couldlead to new data collection and experiments [4][8]. Data types: Datatype refers to the type of data- int,object, float are the basic datatypes in python. Printingthe types of data of all the columns in the dataset usingdtypesmydata.dtypesFig. 4. Checking null values in the datasetAccording to our needs for the analysis, we can extract someparticular rows or records from the dataset. Here is an exampleto extract the top most and last rows from the dataset.#head() is used to extract the top-most data in the dataset. 5 isthe default value of the head(). Here, top 10 rows from thedataset is taken.headdata mydata.head(10) headdataFig. 7. Datatypes of the whole coumns in the dataset Describing the dataset: Describing data of a datasetmeans extracting the summary of the given dataframesuch as mean, count, min, max, etc. It can be doneusing describe() function-For the whole dataset: mydata.describe()Fig. 5. Top 10 rows of the dataset#tail() is used to extract the last rows in the dataset. 5 is thedefault value of the tail(). taildata mydata.tail(10) taildataFig. 8. Summary of the whole datasetFig. 6. Last 10 rows of the datasetIJERTV10IS070241www.ijert.org(This work is licensed under a Creative Commons Attribution 4.0 International License.)465