Data Analysis by Web Scraping using Python

ABSTRACT

The standard information investigations are built on the root and impact relationship, shaped
as an example minuscule examination, subjective and quantitative examination, the rationality
approach of creating extrapolation examination. The Web Scraper’s conniving ethics and
procedures are juxtaposed, it explains about the working of how the scraper is premeditated.
The technique of it is allocated into three fragments: the web scraper draws the desired links
from the web, and then the data is extracted to get the data from the source links and finally
stowing that data into a csv file. The Python language is implemented for the carrying out. By
doing so, linking all these with the moral knowledge of libraries and working know-how, we can
have an adequate Scraper in our hand to produce the desired result. Due to an enormous
community and library resources for Python and the exquisiteness of coding chic of python
language, it is most appropriate one for Scraping desired data from the desired website.
 INTRODUCTION

Data analysis is the method of extracting solutions to the problems via interrogation and
interpretation of data. The analysis process consists of discovering problems, resolving the
accessibility of suitable data, determining which method can help in finding the solution to the
interesting problem and convey the result. For the purpose of analysis, the data has to
segregate into various steps further on such as starting with its specification assembling,
organizing, cleaning, re-analyzing, applying models and algorithms and the final result. Web
information scraping [1] and publicly supporting are outstanding strategies for naturally
creating substance on the web. A considerable amount of individuals utilized these strategies in
research and business for creating substance or offering criticisms to expand the exactness of
business advertising that enables individuals to deliver resources in advancing and developing
the business [3]

By and large, web scraping is notable for a "Screen Scraping", "Web Data Extraction". The web
scrubber programming is planned to be exhaustive for all noteworthy data from different
online stores and mining, and collecting it into the new website. The scraper tool for the web is
utilized for derived information from the web host, and as a portion of uses used for web
orders, web mining and data mining, online esteem change observing and value correlation,
element survey scratching (to watch the challenge), gathering land postings, atmosphere data
checking, webpage change area, inspect, following on the web closeness and reputation, web
mashup and, web data joining. [2]Pages are manufactured utilizing content-based increase
dialects (HTML and XHTML), and much of the time contain a profusion of cooperative info in
the content structure. Be that it may be as most website pages are anticipated for human end
users and not for minimalism of robotized use. Thus the toolbox that scrapes web info was
made.

IMPLEMENTATION:

MODULES:
● User
● Admin
● web scraping
● python
MODULES DESCRIPTION:

User:
The User can register the first. While registering he required a valid user email and password
for further communications. Once the user registers, then admin can activate the customer.
Once the admin activates the customer then the customer can login into our system. After login
he can search all the company's details. For searching the company details we will get company
rating and reviews and total no.of employees based on our dataset.After login if we click on
web scraping we can find the job portal based on our title and job location.in the job portal
completely it provides job description and requirements of the particular company.
Admin:
Admin can login with his credentials. Once he logs in he can activate the users. The activated
user only login in our applications. The admin can set the data set by
the company details.. In this report the data is considered as company reviews and company
rating and hq and total number of employees.. The admin can add new data to the dataset. So
this data user can perform the testing process.

web scraping:
Web scraping is a term used to describe the use of a program or algorithm to extract and
process large amounts of data from the web. .Whether you are a data scientist, engineer, or
anybody who analyzes large amounts of datasets, the ability to scrape data from the web is a
useful skill to have.
Web scraping is used to collect large information from websites. But why does someone have
to collect such large data from websites? To know about this, let’s look at the applications of
web scraping:
When you run the code for web scraping, a request is sent to the URL that you have mentioned.

As a response to the request, the server sends the data and allows you to read the HTML or

XML page. The code then, parses the HTML or XML page, finds the data and extracts it.

To extract data using web scraping with python, you need to follow these basic steps:

Find the URL that you want to scrape.Inspecting the Page.Find the data you want to
extract.Write the code.Run the code and extract the data.Store the data in the required
format .

Python and data-analysis:

Python is an increasingly popular tool for data analysis. In recent years, a number of libraries
have reached maturity, allowing R and Stata users to take advantage of the beauty, flexibility,
and performance of Python without sacrificing the functionality these older programs have
accumulated over the years.python focus on simplicity and readability, python it boasts a
gradual and relatively low learning curve. This ease of learning makes an ideal tool for
beginning programmers. Python offers programmers the advantage of using fewer lines of code
to accomplish tasks than one needs when using older languages.

SYSTEM SPECIFICATION:

HARDWARE REQUIREMENTS:

 System : Pentium IV 2.4 GHz.

 Hard Disk : 40 GB.

 Floppy Drive : 1.44 Mb.

 Monitor : 14’ Colour Monitor.

  Mouse : Optical Mouse.

 Ram : 512 Mb.

SOFTWARE REQUIREMENTS:

 Operating system : Windows 7 Ultimate.

 Coding Language : Python.

 Front-End : Python.

 Designing : Html,css,javascript.

 Data Base : MySQL.

CONCLUSION

The extraction of hidden web data is a major challenge nowadays because of the autonomous
and heterogeneous nature of hidden web content traditional stress engines have now become
an ineffective way to search this kind of data. The main outcomes of this project were user
friendly search interface, indexing, query processing, and effective data extraction technique
based on web structure, form submission analysis and new submission plan. Hidden web data
need synthetic and semantic matching to fully achieve automatic integration in this thesis fully
automatic and domain dependent prototype system is proposed that extract and integrate the
data lying behind the search form.

REFERENCES
1.”Renita Crystal Pereira, Vanitha T. “Web Scraping of Social Networks.” International Journal of
Innovative Research in Computer and Communication Engineering, vol. 3, pp.237-239, Oct. 7,
2018”

2.”Ghazvinian, Holbert, Viswanathan. “Simple WebScraping.”Internet:https://seanholbert.wor

dpress.com/2011/07/15/scrappy-simple-webscraping/, Jun. 2015”

3.”Bellarosey.“Crowdsourcing-Definition.” Internet:http://crowdsourcing.typepad.com/cs/
2006/06/crowdsourcing_a.html, Jun. 02, 2006”

4.“Naveen Ashish and Craig Knoblock. ”Wrapper Generation for semi-structured Internet
Sources. In Proc” ACM SIGMOD Workshop on Management of Semi Structured Data, Tucson,
Arizona, May 1997.”

5. “Datahen."3 Advantages of web scraping for yourenterprise"Internet:https://www.datahen.c

om/3-advantages-web-scraping enterprise/,May.17,2017””

6. “https://en.wikipedia.org/wiki/Web_scraping”

7.”https://www.webharvy.com/articles/whatis-web-scraping.html”

8.”http://resources.distilnetworks.com/h/i/538 22104-is-web scraping- illegal-depends-on-

what the-meaning-of-the word-is-is/181642”

9.”https://www.quora.com/What-is-thelegality-of-web-scraping”

10. https://en.wikipedia.org/wiki/Web_crawler

11.”Kolari, Pand Joshi A. ,“Web mining : research and practice , Computing in Science &
Engineering”, IEEE Transactions on Knowledge and Data Engineering, vol. 6, no. 2,Vol. 6 , No. 4 ,
2004”

12.“Pythonversion3.6,http://www.python.org.”

13.”Kengtel,W:Wagner,M.Proteins 1999,37,334- 345.”

14. “BrightPLanet.com Deep web White Paper. http://www.completeplanet.com/Tutorials/Dee

pWeb/index.asp.”