Skip to content

Commit effc1c9

Browse files
jshtawayjshtaway
committed
asd
0 parents  commit effc1c9

File tree

10 files changed

+350
-0
lines changed

10 files changed

+350
-0
lines changed

‎.DS_Store

6 KB
Binary file not shown.

‎README.md

+105
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,105 @@
1+
# Mission to Mars
2+
3+
![mission_to_mars](Images/mission_to_mars.jpg)
4+
5+
In this assignment, you will build a web application that scrapes various websites for data related to the Mission to Mars and displays the information in a single HTML page. The following outlines what you need to do.
6+
7+
## Step 1 - Scraping
8+
9+
Complete your initial scraping using Jupyter Notebook, BeautifulSoup, Pandas, and Requests/Splinter.
10+
11+
* Create a Jupyter Notebook file called `mission_to_mars.ipynb` and use this to complete all of your scraping and analysis tasks. The following outlines what you need to scrape.
12+
13+
### NASA Mars News
14+
15+
* Scrape the [NASA Mars News Site](https://mars.nasa.gov/news/) and collect the latest News Title and Paragraph Text. Assign the text to variables that you can reference later.
16+
17+
```python
18+
# Example:
19+
news_title = "NASA's Next Mars Mission to Investigate Interior of Red Planet"
20+
21+
news_p = "Preparation of NASA's next spacecraft to Mars, InSight, has ramped up this summer, on course for launch next May from Vandenberg Air Force Base in central California -- the first interplanetary launch in history from America's West Coast."
22+
```
23+
24+
### JPL Mars Space Images - Featured Image
25+
26+
* Visit the url for JPL's Featured Space Image [here](https://www.jpl.nasa.gov/spaceimages/?search=&category=Mars).
27+
28+
* Use splinter to navigate the site and find the image url for the current Featured Mars Image and assign the url string to a variable called `featured_image_url`.
29+
30+
* Make sure to find the image url to the full size `.jpg` image.
31+
32+
* Make sure to save a complete url string for this image.
33+
34+
```python
35+
# Example:
36+
featured_image_url = 'https://www.jpl.nasa.gov/spaceimages/images/largesize/PIA16225_hires.jpg'
37+
```
38+
39+
### Mars Weather
40+
41+
* Visit the Mars Weather twitter account [here](https://twitter.com/marswxreport?lang=en) and scrape the latest Mars weather tweet from the page. Save the tweet text for the weather report as a variable called `mars_weather`.
42+
43+
```python
44+
# Example:
45+
mars_weather = 'Sol 1801 (Aug 30, 2017), Sunny, high -21C/-5F, low -80C/-112F, pressure at 8.82 hPa, daylight 06:09-17:55'
46+
```
47+
48+
### Mars Facts
49+
50+
* Visit the Mars Facts webpage [here](http://space-facts.com/mars/) and use Pandas to scrape the table containing facts about the planet including Diameter, Mass, etc.
51+
52+
* Use Pandas to convert the data to a HTML table string.
53+
54+
### Mars Hemispheres
55+
56+
* Visit the USGS Astrogeology site [here](https://astrogeology.usgs.gov/search/results?q=hemisphere+enhanced&k1=target&v1=Mars) to obtain high resolution images for each of Mar's hemispheres.
57+
58+
* You will need to click each of the links to the hemispheres in order to find the image url to the full resolution image.
59+
60+
* Save both the image url string for the full resolution hemisphere image, and the Hemisphere title containing the hemisphere name. Use a Python dictionary to store the data using the keys `img_url` and `title`.
61+
62+
* Append the dictionary with the image url string and the hemisphere title to a list. This list will contain one dictionary for each hemisphere.
63+
64+
```python
65+
# Example:
66+
hemisphere_image_urls = [
67+
{"title": "Valles Marineris Hemisphere", "img_url": "..."},
68+
{"title": "Cerberus Hemisphere", "img_url": "..."},
69+
{"title": "Schiaparelli Hemisphere", "img_url": "..."},
70+
{"title": "Syrtis Major Hemisphere", "img_url": "..."},
71+
]
72+
```
73+
74+
- - -
75+
76+
## Step 2 - MongoDB and Flask Application
77+
78+
Use MongoDB with Flask templating to create a new HTML page that displays all of the information that was scraped from the URLs above.
79+
80+
* Start by converting your Jupyter notebook into a Python script called `scrape_mars.py` with a function called `scrape` that will execute all of your scraping code from above and return one Python dictionary containing all of the scraped data.
81+
82+
* Next, create a route called `/scrape` that will import your `scrape_mars.py` script and call your `scrape` function.
83+
84+
* Store the return value in Mongo as a Python dictionary.
85+
86+
* Create a root route `/` that will query your Mongo database and pass the mars data into an HTML template to display the data.
87+
88+
* Create a template HTML file called `index.html` that will take the mars data dictionary and display all of the data in the appropriate HTML elements. Use the following as a guide for what the final product should look like, but feel free to create your own design.
89+
90+
![final_app_part1.png](Images/final_app_part1.png)
91+
![final_app_part2.png](Images/final_app_part2.png)
92+
93+
- - -
94+
95+
## Hints
96+
97+
* Use Splinter to navigate the sites when needed and BeautifulSoup to help find and parse out the necessary data.
98+
99+
* Use Pymongo for CRUD applications for your database. For this homework, you can simply overwrite the existing document each time the `/scrape` url is visited and new data is obtained.
100+
101+
* Use Bootstrap to structure your HTML template.
102+
103+
## Copyright
104+
105+
Trilogy Education Services © 2017. All Rights Reserved.

‎__pycache__/scrape_mars.cpython-36.pyc

2.62 KB
Binary file not shown.

‎app.py

+60
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,60 @@
1+
import pandas as pd
2+
import datetime as dt
3+
import scrape_mars
4+
import pymongo
5+
from bson.json_util import loads
6+
7+
8+
from flask import (
9+
Flask,
10+
render_template,
11+
jsonify,
12+
redirect)
13+
14+
conn = 'mongodb://localhost:27017'
15+
# Pass connection to the pymongo instance.
16+
client = pymongo.MongoClient(conn)
17+
# Connect to a database. Will create one if not already available.
18+
db = client.space_db
19+
20+
#################################################
21+
# Flask Setup
22+
#################################################
23+
app = Flask(__name__)
24+
25+
@app.route('/')
26+
#merged scrape and home
27+
def home():
28+
mars_data = db.mars.find().sort("_id", -1).limit(1)[0]
29+
# mars_data = db.collection.find().limit(1).sort({$natural:-1})
30+
with open('templates/table.html','w+') as f:
31+
f.write(mars_data['mars_facts'])
32+
# print(mars_data['mars_facts'])
33+
return render_template('index2.html', mars=mars_data)
34+
# except:
35+
# print('scraping data')
36+
# return redirect("http://localhost:5000/scrape", code=302)
37+
38+
@app.route('/scrape')
39+
def scrape():
40+
scrapedata = scrape_mars.scrape()
41+
data = {
42+
"news": scrapedata["news"],
43+
"space_images": scrapedata["space_images"],
44+
"featured_image_url": scrapedata["mars_weather"],
45+
"mars_facts": scrapedata["mars_facts"],
46+
"mars_hemispheres": scrapedata["mars_hemispheres"],
47+
}
48+
print('space_images: ',data['space_images'])
49+
collection = db.mars
50+
# Insert data into database
51+
collection.update(
52+
data,
53+
data,
54+
upsert=True)
55+
# Redirect back to home page
56+
return redirect("http://localhost:5000/", code=302)
57+
58+
59+
if __name__ == '__main__':
60+
app.run(debug=True)

‎js/app.js

+22
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,22 @@
1+
// Plot the default route once the page loads
2+
const defaultURL = "/emoji_char";
3+
d3.json(defaultURL).then(function(data) {
4+
var data = [data];
5+
var layout = { margin: { t: 30, b: 100 } };
6+
Plotly.plot("bar", data, layout);
7+
});
8+
9+
// Update the plot with new data
10+
function updatePlotly(newdata) {
11+
Plotly.restyle("bar", "x", [newdata.x]);
12+
Plotly.restyle("bar", "y", [newdata.y]);
13+
}
14+
15+
// Get new data whenever the dropdown selection changes
16+
function getData(route) {
17+
console.log(route);
18+
d3.json(`/${route}`).then(function(data) {
19+
console.log("newdata", data);
20+
updatePlotly(data);
21+
});
22+
}

‎scrape_mars.py

+90
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,90 @@
1+
from bs4 import BeautifulSoup
2+
from urllib.request import urlopen
3+
import re
4+
from pprint import pprint
5+
6+
def scrape():
7+
return {
8+
'news': news(),
9+
'space_images': space_images(),
10+
'mars_weather': space_twitter(),
11+
'mars_facts': mars_facts(),
12+
'mars_hemispheres': mars_hemispheres()
13+
}
14+
15+
def news():
16+
url = 'https://mars.nasa.gov/news/'
17+
18+
content = urlopen(url)
19+
c=content.read()
20+
soup = BeautifulSoup(c, 'html.parser')
21+
slides = soup.find_all(class_='slide')
22+
news = []
23+
for slide in slides:
24+
link = slide.a.get('href')
25+
description = slide.find(class_='rollover_description_inner').text
26+
title = slide.find(class_='content_title').text
27+
news.append({'link': link,
28+
'description': description,
29+
'title': title})
30+
31+
return news
32+
33+
def space_images():
34+
url = 'https://www.jpl.nasa.gov/spaceimages/?search=&category=Mars'
35+
jpl_base_url = 'https://www.jpl.nasa.gov'
36+
from splinter import Browser
37+
38+
with Browser('chrome') as browser:
39+
# Visit URL
40+
browser.visit(url)
41+
html = browser.html
42+
soup = BeautifulSoup(html, 'lxml')
43+
footer = soup.body.find(class_="carousel_item").attrs['style'].split("\'")[1]
44+
featured_image_url = jpl_base_url + footer
45+
print(featured_image_url)
46+
return featured_image_url
47+
48+
49+
def space_twitter():
50+
#TWITTER
51+
url = 'https://twitter.com/marswxreport?lang=en'
52+
53+
content = urlopen(url)
54+
c=content.read()
55+
soup = BeautifulSoup(c, 'html.parser')
56+
slides = soup.find(class_='js-tweet-text-container')
57+
mars_weather = slides.text
58+
return mars_weather
59+
60+
def mars_facts():
61+
#MARS FACCTS
62+
import pandas as pd
63+
url = 'https://space-facts.com/mars/'
64+
dfs = pd.read_html(url)
65+
mars_facts = dfs[0]
66+
mars_facts.columns = ['Description','Value']
67+
mars_facts.set_index('Description', inplace=True)
68+
mars_facts_html = mars_facts.to_html()
69+
mars_facts_html = mars_facts_html.replace('\n', '')
70+
return mars_facts_html
71+
72+
def mars_hemispheres():
73+
#MARS HEMISPHERES
74+
url = 'https://astrogeology.usgs.gov/search/results?q=hemisphere+enhanced&k1=target&v1=Mars'
75+
76+
content = urlopen(url)
77+
c=content.read()
78+
soup = BeautifulSoup(c, 'html.parser')
79+
items = soup.find_all(class_='item')
80+
hemisphere_image_urls = []
81+
for item in items:
82+
link = ''.join(url.split('search')[0])+ item.a.get('href')
83+
content = urlopen(link)
84+
c=content.read()
85+
soup = BeautifulSoup(c, 'html.parser')
86+
title = soup.find(class_='title').text
87+
hemisphere_image_urls.append({'title': title, 'url': link})
88+
89+
return hemisphere_image_urls
90+

‎static/js/app.js

+22
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,22 @@
1+
// Plot the default route once the page loads
2+
const defaultURL = "/emoji_char";
3+
d3.json(defaultURL).then(function(data) {
4+
var data = [data];
5+
var layout = { margin: { t: 30, b: 100 } };
6+
Plotly.plot("bar", data, layout);
7+
});
8+
9+
// Update the plot with new data
10+
function updatePlotly(newdata) {
11+
Plotly.restyle("bar", "x", [newdata.x]);
12+
Plotly.restyle("bar", "y", [newdata.y]);
13+
}
14+
15+
// Get new data whenever the dropdown selection changes
16+
function getData(route) {
17+
console.log(route);
18+
d3.json(`/${route}`).then(function(data) {
19+
console.log("newdata", data);
20+
updatePlotly(data);
21+
});
22+
}

‎templates/.DS_Store

6 KB
Binary file not shown.

‎templates/index2.html

+50
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,50 @@
1+
<!DOCTYPE html>
2+
<html lang="en">
3+
<head>
4+
<meta charset="UTF-8">
5+
<meta name="viewport" content="width=device-width, initial-scale=1.0">
6+
<meta http-equiv="X-UA-Compatible" content="ie=edge">
7+
<title>Weather By Region</title>
8+
<script src="https://cdnjs.cloudflare.com/ajax/libs/d3/5.5.0/d3.min.js"></script>
9+
<script src="https://cdn.plot.ly/plotly-latest.min.js"></script>
10+
11+
</head>
12+
<body>
13+
<div class='description jumbotron row' >
14+
<div class="container" >
15+
<h1 class="display-3">Mars</h1>
16+
<p>
17+
All you need to know for your next trip to Mars, in one place.
18+
</p>
19+
</div>
20+
<style>
21+
body {background-color:'#cccccc';}
22+
.description {
23+
background: url('{{ mars['space_images'] }}');
24+
background-size: contain;
25+
color:white;
26+
}
27+
</style>
28+
</div>
29+
<div>
30+
<div>
31+
<h2>Latest tweet from Mars:</h2>
32+
<p>{{ mars['featured_image_url'] }}</p>
33+
</div>
34+
<!-- hemisphere_image_urls.append({'title': title, 'url': link}) -->
35+
<div>
36+
<h2>Images of Mars</h2>
37+
{% for image in mars['mars_hemispheres'] %}
38+
<p><a class="btn btn-secondary" href="{{ image.url }}" role="button">{{ image.title }} &raquo;</a></p>
39+
{% endfor %}
40+
</div>
41+
</div>
42+
43+
<div>
44+
<h2>Fun Facts!</h2>
45+
{% include 'table.html' %}
46+
</div>
47+
48+
49+
</body>
50+
</html>

‎templates/table.html

+1
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
<table border="1" class="dataframe"> <thead> <tr style="text-align: right;"> <th></th> <th>Value</th> </tr> <tr> <th>Description</th> <th></th> </tr> </thead> <tbody> <tr> <th>Equatorial Diameter:</th> <td>6,792 km</td> </tr> <tr> <th>Polar Diameter:</th> <td>6,752 km</td> </tr> <tr> <th>Mass:</th> <td>6.42 x 10^23 kg (10.7% Earth)</td> </tr> <tr> <th>Moons:</th> <td>2 (Phobos &amp; Deimos)</td> </tr> <tr> <th>Orbit Distance:</th> <td>227,943,824 km (1.52 AU)</td> </tr> <tr> <th>Orbit Period:</th> <td>687 days (1.9 years)</td> </tr> <tr> <th>Surface Temperature:</th> <td>-153 to 20 °C</td> </tr> <tr> <th>First Record:</th> <td>2nd millennium BC</td> </tr> <tr> <th>Recorded By:</th> <td>Egyptian astronomers</td> </tr> </tbody></table>

0 commit comments

Comments
 (0)