Skip to main content

Web Scraping

 Web Scraping

Some websites don't have lovely APIs for us to interface with.


If we want data from these pages, we have to use a tecnique called scraping. This means downloading the whole webpage and poking at it until we can find the information we want.


You're going to use scraping to get the top ten restaurants near you.


Get started

👉 Go to a website like Yelp and search for the top 10 reastaurants in your location. Copy the URL.

 


url = "https://www.yelp.co.uk/search?find_desc=Restaurants&find_loc=San+Francisco%2C+CA%2C+United+States"  


Import libraries
👉 Import your libraries. Beautiful soup is a specialist library for extracting the contents of HTML and helping us parse them. Run the Repl once your imports are sorted because we want the Beautiful Soup library to be installed (it'll run quicker this way).

import requests
from bs4 import BeautifulSoup

url = "https://www.yelp.co.uk/search?find_desc=Restaurants&find_loc=San+Francisco%2C+CA%2C+United+States"  

Webpage as text
👉 Use requests to get the webpage as text. When printing the html, we get back just how much info there is contained in the page.

import requests
from bs4 import BeautifulSoup
url = "https://www.yelp.co.uk/search?find_desc=Restaurants&find_loc=San+Francisco%2C+CA%2C+United+States"
response = requests.get(url)
html = response.text
print(html)  

More Scraping
👉 Next, we pass the html to Beautiful Soup to make more sense of it.

html.parser will scan through the HTML recognizing tokens in the text and breaking it down into something more meaningful.  

import requests
from bs4 import BeautifulSoup

url = "https://www.yelp.co.uk/search?find_desc=Restaurants&find_loc=San+Francisco%2C+CA%2C+United+States"

response = requests.get(url)
html = response.text

soup = BeautifulSoup(html, 'html.parser')  

Inspect
👉 Back on Yelp, right click the first (non-sponsored) restaurant on the list and inspect it.

Right click and copy element for the whole URL, then paste it into your repl as a temporary measure.  




import requests
from bs4 import BeautifulSoup

url = "https://www.yelp.co.uk/search?find_desc=Restaurants&find_loc=San+Francisco%2C+CA%2C+United+States"

response = requests.get(url)
html = response.text

soup = BeautifulSoup(html, 'html.parser')

<a href="/biz/marufuku-ramen-san-francisco-5?osq=Restaurants" class="css-1m051bw" target="_blank" name="Marufuku Ramen" rel="noopener">Marufuku Ramen</a>  


Inspecting the link gives us clues about what we want beautiful soup to look for. In this case, I want it to look for <a> tags and the class css-1m051bw.

Store results
👉 I've created a new variable to store the result of the beautiful soup search.

find_all takes two arguments. The first is the a tag. The second is a dictionary that tells it what class to search for. This effectively says 'find me all the a tags with this class in them.'
I've printed the len of those results to see how many I get back.
import requests
from bs4 import BeautifulSoup
url = "https://www.yelp.co.uk/search?find_desc=Restaurants&find_loc=San+Francisco%2C+CA%2C+United+States"
response = requests.get(url)
html = response.text
soup = BeautifulSoup(html, 'html.parser')
myLinks = soup.find_all("a", {"class":"css-1m051bw"})
print(len(myLinks))


Loop it
Now I'll use a loop to output all the links.  

import requests
from bs4 import BeautifulSoup

url = "https://www.yelp.co.uk/search?find_desc=Restaurants&find_loc=San+Francisco%2C+CA%2C+United+States"

response = requests.get(url)
html = response.text

soup = BeautifulSoup(html, 'html.parser')

myLinks = soup.find_all("a", {"class":"css-1m051bw"})

print(len(myLinks))

for link in myLinks:
    print(link.text)  

👉 You'll see that the same tag has been used in the info about the location and category, so they're included in the results.

I'm going to use a loop counter to start the print loop at item 3 to leave them off the output.
I'm also going to include the link to the restaurant in the output by using a dictionary.  

 



counter = 0
for link in myLinks:
  if counter > 1:
    print(link.text)
    print(link["href"])
  counter +=1  


Add an fstring
The web links are local, they aren't relative to the site I'm using, so I've formatted the print(link["href"]) as an fString to add the relative address (that I found in the Yelp inspect code).

counter = 0
for link in myLinks:
  if counter > 1:
    print(link.text)
    print(f"""https://www.yelp.com{link["href"]}""")
  counter +=1  








Comments

Post a Comment

Popular posts from this blog

FOR LOOP , RANGE

  FOR LOOP  A  while  loop is perfect to use when we  don't  know how many times we want the loop to repeat.  If we have an idea of how many times we want the loop to repeat, we can use a  for  loop to loop code in exactly the same way the  while  loop did.  EXAMPLE :  for counter in range(10):   print(counter) RANGE  The  range  function creates a list of numbers in the range you create. If you only give it one number, it will start at  0  and move to a state where the final number is  one less  than the number in the brackets. In this case, the final number would be  9 .  EXAMPLE :  total = 0 for number in range(100) :   total += number   print(total)
Post a Comment
Read more

OS LIBRARY , TIME LIBRARY

  What is the os library? It allows us to "talk" to the console. One of the most powerful things we can do with this library is allow it to clear the console EXAMPLE: import os print("Welcome") print("to") print("Replit") os.system("clear") username = input("Username: ")  Time Library We can import a second library by placing a  ,  after the name of the first library. EXAMPLE: import os, time print("Welcome") print("to") print("Replit") time.sleep(10) os.system("clear") username = input("Username: ")  NOTE:  from replit import audio import os, time def play():   source = audio.play_file('audio.wav')   source.paused = False # unpause the playback   while True:     stop_playback = int(input("Press 2 anytime to stop playback and go back to the menu : ")) # giving the user the option to stop playback     if stop_playback == 2:       source.paused = True # let'...
Post a Comment
Read more

ALL ABOUT WHILE LOOP

 WHILE LOOP  A  while  loop allows your code to repeat itself based on a condition you set.   EXAMPLE : counter = 0 while counter < 10:   print(counter)   counter +=1 Infinite Loop  You have to be  really  careful that you don't accidentally invoke an infinite loop! This is where the computer will loop code until the end of time. Without a break. Forever.  This is just saying "count to 10 by 1 each time." to make the loop end. Don't forget, if your  condition  is a  >  then you might need to  -= . This will subtract from the variable instead of adding to it.   EXAMPLE : counter = 0 while counter < 10:   print(counter)    counter += 1
Post a Comment
Read more