Extract all links from a webpage using Python and Beautiful Soup 4

Last updated on February 9, 2021 The Shadow Loading... Post a comment

VS Code: How to comment out a block of Python code

January 7, 2021

Python Hello World with VS Code

June 19, 2020

Python: Count words in a string using Regular Expression

November 3, 2020

Examples of using Lambda Functions in Python 3

February 10, 2021

Check Python versions on Mac

October 23, 2020

Examples of using map() function in Python 3

March 4, 2021

2 Ways to Measure the Code Execution Time in Google Colab

November 2, 2021

Examples of numpy.linspace() in Python

December 12, 2020

This article shows you how to get all links from a webpage using Python 3, the Requests module, and the Beautiful Soup 4 module. For the demonstration purpose, I will scrape and extract the main page of Wikipedia:

https://en.wikipedia.org/wiki/Main_Page

Please note that not all websites allow you to crawl content from them.

Getting Started

Install the required modules by running the following commands:

pip install requests

and:

pip install beautifulsoup4

If you’re using a Mac, you may need to type pip3 instead of pip.

The code with explanation:

import requests

# BeautifulSoup is imported with the name bas4 
import bs4

URL = 'https://en.wikipedia.org/wiki/Main_Page'

# Fetch all the HTML source from the url
response = requests.get(URL)


soup = bs4.BeautifulSoup(response.text, 'html.parser')
links = soup.select('a')

# Print out the result
for link in links:
  print(link.get_text())
  if link.get('href') != None:
    if 'https://' in link.get('href'):
      print(link.get('href'))
    else:
      print('https://en.wikipedia.org' + link.get('href')) # Convert relative URL to absolute URL

  print('----------------------------') # Just a line break

When run that program, you should see something like the following:

Main page
https://en.wikipedia.org/wiki/Main_Page
---------------------------
Contents
https://en.wikipedia.org/wiki/Wikipedia:Contents
----------------------------
Current events
https://en.wikipedia.org/wiki/Portal:Current_events
----------------------------
Random article
https://en.wikipedia.org/wiki/Special:Random
----------------------------
About Wikipedia
https://en.wikipedia.org/wiki/Wikipedia:About
----------------------------
Contact us
https://en.wikipedia.org//en.wikipedia.org/wiki/Wikipedia:Contact_us
----------------------------
Donate
https://donate.wikimedia.org/wiki/Special:FundraiserRedirector?utm_source=donate&utm_medium=sidebar&utm_campaign=C13_en.wikipedia.org&uselang=en
----------------------------
Help
https://en.wikipedia.org/wiki/Help:Contents
----------------------------
Learn to edit
https://en.wikipedia.org/wiki/Help:Introduction
----------------------------
Community portal
https://en.wikipedia.org/wiki/Wikipedia:Community_portal
----------------------------
Recent changes
https://en.wikipedia.org/wiki/Special:RecentChanges
----------------------------
Upload file
https://en.wikipedia.org/wiki/Wikipedia:File_Upload_Wizard
----------------------------
What links here
https://en.wikipedia.org/wiki/Special:WhatLinksHere/Main_Page
----------------------------
Related changes
https://en.wikipedia.org/wiki/Special:RecentChangesLinked/Main_Page
----------------------------
Upload file
https://en.wikipedia.org/wiki/Wikipedia:File_Upload_Wizard
----------------------------
Special pages
https://en.wikipedia.org/wiki/Special:SpecialPages
----------------------------
Permanent link
https://en.wikipedia.org/w/index.php?title=Main_Page&oldid=987965326
----------------------------
Page information
https://en.wikipedia.org/w/index.php?title=Main_Page&action=info
----------------------------
Cite this page
https://en.wikipedia.org/w/index.php?title=Special:CiteThisPage&page=Main_Page&id=987965326&wpFormIdentifier=titleform
----------------------------
Wikidata item
https://www.wikidata.org/wiki/Special:EntityPage/Q5296
----------------------------
Download as PDF
https://en.wikipedia.org/w/index.php?title=Special:DownloadAsPdf&page=Main_Page&action=show-download-screen
----------------------------
Printable version
https://en.wikipedia.org/w/index.php?title=Main_Page&printable=yes
----------------------------
Wikimedia Commons
https://commons.wikimedia.org/wiki/Main_Page
----------------------------
MediaWiki
https://www.mediawiki.org/wiki/MediaWiki
----------------------------
Meta-Wiki
https://meta.wikimedia.org/wiki/Main_Page
----------------------------
Wikispecies
https://species.wikimedia.org/wiki/Main_Page
----------------------------
Wikibooks
https://en.wikibooks.org/wiki/Main_Page
----------------------------
Wikidata
https://www.wikidata.org/wiki/Wikidata:Main_Page
----------------------------
Wikimania
https://wikimania.wikimedia.org/wiki/Wikimania
----------------------------
Wikinews
https://en.wikinews.org/wiki/Main_Page
----------------------------
Wikiquote
https://en.wikiquote.org/wiki/Main_Page
----------------------------
Wikisource
https://en.wikisource.org/wiki/Main_Page
----------------------------
Wikiversity
https://en.wikiversity.org/wiki/Wikiversity:Main_Page
----------------------------
Wikivoyage
https://en.wikivoyage.org/wiki/Main_Page
----------------------------
Wiktionary
https://en.wiktionary.org/wiki/Wiktionary:Main_Page
----------------------------
العربية
https://ar.wikipedia.org/wiki/
----------------------------
Български
https://bg.wikipedia.org/wiki/
----------------------------
Bosanski
https://bs.wikipedia.org/wiki/
----------------------------
Català
https://ca.wikipedia.org/wiki/
----------------------------
Čeština
https://cs.wikipedia.org/wiki/
----------------------------
Dansk
https://da.wikipedia.org/wiki/
----------------------------
Deutsch
https://de.wikipedia.org/wiki/
----------------------------
Eesti
https://et.wikipedia.org/wiki/
----------------------------
Ελληνικά
https://el.wikipedia.org/wiki/
----------------------------
Español
https://es.wikipedia.org/wiki/
----------------------------
Esperanto
https://eo.wikipedia.org/wiki/
----------------------------
Euskara
https://eu.wikipedia.org/wiki/
----------------------------
فارسی
https://fa.wikipedia.org/wiki/
----------------------------
Français
https://fr.wikipedia.org/wiki/
----------------------------
Galego
https://gl.wikipedia.org/wiki/
----------------------------
한국어
https://ko.wikipedia.org/wiki/
----------------------------
Hrvatski
https://hr.wikipedia.org/wiki/
----------------------------
Bahasa Indonesia
https://id.wikipedia.org/wiki/
----------------------------
Italiano
https://it.wikipedia.org/wiki/
----------------------------
עברית
https://he.wikipedia.org/wiki/
----------------------------
ქართული
https://ka.wikipedia.org/wiki/
----------------------------
Latviešu
https://lv.wikipedia.org/wiki/
----------------------------
Lietuvių
https://lt.wikipedia.org/wiki/
----------------------------
Magyar
https://hu.wikipedia.org/wiki/
----------------------------
Македонски
https://mk.wikipedia.org/wiki/
----------------------------
Bahasa Melayu
https://ms.wikipedia.org/wiki/
----------------------------
Nederlands
https://nl.wikipedia.org/wiki/
----------------------------
日本語
https://ja.wikipedia.org/wiki/
----------------------------
Norsk bokmål
https://no.wikipedia.org/wiki/
----------------------------
Norsk nynorsk
https://nn.wikipedia.org/wiki/
----------------------------
Polski
https://pl.wikipedia.org/wiki/
----------------------------
Português
https://pt.wikipedia.org/wiki/
----------------------------
Română
https://ro.wikipedia.org/wiki/
----------------------------
Русский
https://ru.wikipedia.org/wiki/
----------------------------
Simple English
https://simple.wikipedia.org/wiki/
----------------------------
Slovenčina
https://sk.wikipedia.org/wiki/
----------------------------
Slovenščina
https://sl.wikipedia.org/wiki/
----------------------------
Српски / srpski
https://sr.wikipedia.org/wiki/
----------------------------
Srpskohrvatski / српскохрватски
https://sh.wikipedia.org/wiki/
----------------------------
Suomi
https://fi.wikipedia.org/wiki/
----------------------------
Svenska
https://sv.wikipedia.org/wiki/
----------------------------
ไทย
https://th.wikipedia.org/wiki/
----------------------------
Türkçe
https://tr.wikipedia.org/wiki/
----------------------------
Українська
https://uk.wikipedia.org/wiki/
----------------------------
Tiếng Việt
https://vi.wikipedia.org/wiki/
----------------------------
中文
https://zh.wikipedia.org/wiki/
----------------------------
Creative Commons Attribution-ShareAlike License
https://en.wikipedia.org//en.wikipedia.org/wiki/Wikipedia:Text_of_Creative_Commons_Attribution-ShareAlike_3.0_Unported_License
----------------------------

https://en.wikipedia.org//creativecommons.org/licenses/by-sa/3.0/
----------------------------
Terms of Use
https://en.wikipedia.org//foundation.wikimedia.org/wiki/Terms_of_Use
----------------------------
Privacy Policy
https://en.wikipedia.org//foundation.wikimedia.org/wiki/Privacy_policy
----------------------------
Wikimedia Foundation, Inc.
https://en.wikipedia.org//www.wikimediafoundation.org/
----------------------------
Privacy policy
https://foundation.wikimedia.org/wiki/Privacy_policy
----------------------------
About Wikipedia
https://en.wikipedia.org/wiki/Wikipedia:About
----------------------------
Disclaimers
https://en.wikipedia.org/wiki/Wikipedia:General_disclaimer
----------------------------
Contact Wikipedia
https://en.wikipedia.org//en.wikipedia.org/wiki/Wikipedia:Contact_us
----------------------------
Mobile view
https://en.wikipedia.org//en.m.wikipedia.org/w/index.php?title=Main_Page&mobileaction=toggle_view_mobile
----------------------------
Developers
https://www.mediawiki.org/wiki/Special:MyLanguage/How_to_contribute

Note that the content of the Wikipedia’s main page may change over time so it’s totally fine if your output is different from mine.