NoBIGTech Wiki Técnico

This page is read only. You can view the source, but not change it. Ask your administrator if you think this is wrong.

====  Simple guide for HTML Web Scraping ====

\\
Install the packages required.

<code bash>
# Debian-based
apt install python3-bs4
apt install python3-requests

# using pip
pip install bs4
pip install requests
</code>

\\
Sample code for parsing:

<code python>
# obtain html using requests
response = requests.get('http://example.org')
html = BeautifulSoup(response.text, 'html.parser')

# get page title
print(html.title)

# select using DOM selector (list of elements)
elements = html.select('#your-id .your-class a[href="value"]')

# examples on findings
if len(elements) > 0:
    # get "href" or "src"
    print(elements[0].get('href'))
    print(elements[0].get('src'))

    # or get using dictionary:
    print(elements[0]['class'])
    print(elements[0]['style'])

    # get text of DOM
    print(elements[0].get_text())
    print(elements[0].string)
</code>

Documentation on BeautifulSoup: https://www.crummy.com/software/BeautifulSoup/bs4/doc/

NoBIGTech Wiki Técnico

User Tools

Site Tools

Page Tools