Python Web Scraping Guide: requests, BeautifulSoup, Pagination, CSV Export, robots.txt and Legal Basics

Web scraping means extracting information from web pages with code. It saves hours of copying data by hand and powers price trackers, research datasets and monitoring tools. It also carries legal and ethical responsibilities, so this guide covers both technique and limits.

Check permission first

• Read the site's terms of service. Many forbid automated scraping.
• Check https://site.com/robots.txt for rules about which paths bots may visit.
• Look for an official API, which is faster, more stable and often permitted.
• Do not scrape personal data, login-protected pages or copyrighted content for republishing. Data protection law, including India's Digital Personal Data Protection Act, may apply.
• Be gentle: add delays and avoid hammering servers.

Setup

pip install requests beautifulsoup4 pandas lxml

You should know basic Python. If not, read Python basics first, and HTML fundamentals help too.

Fetch a page

import requests

url = "https://example.com/products"
headers = {"User-Agent": "MyResearchBot/1.0 (contact: you@example.com)"}
resp = requests.get(url, headers=headers, timeout=15)
resp.raise_for_status()
html = resp.text
print(resp.status_code, len(html))

Identify your bot honestly, set a timeout and check the status code. A 403 or 429 means the site is blocking or rate-limiting you, and you should stop rather than disguise your requests.

Parse with BeautifulSoup

from bs4 import BeautifulSoup

soup = BeautifulSoup(html, "lxml")
for card in soup.select("div.product"):
    name = card.select_one("h2.title").get_text(strip=True)
    price = card.select_one("span.price").get_text(strip=True)
    link = card.select_one("a")["href"]
    print(name, price, link)

Use your browser's developer tools (right-click, Inspect) to find tags and classes. select uses CSS selectors, and find_all uses tag and attribute filters. Always handle missing elements, since pages change.

Handle missing data safely

def text_or_none(el):
    return el.get_text(strip=True) if el else None

rows = []
for card in soup.select("div.product"):
    rows.append({
        "name": text_or_none(card.select_one("h2.title")),
        "price": text_or_none(card.select_one("span.price")),
    })

Pagination and delays

import time

all_rows = []
for page in range(1, 6):
    r = requests.get(f"https://example.com/products?page={page}",
                     headers=headers, timeout=15)
    if r.status_code != 200:
        break
    s = BeautifulSoup(r.text, "lxml")
    items = s.select("div.product")
    if not items:
        break
    all_rows.extend(parse(items))
    time.sleep(2)

Pause between requests and stop when pages run out. Cache downloaded pages while developing so you do not request the same URL repeatedly.

Clean and save the data

import pandas as pd

df = pd.DataFrame(all_rows)
df["price"] = (df["price"].str.replace("₹", "", regex=False)
                          .str.replace(",", "", regex=False).astype(float))
df.to_csv("products.csv", index=False, encoding="utf-8-sig")

Use utf-8-sig so Excel opens rupee symbols and Indian-language text correctly. Visualise the results with Python data visualization or store them in a database with SQL.

JavaScript-heavy sites

If data loads after the page renders, requests will not see it. Open the browser's Network tab to find a JSON endpoint the page calls, which is often easier to use. Otherwise, headless browser tools such as Playwright or Selenium can render pages, though they are slower and carry the same legal questions.

Common problems

• Empty results: selectors are wrong or content loads with JavaScript.
• Blocked requests: too many requests too fast. Slow down or use the official API.
• Garbled text: check encoding, set resp.encoding = "utf-8".
• Changing layouts: write tests and monitor for failures.

Practical uses

• Tracking prices of products you plan to buy.
• Collecting public job listings for research.
• Gathering public statistics from government pages, with a check for downloadable datasets first.
• Building a small dataset for a class project.

For career applications, see freelance developer tips. For a framework to serve your results, read Flask for beginners.

A complete small project

Suppose you want to track the price of a book you plan to buy. Check the shop's terms and robots file first. Write a script that requests the product page once a day, extracts the title and price with BeautifulSoup, and appends the date and price to a CSV file. Schedule it with cron on Linux or Task Scheduler on Windows. After a few weeks, plot the prices with matplotlib, as shown in our visualization guide, and buy when the price dips. Keep the request volume tiny, one request per day, which is respectful to the site. Handle errors by logging them to a file and skipping a day rather than retrying aggressively. This habit of small, polite, well-logged scripts is what separates a useful scraper from one that gets blocked.

Storing and reusing data

For anything beyond a few thousand rows, use SQLite or a proper database and add a unique key such as product id plus date, so repeated runs do not create duplicates. Record the source URL and timestamp with every row so you can explain later where the data came from.

Frequently asked questions

Is web scraping legal in India?

It depends on the site terms, the data type and the use. Avoid personal data and copyrighted content, and prefer official APIs.

Which library is best for beginners?

requests with BeautifulSoup.

What is robots.txt?

A file that tells crawlers which parts of a site they may access.

Why do I get an empty result?

Your selector may be wrong or the content may load via JavaScript.

How do I avoid being blocked?

Respect limits, add delays, identify your bot and use official APIs when available.

Can I scrape sites that need login?

Generally not without permission, since it often breaks terms of service.

Last reviewed 8 November 2026. Rates, limits and rules change, so confirm figures with the official source before you act.