How to Scrape Walmart With Python

I tried to scrape Walmart with Python the obvious way first: one requests.get against a product page, then BeautifulSoup to read the JSON. The request came back 200, so for a second I thought it had worked. Then I printed the page title. It said Robot or human?.

That captcha page is the real starting point for scraping Walmart with Python, because it is what almost every tutorial skips. Below is the code I ran, the exact response Walmart sent back, and the two setups that actually return Walmart data: product details, price, and reviews.

What do you need to scrape Walmart with Python?

To scrape Walmart with Python you need three libraries and one piece of knowledge about how the page is built. The libraries are requests to fetch the HTML, BeautifulSoup (from bs4) to find the right tag, and the built-in json module to load the data. The knowledge is that Walmart is a Next.js site, so the product data is already sitting in the page source as JSON.

pip install requests beautifulsoup4

Walmart renders with Next.js, which means the server embeds the page state into a <script id="__NEXT_DATA__" type="application/json"> tag so React can hydrate on the client. The Next.js docs spell this out: anything passed through getServerSideProps is visible in that script tag in the initial HTML. For scraping that is the whole game. You do not build a CSS selector for the price, then another for the title, then another for the rating. You grab one JSON object and read the fields you want from props.pageProps.initialData.

That is the clean theory. The next section is what happens when you actually send the request.

Why does Walmart block Python scrapers?

Walmart blocks Python scrapers with PerimeterX (now HUMAN Security Bot Defender), which answers automated requests with a press-and-hold captcha page in place of the product data. When I sent a single requests call with a normal Chrome User-Agent to a live product page in June 2026, here is exactly what came back:

The status code is the trap. A 200 looks like success, so a naive script keeps going and then fails at the parse step with a confusing NoneType error, because soup.find("script", {"id": "__NEXT_DATA__"}) returns nothing. The page came back fine. The body was the captcha page, and the product JSON was never in it.

This matches what the anti-bot vendors describe. PerimeterX drops a _px3 cookie on first visit and fingerprints the browser through Canvas, WebGL, fonts, and sensor data, as ScrapingBee documents in its PerimeterX teardown. Walmart layers in TLS-fingerprint and IP-reputation checks, so any HTTP-only client from a datacenter IP gets flagged fast. I sent three identical requests in a row and got the captcha page all three times, which lines up with the rule of thumb that an unspoofed scraper is caught within 10 to 20 requests.

Worth noting on the rules: Walmart’s robots.txt sets Disallow: /search, Disallow: /api/, and Disallow: /account/, while it adds Allow: /reviews/product/ and Allow: /reviews/seller/. The only Crawl-delay is 5, and it applies just to Yahoo’s Slurp crawler. So the review pages are the one product surface Walmart marks crawlable, which is useful context for the reviews section later.

The block decision happens before you ever see product JSON, so the fix has to change how the page is fetched. The next section shows the parse code first, because once you do get a real page, this part does not change.

How do you scrape a Walmart product page with Python?

You scrape a Walmart product page with Python by loading the __NEXT_DATA__ JSON and reading the product object at props.pageProps.initialData.data.product. Here is the full parser. It is the code I run once a real product page (not the captcha page) is in hand:

import json
import requests
from bs4 import BeautifulSoup

UA = ("Mozilla/5.0 (Windows NT 10.0; Win64; x64) "
      "AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36")

def parse_walmart_product(html: str) -> dict:
    soup = BeautifulSoup(html, "html.parser")
    tag = soup.find("script", {"id": "__NEXT_DATA__"})
    if tag is None:
        raise RuntimeError("No __NEXT_DATA__ found. You likely got the captcha page.")
    data = json.loads(tag.string)
    product = data["props"]["pageProps"]["initialData"]["data"]["product"]
    price = product.get("priceInfo", {}).get("currentPrice", {}).get("price")
    return {
        "id": product.get("usItemId"),
        "title": product.get("name"),
        "brand": product.get("brand"),
        "price": price,
        "rating": product.get("averageRating"),
        "in_stock": product.get("availabilityStatus") == "IN_STOCK",
        "image": product.get("imageInfo", {}).get("thumbnailUrl"),
    }

resp = requests.get("https://www.walmart.com/ip/587451676",
                    headers={"User-Agent": UA}, timeout=25)
print(parse_walmart_product(resp.text))

The RuntimeError guard is the line that saved me hours. Without it the script throws a TypeError: 'NoneType' object is not subscriptable deep in the parse, and you waste time debugging your selectors when the real problem is that you got blocked. With the guard, the failure says what is actually happening.

The fields under product cover most of the product details people scrape: the product title in name, plus brand, usItemId, priceInfo.currentPrice.price, averageRating, numberOfReviews, availabilityStatus, and the image URLs on the i5.walmartimages.com CDN. Because it is one JSON object, adding a field is one more product.get(...) line. No new selector to write or maintain. To save the product information for later, dump the dict straight to a CSV file with the built-in csv module:

import csv

rows = [parse_walmart_product(resp.text)]
with open("walmart_products.csv", "w", newline="", encoding="utf-8") as f:
    writer = csv.DictWriter(f, fieldnames=rows[0].keys())
    writer.writeheader()
    writer.writerows(rows)

When I ran the requests.get above against the live URL on my own machine, resp.text was the Robot or human? page, so parse_walmart_product raised the guard error exactly as designed. The parser is correct. The fetch is what Walmart stops, which is the problem the last two sections solve. First, reviews, since they live in the same JSON.

How do you scrape Walmart reviews with Python?

You scrape Walmart reviews with Python from the same __NEXT_DATA__ JSON, at the path props.pageProps.initialData.data.reviews.customerReviews. Each entry in that list carries the fields you need for review analysis:

def parse_walmart_reviews(html: str) -> list[dict]:
    soup = BeautifulSoup(html, "html.parser")
    tag = soup.find("script", {"id": "__NEXT_DATA__"})
    if tag is None:
        raise RuntimeError("No __NEXT_DATA__ found. You likely got the captcha page.")
    data = json.loads(tag.string)
    reviews = (data["props"]["pageProps"]["initialData"]
               ["data"]["reviews"]["customerReviews"])
    return [{
        "rating": r.get("rating"),
        "title": r.get("reviewTitle"),
        "text": r.get("reviewText"),
        "author": r.get("userNickname"),
        "date": r.get("reviewSubmissionTime"),
    } for r in reviews]

Reviews are the one Walmart surface that the site marks crawlable: robots.txt carries Allow: /reviews/product/. They are also the data most teams want at volume, because a few thousand reviews per product is enough to run sentiment analysis and spot recurring complaints. The catch is pagination. The __NEXT_DATA__ blob only holds the first page of reviews, so pulling the full history means walking the review pages, and every one of those requests faces the same PerimeterX layer that blocked the product page.

For a single product’s first review page you can write this yourself once you clear the captcha. For the full review set across many products, the request volume is exactly what triggers the block, which is the scaling problem the next section handles. I keep the deeper review-extraction walkthrough in the Walmart review scraper guide.

How do you scrape Walmart at scale without getting blocked?

You scrape Walmart at scale by moving the fetch off your own machine, either by running a headless browser through residential proxies yourself or by sending the URL to a Walmart scraping API that returns parsed JSON. The two paths trade money against engineering time:

The headless-browser route works because a real browser executes the JavaScript and posts the sensor data PerimeterX expects. The cost is real: you buy a residential proxy pool, rotate it, run and patch the browser farm, and solve the press-and-hold captcha when the score drops. That is a standing maintenance project, and the technical detail lives in my guide on how to avoid getting blocked scraping Walmart.

The API route collapses all of that into one request. You send a Walmart URL and a key, the proxies and captcha solving happen on the server, and you get the parsed product or review JSON back. With ChocoData the call is a single GET against the product endpoint:

curl "https://chocodata.com/api/v1/walmart/product?url=https://www.walmart.com/ip/587451676&api_key=$CHOCO_API_KEY"

In Python that becomes one request and a JSON parse, with no browser and no proxy management:

import os
import requests

API_KEY = os.environ["CHOCO_API_KEY"]

resp = requests.get(
    "https://chocodata.com/api/v1/walmart/product",
    params={"url": "https://www.walmart.com/ip/587451676", "api_key": API_KEY},
    timeout=60,
)
product = resp.json()
print(product["title"], product["price"])

Swap /walmart/product for /walmart/reviews to get the customer reviews, /walmart/search to scrape a Walmart search results page, or /walmart/price-monitoring for real-time price tracking. The shape stays the same: one URL in, structured data out. For a one-off pull of a few pages, the naive script is fine and free. For continuous data collection across thousands of products, a scraper API is usually cheaper once you price in the hours the proxy-and-captcha stack would cost you. A side-by-side of the managed Walmart scraper API options sits in the best Walmart scrapers roundup.

There is also an official door for some of this data. The Walmart I/O Affiliate API and the Walmart Marketplace APIs return product and catalog data through approved programs. Access is gated to affiliates, sellers, and solution providers, so it stays closed to anyone with just a script.

Is it legal to scrape Walmart with Python?

Scraping publicly visible Walmart pages with Python sits in the same gray area as most public-web scraping in the US. In hiQ Labs v. LinkedIn, the Ninth Circuit reaffirmed in April 2022 that scraping data from a public website is unlikely to violate the Computer Fraud and Abuse Act, because public pages have no access barrier to break. The court was clear that data behind a login is a different question, and the case later settled with a judgment and injunction against hiQ, so the picture is not a blanket green light.

Walmart’s own terms and its robots.txt still apply on top of that, and they are what govern the relationship between you and the site. I keep the full breakdown of terms, robots.txt, and policy in is scraping Walmart legal. Treat that section as the prerequisite before you collect anything at volume.