How to Scrape Walmart: A Step-by-Step Guide

I tried to scrape Walmart the lazy way first: one requests.get at a live product page (/ip/587451676) with a normal Chrome User-Agent. It came back 200 OK, which looked like a win for about two seconds. The body was 15,190 bytes, had no __NEXT_DATA__ in it, and contained the string “Robot or human?”. Walmart had served me its PerimeterX CAPTCHA page with a success status code, so my parser would have happily extracted nothing.

That failure is what this guide is built around, because it is what almost everyone hits on the first try, and the standard advice (set a better User-Agent) did nothing for me. Below is what data Walmart actually exposes, where the block comes from, and the three setups that get real JSON back, with code I ran in June 2026.

What data can you scrape from Walmart?

You can scrape Walmart product data, prices, ratings, review counts, stock status, and seller details, because Walmart ships all of it inside a single JSON object on every product page. Walmart runs on Next.js, so it embeds the full product record in a <script id="__NEXT_DATA__"> tag and leaves the visible HTML elements as a thin shell. Parse that one blob and you have the structured data.

Here are the fields I pull most often and where they sit in the JSON:

The full path to the product object is props.pageProps.initialData.data.product. This is a standard Next.js pattern: the framework’s own getServerSideProps documentation confirms that server-fetched data is serialized into __NEXT_DATA__ under the pageProps key so the page can hydrate on the client. The practical point is that you do not write a dozen brittle CSS selectors that break every time Walmart reshuffles its markup. You grab one script tag and read keys. That holds across product pages, search results pages, and the grocery catalog, all three of which carry a __NEXT_DATA__ payload.

Knowing the data is one JSON object away makes the real obstacle obvious: it is not parsing, it is getting a non-CAPTCHA response in the first place. Before that, a quick word on whether you are allowed to.

Is it legal to scrape Walmart data?

Scraping publicly visible Walmart product pages sits in a legal gray area in the US, shaped by the gap between what the Computer Fraud and Abuse Act covers and what Walmart’s own terms say. The Ninth Circuit in hiQ Labs v. LinkedIn held, on April 18, 2022, that scraping data that is publicly available (no login, no password gate) likely does not count as access “without authorization” under the CFAA. Walmart product pages are public, which puts straightforward product scraping on the safer side of that line.

The catch is Walmart’s contract. Walmart’s Terms of Use prohibit using “any robot, spider, site search/retrieval application or other manual or automatic device to retrieve, index, scrape, data mine or otherwise gather” site materials without express prior written consent. In hiQ, the same court that allowed the scraping also found hiQ had breached LinkedIn’s user agreement by doing it. So the realistic read for Walmart is: public-page scraping is unlikely to be a CFAA (criminal-style) problem, while it can still be a terms-of-service (contract) matter that gets your IPs or accounts blocked.

A few hard lines stay off-limits regardless: anything behind a Walmart login, personal data on other users, and the robots.txt disallowed paths covered in the next section. I go deeper on the case law, robots.txt, and policy in my full write-up on whether scraping Walmart is legal. The reason any of this matters technically is that Walmart enforces its stance with one of the more aggressive anti-bot stacks in retail.

Why does Walmart block scrapers?

Walmart blocks scrapers with PerimeterX (now HUMAN) bot detection that scores the request’s IP reputation, HTTP fingerprint, and JavaScript environment, then serves a “Robot or human?” CAPTCHA page in place of the product. The tell that throws people off is the status code. Walmart returns this block with HTTP 200, the same code a real page uses, so a naive script treats it as success.

I tested the same live product URL three ways in June 2026 from an ordinary datacenter IP:

Every variant returned the byte-identical 15 KB CAPTCHA shell. A real Walmart product page is hundreds of kilobytes and contains the __NEXT_DATA__ blob; a 15 KB body with “Robot or human?” in it is the PerimeterX interstitial. The User-Agent made zero difference, which fits how this vendor works. PerimeterX (rebranded HUMAN after its 2022 merger) scores traffic with “intelligent fingerprinting, behavioral signals, and predictive analysis,” per its own product description, so a clean header on a flagged IP still fails.

Walmart’s robots.txt reinforces this at the policy layer. It disallows /search, /account/, /orders, /api/, and /typeahead/, and sets a Crawl-delay: 5 for the bots it does name. Product pages under /ip/ are not disallowed, but the search path that most scrapers want is. The fix has to change where the request comes from and whether JavaScript runs, which is exactly what the working methods below do.

How do you scrape Walmart product data with Python?

The way to scrape Walmart product data with Python is to fetch the page, pull the __NEXT_DATA__ script tag, and read the product fields from its JSON, using requests and BeautifulSoup. The code below is the parser you want. The honest caveat from my testing: this exact script returns the CAPTCHA page from a plain datacenter IP, so it works once the request reaches a real product page (through a residential proxy or the API route in the next sections).

First, the naive call so you can recognize the failure when you see it:

import requests

ua = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0 Safari/537.36"
r = requests.get("https://www.walmart.com/ip/587451676",
                 headers={"User-Agent": ua}, timeout=25)

print(r.status_code)                 # -> 200 (looks fine, is not)
print(len(r.text))                   # -> ~15190 (a real page is much larger)
print("__NEXT_DATA__" in r.text)     # -> False
print("Robot or human" in r.text)    # -> True  (PerimeterX CAPTCHA)

That 200 with no __NEXT_DATA__ is the trap. Now the parser that turns a real Walmart page into structured data:

import json
import requests
from bs4 import BeautifulSoup

def scrape_walmart_product(url):
    ua = ("Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
          "(KHTML, like Gecko) Chrome/124.0 Safari/537.36")
    r = requests.get(url, headers={"User-Agent": ua}, timeout=25)

    soup = BeautifulSoup(r.text, "html.parser")
    tag = soup.find("script", id="__NEXT_DATA__")
    if tag is None:
        raise RuntimeError("No __NEXT_DATA__ found - likely the CAPTCHA page")

    data = json.loads(tag.string)
    product = data["props"]["pageProps"]["initialData"]["data"]["product"]

    return {
        "id": product.get("usItemId"),
        "title": product.get("name"),
        "price": product.get("priceInfo", {}).get("currentPrice", {}).get("price"),
        "rating": product.get("reviews", {}).get("averageRating"),
        "reviews": product.get("reviews", {}).get("numberOfReviews"),
        "in_stock": product.get("availabilityStatus"),
    }

print(scrape_walmart_product("https://www.walmart.com/ip/587451676"))

The structure is the whole trick: one find("script", id="__NEXT_DATA__"), one json.loads, then key access. No price-span selectors to maintain. From there you collect the dicts into a list and write them to a CSV with pandas.DataFrame(rows).to_csv("walmart.csv") for analysis. To run this for more than a handful of products you add residential proxies and proxy rotation, and you slow the request rate, because a bare requests loop from one IP gets the CAPTCHA every time, as my table above showed. Python is the common starting point, but the same JSON tag is reachable from Node.js, which has one advantage worth covering next.

How do you scrape Walmart with Node.js?

You scrape Walmart with Node.js by driving a headless browser with Puppeteer, letting the page render, then reading the __NEXT_DATA__ script tag from the live DOM. This is the route the “web scraping Walmart with Node JS” question usually wants, and Puppeteer’s edge over a raw HTTP client is that it executes the JavaScript Walmart’s PerimeterX layer checks for, so it looks more like a real browser.

// npm install puppeteer
const puppeteer = require("puppeteer");

async function scrapeWalmartProduct(url) {
  const browser = await puppeteer.launch({ headless: true });
  const page = await browser.newPage();
  await page.setUserAgent(
    "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 " +
    "(KHTML, like Gecko) Chrome/124.0 Safari/537.36"
  );
  await page.goto(url, { waitUntil: "networkidle2", timeout: 45000 });

  const product = await page.evaluate(() => {
    const tag = document.querySelector("#__NEXT_DATA__");
    if (!tag) return null; // CAPTCHA page has no __NEXT_DATA__
    const data = JSON.parse(tag.textContent);
    const p = data.props.pageProps.initialData.data.product;
    return {
      id: p.usItemId,
      title: p.name,
      price: p.priceInfo?.currentPrice?.price,
      rating: p.reviews?.averageRating,
      reviews: p.reviews?.numberOfReviews,
    };
  });

  await browser.close();
  return product;
}

scrapeWalmartProduct("https://www.walmart.com/ip/587451676").then(console.log);

The mechanics mirror the Python version: query #__NEXT_DATA__, JSON.parse the text content, read keys. The Puppeteer documentation covers page.evaluate, which runs that snippet inside the rendered page context. Wrap the whole call in retry logic with a MAX_RETRIES counter, because a single render still fails when the IP is flagged. A headless browser raises your success rate, and it does not change Walmart’s IP scoring, so at volume you still feed it residential proxies. Running a proxy pool plus a browser farm is the maintenance cost that pushes most teams toward an API, which removes both pieces.

How do you scrape Walmart at scale without getting blocked?

You scrape Walmart at scale without getting blocked by sending the product or search URL to a scraper API that rotates residential proxies, renders JavaScript, and returns the parsed JSON, so the PerimeterX CAPTCHA and the proxy pool become the server’s problem. You make one request and get structured data, with no __NEXT_DATA__ extraction and no 200-status CAPTCHA to detect on your end.

The ChocoData Walmart Product API follows the shape I used in testing. You pass a Walmart URL and your key:

curl "https://chocodata.com/api/v1/walmart/product?url=https://www.walmart.com/ip/587451676&api_key=$CHOCO_API_KEY"

The Python equivalent for a batch of product IDs:

import requests

API = "https://chocodata.com/api/v1/walmart/product"
KEY = "your_choco_api_key"

def fetch(item_url):
    r = requests.get(API, params={"url": item_url, "api_key": KEY}, timeout=60)
    r.raise_for_status()
    return r.json()   # parsed product: id, price, rating, reviews, in_stock

for item in [
    "https://www.walmart.com/ip/587451676",
    "https://www.walmart.com/ip/10295722",
]:
    data = fetch(item)
    print(data["id"], data["price"], data["rating"])

Same idea for search and category collection through the Walmart Search API, which returns the result items already parsed so you skip the 25-page CAPTCHA gauntlet by hand. For continuous price tracking, the Walmart price monitoring API reruns these pulls on a schedule. The tradeoff is the usual one: for a one-off pull of a few dozen products, the Python parser above is fine once you have a proxy. For continuous collection across thousands of SKUs, offloading the blocking and rotation is the cheaper path once you price in your own time. You can get an API key here to test it against your own URLs.

This URL-in, JSON-out pattern is also what makes the hardest Walmart collection job, every grocery product URL, tractable.

How do you scrape all Walmart grocery product URLs?

You scrape all Walmart grocery product URLs by walking the grocery category tree and collecting item links page by page, because Walmart caps any single search or category result at 25 pages, roughly 1,000 products. That ceiling is a known Walmart constraint, and it means one broad “groceries” query will never return the whole catalog. You subdivide until each leaf category fits under that ceiling.

The workflow that gets full coverage:

• Start at the grocery department, then drill into categories. Walmart’s /search path is disallowed in robots.txt and is the most aggressively gated. Category and browse pages under the grocery taxonomy carry the same __NEXT_DATA__ item lists and draw less heat.
• Recurse into subcategories. Split “Pantry” into “Canned Goods”, “Pasta”, “Snacks”, and so on, until each category’s result count is under the ~1,000-item cap. This is the only way past the 25-page limit.
• Read item URLs from __NEXT_DATA__. Each results page lists its products in the JSON blob with their canonicalUrl. Collect those, dedupe by usItemId, and you have clean product URLs to feed the product endpoint.
• Throttle hard. Grocery crawls are large, so a Crawl-delay-respecting pace (Walmart’s robots.txt asks for 5 seconds) and rotating residential IPs keep the run alive.

The ChocoData Walmart Grocery API does this category walk server-side and returns the product URLs and data directly, which is the difference between a crawl you babysit and a list you request. Whichever route you take, the grocery job is a breadth problem (many small category pulls) layered on the same blocking problem the rest of this guide solves. Once you can get one page back reliably, scaling to the whole catalog is mostly bookkeeping. If the blocks are your main pain, my guide on how to avoid getting blocked scraping Walmart goes deeper on proxy and fingerprint setup, and the best Walmart scrapers in 2026 compares the tools head to head.