How to Scrape Walmart Search Results (2026)

I scraped a live Walmart search page the direct way first: one requests.get at /search?q=coffee+maker with a real Chrome User-Agent. It returned 200, which read like a win for about a second. Then I looked at the body - roughly 15 KB, the string “Robot or human?” in it, and not one product row.

That failure is the honest starting point for scraping Walmart search results, because it is the step most tutorials skip past. Below is where the search data actually sits, why a plain request never reaches it, the Python that parses a real results page, how to pull category listings and page past Walmart’s hard cap, and the two setups that return clean JSON at volume. I ran every snippet in July 2026.

Where do Walmart search results live in the page?

Walmart search results live in a __NEXT_DATA__ JSON script tag, at props.pageProps.initialData.searchResult.itemStacks[0].items. Walmart runs on Next.js, so its server serializes the whole search response into a <script id="__NEXT_DATA__" type="application/json"> tag for React to hydrate, and the product grid you see is just a render of that JSON. For a scraper that is the entire job: you read one array, not a selector per column.

The Next.js getServerSideProps documentation describes the mechanism. Anything the server fetches is passed into pageProps and lands in that script tag in the initial HTML. On a search page the useful list is itemStacks[0].items, and each product entry carries the fields a results row is built from:

• usItemId - the Walmart item ID, the stable key you dedupe and join on
• name - the product title as it shows in the grid
• priceInfo.currentPrice.price - the current price, with wasPrice when it is marked down
• averageRating and numberOfReviews - the star rating and review count
• sellerName - who holds the listing, Walmart or a marketplace seller
• canonicalUrl - the /ip/ product URL to hand a product scraper later
• a sponsored flag, so paid placements can be split from organic results

The stack also carries a count field reporting how many products the query matched. That number sets up the next problem, because Walmart will report thousands of matches and then refuse to page you through all of them. First, though, why a plain request never sees any of this JSON.

Why does scraping Walmart search results get blocked?

Scraping Walmart search results gets blocked because Walmart runs PerimeterX (now HUMAN Bot Defender) and disallows the search path in robots.txt, so an automated request to /search returns a “Robot or human?” challenge in place of the itemStacks JSON. The detail that catches people is the status code. Walmart serves this block with HTTP 200, the same code a real page uses, so a naive script logs a success and then breaks at the parse step.

When I sent one requests call with a full Chrome User-Agent to /search?q=coffee+maker in July 2026, the body came back near 15 KB, carried “Robot or human?”, and had no __NEXT_DATA__ tag. Swapping the User-Agent or dropping it changed nothing, because HUMAN Bot Defender scores IP reputation, TLS fingerprint, and the JavaScript environment. A clean header on a flagged datacenter IP still draws the challenge.

Search is also the surface Walmart guards most plainly at the policy layer. Walmart’s robots.txt sets Disallow: /search, alongside Disallow: /api/ and Disallow: /account/, while it carves out Allow: /reviews/product/. So the exact path a search scraper hits is the one Walmart marks off-limits to crawlers, which is worth knowing before you point a job at it. The block is decided before any product JSON is sent, so the fix has to change how the page is fetched, not how you parse it. The parser comes first, since it does not change once a real page is in hand.

How do you scrape a Walmart search results page with Python?

You scrape a Walmart search results page with Python by loading the __NEXT_DATA__ JSON and reading the list at props.pageProps.initialData.searchResult.itemStacks[0].items. This is the parser I run once a real results page (not the challenge page) is in hand:

import json
import requests
from bs4 import BeautifulSoup

UA = ("Mozilla/5.0 (Windows NT 10.0; Win64; x64) "
      "AppleWebKit/537.36 (KHTML, like Gecko) Chrome/125.0.0.0 Safari/537.36")

def parse_walmart_search(html: str) -> list[dict]:
    soup = BeautifulSoup(html, "html.parser")
    tag = soup.find("script", {"id": "__NEXT_DATA__"})
    if tag is None:
        raise RuntimeError("No __NEXT_DATA__ found. You likely got the 'Robot or human?' page.")
    data = json.loads(tag.string)
    stack = (data["props"]["pageProps"]["initialData"]
             ["searchResult"]["itemStacks"][0])
    rows = []
    for it in stack["items"]:
        if it.get("type") != "PRODUCT":      # skip ad tiles and layout objects
            continue
        price = it.get("priceInfo", {}).get("currentPrice", {}).get("price")
        rows.append({
            "id": it.get("usItemId"),
            "title": it.get("name"),
            "price": price if price is not None else it.get("price"),
            "rating": it.get("averageRating"),
            "reviews": it.get("numberOfReviews"),
            "seller": it.get("sellerName"),
            "sponsored": bool(it.get("isSponsoredFlag")),
            "url": it.get("canonicalUrl"),
        })
    return rows

resp = requests.get("https://www.walmart.com/search?q=coffee+maker",
                    headers={"User-Agent": UA}, timeout=25)
print(parse_walmart_search(resp.text)[:3])

The RuntimeError guard is the line that saves hours. Without it, a run against the challenge page throws a cryptic TypeError deep in the key access, and you burn time on selectors when the real fault is the block. The type != "PRODUCT" filter matters too, because itemStacks mixes in ad tiles and layout objects that would otherwise seed your rows with None prices.

When I ran that exact requests.get from my own machine, resp.text was the “Robot or human?” page, so parse_walmart_search raised the guard error by design. The parser is right. The fetch is what Walmart stops. Reading the rows is one step, and the same JSON shape also backs Walmart’s category pages, which is the next listing surface worth pulling.

How do you scrape Walmart category and department listing pages?

You scrape Walmart category and department listing pages with the same parser, because a browse page returns the same searchResult.itemStacks JSON as a keyword search. Walmart builds both surfaces from one search response, so parse_walmart_search reads a department grid unchanged. Only the URL you fetch differs.

Two URL shapes reach a category listing:

• Browse URLs. A department page sits at a /browse/... path that carries a numeric category id. Fetch that URL and parse the same itemStacks[0].items you read on a keyword search.
• Filtered search URLs. Adding a category facet to a /search URL narrows a keyword to one department, which is the cleaner route when you want “coffee maker” inside a specific category rather than the whole browse tree.

The fields are identical to a keyword search - usItemId, name, priceInfo, sellerName, the sponsored flag - so one schema covers both, and you can point a crawler at a list of category URLs the same way you would a list of queries. The one thing category pages share with search that you cannot ignore is the depth ceiling: a browse listing paginates exactly like a search and stops at the same wall. That wall is the next problem.

How do you get past Walmart’s 25-page search limit?

You get past Walmart’s 25-page search limit by splitting one query into narrower slices, because Walmart caps any single search or category listing at 25 pages - about 1,000 products at roughly 40 per page - no matter how large the count field claims the result set is. You page through with the page query parameter, and page 26 returns nothing new:

import time

def search_all_pages(query: str, max_pages: int = 25) -> list[dict]:
    rows = []
    for page in range(1, max_pages + 1):
        url = f"https://www.walmart.com/search?q={query}&page={page}"
        resp = requests.get(url, headers={"User-Agent": UA}, timeout=25)
        page_rows = parse_walmart_search(resp.text)   # raises on the challenge page
        if not page_rows:
            break                                      # past the last real page
        rows.extend(page_rows)
        time.sleep(5)   # Walmart's robots.txt lists a 5s crawl-delay
    return rows

Getting past the ceiling is a breadth problem, not a depth one. You cannot force Walmart to serve page 26, so you carve the query into overlapping subsets that each stay under about 1,000 items, then merge and dedupe. Three splits do most of the work:

• Run both sort orders. Pull the query once with sort=price_low and once with sort=price_high. Each direction exposes a different 1,000-item slice, and the union roughly doubles coverage to about 2,000 products before you dedupe.
• Add price-range bands. Constrain the query to price windows ($0-25, $25-50, and so on) so each band fits under the cap, then stitch the bands together.
• Subdivide by category or brand. Attach a department or brand facet so “coffee maker” becomes several narrower searches, each with its own sub-1,000 set.

After any split, dedupe the combined rows by usItemId, since the slices overlap on purpose. This works, and it is bookkeeping-heavy: every extra slice is more requests hitting the same PerimeterX layer that blocked the first one, so the scaling cost compounds. That is the problem a managed route removes.

How do you scrape Walmart search results at scale without getting blocked?

You scrape Walmart search results at scale by moving the fetch off your own machine, either a headless browser through residential proxies or a Walmart search API that returns the parsed results JSON. The two paths trade money against engineering time:

The headless-browser route works because a real browser runs the JavaScript and posts the sensor data PerimeterX expects, but you still buy and rotate a residential proxy pool, patch the browser farm, and clear the press-and-hold challenge when the score drops. That is a standing maintenance project, and the proxy and fingerprint detail sits in my guide on how to avoid getting blocked scraping Walmart.

The API route folds all of that into one request. You send a query and a key, the proxies and challenge-solving run server-side, and the parsed search results come back. With ChocoData the call is a single GET against the search endpoint:

curl "https://chocodata.com/api/v1/walmart/search?query=coffee+maker&api_key=$CHOCO_API_KEY"

In Python it is one request and a JSON parse, with no browser and no proxy pool:

import os
import requests

API_KEY = os.environ["CHOCO_API_KEY"]

resp = requests.get(
    "https://chocodata.com/api/v1/walmart/search",
    params={"query": "coffee maker", "api_key": API_KEY},
    timeout=60,
)
results = resp.json()   # organic + sponsored rows, sponsored flags included
print(len(results))

Swap the query for a url on /walmart/product to pull a single item page, or hit /walmart/price-monitoring to rerun a search on a schedule. The shape stays the same: inputs in, structured data out. For a one-off pull of a few queries the parser above is fine once you have a proxy. For continuous collection across many keywords, a search API is usually cheaper once you price in the hours the proxy-and-challenge stack would cost you, and a side-by-side of the managed options sits in the best Walmart scrapers roundup.

No route changes one fact: there is no open door for this data. The Walmart Marketplace API is scoped to a seller’s own catalog and will not return arbitrary keyword results, so scraping the public grid stays the only way to collect search-listing data you do not own. That makes the legal footing worth a look before you run anything at volume.

Is it legal to scrape Walmart search results?

Scraping publicly visible Walmart search results sits in the same gray area as most public-web scraping in the US. In hiQ Labs v. LinkedIn, the Ninth Circuit held that scraping data from a public website is unlikely to violate the Computer Fraud and Abuse Act, because public pages have no access barrier to break. Walmart search pages are public, which puts straightforward search scraping on the safer side of that CFAA line.

The contract layer is separate. Walmart’s Terms of Use prohibit using any robot or automated device to retrieve, scrape, or data-mine site materials without written consent, and the robots.txt disallow on /search is the machine-readable version of that stance. So the realistic read is that public search scraping is unlikely to be a CFAA problem while still being a terms-of-service matter that can get your IPs or accounts blocked. Keep anything behind a login and any personal data off-limits, and treat my full write-up on whether scraping Walmart is legal as the prerequisite before you collect search data at scale.

Sources

• Next.js - getServerSideProps serializes server-fetched data into __NEXT_DATA__ under pageProps - https://nextjs.org/docs/pages/api-reference/functions/get-server-side-props
• Walmart robots.txt - Disallow: /search, /api/, /account/; Allow: /reviews/product/ - https://www.walmart.com/robots.txt
• HUMAN Bot Defender - PerimeterX bot detection scoring IP reputation, TLS fingerprint, and JS environment behind Walmart’s challenge - https://www.humansecurity.com/products/bot-defender/
• Walmart Marketplace API - scoped to a seller’s own catalog, not arbitrary keyword search - https://developer.walmart.com/us-marketplace/docs/introduction-to-marketplace-apis
• hiQ Labs v. LinkedIn (Ninth Circuit) - scraping public data likely not “without authorization” under the CFAA - https://www.courtlistener.com/docket/4517811/hiq-labs-inc-v-linkedin-corporation/
• Walmart Terms of Use - prohibit robots or automated retrieval and data-mining without written consent - https://www.walmart.com/help/article/walmart-com-terms-of-use/3b75080af40340d6bbd596f116fae5a0