Crawl4AI JavaScript rendering turns blank pages into complete, usable data. Many modern sites send almost no content in the first HTML response. The real text, prices, and listings appear only after JavaScript runs in the browser.
Crawl4AI JavaScript rendering solves that problem by driving a real headless browser. It loads the page in Playwright Chromium, lets scripts execute, waits for the content you need, and then converts the rendered page into clean Markdown.
How Crawl4AI Renders JavaScript Pages
Crawl4AI is an open-source Python crawler built around AsyncWebCrawler and its arun() method. By default, it opens each URL in a headless Chromium browser instead of fetching raw HTML only.
Two configuration objects control the process. BrowserConfig controls the browser itself, including headless mode and viewport. CrawlerRunConfig controls each crawl, including caching, waits, JavaScript code, and extraction settings.
Basic JavaScript Rendering Setup
The simplest setup needs only a few lines. Create a browser config and a run config, open the crawler with async with, and call arun() with your URL.
import asyncio
from crawl4ai import AsyncWebCrawler, BrowserConfig, CrawlerRunConfig, CacheMode
async def main():
browser_config = BrowserConfig(headless=True)
run_config = CrawlerRunConfig(cache_mode=CacheMode.BYPASS)
async with AsyncWebCrawler(config=browser_config) as crawler:
result = await crawler.arun(url="https://example.com/products", config=run_config)
print(result.markdown)
asyncio.run(main())Use CacheMode.BYPASS while you test a dynamic page. It forces a fresh render on every run, so cached results cannot hide a change.
Wait for Content Before Extraction
Rendering takes time, and extraction must not start too early. A page can return a shell layout in milliseconds, while its product grid or search results load one or two seconds later.
The wait_for option in CrawlerRunConfig fixes this. It pauses extraction until a CSS selector appears or specific text is present. Choose a selector that exists only when the target content has loaded.
Wait for a CSS Selector
Use a stable selector from the content area, such as a product card or results container. Avoid generic selectors like body, because they appear before the data loads.
run_config = CrawlerRunConfig(
cache_mode=CacheMode.BYPASS,
wait_for="css:.product-grid",
)If the selector never appears, check it in your browser developer tools first. The class name may change after hydration, or the content may load only after a click or scroll.
Run Custom JavaScript With js_code
Some pages need more than a wait. They need a click, a scroll, a filter selection, or a tab change before the data appears. The js_code option runs your own JavaScript inside the page before extraction.
Pass js_code as a list of JavaScript strings. Each string runs in the rendered page, so you can use normal DOM methods such as querySelector and click.
Scroll to Load More Content
Infinite scroll pages load new items only when the user scrolls down. A single scroll to the bottom often triggers the next batch. Repeat the scroll if the list loads in several batches.
scroll_js = """
(async () => {
window.scrollTo(0, document.body.scrollHeight);
await new Promise(r => setTimeout(r, 1500));
})();
"""
run_config = CrawlerRunConfig(
cache_mode=CacheMode.BYPASS,
js_code=[scroll_js],
wait_for="css:.product-grid",
)Combine scrolling with scan_full_page for lazy-loaded images and long articles. That option scrolls through the whole page before extraction, so use it only when the page needs it.
Scrape React, Vue, and Next.js SPAs
Single-page applications built with React, Vue, or Next.js often ship a small HTML shell and a large JavaScript bundle. The router then renders the view in the browser. A plain HTTP request sees the shell, while Crawl4AI sees the finished view.
For SPAs, wait for a route-specific element rather than a shared layout element. A header loads on every route, so it proves nothing. A product title or post list proves the correct view rendered.
Handle Client-Side Navigation
Some SPAs change content without a full page reload. If a deep link does not render the view, use js_code to click the navigation link and wait for the new view.
Reuse a session with the same session_id when several steps share cookies and page state. For a full walkthrough of the basic crawl flow, see our Crawl4AI Python tutorial.
Static HTML Scraping vs Browser Rendering
Browser rendering is powerful, but it is not free. It uses more CPU and memory than a plain HTTP request, and it takes longer per page. Use it when the page needs JavaScript, and use a lighter method when the content is already in the HTML.
Check the page source before you choose. If the text you need is in the source, a lighter crawl may be enough. If the source is mostly scripts and empty containers, browser rendering is the right tool.
| Approach | Speed | Handles JavaScript | Best use |
|---|---|---|---|
| Static HTML request | Fast | No | Server-rendered blogs and docs |
| Crawl4AI browser rendering | Slower | Yes | SPAs, dynamic listings, lazy-loaded pages |
When to Choose Browser Rendering
Our Crawl4AI vs Playwright comparison explains when to use the full crawler and when raw Playwright fits better.
Limits: Rendering Is Not an Anti-Bot Bypass
Browser rendering does not remove bot protection. Cloudflare challenges, CAPTCHA pages, and login walls still block automated access. A rendered browser can look like a normal visitor, but it cannot pass a challenge meant for humans.
For protected pages, use authenticated sessions, saved cookies, or a proxy where the site allows it. Respect robots rules and terms of service, and keep request rates reasonable. For large multi-page jobs, plan the crawl first with our Crawl4AI deep crawling guide.
FAQ
Does Crawl4AI run JavaScript by default?
Yes. Crawl4AI loads pages in a real Chromium browser, so JavaScript runs before extraction. You can disable it with java_script_enabled=False if a page does not need it.
How do I wait for dynamic content in Crawl4AI?
Set wait_for in CrawlerRunConfig to a CSS selector or text value. Extraction starts only after that content appears. Use a selector from the data area, not the shared header.
Can Crawl4AI scrape React and Next.js SPAs?
Yes. It renders the app in Chromium and waits for the route content to appear. Wait for a view-specific element, and use js_code for clicks or navigation when a direct link does not render the view.
How do I scrape infinite scroll pages?
Scroll with js_code, wait after each scroll, and repeat until no new items load. Add scan_full_page for lazy-loaded sections, and set a limit to keep the crawl bounded.
Why is my Crawl4AI result empty on a JavaScript page?
The usual causes are a wrong wait selector, content behind a click or scroll, a login wall, or a cached empty render. Check the selector, add the missing trigger, and rerun with CacheMode.BYPASS. For agent workflows that need rendered data, see our Crawl4AI guide for AI agents.
Conclusion
Crawl4AI JavaScript rendering gives you the finished page, not the empty shell. Load the page in Chromium, wait for the content that matters, scroll or click when the page needs it, and extract only after the view is complete.
Start with the basic setup, add one wait at a time, and test each change on a single URL. That small loop turns most dynamic pages and SPAs into clean, reliable Markdown.