How to scrape an ajax page with load more button? Posted on : 2016-07-13

How to scrape data from a website that has load more button?

Scraping websites which use javascript to load content can sometimes be tricky. In this case you require tool which will mimic a real browser and run the required javascript. More and more sites are implementing dynamic updates of their contents. New items are added as the user scrolls down. Twitter is one of these sites. Twitter only displays a certain number of news items initially, loading additional ones on demand. How can sites with this behavior be scraped?

Visual scraper provides you a SaaS Web scraping software that enables user to mimic exactly like the browser. 

Download and use the visualscraper software now: http://www.visualscraper.com

1/ see the tutorial
2/ make sure you click on the load more button
3/ set the "jump on action" property

 

4/ click the selected position where you want it to jump to "after the click button is fired"

 

How to crawl the whole website AJAX pages, if data append with Jquery GET/POST after scrolling down in browser or sometime a button appeared and ask to Load more. Can I make this happen with my scraping agent?

Also can, however in that case you need to choose web scraping service of visual scraper.

This video may related to these search terms:
How do I scrape data in a page which has jQuery button click to load more items dynamically (Shopping website)?
How do I crawl an infinite-scrolling AJAX website?
How do i scrape data and click on load button or next button forever?
How to scrape ANY websites with AJAX content?