URL Variations - Progress

What do you suggest? What should be improve? We are all ears.
Post Reply
Posts: 3
Joined: Sun Mar 18, 2012 9:26 pm

URL Variations - Progress

Post by Jake » Sat Mar 31, 2012 2:11 pm

Just bought Helium Scraper - It's an amazing product!

That being said - Using the URL Variations online template is incredibly useful - But the one thing that you can't see is the current page you're scraping when using it.
For example, It would be nice to see something like:

Execute Tree: URL Variations - Page 4 of 10

in the actions tree.

Maybe this is possible through scripting?

Site Admin
Posts: 521
Joined: Mon Dec 06, 2010 8:39 am

Re: URL Variations - Progress

Post by webmaster » Sun Apr 01, 2012 7:21 pm

Hi Jake,

This is a cool idea. I've just added it to the our TODO list. For now, though, you can use the log. Here is an example. Just go to the URL Variations tree, open Execute JS (Do Navigations) and replace the existing code by this:

Code: Select all

if(Tree.UserData.currentPage > Tree.UserData.lastPage) 
	return false;
	Global.Log("Page " + Tree.UserData.currentPage + " of " + Tree.UserData.lastPage);
	var url = Tree.UserData.url.replace(new RegExp("\\$", "g"), Tree.UserData.currentPage);
	window.location.href = url;
	Tree.UserData.currentPage += Tree.UserData.stepSize;
	return true;	
As you can see, this code has just one extra line. You can then see the current page at Project -> View Log. Note that you can use the buttons above to filter the kind of messages shown. You might want to keep only User Events active so that it shows only your current page.
Juan Soldi
The Helium Scraper Team

Post Reply