Adding Rows to database where last left off

Questions and answers about anything related to Helium Scraper
Post Reply
kfarsad
Posts: 3
Joined: Tue Apr 17, 2012 9:46 am

Adding Rows to database where last left off

Post by kfarsad » Fri Apr 20, 2012 8:04 am

So I am running through multiple pages and extracting the information of each page to a database, but instead of the rows picking up where they left off they are overwriting the existing lines starting from row 1 with each new page. I was wondering if there was an obvious way around this........

Either to have the new data extracted with each new pass to the next empty row in the existing database

Or possibly export the database to a separate file using JS at the end of each pass(with incrementally increasing file name or something so it doesnt get overwritten with each pass)

Or even have a new table created with each pass(dummy1, dummy2, etc)

thanks

vodkasushi
Posts: 11
Joined: Wed Apr 18, 2012 2:57 pm

Re: Adding Rows to database where last left off

Post by vodkasushi » Fri Apr 20, 2012 8:30 am

Wait... how are you overwriting the table rows??? I'm adding onto the ones I have at the moment using "Extract to <table>" action. How do you overwrite them?

kfarsad
Posts: 3
Joined: Tue Apr 17, 2012 9:46 am

Re: Adding Rows to database where last left off

Post by kfarsad » Fri Apr 20, 2012 8:44 am

I dont think its overwriting the table line by line, I think its just replace my existing table with a new one, but im using the same 'extract to [table] function to achieve this

vodkasushi
Posts: 11
Joined: Wed Apr 18, 2012 2:57 pm

Re: Adding Rows to database where last left off

Post by vodkasushi » Fri Apr 20, 2012 12:03 pm

Interesting... I'd like the overwrite function when I restart a scrape - at the moment it simply appends to the existing data.

webmaster
Site Admin
Posts: 521
Joined: Mon Dec 06, 2010 8:39 am
Contact:

Re: Adding Rows to database where last left off

Post by webmaster » Sat Apr 21, 2012 2:11 am

Have you set any of your columns to Unique when creating your extract action? If so, rows with the same values in Unique columns will be updated if the Unique columns are the same as the values that are about to be extracted.

Also, note that the table will be recreated if you edit your Extract action.
Juan Soldi
The Helium Scraper Team

Post Reply