So I am running through multiple pages and extracting the information of each page to a database, but instead of the rows picking up where they left off they are overwriting the existing lines starting from row 1 with each new page. I was wondering if there was an obvious way around this........
Either to have the new data extracted with each new pass to the next empty row in the existing database
Or possibly export the database to a separate file using JS at the end of each pass(with incrementally increasing file name or something so it doesnt get overwritten with each pass)
Or even have a new table created with each pass(dummy1, dummy2, etc)
thanks
Adding Rows to database where last left off
-
- Posts: 11
- Joined: Wed Apr 18, 2012 2:57 pm
Re: Adding Rows to database where last left off
Wait... how are you overwriting the table rows??? I'm adding onto the ones I have at the moment using "Extract to <table>" action. How do you overwrite them?
Re: Adding Rows to database where last left off
I dont think its overwriting the table line by line, I think its just replace my existing table with a new one, but im using the same 'extract to [table] function to achieve this
-
- Posts: 11
- Joined: Wed Apr 18, 2012 2:57 pm
Re: Adding Rows to database where last left off
Interesting... I'd like the overwrite function when I restart a scrape - at the moment it simply appends to the existing data.
Re: Adding Rows to database where last left off
Have you set any of your columns to Unique when creating your extract action? If so, rows with the same values in Unique columns will be updated if the Unique columns are the same as the values that are about to be extracted.
Also, note that the table will be recreated if you edit your Extract action.
Also, note that the table will be recreated if you edit your Extract action.
Juan Soldi
The Helium Scraper Team
The Helium Scraper Team