I'm scraping Fandango for movie times for an app I'm working on, and I'm searching incrementally by zip codes. Not all Zip codes are associated with theaters, though, and those that aren't show a webpage notifying you that there are no results. 97% of the time that the scraper encounters these pages, it breaks, claiming to indefinitely load my JavaScript (three lines that navigate to the next zip code).
I haven't been able to find a way around this, and help would be greatly appreciated.
Here's a copy of the log when navigation fails:
Code: Select all
Navigation Navigation Started 11:12:56 PM
Navigation Partial Navigation Started http://www.fandango.com/36694_movietimes 11:12:56 PM
Navigation Partial Navigation Started http://edge.sharethis.com/share4x/index.46f5bc5fc0e85c1367fbe18fbd1b1792.html#widget/publisherGA=UA-7944652-1 11:12:56 PM
Navigation Partial Navigation Completed http://edge.sharethis.com/share4x/index.46f5bc5fc0e85c1367fbe18fbd1b1792.html#widget/publisherGA=UA-7944652-1 11:12:56 PM
Navigation Partial Navigation Started http://edge.sharethis.com/share4x/index.46f5bc5fc0e85c1367fbe18fbd1b1792.html# 11:12:57 PM
Navigation Partial Navigation Completed http://edge.sharethis.com/share4x/index.46f5bc5fc0e85c1367fbe18fbd1b1792.html# 11:12:57 PM
Navigation Partial Navigation Started http://edge.sharethis.com/share4x/index.46f5bc5fc0e85c1367fbe18fbd1b1792.html#data/bInit=true/publisherKeySet=true/pageInfo={"fpc":"bfcd1ff-137a54c2579-7f345145-134","sessionID":"1338523974755.19808","sourceURL":"http%253A%252F%252Fwww.fandango.com%252F36693_movietimes","hostname":"www.fandango.com","location":"%252F36693_movietimes","publisher":"9cfd4f08-0ecf-4989-a9da-19c9693cda97","shareHash":"sthash.aXW7bu7L","incomingHash":"","refDomain":"www.fandango.com","refQuery":"36692_movietimes"}/shareInfo={"url":"http%253A%252F%252Fwww.fandango.com%252F36693_movietimes","sharURL":"","source":"share4x","title":"Movie%2520Times%2520and%2520Movie%2520Theaters%2520in%252036693%2520-%2520Fandango.com","ts1338523974778.0":""} 11:12:57 PM
Navigation Partial Navigation Completed http://edge.sharethis.com/share4x/index.46f5bc5fc0e85c1367fbe18fbd1b1792.html#data/bInit=true/publisherKeySet=true/pageInfo={"fpc":"bfcd1ff-137a54c2579-7f345145-134","sessionID":"1338523974755.19808","sourceURL":"http%253A%252F%252Fwww.fandango.com%252F36693_movietimes","hostname":"www.fandango.com","location":"%252F36693_movietimes","publisher":"9cfd4f08-0ecf-4989-a9da-19c9693cda97","shareHash":"sthash.aXW7bu7L","incomingHash":"","refDomain":"www.fandango.com","refQuery":"36692_movietimes"}/shareInfo={"url":"http%253A%252F%252Fwww.fandango.com%252F36693_movietimes","sharURL":"","source":"share4x","title":"Movie%2520Times%2520and%2520Movie%2520Theaters%2520in%252036693%2520-%2520Fandango.com","ts1338523974778.0":""} 11:12:57 PM
Navigation Partial Navigation Started http://static.ak.facebook.com/connect/xd_arbiter.php?version=6#channel=f1626918b243763&origin=http://www.fandango.com&channel_path=/36694_movietimes?fb_xd_fragment%23xd_sig=f344adbf6b4dffc&&transport=postmessage 11:12:57 PM
Navigation Partial Navigation Started https://s-static.ak.facebook.com/connect/xd_arbiter.php?version=6#channel=f1626918b243763&origin=http://www.fandango.com&channel_path=/36694_movietimes?fb_xd_fragment%23xd_sig=f344adbf6b4dffc&&transport=postmessage 11:12:57 PM
Navigation Partial Navigation Completed http://static.ak.facebook.com/connect/xd_arbiter.php?version=6#channel=f1626918b243763&origin=http://www.fandango.com&channel_path=/36694_movietimes?fb_xd_fragment%23xd_sig=f344adbf6b4dffc&&transport=postmessage 11:12:57 PM
Navigation Partial Navigation Completed https://s-static.ak.facebook.com/connect/xd_arbiter.php?version=6#channel=f1626918b243763&origin=http://www.fandango.com&channel_path=/36694_movietimes?fb_xd_fragment%23xd_sig=f344adbf6b4dffc&&transport=postmessage 11:12:57 PM
Navigation Partial Navigation Started http://www.facebook.com/dialog/oauth?api_key=5885186655&app_id=5885186655&channel_url=http://static.ak.facebook.com/connect/xd_arbiter.php?version=6%23cb=f36f08b19d67be4&origin=http%253A%252F%252Fwww.fandango.com%252Ff1626918b243763&domain=www.fandango.com&relation=parent.parent&client_id=5885186655&display=none&domain=www.fandango.com&locale=en_US&origin=1&redirect_uri=http://static.ak.facebook.com/connect/xd_arbiter.php?version=6%23cb=f1fa612c8e2c715&origin=http%253A%252F%252Fwww.fandango.com%252Ff1626918b243763&domain=www.fandango.com&relation=parent&frame=f36fcba49b18094&response_type=token,signed_request,code&sdk=joey 11:12:57 PM
Navigation Partial Navigation Completed http://www.facebook.com/dialog/oauth?api_key=5885186655&app_id=5885186655&channel_url=http://static.ak.facebook.com/connect/xd_arbiter.php?version=6%23cb=f36f08b19d67be4&origin=http%253A%252F%252Fwww.fandango.com%252Ff1626918b243763&domain=www.fandango.com&relation=parent.parent&client_id=5885186655&display=none&domain=www.fandango.com&locale=en_US&origin=1&redirect_uri=http://static.ak.facebook.com/connect/xd_arbiter.php?version=6%23cb=f1fa612c8e2c715&origin=http%253A%252F%252Fwww.fandango.com%252Ff1626918b243763&domain=www.fandango.com&relation=parent&frame=f36fcba49b18094&response_type=token,signed_request,code&sdk=joey 11:12:57 PM
Navigation Partial Navigation Started http://view.atdmt.com/AD0/iview/400514306/direct;wi.728;hi.90/01/1603871?click=http://ad.doubleclick.net/click;h=v8/3c87/3/0/*/c;257776846;0-0;0;10934597;3454-728/90;48482089/48481582/1;u=tile*1^size*728x90,960x150^cid*0^pm*1;~okv=;tile=1;entry=no;dcopt=ist;gal=;mv=;dma=mobilepensacola;zip=36693;cid=0;tid=;genre=;rt=;pname=;sweeps=;promo=;env=production;entry=no;sz=728x90,960x150;pm=1;abtest=0;u=tile*1^size*728x90,960x150^cid*0^pm*1;bsg=102437;bsg=103268;bsg=103666;bsg=103670;bsg=111304;;~sscs=? 11:12:58 PM
Navigation Partial Navigation Started http://www.facebook.com/widgets/like.php?width=90&action=like&show_faces=false&layout=button_count&colorscheme=light&height=21&href=http://www.facebook.com/fandango 11:12:58 PM
Navigation Partial Navigation Started http://view.atdmt.com/NYC/iview/399266976/direct/01/1604012?click= 11:12:58 PM
Navigation Partial Navigation Completed http://view.atdmt.com/AD0/iview/400514306/direct;wi.728;hi.90/01/1603871?click=http://ad.doubleclick.net/click;h=v8/3c87/3/0/*/c;257776846;0-0;0;10934597;3454-728/90;48482089/48481582/1;u=tile*1^size*728x90,960x150^cid*0^pm*1;~okv=;tile=1;entry=no;dcopt=ist;gal=;mv=;dma=mobilepensacola;zip=36693;cid=0;tid=;genre=;rt=;pname=;sweeps=;promo=;env=production;entry=no;sz=728x90,960x150;pm=1;abtest=0;u=tile*1^size*728x90,960x150^cid*0^pm*1;bsg=102437;bsg=103268;bsg=103666;bsg=103670;bsg=111304;;~sscs=? 11:12:59 PM
Navigation Partial Navigation Completed http://www.facebook.com/widgets/like.php?width=90&action=like&show_faces=false&layout=button_count&colorscheme=light&height=21&href=http://www.facebook.com/fandango 11:12:59 PM
Navigation Partial Navigation Completed http://view.atdmt.com/NYC/iview/399266976/direct/01/1604012?click= 11:12:59 PM