Fetching a lot of urls in python with google app engine -

-

In my sub-category of RequestHandler, I am trying to get the URL range: 

  class GetStats (webapp2.RequestHandler): Deaf Post (self): Last page for page in category = 50 (1, final page): tmpurl = url + str (page) response = urllib2.urlopen (tmpurl, timeout = 5) html = response (read) # some parsing HTML heap.append (result_of_parsing) self.response.write (heap)

but it works with ~ 30 urls (page Is getting taller but it works). In more than 30 cases, I get an error:

Error: Server Error

The server encountered an error and did not complete your request Could go

Please try again in 30 seconds.

Is there any way to get too many URLs? Can be more optimal or mouth? To hundreds of pages?

Update:

I'm using prettySupup to parse every single page. I found this traceback in the GAI log: 

  traceback (most recent call final): file "/ base / data / home / runtime / python 27 / python 27_ lib / version / 1 / Google / appenagine / runtime /wsgi.py ", line 267, handle result = handler (dict.self._environ), self._StartResponse) file" / base / data / home / runtimes / python27 / python27_lib / versions / third_party / Webapp2-2.5.2 /webapp2.py ", line 1529, __call__ rv = self.router.dispatch (request, response) in the file" /base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.5 .2 / webapp2. Py ", line 1278, default_dispatcher return path.handler_adapter (un Occlusion, feedback) "/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.5.2/webapp2.py" file, line 1102, in __call__ returns handler. Dispatch () file "/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.5.2/webapp2.py", line 570, (* args, * * Kwargs) in the sending withdrawal method " /base/data/home/apps/s~gae/1.379703839015039430/main.py ", line 68, later hept = get_times (tmp_url, 160) file" /base/data/home/apps/s~gae/1.379703839015039430/ "Libs / bs4 / __ init__.py" in line 168, __init__ self._feed () file, line_6, in line 106, get_times soup = beautiful file (html) "libs / bs4 / _init__.py" 181, _feed self.builder.feed (self.markup) file "libs / bs4 / builder / _htmlparser.py", in line 4, Feed Super (HTMLParserTreeBuilder, itself) .feed (markup) file "/ base / data /home/runtimes/python27/python27_dist/lib/python2.7/HTMLParser.py ", line 114, feed (0) file in the self.goahead" / base / data / home / runtimes / python27 / python27_dist / lib / python2. 7 / HTMLParser.py ", line 155, goahead startswith = rawdata.startswith deadlineExceededError

Because you only have 60 seconds to return a response to the user it is to be unsuccessful and I think that going to do it is now taking that

You use this Want to:

There is a 10 minute time that to make a work out. You can then return to the user immediately and they can "lift" the subsequent results through the other handler (which you make). If all the URLs are collected then it takes 10 minutes, then you have to divide them into further tasks.

See this:

The reason for understanding is that you can not go for 60 seconds for a long time.


Comments

Post a Comment

Popular posts from this blog

java - Can't add JTree to JPanel of a JInternalFrame -

-
Image
There are two panels in my JInternalframe. I want to add TopPanel to Jtree to nominate another. But I can not add Jaitari to the top of the panel. Please help me in this piece of code: DefaultMutableTreeNode root = new DefaultMutableTreeNode ("deck"); DefaultMutableTreeNode Item Clubs = New DefaultMutableTreeNode ("Club"); AddAllCard (itemClubs); Root.add (itemClubs); DefaultMutableTreeNode item = New DefaultMutableTreeNode ("Diamonds"); AddAllCard (itemDiamonds); Root.add (itemDiamonds); DefaultMutableTreeNode Itemspads = New DefaultMutableTreeNode ("Spades"); AddAllCard (itemSpades); Root.add (itemSpades); DefaultMutableTreeNode item increases = new DefaultMutableTreeNode ("heart"); AddAllCard (itemHearts); Root.add (itemHearts); Default Trimodel Tree Model = New DefaultTreeModel (Route); Tree = new jetty (tree modal); ScrollPane = new JScrollPane (tree); // scrollPane.setViewportView (tree); . ScrollPane.getViewport () add (tr
Read more

javascript - data.match(var) not working it seems -

-
OK, I have this small block code, which is to scanned a number on the website, but for some reason There is a hip that it is not scanning at all. var regenges = / \ & lt; Span class = "currency-robux" data-to-"item-private-value-price" & gt; ([\ D,]) \ & LT; \ / Span \ & gt; /; Var ValueCiling = Data. Match (reggae); Value-song = number (value slicing location (",", "")); Is there anything wrong with that? This is related to the if statement if (value sling and lieutenant; = price watching) { < P> A function calls to move from there, but for some reason it does not seem to run at all. So I think the Regex is wrong but I'm not sure how. (There is a variable in PriceWinking , this is a snippet from the code.) On the website, I'm trying to remove it. & lt; Span class = "robux" data-plus = "item-privatesale-price" & gt; 115 & lt; / Span & gt; Keep in mind
Read more

javascript - How can I pause a jQuery .each() loop, while waiting for user input? -

-
Before I continue to get a jQuery .each loop to wait for user input There is a problem in repeatedly repeating This is my code (I have commented on what I would like to do): & lt; Html xmlns = "http://www.w3.org/1999/xhtml" & gt; & Lt; Top & gt; & Lt; Link rel = "stylesheet" type = "text / css" href = "common_components.css" /> & Lt; Meta http-equiv = "content-type" content = "text / html; charset = UTF-8" /> & Lt; Link rel = "stylesheet" href = "slider.css" & gt; & Lt ;? Php include_once 'common_components.php'; $ Query = $ dbh- & gt; Queries ("SELECT movies_dir FROM settings"); $ Line = $ query- & gt; Obtain (PDO :: FETCH_ASSOC); Date_default_timezone_set ('UTC'); Echo "& lt; title & gt; movie scanner & lt; / title & gt;"; ? & Gt; & Lt; Script src = "jquery.min.js" & gt; &am
Read more