Fetching a lot of urls in python with google app engine -

-

In my sub-category of RequestHandler, I am trying to get the URL range: 

  class GetStats (webapp2.RequestHandler): Deaf Post (self): Last page for page in category = 50 (1, final page): tmpurl = url + str (page) response = urllib2.urlopen (tmpurl, timeout = 5) html = response (read) # some parsing HTML heap.append (result_of_parsing) self.response.write (heap)

but it works with ~ 30 urls (page Is getting taller but it works). In more than 30 cases, I get an error:

Error: Server Error

The server encountered an error and did not complete your request Could go

Please try again in 30 seconds.

Is there any way to get too many URLs? Can be more optimal or mouth? To hundreds of pages?

Update:

I'm using prettySupup to parse every single page. I found this traceback in the GAI log: 

  traceback (most recent call final): file "/ base / data / home / runtime / python 27 / python 27_ lib / version / 1 / Google / appenagine / runtime /wsgi.py ", line 267, handle result = handler (dict.self._environ), self._StartResponse) file" / base / data / home / runtimes / python27 / python27_lib / versions / third_party / Webapp2-2.5.2 /webapp2.py ", line 1529, __call__ rv = self.router.dispatch (request, response) in the file" /base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.5 .2 / webapp2. Py ", line 1278, default_dispatcher return path.handler_adapter (un Occlusion, feedback) "/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.5.2/webapp2.py" file, line 1102, in __call__ returns handler. Dispatch () file "/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.5.2/webapp2.py", line 570, (* args, * * Kwargs) in the sending withdrawal method " /base/data/home/apps/s~gae/1.379703839015039430/main.py ", line 68, later hept = get_times (tmp_url, 160) file" /base/data/home/apps/s~gae/1.379703839015039430/ "Libs / bs4 / __ init__.py" in line 168, __init__ self._feed () file, line_6, in line 106, get_times soup = beautiful file (html) "libs / bs4 / _init__.py" 181, _feed self.builder.feed (self.markup) file "libs / bs4 / builder / _htmlparser.py", in line 4, Feed Super (HTMLParserTreeBuilder, itself) .feed (markup) file "/ base / data /home/runtimes/python27/python27_dist/lib/python2.7/HTMLParser.py ", line 114, feed (0) file in the self.goahead" / base / data / home / runtimes / python27 / python27_dist / lib / python2. 7 / HTMLParser.py ", line 155, goahead startswith = rawdata.startswith deadlineExceededError

Because you only have 60 seconds to return a response to the user it is to be unsuccessful and I think that going to do it is now taking that

You use this Want to:

There is a 10 minute time that to make a work out. You can then return to the user immediately and they can "lift" the subsequent results through the other handler (which you make). If all the URLs are collected then it takes 10 minutes, then you have to divide them into further tasks.

See this:

The reason for understanding is that you can not go for 60 seconds for a long time.


Comments

Post a Comment

Popular posts from this blog

mysql - How to enter php data into a html multiple select box -

-
I am working on a page that will create a team. I just want to create a selection box which will list all the users of all the users. After that, the user can select more than one friend (i.e. 5) and can send the request to these 5 people to join the team but I am having problems I did this: Select $ sql = "Select i. *, Join * Connect with M * m.id = i.clique_id where adder_id =: id"; $ Stmt = $ db- & gt; Query ($ sql, array ('id' = & gt; $ _SESSION ['id'])); While ($ record = $ stmt-> fetch ()) {$ surname = $ record ['surname']; } and this: & lt; Select multiple = "true" name = "category" well id = "category" name = "category" category = "" & gt; & Lt; Option value = "& lt ;? php echo $ surname ;? & gt;" & Gt; & Lt ;? Php ek $ nickname; ? & Gt; & Lt; / Options & gt; & Lt; / Select & gt; But only a friend's nickname...
Read more

java - Can't add JTree to JPanel of a JInternalFrame -

-
Image
There are two panels in my JInternalframe. I want to add TopPanel to Jtree to nominate another. But I can not add Jaitari to the top of the panel. Please help me in this piece of code: DefaultMutableTreeNode root = new DefaultMutableTreeNode ("deck"); DefaultMutableTreeNode Item Clubs = New DefaultMutableTreeNode ("Club"); AddAllCard (itemClubs); Root.add (itemClubs); DefaultMutableTreeNode item = New DefaultMutableTreeNode ("Diamonds"); AddAllCard (itemDiamonds); Root.add (itemDiamonds); DefaultMutableTreeNode Itemspads = New DefaultMutableTreeNode ("Spades"); AddAllCard (itemSpades); Root.add (itemSpades); DefaultMutableTreeNode item increases = new DefaultMutableTreeNode ("heart"); AddAllCard (itemHearts); Root.add (itemHearts); Default Trimodel Tree Model = New DefaultTreeModel (Route); Tree = new jetty (tree modal); ScrollPane = new JScrollPane (tree); // scrollPane.setViewportView (tree); . ScrollPane.getViewport () add (tr...
Read more

c++ - Cassandra datastax cpp driver - avoiding unnecessary copies -

-
I use the following method to create my insert statement for Cassandra (using datastax cpp driver) : ssStmt1 & lt; & Lt; "Insert some table (...) value (....)" string sStatement1 = ssStmt1.str (); Castestring cass1 = cass_string_init (sStatement1.c_str ()); CassStatement * pStmt1 = cass_statement_new (cass1, 0); I have also tried the following: CassString cass1 = cass_string_init ("Enter some table (id, date, c) value (? ,?,?); "); CassStatement * pStmt1 = cass_statement_new (cass1, 3); Cass_statement_bind_uuid (pStmt1, 0, uuidKey); Cass_statement_bind_int64 (PSTMT1, 1, timestamp); Cass_statement_bind_string (pStmt1, 2, sSomething); The first pattern works with mixed keys, the second is not. The key has a timestamp and a UUID. Timestamp is one of two errors in the field (based on the used bond statement). If bound in this way: cass_statement_bind_int64_by_name (pStmt1, "date", timestamp); I get the following error: Inva...
Read more