How to define an input for python multiprocessing function to take all the files in a directory? -

-

This question sounds basic because I do not know much about multiprocessing, I'm just learning.

I have a Pyro code that processes a bunch of files in a directory. 

  with bridge (process = core) as PP: pp.map (function, list)

Here's my code: 

  path = '/ data / personal' print ("running with PID:% d"% getpid ()) psl = publicsuffixList () d = defaultdict (set) start = time ( ) # Files_list = glob (path) for file name in file: print (filename) for f = open (filename, 'r') n, enumerate the line (f): line = line [: - 1] ip , Reversed_domain_1 = line.split ('|') reversed_domain_2 = Reversed_domain_1.split ('.') Reversed_domain_3 = list (reverse (reversed_domain_2)) domain = ('.joined (reversed_domain_3)) domain = psl.get_public_suffix (Domain) d [IP] .add (domain) ### For domains in domain for IP domain: print (IP, domain)

I will call it in a multiprocessing pool How can i

You can process each file in a different process like this:

< Import from IP Import from OS import from pre> , Import from Globe Import Globe with Multprossing Import Partial path = '/ data / personal' print (with "PID:% d"% getpid ()) ) Import def process_file (psl, filename): print (filename) f = open (filename, 'r') for n, line (line) = ([= 1] ip, reverse_domain_1 = line.plit ('| '') 'Reversed_domain_2 = Reversed_domain_1.split ('.') Reversed_domain_3 = list (reverse (opposite) =) domain = ('domain = psd = public_suffix (domain) return = iptables) domain = psl.get_public_suffix (domain) return ip, domain if __name__ = = "__main__": psl = publicsuffixList () d = defaultdict (set) start = time () files_list = glob (path) pp = pool (process = core) func = partial (process_file, psl) results = pp.imap_unordered (func , Files_list) for P, domain results in: D [IP] .add (domain) p.close () p.join () for IP, domain. In domain: domain for domain: print (IP, domain)

Note that defaultdict is populated in the original process, because you are actually the same < Code> defaultdict can not be shared, code> multiprocessing.Manager . If you want you can do it here, but I do not need it. Instead, as soon as the result of any child is available, we add it to the parent defaultdict instead of map instead of imap_unordered Instead of waiting for us to be ready for all of them, we will be able to get results on demand. The only other noteworthy thing is that in addition to one item from files_list in all child processes, Psl to enable list partial is used to imap_unordered .

Here is an important note: Using the multiprocessing for this kind of operation does not really improve the performance. You are reading a lot of work from your disk, which Can not be extended through many processes; Your hard drive can only work one read at a time. The reading requests for individual files from a group of processes are slowed down in real-time at a slow pace, so that they can be sequentially, because possibly reading a new line from each file For different areas of the physical disk, you have to switch continuously, now it is possible that you are working CPU-bound with each line to dominate I / O time. It is quite expensive for you, in this situation you will get momentum.


Comments

Post a Comment

Popular posts from this blog

mysql - How to enter php data into a html multiple select box -

-
I am working on a page that will create a team. I just want to create a selection box which will list all the users of all the users. After that, the user can select more than one friend (i.e. 5) and can send the request to these 5 people to join the team but I am having problems I did this: Select $ sql = "Select i. *, Join * Connect with M * m.id = i.clique_id where adder_id =: id"; $ Stmt = $ db- & gt; Query ($ sql, array ('id' = & gt; $ _SESSION ['id'])); While ($ record = $ stmt-> fetch ()) {$ surname = $ record ['surname']; } and this: & lt; Select multiple = "true" name = "category" well id = "category" name = "category" category = "" & gt; & Lt; Option value = "& lt ;? php echo $ surname ;? & gt;" & Gt; & Lt ;? Php ek $ nickname; ? & Gt; & Lt; / Options & gt; & Lt; / Select & gt; But only a friend's nickname...
Read more

java - Can't add JTree to JPanel of a JInternalFrame -

-
Image
There are two panels in my JInternalframe. I want to add TopPanel to Jtree to nominate another. But I can not add Jaitari to the top of the panel. Please help me in this piece of code: DefaultMutableTreeNode root = new DefaultMutableTreeNode ("deck"); DefaultMutableTreeNode Item Clubs = New DefaultMutableTreeNode ("Club"); AddAllCard (itemClubs); Root.add (itemClubs); DefaultMutableTreeNode item = New DefaultMutableTreeNode ("Diamonds"); AddAllCard (itemDiamonds); Root.add (itemDiamonds); DefaultMutableTreeNode Itemspads = New DefaultMutableTreeNode ("Spades"); AddAllCard (itemSpades); Root.add (itemSpades); DefaultMutableTreeNode item increases = new DefaultMutableTreeNode ("heart"); AddAllCard (itemHearts); Root.add (itemHearts); Default Trimodel Tree Model = New DefaultTreeModel (Route); Tree = new jetty (tree modal); ScrollPane = new JScrollPane (tree); // scrollPane.setViewportView (tree); . ScrollPane.getViewport () add (tr...
Read more

c++ - Cassandra datastax cpp driver - avoiding unnecessary copies -

-
I use the following method to create my insert statement for Cassandra (using datastax cpp driver) : ssStmt1 & lt; & Lt; "Insert some table (...) value (....)" string sStatement1 = ssStmt1.str (); Castestring cass1 = cass_string_init (sStatement1.c_str ()); CassStatement * pStmt1 = cass_statement_new (cass1, 0); I have also tried the following: CassString cass1 = cass_string_init ("Enter some table (id, date, c) value (? ,?,?); "); CassStatement * pStmt1 = cass_statement_new (cass1, 3); Cass_statement_bind_uuid (pStmt1, 0, uuidKey); Cass_statement_bind_int64 (PSTMT1, 1, timestamp); Cass_statement_bind_string (pStmt1, 2, sSomething); The first pattern works with mixed keys, the second is not. The key has a timestamp and a UUID. Timestamp is one of two errors in the field (based on the used bond statement). If bound in this way: cass_statement_bind_int64_by_name (pStmt1, "date", timestamp); I get the following error: Inva...
Read more