python - Remove multiple items from a numy.narray without numpy.delete -

-

I am using a large nump.narray (11.000x3180) to develop an active learning algorithm (text mining) . In this algorithm, I have to delete every Ireshian 16 samples (line vectors) in my dataset, and then have to integrate them into the training set (this increment increases in 16 samples). After performing this process for 60 iterations (approximately), the same process is started from the beginning of the algorithm repeatedly starting from 100

In my data set, set of 16 elements To delete, I use the method numpy.delete (dataset [listfoundx], axis = 0) , where [ListifoIndex] indexes of selected objects to be removed Corresponds to

This method works for the first time (1 out of 100), but again the algorithm starts again, I have the following error: 

  new = Empty (newshape, arr .dtype, arr.flags.fnc) Random

Obviously numpy.delete metod creates a copy of my database for each index (16x1.2GB), which is more than the amount of memory on my computer.

The question is how can I get the item removed from a numpy.narray, not to use too much memory and without excessive execution time?

PD1: I have done the reverse process, where I add the elements that are not in the index list to remove but the process is very slow. PD2: Sometimes the error occurs before the start of the algorithm (before the recurrence number 60)

It can help you understand exactly what np.delete does in your case 

  newset = np.delete (dataset, listfoundx, axis = 0) # Correct

In short it does: 

  keep = np.ones (dataset.shape [0], dtype = bool) #The truth of the mail Matching first index [ListifoIndex] = False newset = Dataset [keep:]

If I run to run 

  dataset = np.delete (dataset, listfoundx), then its boolean index builds in other words. , Axis = 0)

There is no accumulation of frequent, intermediate arrays in an interactive shell. This will temporarily be a new copy of the array and dataset to keep it going while moving . But with the assignment, the old copy disappears.

Are you sure that this is delete rising memory usage, which is in opposition to increasing training set? For speed in terms of speed, you can improve that by actually retaining the 'mask' of all 'removed' rows instead of removing anything, how it depends on how ListifoIndex overlaps with the previous extinction, updating this mask can cause more trouble than it. It is also likely to have more error prone.


Comments

Post a Comment

Popular posts from this blog

mysql - How to enter php data into a html multiple select box -

-
I am working on a page that will create a team. I just want to create a selection box which will list all the users of all the users. After that, the user can select more than one friend (i.e. 5) and can send the request to these 5 people to join the team but I am having problems I did this: Select $ sql = "Select i. *, Join * Connect with M * m.id = i.clique_id where adder_id =: id"; $ Stmt = $ db- & gt; Query ($ sql, array ('id' = & gt; $ _SESSION ['id'])); While ($ record = $ stmt-> fetch ()) {$ surname = $ record ['surname']; } and this: & lt; Select multiple = "true" name = "category" well id = "category" name = "category" category = "" & gt; & Lt; Option value = "& lt ;? php echo $ surname ;? & gt;" & Gt; & Lt ;? Php ek $ nickname; ? & Gt; & Lt; / Options & gt; & Lt; / Select & gt; But only a friend's nickname...
Read more

java - Can't add JTree to JPanel of a JInternalFrame -

-
Image
There are two panels in my JInternalframe. I want to add TopPanel to Jtree to nominate another. But I can not add Jaitari to the top of the panel. Please help me in this piece of code: DefaultMutableTreeNode root = new DefaultMutableTreeNode ("deck"); DefaultMutableTreeNode Item Clubs = New DefaultMutableTreeNode ("Club"); AddAllCard (itemClubs); Root.add (itemClubs); DefaultMutableTreeNode item = New DefaultMutableTreeNode ("Diamonds"); AddAllCard (itemDiamonds); Root.add (itemDiamonds); DefaultMutableTreeNode Itemspads = New DefaultMutableTreeNode ("Spades"); AddAllCard (itemSpades); Root.add (itemSpades); DefaultMutableTreeNode item increases = new DefaultMutableTreeNode ("heart"); AddAllCard (itemHearts); Root.add (itemHearts); Default Trimodel Tree Model = New DefaultTreeModel (Route); Tree = new jetty (tree modal); ScrollPane = new JScrollPane (tree); // scrollPane.setViewportView (tree); . ScrollPane.getViewport () add (tr...
Read more

c++ - Cassandra datastax cpp driver - avoiding unnecessary copies -

-
I use the following method to create my insert statement for Cassandra (using datastax cpp driver) : ssStmt1 & lt; & Lt; "Insert some table (...) value (....)" string sStatement1 = ssStmt1.str (); Castestring cass1 = cass_string_init (sStatement1.c_str ()); CassStatement * pStmt1 = cass_statement_new (cass1, 0); I have also tried the following: CassString cass1 = cass_string_init ("Enter some table (id, date, c) value (? ,?,?); "); CassStatement * pStmt1 = cass_statement_new (cass1, 3); Cass_statement_bind_uuid (pStmt1, 0, uuidKey); Cass_statement_bind_int64 (PSTMT1, 1, timestamp); Cass_statement_bind_string (pStmt1, 2, sSomething); The first pattern works with mixed keys, the second is not. The key has a timestamp and a UUID. Timestamp is one of two errors in the field (based on the used bond statement). If bound in this way: cass_statement_bind_int64_by_name (pStmt1, "date", timestamp); I get the following error: Inva...
Read more