python - Remove multiple items from a numy.narray without numpy.delete -

-

I am using a large nump.narray (11.000x3180) to develop an active learning algorithm (text mining) . In this algorithm, I have to delete every Ireshian 16 samples (line vectors) in my dataset, and then have to integrate them into the training set (this increment increases in 16 samples). After performing this process for 60 iterations (approximately), the same process is started from the beginning of the algorithm repeatedly starting from 100

In my data set, set of 16 elements To delete, I use the method numpy.delete (dataset [listfoundx], axis = 0) , where [ListifoIndex] indexes of selected objects to be removed Corresponds to

This method works for the first time (1 out of 100), but again the algorithm starts again, I have the following error: 

  new = Empty (newshape, arr .dtype, arr.flags.fnc) Random

Obviously numpy.delete metod creates a copy of my database for each index (16x1.2GB), which is more than the amount of memory on my computer.

The question is how can I get the item removed from a numpy.narray, not to use too much memory and without excessive execution time?

PD1: I have done the reverse process, where I add the elements that are not in the index list to remove but the process is very slow. PD2: Sometimes the error occurs before the start of the algorithm (before the recurrence number 60)

It can help you understand exactly what np.delete does in your case 

  newset = np.delete (dataset, listfoundx, axis = 0) # Correct

In short it does: 

  keep = np.ones (dataset.shape [0], dtype = bool) #The truth of the mail Matching first index [ListifoIndex] = False newset = Dataset [keep:]

If I run to run 

  dataset = np.delete (dataset, listfoundx), then its boolean index builds in other words. , Axis = 0)

There is no accumulation of frequent, intermediate arrays in an interactive shell. This will temporarily be a new copy of the array and dataset to keep it going while moving . But with the assignment, the old copy disappears.

Are you sure that this is delete rising memory usage, which is in opposition to increasing training set? For speed in terms of speed, you can improve that by actually retaining the 'mask' of all 'removed' rows instead of removing anything, how it depends on how ListifoIndex overlaps with the previous extinction, updating this mask can cause more trouble than it. It is also likely to have more error prone.


Comments

Post a Comment

Popular posts from this blog

java - Can't add JTree to JPanel of a JInternalFrame -

-
Image
There are two panels in my JInternalframe. I want to add TopPanel to Jtree to nominate another. But I can not add Jaitari to the top of the panel. Please help me in this piece of code: DefaultMutableTreeNode root = new DefaultMutableTreeNode ("deck"); DefaultMutableTreeNode Item Clubs = New DefaultMutableTreeNode ("Club"); AddAllCard (itemClubs); Root.add (itemClubs); DefaultMutableTreeNode item = New DefaultMutableTreeNode ("Diamonds"); AddAllCard (itemDiamonds); Root.add (itemDiamonds); DefaultMutableTreeNode Itemspads = New DefaultMutableTreeNode ("Spades"); AddAllCard (itemSpades); Root.add (itemSpades); DefaultMutableTreeNode item increases = new DefaultMutableTreeNode ("heart"); AddAllCard (itemHearts); Root.add (itemHearts); Default Trimodel Tree Model = New DefaultTreeModel (Route); Tree = new jetty (tree modal); ScrollPane = new JScrollPane (tree); // scrollPane.setViewportView (tree); . ScrollPane.getViewport () add (tr
Read more

javascript - data.match(var) not working it seems -

-
OK, I have this small block code, which is to scanned a number on the website, but for some reason There is a hip that it is not scanning at all. var regenges = / \ & lt; Span class = "currency-robux" data-to-"item-private-value-price" & gt; ([\ D,]) \ & LT; \ / Span \ & gt; /; Var ValueCiling = Data. Match (reggae); Value-song = number (value slicing location (",", "")); Is there anything wrong with that? This is related to the if statement if (value sling and lieutenant; = price watching) { < P> A function calls to move from there, but for some reason it does not seem to run at all. So I think the Regex is wrong but I'm not sure how. (There is a variable in PriceWinking , this is a snippet from the code.) On the website, I'm trying to remove it. & lt; Span class = "robux" data-plus = "item-privatesale-price" & gt; 115 & lt; / Span & gt; Keep in mind
Read more

javascript - How can I pause a jQuery .each() loop, while waiting for user input? -

-
Before I continue to get a jQuery .each loop to wait for user input There is a problem in repeatedly repeating This is my code (I have commented on what I would like to do): & lt; Html xmlns = "http://www.w3.org/1999/xhtml" & gt; & Lt; Top & gt; & Lt; Link rel = "stylesheet" type = "text / css" href = "common_components.css" /> & Lt; Meta http-equiv = "content-type" content = "text / html; charset = UTF-8" /> & Lt; Link rel = "stylesheet" href = "slider.css" & gt; & Lt ;? Php include_once 'common_components.php'; $ Query = $ dbh- & gt; Queries ("SELECT movies_dir FROM settings"); $ Line = $ query- & gt; Obtain (PDO :: FETCH_ASSOC); Date_default_timezone_set ('UTC'); Echo "& lt; title & gt; movie scanner & lt; / title & gt;"; ? & Gt; & Lt; Script src = "jquery.min.js" & gt; &am
Read more