java - Spark data processing with grouping -

-

I need a group group of CSV lines through a specific column and do some processing on each group.

JavaRDD & lt; String & gt; Rows = sc .textfile ("somefile.csv"); JavaPairRDD & lt; String, string & gt; PairRDD = lines.mapToPair (new SomeParser ()); & Lt; String & gt; Keys = pairRDD.keys (). Distinct (). collection (); (String key: keys) {list & lt; String & gt; Rows = jodi rdd Lookup (key); NoOfVisits = rows.size (); Country = COMMA.split (rows.get (0)) [6]; AccessDuration = getAccessDuration (rows, time format); Click the & lt; String, integer & gt; Calculation = gets (rows); Whitepapers = counts.get ("white paper"); Tutorial = counts.get ("Tutorial"); Workshops = Calculations ("Workshop"); Accidental = Calculations ("Concise"); Product page = Calculations ("Product page"); } Private Static Long Dade Pursure (String Destestering) throws ParseException {Simple Data Format Format = New SimpleDateform ("MMM DD Yyyy HH: MMA"); Date date = format PRS (datestring); Return date .getTime (); } Is called dateParser for each row, then it is calculated to get the minimum and maximum usage duration for the group. There are other string matches

pairRDD.lookup is extremely slow .. Is this a better way to do with spark.

I think you can use that column as a key and a GroupByKey . There is no mention on the operation on those lines. If it is an action that combines those lines in some way, then you can also reduce lesser .

Something like this: 

  import org.apache.spark sparkcontact._ // the built-in pair functional pairing = lines.map (parser _) well grouped = pairs.groupByKey / / Here is grouped: (key, iterator [string])

* edit After looking at this process, I think every line Will be more able to contribute, which contributes to it, and then to reduce all of them to the totality Use total. CombOp: 2 functions and one zero: 

  def consolidated Becky [u: class tag] (zeroview: u) (U, U) = & gt; U): RDD [(K, U)]

The first task is a partition aggregator and it will run efficiently through local partitions, local copy Partition Segmented Partial

val line = sc.textFile ("somefile.csv") Something like this:) // Parsed gives a key and a decomposed record of the values ​​of the track: (Key, record ("country", timestamp, "whitepaper", ...)) val records = lines.map (parse (_)) val totals = Records.aggregateByKey ((0, set [string]. Fair, long (Record, Count, Desert, Minetime, Maximum Time, Countermap)) => (Count + 1, countrySet + record.country, mathematics). .min (minTime, record.timestamp), math.max (maxTime, record.timestamp), ...) (cumm1, cumm2) = & gt; ??? //


Comments

Post a Comment

Popular posts from this blog

java - Can't add JTree to JPanel of a JInternalFrame -

-
Image
There are two panels in my JInternalframe. I want to add TopPanel to Jtree to nominate another. But I can not add Jaitari to the top of the panel. Please help me in this piece of code: DefaultMutableTreeNode root = new DefaultMutableTreeNode ("deck"); DefaultMutableTreeNode Item Clubs = New DefaultMutableTreeNode ("Club"); AddAllCard (itemClubs); Root.add (itemClubs); DefaultMutableTreeNode item = New DefaultMutableTreeNode ("Diamonds"); AddAllCard (itemDiamonds); Root.add (itemDiamonds); DefaultMutableTreeNode Itemspads = New DefaultMutableTreeNode ("Spades"); AddAllCard (itemSpades); Root.add (itemSpades); DefaultMutableTreeNode item increases = new DefaultMutableTreeNode ("heart"); AddAllCard (itemHearts); Root.add (itemHearts); Default Trimodel Tree Model = New DefaultTreeModel (Route); Tree = new jetty (tree modal); ScrollPane = new JScrollPane (tree); // scrollPane.setViewportView (tree); . ScrollPane.getViewport () add (tr
Read more

javascript - data.match(var) not working it seems -

-
OK, I have this small block code, which is to scanned a number on the website, but for some reason There is a hip that it is not scanning at all. var regenges = / \ & lt; Span class = "currency-robux" data-to-"item-private-value-price" & gt; ([\ D,]) \ & LT; \ / Span \ & gt; /; Var ValueCiling = Data. Match (reggae); Value-song = number (value slicing location (",", "")); Is there anything wrong with that? This is related to the if statement if (value sling and lieutenant; = price watching) { < P> A function calls to move from there, but for some reason it does not seem to run at all. So I think the Regex is wrong but I'm not sure how. (There is a variable in PriceWinking , this is a snippet from the code.) On the website, I'm trying to remove it. & lt; Span class = "robux" data-plus = "item-privatesale-price" & gt; 115 & lt; / Span & gt; Keep in mind
Read more

javascript - How can I pause a jQuery .each() loop, while waiting for user input? -

-
Before I continue to get a jQuery .each loop to wait for user input There is a problem in repeatedly repeating This is my code (I have commented on what I would like to do): & lt; Html xmlns = "http://www.w3.org/1999/xhtml" & gt; & Lt; Top & gt; & Lt; Link rel = "stylesheet" type = "text / css" href = "common_components.css" /> & Lt; Meta http-equiv = "content-type" content = "text / html; charset = UTF-8" /> & Lt; Link rel = "stylesheet" href = "slider.css" & gt; & Lt ;? Php include_once 'common_components.php'; $ Query = $ dbh- & gt; Queries ("SELECT movies_dir FROM settings"); $ Line = $ query- & gt; Obtain (PDO :: FETCH_ASSOC); Date_default_timezone_set ('UTC'); Echo "& lt; title & gt; movie scanner & lt; / title & gt;"; ? & Gt; & Lt; Script src = "jquery.min.js" & gt; &am
Read more