database - Querying large amount of data processed by Hive -

-

Say that I have about 10-20 GB of data in HDFS, which is in the form of a hive table, reducing many maps And after joining on two different datasets it has been received. I need to make the user eligible for this query. What options do I have?

  1. Use to transfer data from HDFS to RGS such as Post GRSL. But I want to spend so much time on data transfers. I have tested HDFS-> RDS only using Sqoop in the same SWS area, and takes 4-8 minutes to 800 MB of data. So you can imagine that ~ 60GB data will be very ineffective. This will be my last resort.
  2. Ask questions directly to my webserver according to user request. I have never used such a hive so I I am doubting about this. It struck me because I just got to know that you can ask a bee table remotely after some port forwarding on the EMR cluster. But being new to the large (ISH) data, I am not sure about the risks associated with this, is it common to do this?
  3. Some other solutions - how do people generally do this kind of thing? Sounds like a very normal work.

Just for completeness, my data looks like this: 

  ID time cat1 cat2 cat3 metrics [200] A 123 1234212133 12 ABC 24 4 , 55,231,34,556,123 .... (~ 200). . .

(time is the era)

And my question looks like this:

choose cat1, corr (metrics [2] , Matrix [3]), core (metrics [2], matrix [4]), core (metrics [2], matrix [5]), core (matrix [2], matrix [6]) / P>

I need correlation function, which is why I chose postgresql on MySQL.

You have the correlation function:

corr (col1, Col2)

Pearson gives coefficient of correlation of a pair of numerical columns in the group.


Comments

Post a Comment

Popular posts from this blog

java - Can't add JTree to JPanel of a JInternalFrame -

-
Image
There are two panels in my JInternalframe. I want to add TopPanel to Jtree to nominate another. But I can not add Jaitari to the top of the panel. Please help me in this piece of code: DefaultMutableTreeNode root = new DefaultMutableTreeNode ("deck"); DefaultMutableTreeNode Item Clubs = New DefaultMutableTreeNode ("Club"); AddAllCard (itemClubs); Root.add (itemClubs); DefaultMutableTreeNode item = New DefaultMutableTreeNode ("Diamonds"); AddAllCard (itemDiamonds); Root.add (itemDiamonds); DefaultMutableTreeNode Itemspads = New DefaultMutableTreeNode ("Spades"); AddAllCard (itemSpades); Root.add (itemSpades); DefaultMutableTreeNode item increases = new DefaultMutableTreeNode ("heart"); AddAllCard (itemHearts); Root.add (itemHearts); Default Trimodel Tree Model = New DefaultTreeModel (Route); Tree = new jetty (tree modal); ScrollPane = new JScrollPane (tree); // scrollPane.setViewportView (tree); . ScrollPane.getViewport () add (tr
Read more

javascript - data.match(var) not working it seems -

-
OK, I have this small block code, which is to scanned a number on the website, but for some reason There is a hip that it is not scanning at all. var regenges = / \ & lt; Span class = "currency-robux" data-to-"item-private-value-price" & gt; ([\ D,]) \ & LT; \ / Span \ & gt; /; Var ValueCiling = Data. Match (reggae); Value-song = number (value slicing location (",", "")); Is there anything wrong with that? This is related to the if statement if (value sling and lieutenant; = price watching) { < P> A function calls to move from there, but for some reason it does not seem to run at all. So I think the Regex is wrong but I'm not sure how. (There is a variable in PriceWinking , this is a snippet from the code.) On the website, I'm trying to remove it. & lt; Span class = "robux" data-plus = "item-privatesale-price" & gt; 115 & lt; / Span & gt; Keep in mind
Read more

javascript - How can I pause a jQuery .each() loop, while waiting for user input? -

-
Before I continue to get a jQuery .each loop to wait for user input There is a problem in repeatedly repeating This is my code (I have commented on what I would like to do): & lt; Html xmlns = "http://www.w3.org/1999/xhtml" & gt; & Lt; Top & gt; & Lt; Link rel = "stylesheet" type = "text / css" href = "common_components.css" /> & Lt; Meta http-equiv = "content-type" content = "text / html; charset = UTF-8" /> & Lt; Link rel = "stylesheet" href = "slider.css" & gt; & Lt ;? Php include_once 'common_components.php'; $ Query = $ dbh- & gt; Queries ("SELECT movies_dir FROM settings"); $ Line = $ query- & gt; Obtain (PDO :: FETCH_ASSOC); Date_default_timezone_set ('UTC'); Echo "& lt; title & gt; movie scanner & lt; / title & gt;"; ? & Gt; & Lt; Script src = "jquery.min.js" & gt; &am
Read more