database - Querying large amount of data processed by Hive -

-

Say that I have about 10-20 GB of data in HDFS, which is in the form of a hive table, reducing many maps And after joining on two different datasets it has been received. I need to make the user eligible for this query. What options do I have?

  1. Use to transfer data from HDFS to RGS such as Post GRSL. But I want to spend so much time on data transfers. I have tested HDFS-> RDS only using Sqoop in the same SWS area, and takes 4-8 minutes to 800 MB of data. So you can imagine that ~ 60GB data will be very ineffective. This will be my last resort.
  2. Ask questions directly to my webserver according to user request. I have never used such a hive so I I am doubting about this. It struck me because I just got to know that you can ask a bee table remotely after some port forwarding on the EMR cluster. But being new to the large (ISH) data, I am not sure about the risks associated with this, is it common to do this?
  3. Some other solutions - how do people generally do this kind of thing? Sounds like a very normal work.

Just for completeness, my data looks like this: 

  ID time cat1 cat2 cat3 metrics [200] A 123 1234212133 12 ABC 24 4 , 55,231,34,556,123 .... (~ 200). . .

(time is the era)

And my question looks like this:

choose cat1, corr (metrics [2] , Matrix [3]), core (metrics [2], matrix [4]), core (metrics [2], matrix [5]), core (matrix [2], matrix [6]) / P>

I need correlation function, which is why I chose postgresql on MySQL.

You have the correlation function:

corr (col1, Col2)

Pearson gives coefficient of correlation of a pair of numerical columns in the group.


Comments

Post a Comment

Popular posts from this blog

mysql - How to enter php data into a html multiple select box -

-
I am working on a page that will create a team. I just want to create a selection box which will list all the users of all the users. After that, the user can select more than one friend (i.e. 5) and can send the request to these 5 people to join the team but I am having problems I did this: Select $ sql = "Select i. *, Join * Connect with M * m.id = i.clique_id where adder_id =: id"; $ Stmt = $ db- & gt; Query ($ sql, array ('id' = & gt; $ _SESSION ['id'])); While ($ record = $ stmt-> fetch ()) {$ surname = $ record ['surname']; } and this: & lt; Select multiple = "true" name = "category" well id = "category" name = "category" category = "" & gt; & Lt; Option value = "& lt ;? php echo $ surname ;? & gt;" & Gt; & Lt ;? Php ek $ nickname; ? & Gt; & Lt; / Options & gt; & Lt; / Select & gt; But only a friend's nickname...
Read more

java - Can't add JTree to JPanel of a JInternalFrame -

-
Image
There are two panels in my JInternalframe. I want to add TopPanel to Jtree to nominate another. But I can not add Jaitari to the top of the panel. Please help me in this piece of code: DefaultMutableTreeNode root = new DefaultMutableTreeNode ("deck"); DefaultMutableTreeNode Item Clubs = New DefaultMutableTreeNode ("Club"); AddAllCard (itemClubs); Root.add (itemClubs); DefaultMutableTreeNode item = New DefaultMutableTreeNode ("Diamonds"); AddAllCard (itemDiamonds); Root.add (itemDiamonds); DefaultMutableTreeNode Itemspads = New DefaultMutableTreeNode ("Spades"); AddAllCard (itemSpades); Root.add (itemSpades); DefaultMutableTreeNode item increases = new DefaultMutableTreeNode ("heart"); AddAllCard (itemHearts); Root.add (itemHearts); Default Trimodel Tree Model = New DefaultTreeModel (Route); Tree = new jetty (tree modal); ScrollPane = new JScrollPane (tree); // scrollPane.setViewportView (tree); . ScrollPane.getViewport () add (tr...
Read more

c++ - Cassandra datastax cpp driver - avoiding unnecessary copies -

-
I use the following method to create my insert statement for Cassandra (using datastax cpp driver) : ssStmt1 & lt; & Lt; "Insert some table (...) value (....)" string sStatement1 = ssStmt1.str (); Castestring cass1 = cass_string_init (sStatement1.c_str ()); CassStatement * pStmt1 = cass_statement_new (cass1, 0); I have also tried the following: CassString cass1 = cass_string_init ("Enter some table (id, date, c) value (? ,?,?); "); CassStatement * pStmt1 = cass_statement_new (cass1, 3); Cass_statement_bind_uuid (pStmt1, 0, uuidKey); Cass_statement_bind_int64 (PSTMT1, 1, timestamp); Cass_statement_bind_string (pStmt1, 2, sSomething); The first pattern works with mixed keys, the second is not. The key has a timestamp and a UUID. Timestamp is one of two errors in the field (based on the used bond statement). If bound in this way: cass_statement_bind_int64_by_name (pStmt1, "date", timestamp); I get the following error: Inva...
Read more