Updating many json files in Azure blob storage? -

-

I have ~ 1,000,000 JSON files, I would like to update every 30 minutes. Update is only enclosing a new array at the end of the existing content.

Uses the same code as a single update: 

  CloudBlockBlob blockBlob = container.GetBlockBlobReference (blobName); Jobbject jObject = null; // If the blob is present, then we may need to update it. If (blockBlob.Exists ()) {MemoryStream memoryStream = new MemoryStream (); BlockBlob.DownloadToStream (MemoryStream); JObject = JsonConvert.DeserializeObject (encoding.UTF8.GetString (MemoryStream. ToArray ()) functioning; } // End of Blease is present if (null == jObject) {jObject = new Jobbject (); JObject.Add (new JProperty ("identifier", identifier)); } // Closing of the block did not exist JArray jsonArray = new JArray (); JObject.Add (New JProperty (string.Format ("Entries {0}", Timestamp. ToString ()), jsonArray); Foreign Currency (Miscellaneous Entries in New Entry) {JSNARRA.ed (new JobTech (new JPPrint ("Some ID", Entry.), New JProperty ("Some Value", Value)); } // the end of the loop string jsonString = JsonConvert.SerializeObject (jObject); // upload blockblock Properties Type Content = "Text / Jason"; Blockblock UploadFomstream (new memorystream (encoding.utf 8.gatebites (jasonstring)); Basically:  
     
  1.  Check if the blob exists,  
  2.  If this happens, download the data and create a Jason object.  
  3.  If it has not, create a new object with the details.  
  4.  Push the blob update.   
     Problems with it is performance demonstrations I have done a lot that I can do to increase the performance (updates run in five parallel threads and I use  ServicePointManager.UseNagleAlgorithm  Is set to false. 
     
     It still runs at a slow pace, however, approximately 100,000 updates can take up to one hour. 
     
     So I was basically guessed , My question will be: 
     
       
    1.  Do I use blue blob for this 
      2.  
    2.  If yes, any suggestion for improving performance? 
      3.  
     
     Note: the original in the file There is a history of events and I can not re-generate the full file based on the existing data. Therefore, content is downloaded prior to updating.


Comments

Post a Comment

Popular posts from this blog

Active Directory Authenticated Proxy Server with Squid or CNTLM -

-
I need to configure a proxy server for a local network that authenticates with Active Directory and gives me access The control list enables you to configure the domain groups within AD. I am using: Static with Squid 2.7: Windows 2008 Server R2 I have followed this guide here: The proxy does not currently ask for any Active Directory accounts, as soon as I add the following lines, it starts denying this connection: ACL localnet proxy_auth Required src 192.168. 0.0 / 24 http_access permission localname This is my squid.conf: pastebin.com/ERuKtirc I see logs and 'none' or 'direct' L It comes together TCP_DENIED message 1415014073.069 1 192.168.0.63 TCP_DENIED / 407/1904 Received / -. None / - Text / html 1415014073.07 9 4 192.168.0.63 TCP_DENIED / 407 2304 Received / - None / - Text / html 1415014073.084 2 192.168.0.63 TCPDAD / 407 1904 Received / - None - - Text / html itemprop = "text"> This proxy returns a 407 request with a browse...
Read more

java - @PluginTarget(Node.class) not registering as neo4j plugin -

-
Simple class below will not show as a plugin package com.danmacias neoneoteplugin ; Import java.util.ArrayList; Import java.util.list; Import org.neo4j.graphdb. *; Import org.neo4j.server.plugins *; Import org.neo4j.graphalgo *; // import org.neo4j.tooling.GlobalGraphOperations; @ Description (expansion of Neo4j server to achieve all nodes or relationships) Public class GetPaths expanded the server plugin {@ name ("breadth_first_paths") @ Description ("first wide travel traversal of specific node"). @plugin target (node ​​class) public shaky & lt; Path & gt; GetPaths (@ Source Node Graphidi) {ArrayList & lt; Path & gt; Path = new arreelist & lt; & Gt; (); Return path; }} curl return {"extension": {"GetAll": {"get_all_nodes": "http: // localhost: 7474 / db / ext / GetAll / graphdb / get_all_nodes "} ...}, But when I Node.class both conditions GraphDatabaseService I change with Class ,...
Read more

python - Calculate STD manually using Groupby Pandas DataFrame -

-
Image
I was trying to write a solution, which is a different and a manual for calculating a mean and STD The way. I have created a = ["Apple", "Banana", "Cherry", "Apple"] B = [3,4,7,3] C = [5,4,1,4] D = [7,8,3,7] Pd DF = PD to import pandals DataFrame (index = class (4), column = list ("ABCD")) DF ["A"] = A DF ["B"] = BDF ["C"] = C DF ["D"] = D Again, I made a list of A duplication. Then I went through the group all the time of the objects and calculated the solution. import as np l = list (set (df.A)) df.groupby ('A', As_index = False) listMean = [0] * len (df.C) ListSTD = [0] * L in the LAN (df.C) X: s = np.mean (df [df ['A'] == x] for C =.) = Z = [index for index, enumerate In the object (df ['a']. Value] x == item i for z: listMean [i] = s in: s = np.std (df [df ['a'] == X] .cvalues) z = index for index, enumerate item (df ['a']. Value) if x ==...
Read more