Recent Blog Posts

  • How to manage Microsoft Azure Blob Storage with Node.js
    POSTED ON: 08/01/2014 10:14 AM  BY:  [Infragistics] Mihail Mateev
    Node.js is one one of the the most popular growing platforms for development. The language, used with Node.js is JavaScript - he #1 most-used language on GitHub, and this trend is only going to increase (Forbes, 14 of July, 2014) :      I started  several articles, describing some details how to use Node.js with Microsoft SQL Server ( you can see part 1 and part 2 ) and   Microsoft Azure Storage (the post about Azure Table Storage is available here) .   This blog is about how you can use Node.js and Microsoft Azure Blob Storage .  You will learn how to start with Node and Blob Storage, covering the most often seen cases.   Microsoft Azure Blob Storage Microsoft Azure storage services allow us to store/retrieve the NON RELATIONAL data to/from Windows Cloud environment. In Microsoft Azure Storage, the data can be stored in 4 different formats (v.i.z. Blobs, Tables and Queues, File Storage (in preview )). More details about what is Microsoft Azure Storage in general you can read in my previous post.   What is a Blob? Azure Blob storage is a service for storing large amounts of unstructured data that can be accessed from anywhere in the world via HTTP or HTTPS. A single blob can be hundreds of gigabytes in size. The Blob service contains the following components: Storage Account: All access to Azure Storage is done through a storage account. Container: A container provides a grouping of a set of blobs. All blobs must be in a container. An account can contain an unlimited number of containers. A container can store an unlimited number of blobs. Blob: A file of any type and size. There are two types of blobs that can be stored in Azure Storage: block and page blobs. Most files are block blobs. A single block blob can be up to 200 GB in size. This tutorial uses block blobs. Page blobs, another blob type, can be up to 1 TB in size, and are more efficient when ranges of bytes in a file are modified frequently.   Blob stands for 'binary large object', which is an array of raw bytes. The following figure shows the hierarchical structure used for the Blob storage in Windows Azure:   Azure Storage Account can have multiple Containers. A Container can be considered as an array or collection of one or more Blobs. Also, each Blob can have one or more metadata properties to define what the Blob content is all about. The metadata properties are Name-Value collection of strings. As mentioned before, the contents of every Blob on Microsoft Azure can be accessed by browsing its corresponding URI (REST). The URI is usually of the following format: https://<Account>.blob.core.windows.net/<Container>/<BlobName>   How to use Blob Storage and Node.JS? You can use different NodeJS packages to handle Azure Blob Storage. In this post we will cover azure Node package ( an official Microsoft Azure SDK ).   Microsoft Azure SDK for Node.js The  official Microsoft Azure SDK (npm page:  https://www.npmjs.org/package/azure , homepage: http://github.com/WindowsAzure/azure-sdk-for-node  ) for Node.js.  provides a Node.js package that makes it easy to consume and manage Microsoft Azure Services. The listed snippets below demonstrate how to manage Azure Blob service using Microsoft Azure SDK for Node.js.   Create an Azure Blob Service 1: var azure = require('azure'); 2:  3: var accessKey = '[accountKey]'; 4: var storageAccount = '[accountName]'; 5: var containerName = 'nodejs'; 6:  7: //create a blob service set explicit credentials 8: var blobService = azure.createBlobService(storageAccount, accessKey); You can create Azure Blob Service using the environment variables to store the credentials  in the same way like for Azure Table service ( look at this post ).   List blobs in Azure Blob Storage   JavaScript code: 1: var azure = require('azure'); 2:  3: exports.blobs = function (request, response) { 4:  5: var accessKey = '[accountKey]'; 6: var storageAccount = '[accountName]'; 7: var container = 'nodejs'; 8:  9: var blobService = azure.createBlobService(storageAccount, accessKey); 10: //render blobs with blobs.jade view 11: blobService.listBlobs(container, function (error, blobs) { 12: response.render('blobs', { 13: error: error, 14: container: container, 15: blobs: blobs 16: }); 17: }); 18: }   A jade view ( using Express.js ) 1: extends layout 2: 3: block content 4: h1 Blob listing for #{container} 5: 6: if error 7: h3= error 8: 9: if blobs 10: h3= container 11: table 12: tr 13: th Name 14: th Properties 15: - each blob in blobs 16: tr 17: td= blob.name 18: td 19: ul 20: - each value, name in blob.properties 21: if value 22: li= name + ":" + value     Upload file to Azure Blob Storage One approach is to use multiparty node module. It is multipart/form-data parser which supports streaming.  Using Multiparty(npm install multiparty), a fork of Formidable, we can access the multipart data. This module will not stream the file to disk unless you tell it to. 1: //upload a file to azure blob storage 2: app.get('/upload', function (req, res) { 3: res.send( 4: '<form action="/upload" method="post" enctype="multipart/form-data">' + 5: '<input type="file" name="snapshot" />' + 6: '<input type="submit" value="Upload" />' + 7: '</form>' 8: ); 9: }); 10:  11: app.post('/upload', function (req, res) { 12: var multiparty = require('multiparty'); 13: var accessKey = '[accountKey]'; 14: var storageAccount = '[accountName]'; 15:  16: var container = 'nodejs'; 17: var blobService = azure.createBlobService(storageAccount, accessKey); 18: var form = new multiparty.Form(); 19: 20: form.on('part', function (part) { 21: if (part.filename) { 22: 23: var size = part.byteCount - part.byteOffset; 24: var name = part.filename; 25: 26: blobService.createBlockBlobFromStream(container, name, part, size, function (error) { 27: if (error) { 28: res.send(' Blob create: error '); 29: } 30: }); 31: } else { 32: form.handlePart(part); 33: } 34: }); 35: form.parse(req); 36: res.send('OK'); 37: }); 38: //end of upload a file to azure blob storage   Screens below demonstrate how to upload file to Azure Blob Storage using the sample demo  application.     Download files from Azure Blob Storage To download the blob and write it to the file system, a similar getBlob or getBlobToFile  methods can be used.  The snipped below demonstrates how to use  getBlob: 1: var azure = require('azure'); 2:  3: var blobService = azure.createBlobService(); 4:  5: //using getBlob to save a file from Azure Blob 6: blobService.getBlob('[containerName]', '[blobName]').pipe(fs.createWriteStream('[myFileName]'));   Code below is part of the demo application, demonstrating how to download file from Azure Blob Storage using getBlobToFile : 1:  2: //download azure blob storage cotent 3: app.get('/downloadBlob', function (req, res) { 4: res.send( 5: '<form action="/downloadBlob" method="post" >' + 6: '<input type="text" name="blobFile" value="C:\\temp" />' + 7: '<input type="submit" value="Download" />' + 8: '</form>' 9: ); 10: }); 11:  12: app.post('/downloadBlob', function (req, res) { 13: var fs = require('fs'); 14: 15: if (!fs.existsSync) { 16: fs.existsSync = require('path').existsSync; 17: } 18: var destinationDirectoryPath = req.body.blobFile; 19: var accessKey = '[accountKey]'; 20: var storageAccount = '[accountName]'; 21: var containerName = 'nodejs'; 22:  23: var blobService = azure.createBlobService(storageAccount, accessKey); //ok 24: 25: if (!fs.existsSync(destinationDirectoryPath)) { 26: console.log(destinationDirectoryPath + ' is an invalid directory path.'); 27: } else { 28: downloadFilesParallel(res, blobService, containerName, destinationDirectoryPath); 29: } 30:  31: });   downloadFilesParallel function, used in the code above: 1: function downloadFilesParallel(res, blobService, containerName, destinationDirectoryPath) { 2: blobService.listBlobs(containerName, function (error, blobs) { 3: if (error) { 4: console.log(error); 5: } else { 6: var blobsDownloaded = 0; 7: res.writeHead(200, { 'Content-Type': 'text/plain' }) 8: blobs.forEach(function (blob) { 9: blobService.getBlobToFile(containerName, blob.name, destinationDirectoryPath + '/' + blob.name, function (error2) { 10: blobsDownloaded++; 11: 12: if (error2) { 13: console.log(error2); 14: } else { 15: res.write('\nBlob ' + blob.name + ' download finished.'); 16: 17: if (blobsDownloaded === blobs.length) { 18: // Wait until all workers complete and the blobs are downloaded 19: res.end('\nAll files downloaded'); 20: } 21: } 22: }); 23: }); 24: } 25: }); 26: }   Screenshots from the sample app, demonstrating how to download a file from Blob Storage.       Specialists can learn much more details about Azure Blob services and Node.  Microsoft continues to grow and improve blob storage. This post describes how to start, covering the base cases and most popular Node.js modules for Azure Blob Storage. It will be useful for both – JavaScript developers who don’t have experience with Microsoft Azure and Azure developers who have less experience with JavaScript and/or Node.js. Developers on the Microsoft Azure Platform should become familiar with how Azure Blob Storage works and how to handle unstructured data in the Microsoft cloud. Knowing how blob storage works will help you determine if it is a good fit for your particular requirements.   You can download source code from Git repository .   If you want more information about how to use Microsoft Azure Storage  & Node.js feel free to contact me at mmateev@infragistics.com You can learn more about Node.js , Microsoft Azure and related events like Azure Bootcamp Bulgaria if you follow us on Twitter @mihailmateev  and @Infragistics and stay in touch on Facebook, Google+, LinkedIn and Infragistics Friends User Group !
  • How to manage Microsoft Azure Table Storage with Node.js
    POSTED ON: 07/31/2014 6:38 AM  BY:  [Infragistics] Mihail Mateev
    Node.js is one one of the the most popular growing platforms for development.  I started a series of posts dedicated to Node.js , starting with two articles about Node.js and Microsoft SQL Server ( you can see part 1 and part 2 ) .  This blog discusses how you can use Node.js and Microsoft Azure Table Storage . Open source solutions (OSS),  are very suitable for the implementation of platform independent and/or cloud applications. Node.js is widely used in the actual implementation of such cloud applications. I will try demonstrate in several blogs some details how to use Node.js with  Microsoft Azure Storage .   What is Azure Storage? Microsoft Azure storage services allow us to store/retrieve the NON RELATIONAL data to/from Microsoft Cloud environment. (For relational data, SQL Azure services are used). In Microsoft Azure Storage, the data can be stored in 4 different formats (v.i.z. Blobs, Tables and Queues, File Storage (in preview )). The retrieval/storage of the above data is done in RESTful way. Blob storage stores file data. A blob can be any type of text or binary data, such as a document, media file, or application installer. Table storage stores structured datasets. Table storage is a NoSQL key-attribute data store, which allows for rapid development and fast access to large quantities of data. Queue storage provides reliable messaging for workflow processing and for communication between components of cloud services. File storage offers shared storage for legacy applications using the standard SMB 2.1 protocol. Azure virtual machines and cloud services can share file data across application components via mounted shares, and on-premise applications can access file data in a share via the File service REST API.     This article is focused on how to handle Azure Table services with Node.js   Table Storage The Azure Table storage service stores large amounts of structured data. The service is a NoSQL datastore which accepts authenticated calls from inside and outside the Azure cloud. The Table service contains the following components:     Table Entities: Table entities represent the units of data stored in a table and are similar to rows in a typical relational database table. Each entity defines a collection of properties. Each property is key/value pair defined by its name, value, and the value's data type. Entities must define the following three system properties as part of the property collection: PartitionKey – The PartitionKey property stores string values that identify the partition that an entity belongs to. This means that entities with the same PartitionKey values belong in the same partition. Partitions, as discussed later, are integral to the scalability of the table. RowKey – The RowKey property stores string values that uniquely identify entities within each partition. Timestamp – The Timestamp property provides traceability for an entity. A timestamp is a DateTime value that tells you the last time the entity was modified. A timestamp is sometimes referred to as the entity's version. Modifications to timestamps are ignored because the table service maintains the value for this property during all inserts and update operations.   Consider PartitionKey, RowKey in your design. Think of PartitionKey and RowKey as being a primary index.   Table Partitions: Azure Tables use keys that enable efficient querying, and you can employ one—the PartitionKey—for load balancing when the table service decides it’s time to spread your table over multiple servers. A table doesn’t have a specified schema.     Partitions represent a collection of entities with the same PartitionKey values. Partitions are always served from one partition server and each partition server can serve one or more partitions.     Dealing with an Azure Table Storage You can use different NodeJS packages to handle Azure Table Storage. In this post we will cover azure and azure-table-node Node packages.   Microsoft Azure SDK for Node.js It is an official Microsoft Azure SDK for Node.js. This project provides a Node.js package that makes it easy to consume and manage Microsoft Azure Services. The listed snippets below demonstrate how to manage Azure Table service using Microsoft Azure SDK for Node.js.   This is easy to do since the Azure SDK will look for credentials using environment variables first. The magical environment variable names are AZURE_STORAGE_ACCOUNT and AZURE_STORAGE_ACCESS_KEY.   Create a table service Set credentials using environment variables 1: var azure = require('azure'); 2:  3: //using enviroment variables for credentials 4: var tableService = azure.createTableService(); // implicitly use env variables 5: 6: tableService = azure.createTableService( 7: process.env.AZURE_STORAGE_ACCOUNT, 8: process.env.AZURE_STORAGE_ACCESS_KEY); // explicit 9:    Set credentials explicitly using local variables 1: var accessKey = '[accountKey]'; 2: var storageAccount = '[accountName]'; 3: 4: var tableService = azure.createTableService( 5: accessKey, storageAccount); // explicit   Insert an entity 1: var tableService = azure.createTableService(); 2:  3: //insert an entity 4: var task1 = { 5: PartitionKey : 'myPartitionKey', 6: RowKey: '1', 7: Description: 'Row description', 8: DueDate: new Date(2011, 12, 14, 12) 9: }; 10: 11: tableService.insertEntity('tasktable', task1, function(error){ 12: if(!error){ 13: // Entity inserted 14: } 15: });   Query entities 1: //query an entity 2: var tableService = azure.createTableService(); 3: tableService.queryEntity('demotable', 'myPartitionKey', '1', function(error, serverEntity){ 4: if(!error){ 5: // Entity available in serverEntity variable 6: } 7: });     azure-table-node: It is a simplified Azure Table Storage client library for Node.js that supported: creating, deleting and listing tables creating, updating, querying and deleting entities batch operation support generating SAS (Shared Access Signature) and using it for authentication   Code samples below show how to use azure-table-node module to work with Azure Table Storage  Set Azure Storage credentials 1: var azureTable = require('azure-table-node') 2:  3: //set azure storage credentials 4: azureTable.setDefaultClient({ 5: accountUrl: 'http://[accountName].table.core.windows.net/', 6: accountName: '[accountName]', 7: accountKey: '[accountKey]' 8: });   Create an Azure Table 1: //create azure table 2: app.get("/createTable", function (req, res) { 3:  4: var client = azureTable.getDefaultClient(); 5: client.createTable('testtable', function (err, data) { 6: }); 7:  8: client.insertEntity('testtable', { 9: PartitionKey: 'tests', 10: RowKey: '1', 11: value1: 'ABCDEFG' 12: }, function (err, data) { 13: res.write("Got error :-( " + err); 14: }); 15:  16: res.end("Table created."); 17: });   Display an Azure Table 1: //display an azure table 2: app.get("/displayTable", function (req, res) { 3:  4: var client = azureTable.getDefaultClient(); 5:  6: client.queryEntities('testtable', { 7: query: azureTable.Query.create('PartitionKey', '==', 'tests') 8:  9: }, function (err, data, continuation) { 10: if (err) { 11: res.writeHead(500, { 'Content-Type': 'text/plain' }); 12: res.write("Got error :-( " + err); 13: res.end(""); 14: return; 15: } 16:  17: var json = JSON.stringify(data); 18: res.writeHead(200, { 'Content-Type': 'text/plain' }) 19:  20: res.end("Table displayed: " + json); 21: }); 22:  23: });     List  all Azure Tables 1: //list all azure tables 2: app.get("/listTables", function (req, res) { 3:  4: var client = azureTable.getDefaultClient(); 5:  6: client.listTables(function (err, data, continuation) { 7: if (err) { 8: res.writeHead(500, { 'Content-Type': 'text/plain' }); 9: res.write("Got error :-( " + err); 10: res.end(""); 11: return; 12: } 13:  14: res.writeHead(200, { 'Content-Type': 'text/plain' }) 15:  16: for (var i = 0; i < data.length; i++) { 17: res.write("Table[" + i + "]: " + data[i] + " " ); 18: } 19:  20: res.end("Tables listed." + data); 21: }); 22:  23: });     Delete an Azure Table 1: //delete azure table 2: app.get("/deleteTable", function (req, res) { 3:  4: var client = azureTable.getDefaultClient(); 5: client.deleteTable('testtable', function (err, data) { 6: }); 7: res.end("Table testtable has been deleted."); 8:  9: });     There’s so much more to learn about Azure Table services and Node.  The features and capabilities of table storage continue to grow. This post is just an intro how to start, covering the base cases and most popular Node.js modules for Azure Tables. It will be useful for both – JavaScript developers who don’t have experience with Microsoft Azure and Azure developers who have less experience with JavaScript and/or Node.js. Developers on the Microsoft Azure Platform should become familiar with how Table Storage works and how it differs from the relational databases they are used to. Knowing how table storage works will help you determine if it is a good fit for your particular requirements.   You can download source code from Git repository .   If you want more information about how to use Microsoft Azure Storage  & Node.js feel free to contact me at mmateev@infragistics.com You can learn more about Node.js , Microsoft Azure and related events like Azure Bootcamp Bulgaria if you follow us on Twitter @mihailmateev  and @Infragistics and stay in touch on Facebook, Google+, LinkedIn and Infragistics Friends User Group !
  • Developer News - What's IN with the Infragistics Community? (7/21-7/27)
    POSTED ON: 07/29/2014 4:09 PM  BY:  Dara Monasch
    News for the week, comin' atcha! 5. A JavaScript Survival Guide (②ality) 4. HTML5 Vs. Native Apps for Mobile (Business Insider) 3. 15 Best and Useful Gadgets for Tech Savvy - Part 3 (Crooked Brains) 2. UI, UX: Who Does What? A Designer's Guide To The Tech Industry (Co.Design) 1. The "Just In Time" Theory of User Behavior (Coding Horror)