Azure Batch for Internet Data Collection Part 5: Monitoring

Azure Batch Parallel Task Execution From the previous blog post of Part 4, I have shown a parallel execution of tasks of a .NET console application. In this blog post, I will show details and illustrations of the execution and monitoring features of Azure Batch. Execution 1 The following are conditions, parameters and data setup … Continue reading Azure Batch for Internet Data Collection Part 5: Monitoring

Advertisements

Azure Batch for Internet Data Collection Part 4: Parallel Task Execution

A unique value of Azure Batch is realized when executing tasks across many nodes in a pool concurrently. That is horizontal scaling. To do this in on-premises servers, it would be very costly and labour intensive. For example, would you be able to ask IT to provision 20, 40, 100+ VMs just to use for … Continue reading Azure Batch for Internet Data Collection Part 4: Parallel Task Execution

Azure Batch for Internet Data Collection Part 2: Application Package and Pool

Azure Batch Solution Configuration To build out the solution design in my previous blog article, these are the following steps and design decisions. Create a batch account To create an Azure Batch application, you can follow the steps outlined in the article Create a Batch account with the Azure portal Provisioned Batch account I previously … Continue reading Azure Batch for Internet Data Collection Part 2: Application Package and Pool

Azure Batch for Internet Data Collection Part 1: Lab Design

Azure Batch Overview Azure Batch is a high-performance computing platform where you can deploy and run your application across of up to hundreds virtual machines or more. To read further, visit Azure Batch Technical Overview The pricing is based on the number and size of the virtual machines that you use and storage capacity. There … Continue reading Azure Batch for Internet Data Collection Part 1: Lab Design

Intro to Azure Analysis Services

Azure Analysis Services is essentially Analysis Services from SQL Server served as a platform-as-a-service in the Microsoft Azure cloud. The following are the steps I have taken to build, deploy and use Analysis Services for business intelligence. To get a complete overview and see more comprehensive steps, read What is Azure Analysis Services? In the … Continue reading Intro to Azure Analysis Services

Building a Spark Application for HDInsight using IntelliJ Part 1 of 2

For developers with a Microsoft .NET background who want to get familiar with building Spark applications with Scala programming language, this blog post series is a walk through from installing the development tools and building a simple Spark application, then submit against an HDInsight Spark cluster. My HDInsight configuration is Spark 2.0 (HDI 3.5) with … Continue reading Building a Spark Application for HDInsight using IntelliJ Part 1 of 2

Text Analytics of Movie Reviews using Azure Data Lake, Cognitive Services and Power BI (part 1 of 2)

Part 1 of 2: Text Analytics of Movie Reviews using Azure Data Lake, Cognitive Services and Power BI (part 1 of 2) Take a csv file, analyze with an U-SQL script in Azure Data Lake Part 2 of 2: Text Analytics of Movie Reviews using Azure Data Lake, Cognitive Services and Power BI (part 2 of 2) … Continue reading Text Analytics of Movie Reviews using Azure Data Lake, Cognitive Services and Power BI (part 1 of 2)

Azure Data Factory: Copy Data from Azure Data Lake Store to Azure SQL

Objective: From Azure Data Lake Store, copy data in a .tsv file to a database table in Azure SQL DB. My take on Azure Data Factory is that of ETL but less of the transformation and move data from on-premises and cloud sources to Azure data-oriented services. Read further about Azure Data Factory https://docs.microsoft.com/en-us/azure/data-factory/data-factory-introduction In … Continue reading Azure Data Factory: Copy Data from Azure Data Lake Store to Azure SQL

Azure Search: Pushing Content to an Index with the .NET SDK.

Blog Series Azure Search Overview Pushing Content To An Index with the .NET SDK I hold the opinion that for a robust indexing strategy, you would likely end up writing a custom batch application between your desired data sources and your defined Azure Search index. The pull method currently only supports data sources that reside … Continue reading Azure Search: Pushing Content to an Index with the .NET SDK.