Monitoring of any system/application's health and performance plays very important role in an Enterprise , as you get insight about what is happening with the applications and can prevent from major losses by enabling better/corrective decision. Microsoft has provided inbuilt Monitoring feature within Azure Data Factory where you gets details about the each Pipeline which is run and also details about the activity inside the pipeline. As can be seen lot of details are made available at both the levels,RunID, Starttime, status, name, duration etc.   and at activity level you can even see  input and output of that particular activity. So the question might arise why need of Custom monitoring table? 1. Currently the pipeline run data retention is only for 45 days , then after data would be no more available. 2. Support for User defined properties -- all the details which we see is system properties but what about other than those, i.e. User defined p...

Why and What is Data Factory Day by day the amount of data collected by any business/enterprise is growing - both the historical and transactional. But is there any need to keep them? Is it useful to business in any way? -- Yes, the data which is collected so far or would be collected on going can be used to deduce value by process of analyzing data and presenting actionable information to help executives, managers and other corporate end users to make informed business decisions. (assisting Business Intelligence) Like it is said Information when processed on turns into Knowledge. To make this possible  Raw Data(information) --> Transformed data(knowledge),  Microsoft introduced data integration service i.e. Azure Data Factory which is capable of extracting-transformation-loading (ETL) or  extracting- loading- transformation(ELT), for transformation activities either of following can be used - HDInsight, Data Bricks, Azure Data lake U-SQL, Machine Lear...

Introduction  It’s great to see how the evolution is taking place in IT world, mostly focused on sharing the existing resources leading to invent of cloud computing where resources are shared and used as a service on demand. Based on the type of resource shared, categorization is done – IAAS, PAAS, SAAS, IPAAS, FAAS etc. Off course, sharing is not free – you pay for whatever you use based on the charges defined per unit as to how long it is used and how much (time and amount). This ultimately helps in reducing time to market (by using already existing resource of others) and monetizing from the available resource (by sharing it). Publishing the API (functionality) is just another thing on the same line – where the intention is to share a piece of function for enabling faster integration with clients or for monetizing from it. API (Application Programming interface) is a set of functions/methods/procedure with a set of defined rules enabling the interaction betwee...