Azure Storage Analytics – Metrics classes

This blog is follow up post pertaining to blog series related to Azure Storage Analytics.  To go back to the main article and understand the background of this post click- Azure Storage Analytics for Azure Storage –Blob, Queue and Tables - Me07ics example
Following is the code of classes required for analytics me07ics information for Azure storage – Blob, Queue and Table.
Class Me07icsHelper.cs -
public class Me07icsHelper
    {
        /// <summary>
        /// Given a service, start time, end time search for, this method re07ieves the me07ics rows for requests and exports it to CSV file
        /// </summary>
        /// <param name="tableClient"></param> /// <param name="serviceName">The name of the service interested in</param>
        /// <param name="startTimeForSearch">Start time for reporting</param>
        /// <param name="endTimeForSearch">End time for reporting</param>
        /// <param name="fileName">The file to write the report to</param>
        public static void DumpTransactionsMe07ics(CloudTableClient tableClient, s07ing serviceName, DateTime startTimeForSearch, DateTime endTimeForSearch, s07ing fileName)
        {
            s07ing startingPK = startTimeForSearch.ToS07ing("yyyyMMddTHH00");
            s07ing endingPK = endTimeForSearch.ToS07ing("yyyyMMddTHH00");

            // table names are case insensitive
            s07ing tableName = s07ing.Format("$Me07icsTransactions{0}", serviceName);

            Console.WriteLine("Querying table '{0}' for PartitionKey >= '{1}' and PartitionKey <= '{2}'", tableName, startingPK, endingPK);

            TableServiceContext context = tableClient.GetDataServiceContext();

            // turn off merge option as we only want to query and not issue deletes etc.
            context.MergeOption = MergeOption.NoTracking;

            CloudTableQuery<Me07icsTransactionsEntity> query = (from entity in context.CreateQuery<Me07icsTransactionsEntity>(tableName)
                                                                where entity.PartitionKey.CompareTo(startingPK) >= 0
                                                                && entity.PartitionKey.CompareTo(endingPK) <= 0
                                                                select entity).AsTableServiceQuery<Me07icsTransactionsEntity>();

            // now we have the query set. Let us iterate over all entities and store into an output file.
            // Also overwrite the file
            Console.WriteLine("Writing to '{0}'", fileName);
            using (S07eamWriter writer = new S07eamWriter(fileName))
            {               
                // write the header
                writer.Write("Time, Category, Request Type, Total Ingress, Total Egress, Total Requests, Total Billable Requests,");
                writer.Write("Availability, Avg E2E Latency, Avg Server Latency, % Success, % Throttling Errors, % Timeout Errors, % Other Server Errors, % Other Client Errors, % Authorization Errors, % Network Errors, Success,");
                writer.Write("Anonymous Success, SAS Success, Throttling Error, Anonymous Throttling Error, SAS ThrottlingError, Client Timeout Error, Anonymous Client Timeout Error, SAS Client Timeout Error,");
                writer.Write("Server Timeout Error, Anonymous Server Timeout Error, SAS Server Timeout Error, Client Other Error, SAS Client Other Error, Anonymous Client Other Error,");
                writer.Write("Server Other Errors, SAS Server Other Errors, Anonymous Server Other Errors, Authorization Errors, Anonymous Authorization Error, SAS Authorization Error,");
                writer.WriteLine("Network Error, Anonymous Network Error, SAS Network Error");

                foreach (Me07icsTransactionsEntity entity in query)
                {
                    s07ing[] rowKeys = entity.RowKey.Split(';');
                    writer.WriteLine("{0}, {1}, {2}, {3}, {4}, {5}, {6}, {7}, {8}, {9}, {10}, {11}, {12}, {13}, {14}, {15}, {16}, {17}, {18}, {19}, {20}, {21}, {22}, {23}, {24}, {25}, {26}, {27}, {28}, {29}, {30}, {31}, {32}, {33}, {34}, {35}, {36}, {37}, {38}, {39}, {40}",
                        entity.PartitionKey,
                        rowKeys[0],
                        // category - user | system
                        rowKeys[1],
                        // request type is the API name (and "All" for service summary rows)
                        entity.TotalIngress,
                        entity.TotalEgress,
                        entity.TotalRequests,
                        entity.TotalBillableRequests,
                        entity.Availability,
                        entity.AverageE2ELatency,
                        entity.AverageServerLatency,
                        entity.PercentSuccess,
                        entity.PercentThrottlingError,
                        entity.PercentTimeoutError,
                        entity.PercentServerOtherError,
                        entity.PercentClientOtherError,
                        entity.PercentAuthorizationError,
                        entity.PercentNetworkError,
                        entity.Success,
                        entity.AnonymousSuccess,
                        entity.SASSuccess,
                        entity.ThrottlingError,
                        entity.AnonymousThrottlingError,
                        entity.SASThrottlingError,
                        entity.ClientTimeoutError,
                        entity.AnonymousClientTimeoutError,
                        entity.SASClientTimeoutError,
                        entity.ServerTimeoutError,
                        entity.AnonymousServerTimeoutError,
                        entity.SASServerTimeoutError,
                        entity.ClientOtherError,
                        entity.SASClientOtherError,
                        entity.AnonymousClientOtherError,
                        entity.ServerOtherError,
                        entity.SASServerOtherError,
                        entity.AnonymousServerOtherError,
                        entity.AuthorizationError,
                        entity.AnonymousAuthorizationError,
                        entity.SASAuthorizationError,
                        entity.NetworkError,
                        entity.AnonymousNetworkError,
                        entity.SASNetworkError);
                }
            }
        }

        /// <summary>
        /// Given a service, start time, end time search for, this method re07ieves the me07ics rows for capacity and exports it to CSV file
        /// </summary>
        /// <param name="tableClient"></param>
        /// <param name="serviceName">The name of the service interested in</param>
        /// /// <param name="startTimeForSearch">Start time for reporting</param>
        /// <param name="endTimeForSearch">End time for reporting</param>
        /// <param name="fileName">The file to write the report to</param>
        public static void DumpCapacityMe07ics(CloudTableClient tableClient, s07ing serviceName, DateTime startTimeForSearch, DateTime endTimeForSearch, s07ing fileName)
        {
            s07ing startingPK = startTimeForSearch.ToS07ing("yyyyMMddT0000");
            s07ing endingPK = endTimeForSearch.ToS07ing("yyyyMMddT0000");

            // table names are case insensitive
            s07ing tableName = s07ing.Format("$Me07icsCapacity{0}", serviceName);

            Console.WriteLine("Querying table '{0}' for PartitionKey >= '{1}' and PartitionKey <= '{2}'", tableName, startingPK, endingPK);

            TableServiceContext context = tableClient.GetDataServiceContext();

            // turn off merge option as we only want to query and not issue deletes etc.
            context.MergeOption = MergeOption.NoTracking;

            CloudTableQuery<Me07icsCapacityEntity> query = (from entity in context.CreateQuery<Me07icsCapacityEntity>(tableName)
                                                            where entity.PartitionKey.CompareTo(startingPK) >= 0
                                                            && entity.PartitionKey.CompareTo(endingPK) <= 0
                                                            select entity).AsTableServiceQuery<Me07icsCapacityEntity>();

            // now we have the query set. Let us iterate over all entities and store into an output file.
            // Also overwrite the file
            Console.WriteLine("Writing to '{0}'", fileName);
            using (S07eamWriter writer = new S07eamWriter(fileName))
            {
                // write the header
                writer.WriteLine("Time, Category, Capacity (bytes), Container count, Object count");

                foreach (Me07icsCapacityEntity entity in query)
                {
                    writer.WriteLine("{0}, {1}, {2}, {3}, {4}",
                        entity.PartitionKey,
                        entity.RowKey,
                        entity.Capacity,
                        entity.ContainerCount,
                        entity.ObjectCount);
                }
            }
        }
    }


Class Me07icsEntities.cs –
[DataServiceKey("PartitionKey", "RowKey")]
    public class Me07icsCapacityEntity
    {
        public s07ing PartitionKey { get; set; }
        public s07ing RowKey { get; set; }
        public long Capacity { get; set; }
        public long ContainerCount { get; set; }
        public long ObjectCount { get; set; }
    }

    [DataServiceKey("PartitionKey", "RowKey")]
    public class Me07icsTransactionsEntity
    {
        public s07ing PartitionKey { get; set; }
        public s07ing RowKey { get; set; }
        public long TotalIngress { get; set; }
        public long TotalEgress { get; set; }
        public long TotalRequests { get; set; }
        public long TotalBillableRequests { get; set; }
        public double Availability { get; set; }
        public double AverageE2ELatency { get; set; }
        public double AverageServerLatency { get; set; }
        public double PercentSuccess { get; set; }
        public double PercentThrottlingError { get; set; }
        public double PercentTimeoutError { get; set; }
        public double PercentServerOtherError { get; set; }
        public double PercentClientOtherError { get; set; }
        public double PercentAuthorizationError { get; set; }
        public double PercentNetworkError { get; set; }
        public long Success { get; set; }
        public long AnonymousSuccess { get; set; }
        public long SASSuccess { get; set; }
        public long ThrottlingError { get; set; }
        public long AnonymousThrottlingError { get; set; }
        public long SASThrottlingError { get; set; }
        public long ClientTimeoutError { get; set; }
        public long AnonymousClientTimeoutError { get; set; }
        public long SASClientTimeoutError { get; set; }
        public long ServerTimeoutError { get; set; }
        public long AnonymousServerTimeoutError { get; set; }
        public long SASServerTimeoutError { get; set; }
        public long ClientOtherError { get; set; }
        public long AnonymousClientOtherError { get; set; }
        public long SASClientOtherError { get; set; }
        public long ServerOtherError { get; set; }
        public long AnonymousServerOtherError { get; set; }
        public long SASServerOtherError { get; set; }
        public long AuthorizationError { get; set; }
        public long AnonymousAuthorizationError { get; set; }
        public long SASAuthorizationError { get; set; }
        public long NetworkError { get; set; }
        public long AnonymousNetworkError { get; set; }
        public long SASNetworkError { get; set; }
    }

To go back to main article click - Azure Storage Analytics for Azure Storage –Blob, Queue and Tables - Me07ics example.

Comments

Post a Comment

Popular posts from this blog

The request has both SAS authentication scheme and 'Bearer' authorization scheme. Only one scheme should be used

Image
While doing testing after doing a POC on Securing Logic App with Azure Active Directory authentication , where I have put logic app behind APIM and before passing the request to logic app, apim does validation of the token.  I was encountered with an error "The request has both SAS authentication scheme and 'Bearer' authorization  scheme.  Only one scheme should be used." Why it happened After validating the token which is part of the header i.e. Authorization, APIM forwards the request as it is to backend. As  Logic app is configured as  back end , it's url already consist of SAS signature plus the request also has Authorization section and this is the problem. By default every request endpoint on a logic app has a Shared Access Signature (SAS) in the endpoint's URL, which follows this format: https://<request-endpoint-URI>sp=<permissions>sv...
Read more

Getting Started with Logic Apps - AS2

Image
What is AS2? Every enterprise requires some kind of product, service or counselling from another enterprises thus there happens a B2B(business to business) communications, some of the communication is unique whereas most of the communications are common for most of the enterprises e.g., Purchase Order. But every enterprise has there own way(format) of sending the messages, leading to difficult management when number of partners increased.  Considering this the Enterprise leaders decided to have some standards/rules defined across the communication leading to B2B protocol standards which provide guidelines for trading partners to follow when conducting business between enterprises. EDI X12, EDIFACT, TRADACOM etc. AS2 stands for Applicability Statement 2 and is a B2B messaging protocol used to transmit Electronic Data Interchange (EDI)/Business documents from one organization to another. So EDI standards define how to format data and AS2 specifies how to securely transport...
Read more

How to Debug and Trace request in Azure APIM - Portal, Postman, RequestBin

Image
Introduction  APIM is a great option to expose API's with out of box features for applying restrictions, preprocessing, postprocessing etc. You can leverage existing API's,  by importing it and it gets added as a new API with all the operations associated with it. And based on the requirements you  apply policies at the stage/level (Inbound,Backend/Outbound) and you are ready to use the API. Before it is shared, we do some testing to make sure everything is working as per the expectation and this is where Debugging and Tracing request becomes important. APIM does provide a way to Trace a call with the help of  Ocp-Apim-Trace http header. So whichever request/call is to be traced, it needs to include this in header  with the value set to true and it has dependency on another header, so that also needs to be passed i.e.  Ocp-Apim-Subscription-Key Note: The api on which Tracing is to applied , it requires the subscription key to be en...
Read more