Introduction

Anyone who has ever done application development in any capacity is almost certain to understand the need to log actions, events, and errors for the application and modules. It is one of those things that does not provide direct value to the end-users but is extremely helpful in error conditions in applications and very critical for best practices. The logs that we push should pinpoint the exact module or component and provide all the necessary information about the error/exception that the application has encountered which will help the engineer figure out the cause and fix the problem appropriately.

Traditionally, all these logs used to be stored in application databases which would be then accessed and understood by the engineer using log reading applications that can present the logs in an understandable format. The problem with this approach is that since the custom logs are stored in the application database/logging mechanism, the log data is generally huge and has application events, user accesses, server actions, etc. logged in the same space along with error information. This is a huge overhead for the engineer who is trying to find the needle in this haystack of log information. Another problem is that then each application has its own space of logging so more often than not, we have to keep tabs on multiple log spaces and then map it to the right application and find errors. In addition, there will be inconsistencies in the way custom logs are pushed as there is no standardized approach and each implementer has done the logging in the way it worked for them. Last but not the least, the engineer needs to have access to all of the log spaces as well. While our hardworking fellow engineers make that work in most environments, it is best if we can somehow introduce standardization, have a single space for collecting and looking at failure logs and minimize the overall maintenance effort for applications and logs.

With this context, we started looking at our possible solutions. Azure offers a very powerful service in Azure Log Analytics which, apart from integrating well with the Azure Monitor service, provides the capability to define, read, write, query, and search logs. For this article, we are going to leverage the API store architecture that we have defined in our earlier article to host a new API that will push our application logs into Log Analytics workspace which we will then view and analyze using the Log Analytics interface. Let’s jump right into it.

Implementation

First, we are going to define a schema that we will use for our custom logs. Each application’s log requirements are going to be different and would want to log details relevant to the application to help us at a later stage to figure out the issues and errors. So, we cannot really have a restricted schema and should give enough flexibility to the consuming application that they can log what they desire, in the format they desire.

For this reason, we are going to define just a basic schema for our Log Analytics Workspace that we will create for custom logs. We are going in with fixed fields described as under,

• log filename: Name of the log file that you have created in your Log Analytics Workspace
• AppName: Unique name/identifier for your application that can be used later to map to application
• ModuleName: Name of the module/function/method for which the logging is being done
• log data: This is a JSON field that can hold any number of attributes and details in a key-value pair (nested or plain). This has to be a valid JSON in order for us to make use of the visual and parsing capabilities in Log Analytics

Okay. Now that we have decided on the schema, let’s go ahead and create our helper class under the Helper folder in our solution and paste the following code in it. We will create three methods in this helper file. The first one to build the signature hash string required for authorizing to the Azure Log Analytics endpoint. The signature hash is built using secret, message, and SHA256 encryption. The second method will be the one that will ingest the log to Azure Log Analytics endpoint using the signature generated and log data. The third and final method is the root method that our custom Azure Function is going to invoke on the trigger to perform the requested action for logging data into the Azure Log Analytics Workspace. We are using the Azure Data Collector API implementation in this.

LogAnalyticsHelper.cs

1. using  System;
2. using  System.Text;
3. using  System.Threading.Tasks;
4. using  System.Net.Http;
5. using  System.Net.Http.Headers;
6. using  System.Security.Cryptography;
7. using  Newtonsoft.Json;
8. using  Microsoft.Extensions.Logging;
9. using  Newtonsoft.Json.Linq;
11. namespace  Reusable.Functions
12. {
13. public  class  LogAnalyticsHelper
14. {
15. /// 
16. /// Get LogAnalytics Specific Details
17. /// 
18. /// 
19. public  static  string  GetTimeStampField()
20. {
21. return  ConstantsHelper.TimeStampField;
22. }
24. /// 
25. /// Validate User Input Json
26. /// 
27. /// 
28. /// 
29. /// 
30. public  static  bool  IsValidJson(string  strInput, ILogger log)
31. {
32. strInput = strInput.Trim();
33. if  ((strInput.StartsWith( “{” ) && strInput.EndsWith( “}” )) ||  //For object
34. (strInput.StartsWith( “[” ) && strInput.EndsWith( “]” )))  //For array
35. {
36. try
37. {
38. var obj = JToken.Parse(strInput);
39. return  true ;
40. }
41. catch  (JsonReaderException jex)
42. {
43. //Exception in parsing json
44. log.LogInformation($ “\n IsValidJson method got Exception \n Time: { DateTime.Now} \n Exception{ jex.Message}” );
45. return  false ;
46. }
47. catch  (JsonException je)
48. {
49. //Exception in parsing json
50. log.LogInformation($ “\n IsValidJson method got Exception \n Time: { DateTime.Now} \n Exception{ je.Message}” );
51. return  false ;
52. }
53. catch  (Exception ex)  //some other exception
54. {
55. //Exception in parsing json
56. log.LogInformation($ “\n IsValidJson method got Exception \n Time: { DateTime.Now} \n Exception{ ex.Message}” );
57. return  false ;
58. }
59. }
60. else
61. {
62. return  false ;
63. }
64. }
66. /// 
67. /// To push logs in Log Analytics Workspace
68. /// 
69. /// 
70. /// 
71. /// 
72. /// 
73. /// 
74. public  static  async Task<bool > PushLogsToLogAnalytics(string  json, string  logFileName, string  workspaceId, string  sharedKey, ILogger log)
75. {
76. try
77. {
78. // Create a hash for the API signature
79. var datestring = DateTime.UtcNow.ToString( “r” );
80. var jsonBytes = Encoding.UTF8.GetBytes(json);
81. string  stringToHash =  “POST\n”  + jsonBytes.Length +  “\napplication/json\n”  +  “x-ms-date:”  + datestring +  “\n/api/logs” ;
82. string  hashedString = LogAnalyticsHelper.BuildSignature(stringToHash, sharedKey, log);
83. log.LogInformation($ “HashedString : {hashedString}” );
84. string  signature =  "SharedKey "  + workspaceId +  “:”  + hashedString;
85. log.LogInformation($ "Signature : "  + signature);
86. bool  ingestionStatus = await LogAnalyticsHelper.IngestToLogAnalytics(signature, datestring, json, logFileName, workspaceId, log);
87. return  ingestionStatus;
88. }
89. catch  (Exception e)
90. {
91. log.LogInformation($ “PushLogsToLogAnalytics got Exception \n  Time: {DateTime.Now} \n Exception{e.Message} and complete Exception:{e}” );
92. return  false ;
93. }
95. }
97. //Build the API signature
98. /// 
99. /// To build signature for log data
100. /// 
101. /// 
102. /// 
103. /// 
104. /// 
105. public  static  string  BuildSignature(string  message, string  secret, ILogger log)
106. {
107. log.LogInformation($ “Begin BuildSignature \n Start Time: {DateTime.Now}” );
108. var encoding = new  System.Text.ASCIIEncoding();
109. byte [] keyByte = Convert.FromBase64String(secret);
110. byte [] messageBytes = encoding.GetBytes(message);
111. using  (var hmacsha256 = new  HMACSHA256(keyByte))
112. {
113. byte [] hash = hmacsha256.ComputeHash(messageBytes);
114. return  Convert.ToBase64String(hash);
115. }
116. }
118. /// 
119. /// To Ingest Into Log Analytics
120. /// 
121. /// 
122. /// 
123. /// 
124. /// 
125. /// 
126. /// 
127. public  static  async Task<bool > IngestToLogAnalytics(string  signature, string  date, string  datajson, string  logFile, string  workspaceId, ILogger log)
128. {
129. try
130. {
131. string  url =  “https://”  + workspaceId +  “.ods.opinsights.azure.com/api/logs?api-version=2016-04-01” ;
132. HttpClient client = new  HttpClient();
133. client.DefaultRequestHeaders.Add( “Accept” ,  “application/json” );
134. client.DefaultRequestHeaders.Add( “Log-Type” , logFile);
135. client.DefaultRequestHeaders.Add( “Authorization” , signature);
136. client.DefaultRequestHeaders.Add( “x-ms-date” , date);
137. client.DefaultRequestHeaders.Add( “time-generated-field” , GetTimeStampField());
139. HttpContent httpContent = new  StringContent(datajson, Encoding.UTF8);
140. httpContent.Headers.ContentType = new  MediaTypeHeaderValue( “application/json” );
142. var response = await client.PostAsync(new  Uri(url), httpContent);
143. if  (response.IsSuccessStatusCode)
144. {
145. HttpContent responseContent = response.Content;
146. var result = await responseContent.ReadAsStringAsync().ConfigureAwait(false );
147. log.LogInformation( "Ingestion of Logs is completed with status code : "  + response.StatusCode);
148. return  true ;
149. }
150. else
151. {
152. HttpContent responseContent = response.Content;
153. string  result = await responseContent.ReadAsStringAsync().ConfigureAwait(false );
154. log.LogInformation( "Ingestion of Logs has failed with status code : "  + response.StatusCode);
155. return  false ;
156. }
157. }
158. catch  (Exception e)
159. {
160. log.LogInformation($ “IngestToLogAnalytics got Exception \n  Time: {DateTime.Now} \n Exception{e.Message} and complete Exception:{e}” );
161. return  false ;
162. }
163. }
164. }
165. }

Next, let’s add a new Azure Function to our existing solution that we have set up in the previous article. We will add this function to the API Management at a later point to provide the API end-point for our consumers via subscription. We will define a HTTP Trigger function for pushing the log data to Log Analytics Workspace. Since our log schema expects the application log details in a JSON format, we will also add a method to validate that the log information being sent is containing a valid JSON. If not, we are not going to log this information as it will interfere with the visual presentation and parsing in Log Analytics Workspace. Once we have validated the JSON, we are going to fetch the Log Analytics Workspace ID and Key from the Azure Key Vault using custom API that we have created in our earlier artile. You may hardcode this but from a security best practice point of view, this should be stored and fetched from Azure Key Vault or any other secure vault of your choice. Once we have these details, we will just invoke the method from our LogAnalyticsHelper class to push the log into Azure Log Analytics.

#coding #api #azure api #azure