Dynatrace Application Performance Monitoring Tool (APM)

Introduction:

As the information technology industry is rapid development and most optimized performance applications needs to be delivered into the market using agile methodologies. To meet end users expectations we have many application performance monitoring tools in the market.

Dynatrace is one of the best industry standard application monitoring tool.

Dynatrace Architecture:

Dynatrace Architecture is client server based architecture which involves the monitoring of the client side metrics, application server based metrics using dynatrace agents and automatically monitors database server based metrics.

　

Everything is agent based collections interms of metrics for

dynatrace, so there is high accuracy for correctness of data for any trusted customers using the application which is highly scalable in the any competitive market.

Dynatrace Architecture has four important components

Dynatrace client

Dynatrace collector

Dynatrace server

Dynatrace Agents

Dynatrace client will give all the metrics of the server health and application performance.

· It is the front end component which will show the end user point of view experience of using the application.

Dynatrace collector will collect all the information from the application server, web server and Database server for sending all the traffic related information.

Dynatrace Server in turn receive all the information from Dynatrace collector to send the information of servers and application health to Dynatrace Client

Dynatrace Agents are the main important to monitor the application server and web server, Database server for bringing out the application level metrics both client side and server side including the application level code issues.

Working Mechanism of Dynatrace

The main focus on working with Dynatrace tool will be on the two key factors

Pure stacks

Pure paths

Pure stacks: In the Pure stacks will follow the bottom to top approach to find out the root cause of the problem in the application.

Bottom to top approach involves identifying the application issue from the server level which will be majorly on the hardware level such as memory, CPU, disk level problems before going into code level application problems.

Important thing we will see in the pure stack is in the application server we will check

(a) Host health: It will show the health of the hardware details of application server

1) CPU Utilization: Always CPU Utilization should be under threshold limit (80%). If we found the CPU Utilization is greater than or equal to 80% then we will analyze the Thread dump for finding out the reason for taking the high CPU utilization.

If you find in any specific period observe high CPU but all the remaining resources are looking acceptable limit, also if no issues found in the thread dump.

We need to go for the CPU Sampling.

In the CPU sampling need to analyze in that specific time period which particular CPU threads are causing the CPU wait or block for making application to consume high CPU utilization.

2) Memory: Memory consumed from the total available memory. Acceptable level should be less than 80% of total memory.

3) Network Utilization: Network utilization should not be having spikes during the load test.

It includes the network parameters like network latency and bandwidth for making the proper channel for maximum bandwidth.

4) Disk space: How the Disk reads and writes are going on in the database can be identified here for proper maintaining file related operations.

5) In the start center , go to the Analyze performance and Analyze the load test results for getting the how the load test or corresponding performance test is performed.

We will get the metrics of Response time and Load vs Response time. To analyze the performance of the test.

(b) Application Health:

In the application Health of all the application server JVM overview, we can see the Heap Memory, it is performed throughout the performance test.

We will also get the GC suspended time details, Thread count.

In the Purepath will follow top to bottom approach to find out the root cause of the problem in the application.

Top to bottom approach involves identifying the application issue from the code level such as which particular methods in the application is taking more response time, finding out the method hotspots causing the application to consume more time to load the pages.

PurePath:

This is multidimensional data related to every transaction which gives the clear visibility across Browser, web apps, web server, app server and database server.

Gives the entire information about a transactions regardless where they originated (Browser/server) from end to end covering all web requests, calls, methods, and database calls etc.

UserActionPurePath:

This is a Dashlet which captures all purepaths which are generated in the browser (or) which are generated based on the user actions (ex: clicking on button, selecting a link).

For this user action generated in browser, there consist of many web requests from end to end which can be traced using the option User Action Pure Path.

Applicare APM tool: Advanced Application performance management tool

Introduction to Applicare tool:

1.      Applicare is one of best application performance management (APM) solution that not only monitors but also improves your enterprise application’s performance, stability and availability.

2.      Applicare helps cut costs by early detection of problems, significantly reducing mean time to repair and preventing/eliminating outages.

3.      Over time, Applicare makes you become proactive and keeps application performance at its peak - because you and your customers cannot afford the drawbacks/cost that comes from being reactive.

4.      Applicare intelligently measures application performance, auto analyzes and auto tunes your environment.

5.      Applicare warns of system downtime symptoms and their probable causes before your customers are impacted.

6.      With Applicare Arcturus Technologies takes enterprise application performance management to the next level, with predictable and peak-performing SOA environments.

Services of Applicare tool:

1.      Automated application performance tuning

2.      Automated Problem Detection

3.      Automated Health Monitoring

4.      Automated Root-cause analysis

5.      InteliSense – Our patent pending technology that auto detects problem transactions.

6.      Monitoring overhead by reducing the # of transactions that need to be monitored significantly.

7.      InteliTrace – Our patent pending technology that auto detects what methods should be monitored at what time.

8.      Reduces monitoring overhead by reducing the amount of code that is monitored at any point of time.

9.      Business Transaction Monitoring

10.  End User Experience Monitoring

11.  Synthetic Transactions

12.  Real User Experience Analysis

Advantages of Applicare tool:

1)     Easy to monitor the all the Application server , database servers in a single place

2)     Monitoring can be done from the application health point to server hardware health including database

3)     End to End monitoring from Network related issues and GUI based metrics.

4)     It will be providing the sample application Jpetstore to play around for better user experience of tool

5)     Multi- level Architecture of the tool gives all the metrics in a single place for better visualization of tool

6)     Code level profiling also can be possible for optimizing the code level issues in the application.

7)     Easy monitor the JVM related problems to give best options in improving application performance

Note:

Practice the Applicare tool using the below link

Register yourself at http://support.arcturustech.com/register.do and get an access to free version of Applicare"

Performance Monitoring of different servers of an Application

When we are running different types of performance tests we need to monitor the servers apart from regular performance metrics like transaction response times of Average and 90th Percentile, we need to look into below components of servers for identifying root cause of problems. 1) Webserver: While monitoring the webserver, we have to check for the Number of hits per sec (directly proportional to Throughput) Number of Active sessions available for your application 2) Application server: CPU utilization -- Indicates the CPU consumption in the application Memory Consumption --Indicates the total memory consumed for application Heap Memory -- Object level memory utilization Thread Dump -- Gives the problems in states of the threads which is related for CPU problems Heap Dump--Gives the problems in states of the objects which is related for Memory problems Garbage collector -- Gives the problems related to cleaning of unused references 3) Database server Query processing time -- Time taken to process the query from client to server and back to client. Number of connections established for Database (Connection pooling)--Number of database connections and its significance Checking index is present for table or not -- For fast processing of data , we use indexes Database level dead locks -- When ever pages are struck in loading due to concurrency issues Database Level: Tables and its Locks -- How the information in the database is maintained Procedures -- Sequence of steps to process the SQL data for getting desired results Triggers -- Automatic processed which is available for timely events Stored Procedures --Pre-compiled data for processing the results when executed Functions --Reusable codes used for fetching the desired results Pre-Requisite of DB Analysis: 1) Before starting the load test it is advisable to take the AWR Report. 2) After completion of the load test it is mandatory (mostly for DBA) to take AWR report. 3) Comparison of Reports will be done in the Report level (Data comparison will be happen)

Interview Questions

1.     What are the different scripting issues you faced so far?

Ans) some of the important real times which I have faced are

(a)  When I am recording the loadrunner script , the script has been not recorded then I have checked the Network options changed the capture level from WinNet level data to Socket level data, also added the new entry Server->All, Port->Any and service->Auto detect, SSL->No

(b) Problem) while generating analysis report the transaction summary has numbers instead of transaction names. This issue is seen mostly in case of test being run in performance Center. 

   Solution)Try the following workaround so that Analysis will display names under transaction names rather than numbers in the Analysis Report.

1. Open a new analysis session and change the option there in Tools ---> Options ---> Result collection --->Choose "Generate summary data only."

2. The default is "Display summary while generating complete data." 

3. Then, go to File ---> Open ----> Change the "Type of File" to "LoadRunner Results."  The default is "Analysis Session Files."

4. Choose the .lrr file and Click on "Open". 

5. The Analysis report will be generated.  Save the .lra file.

6. Open the .lra file and check the Summary report for transaction names.

2.     What are the performance bottlenecks that you found in projects you were working? What are the recommendations made to overcome those issues?

Ans) I have found the many bottlenecks in application, some of them are

(a)  I have found the bottleneck in the Javascript request which is taking more response time than SLA (service level agreement) , when I studied deeply into it , the main cause for taking the high response time is the loading of the css file is very heavy to download and redirection is happening while loading the content. (It is fixed by using angular js instead of simple java script)

(b) For logging into application and also which includes database related transactions are very slow, then I have looked into the tables in the database indexing is not present in the tables, once the indexing is attached for the tables then readability of the very fast when compare to previous results.

3.     Have you applied Little’s law in your project? If so, how?

       Ans) Yes, The law states that the average number of customers in a system (over some time interval), N, is equal to their average arrival rate, X, multiplied by their average time in the system, R.

N = X . R

This law is very important to check whether the load testing tool is not a bottleneck.

For Example, in a shop , if there are always 2 customers available at the counter queue , wherein the customers are entering the shop at the rate of 4 per second , then time taken for the customers to leave from the shop can be calculated as

N = X . R

R = 2/4 = 0.5 seconds

A Simple example of how to use this law to know how many virtual users licenses are required:

Web system that has peak user load per hour = 2000 users

Expected Response time per transaction = 4 seconds

The peak page hits/sec = 80 hits/sec

For carrying out Performance tests for the above web system, we need to calculate how many number of Virtual user licenses we need to purchase.

N = X. R

N = 80 . 4 = 320

Therefore 320 virtual user licenses are enough to carry out the Load Test.

Eg: Any website, Login and check the URL. The URL consist of Session Value which is dynamically generated by the Server and even developer cannot tell that "What the dynamic value is going to generated"

4.     What is your approach for analysis?

Ans) Approach for analysis for finding out problems in the application is always depends on the initial baseline test conducted for application.

If the application is not reaching the SLA (ex: 5secs) then issues may be the Front end technologies like java script, css, angular js, html 5…etc code in the methods may consume higher response time due to the bad code.

The Data base tables are not having proper indexing, also the connection pooling for loading the data may very slow, and data loading is very slow due the improper normalization and duplicate records in database.

5.     What do you monitor while execution?

Ans) While execution of the desired scenarios for the application , we always look for both client side metrics like transaction response time, throughput, hits per second , Running Vusers ..Etc and server side metrics too like CPU Utilization, Memory consumption, garbage collector performance …etc

6.     How to extract server data for test execution and how to analyze that?

Ans) Every time post the execution of the tests we need to exact the data from the server which is the CPU ,Memory , Garbage collector , Heap Memory ..etc which may be in graphical representation we need the third party tools like Dynatrace , JavaMelody , Jvisual VM..Etc to get the exact metrics

7.     How to identify performance bottlenecks?

Ans) Identification of the bottlenecks is looking deeply into the code level why was problem causing not to reach particular transaction with in the SLA (service level agreement).

Based on the existing environment need to check the server and client side code, server’s performance, Architectural level components effecting for the performance in current release. Etc

8.     What are the Challenges that you face during scripting?

Ans) Challenges faced on the scripting

1)    Using of the file functions in the scripting of Vugen

2)    Using of the randomization techniques in scripting

3)    Handling the correlation for the .Net and java applications like view state , Even validation , session ids…etc

9.      What is the Correlation function? How do you do that and how do handle the correlation function? What are the challenges in correlation? How do you know that this is correlation issue? How do you identify the correlation value?

Ans) (a) Correlation is a concept where we can capture the dynamic data when it is generated and used throughout the script.

(b) Correlation function can be added by using the predefined function of the loadrunner (web_reg_save_param)

Ex:

//<input type="hidden" name="userSession" value="118094.434831916zAHtticptQVzzzzHDzttzpiQtQf"/>

web_reg_save_param("sessionId","LB=type=\"hidden\" name=\"userSession\" value=\"","RB=\"/>",LAST);

(c ) Challenges in the correlation is  proper capturing the dynamic value with the boundaries and make sure that script should work for any number of users successfully with any number of iterations.

Randomization Technique for correlation function:

To pick up the random occurrence of our dynamic

char *temp;

web_reg_save_param("Random correlation variable","LB=value","RB=value","ORD=ALL",LAST);

temp=lr_paramarr_random("correlation variable");

lr_save_string(temp,"Random_Correlation");

Now we have to substitute this Random_Correlation variable for the place of dynamic value.

(d) Correlation issue is know when we are running the same script for more than once it fails due to change in the response from the server, which is shown in the below screenshot.

(d) correlation value is identified by recording the same flow of the application twice and comparing the both the scripts in side by side, what are the values changing per each recording which is not user input, the values should populate from the server.

10.  How do you handle the error handling in scripting?

Ans) Error handling can be done in the scripting of the loadrunner , to give the user friendly messages to the users and make sure that script which we are running is navigating through the entire application flow according to the work flow.

Example:

web_reg_find("Text=Login","SaveCount=Text1",LAST);

lr_start_transaction("Demo_Login");

web_url(........)

if(atoi(lr_eval_string("{Text1}"))>0) Note: array to integer.

{

lr_end_transaction("Demo_Login",LR_PASS);

}

else

{

lr_end_transaction("Demo_Login",LR_FAIL);

lr_error_message("%s",lr_eval_string("Login is failed for this user {VuserID} Iteration number {Iteration_Number}"));

lr_exit(LR_EXIT_MAIN_ITERATION_AND_CONTINUE,LR_FAIL);       
 }