LoadRunner Controller Crashes

Suggested debugging steps Controller crashes

If Controller crashes intermediately or shows abnormal behaviour, verify/try the following:

• Make sure that you login to Controller as local administrator

• Make sure that there are sufficient disk space

• Check if the size of the output.mdb file in the results folder is more than 2 GB

• Verify the CPU usage on the machine 

• Try to run the Controller’s batch files

• Try to recreate the Controller’s initialisation files

• Check the temporary environment variables 

• Verify if Hyper Threading is enabled

• Verify the MDAC version

• Reboot the Controller machine

• Verify if there are any information in the Event Viewer 

• Verify if other programs are interfering with Controller

• Shut down all unnecessary processes

• Disable the anti-virus software 

Reinstall LoadRunner

• If Controller is only crashing on large load test, verify/try the following:

• Make sure that there are sufficient disk space

• Make sure that there are sufficient memory available

• Verify the CPU usage on the machine Modify the heartbeat timeout

Known Problem and limitations(links)

Problem ID 26375 - The Controller uses up all the memory though no scenario is running

Problem ID 21925 - The Controller stops responding if a load test scenario issues a high number of errors

Problem ID 22334 - The Controller displays "Loading Data..." but the error text is not shown in the output window

Problem ID 15350 - Generator or vUser or Show output or Error or Transaction window does not open in the Controller  

Make sure that you login as a local administrator

• Log in to the Controller machine with a local administrator account or a domain account that has local administrator privileges.

Make sure that there are sufficient Disk Space

• Make sure that you have enough disk space available on the controller and load generators. During scenario execution, the events are written onto the Load Generator machines and are saved locally until the scenario finished; where results are send back to the Controller. If the machine does not have enough disk space, it can cause problem.

 Check if the size of the output.mdb file in the results folder is more than 2 GB

• If the output.mdb file becomes greater than 2GB during a load test, Controller is unable to write into it anymore and cause a crash. If the size of output.mdb file is larger than 2 GB, divert the output to a text file as per Problem ID 15075 - How to send the Controller output information to a file instead of viewing it from Controller itself

Run the Controller’s batch files to register DLLs

Sometimes, DLLs can become unregistered or the registry can become corrupted to a point where a program's DLLs cannot be found.  The purpose of batch files is to re register them into the system's registry so that the programs can locate them.  Use the following steps to do this:

1.      Shut down the Controller.N

2.      Navigate to the <LoadRunner>\bin directory, and look for the following files: 

·         register_controller.bat

·         set_mon.bat

3.      Create a duplicate copy of the file, in the same location.

4.      Open up the duplicated file. In it, you should see several entries like the following:

·         regsvr32 /s webbrwsr.dll

·         Remove the "/s" from each of these statements, but leave a space between the "regsvr32" and the DLL name

5.      Save the changes.

6.      Double click on the batch file to run it. You should get several pop-up messages similar to the following:

Note: You could get failure messages for escomps.dll and resmonps.dll. You can safely igonore these errors. But you should not get any other failure messages.  If you do, then that DLL is the problem file. Try registering this from the command prompt:

Example: regsvr32 < LoadRunner >\bin\<DLL name>

If this does not work, try copying the DLL from the CD or ask support for the DLL to try registering it manually.  Retry the same action that caused the error.

 

Verify the CPU usage on the machine

Verify the CPU usage on the machines. Ideally, they should be below 80% usages. If necessary, distribute the Vuser load to other machines

 

Try to recreate the Controller’s initialization file

Sometimes, the initialization files can become corrupted (e.g. after a crashed). You will have problem in launching or using the Controller after that.  Use the following steps to do delete the initialization file so that a new copy will be created:

1.      Shut the Controller.

2.      Navigate to the C:\Winnt  ( or C:\Windows for Windows XP machine )

3.      Delete all files that begin with wlrun*. For example, 

      wlrun.ini, wlrun5.ini, wlrun7.dft, wlrun7.hst, wlrun7.ini

 

Check the temporary environment variables

Unlike the earlier window’s versions, Window 2000 and Window XP have the default environment set to c:\Document and Settings\<user-name>\Local Settings\Temp instead of c:\Windows\temp. This long path with a space can cause several problems on LoadRunner. To resolve the issue, change to a directory without empty spaces.

For further information, please refer to:

• Problem ID 15500 - How to set the system's TEMP and TMP directories
• Problem ID KM174286 - How to change the Results file folder in the Controller's Result Settings

Verify if Hyper Threading is enabled

Verify if Hyper Threading is enabled. If so, you will need to disable it. LoadRunner does NOT support Intel Hyper-Threading technology. Hyper-Threading can be disabled in the BIOS. For more information, refer to
http://www.microsoft.com/windows2000/docs/hyperthreading.doc.

Verify the MDAC version

Make sure that you have MDAC 2.6 or higher installed. You can find the MDAC version on your machine by referring to the information on Microsoft Knowledge Base Article 301202 - HOW TO: Check for MDAC Version.  You can download the latest MDAC version from MDAC Downloads

Reboot

A crash of a program leaves the system in an unstable state.  This can cause many other problems that seem to have no apparent reason for happening or has not happened before.  When the system is rebooted, it resets the system into a more stable state.  This should be done after any program crashes.

Verify the information in the event viewer

Sometimes, if a program crashes, it does not give any clues for what had happened.  By using the Windows event viewer, it may be possible to find some clue as to what happened when the crash occurred.  The event viewer can be launched from Start ® Programs ® Administrative Tools ® Event Viewer.

Verify other programs are interfering with Controller

To find out whether hooked DLLs are possibly causing a problem, you can use a third party utility call "Process Explorer."  This utility has the ability to view the DLLs loaded by an application.  It can be downloaded free of charge from the following link:

http://www.sysinternals.com/ntw2k/freeware/procexp.shtml

This can be used to see if LoadRunner loaded any other program's DLLs.  Use the following steps to do this:

1.      Unzip the .zip file, which was downloaded from the above URL, into a directory where you wish to install Process Explorer.

2.      Start the Controller.

3.      Run the Process Explorer (procexp.exe) from the directory into which you unzipped it (step a).

4.      Select wlrun.exe (Controller) from the top section of Process Explorer. 

5.    The bottom section should be displaying a list of DLLs.  If it is showing handles for the application, go to the "View" menu and select "DLLs."  It should look like the screenshot below:

6.   Search through the list to see if any other program's DLLs are loaded.  Normally, only DLLs from the <LoadRunner>\bin directory and standard Microsoft directories are loaded.  For example, if you see wbhook32.dll (McAfee VirusScan hooking DLL) loaded by LoadRunner, then you would want to shut down the anti-virus software

Shut down all unnecessary processes

Some programs are designed to have certain DLLs "hook" or be loaded into another program's memory space.  Normally, this should not have any effect on the application itself.  However, it can interfere with some programs and cause them to behave erratically or crash. (Verify if there are other programs that interfering with Controller )

 For such, it would be recommended to shut down all processes that are not necessary, regardless if they hook into LoadRunner or not.  Any programs that run as an icon in the system tray or on the taskbar are the first candidates for termination.  Also, you can look through the list of processes in the Task Manager (right-click on the taskbar and select "Task Manager").  Some processes are system processes, which may not be able to be shut off, but any processes that can be shut down should be.

Disable anti-virus software

It is known that anti-virus software is intrusive when they are set to look for viruses.  However, in searching for viruses, the software can interfere with a program's proper execution.  This could cause problems and sometimes crashes.  This is why, for debugging purposes, we would recommend turning off the anti-virus software.

The icon for the anti-virus software resides in the system tray (where the clock is located).  Normally, you should be able to right-click on the icon and select "disable."  However, some setups do not allow a user to turn off the anti-virus software.  It is recommended to speak to a system administrator to get the anti-virus program disabled for a short period for debugging the problem.

(Verify if there are other programs that interfering with Controller )

Reinstall

In case that all the above steps fail, the only recourse left would be to try to uninstall LoadRunner.  It is possible that either a previous version of LoadRunner was on the machine before the current installation or that the installation did not go properly although the installation did not give any errors.  It is recommended that a full uninstall be done in this case.  The following steps are for a full uninstall:

1.      Make sure that, all running LoadRunner processes (including the Controller, VuGen, Analysis and the Remote Command Launcher (for 6.x) or the LoadRunner Agent Process/Service (for 7.x) are closed.

2.      Backup any existing scripts that may have been saved in the LoadRunner installation folder (The scripts are sometimes saved in a 'scripts' sub directory under the LoadRunner installation folder.).

3.      Run the uninstall program from the LoadRunner program group (or) use the Windows add/remove programs from the Control Panel. If any prompt is given about removing shared files, remove all the shared dlls that are reported as no longer being in use.  In the very rare instance this causes a problem for some other application it may be necessary to re-install that other app.  This is not generally a problem because every application should have registered which DLLs it needs to run.

4.      Reboot the machine after the Uninstall wizard is complete.  This will complete the basic uninstall procedure.

5.      Delete all LoadRunner Folders. (Including the ones in the start up menu for Remote Command Launcher (LoadRunner 6.x) or Agent Process (LoadRunner 7.x)

6.      If the installation is LoadRunner 6.x then delete the Borland Folder (Usually in c:\Borland or c:\BDE).

7.      Do a search for the following files and remove them from all locations -- they will be replaced during the re-install.

a.       wlrun.*

b.      vugen.*

8.      Bring up the registry editor: (Start à Run à regedit).

9.      Delete the following keys:

a.       Only for LoadRunner 6.x

HKEY_LOCAL_MACHINE\SOFTWARE\BORLAND

b.      If Load Runner is the only HP product on this machine, then delete

HKEY_LOCAL_MACHINE\SOFTWARE\Mercury Interactive.

HKEY_CURRENT_USER\SOFTWARE\Mercury Interactive.

c.       Else delete

HKEY_LOCAL_MACHINE\SOFTWARE\Mercury Interactive\LoadRunner.

HKEY_CURRENT_USER\SOFTWARE\Mercury Interactive\LoadRunner.

10. Empty the Recycle-bin.

After you remove these items, you can re-install LoadRunner.  Also, make sure that you do not have any anti-virus programs running while you are installing LoadRunner.  That has been known to cause some problems with the installation of LoadRunner.

NOTE: Reinstallation should only be done for the following certain circumstances. 

1.      The crash only happens on a particular machine.

2.      Some feature that was previously working is now crashing.

3.      The above options were tried before hand.

 

Modify the heartbeat timeout

In LoadRunner, all hosts manage heartbeat mechanisms with the Controller. By default, each host sends four messages in two minutes with frequency of 30 seconds until heartbeat failure is reached. This could cause a problem when there are a lot of hosts or large data files.

To work around the issue, you need to minimize the heartbeat frequency so that the hosts will send less heartbeat messages to the Controller. Since LoadRunner waits time-out*4 before it reaches heartbeat failure, you can increase the time-out value of the Controller and Load Generator machines (so as to minimize the frequency) or minimize the heartbeat messages.

To minimize the heartbeat messages:
1. Close the Controller, and stop the LoadRunner Agent process/service.
2. Modify the mdrv.dat file in the <LoadRunner>\dat directory.
3. Add the following lines to the end of [Launcher] section:

ExtCmdLine= -HB_TIMEOUT <timeout in milliseconds> //to modify timeout

ExtCmdLine= -HB_TIMEOUT_LIMIT <num_of_trials> //to modify the number of messages

Example:
[Launcher] ExtPriorityType=launcher WINNT_EXT_LIBS=launcher.dll WIN95_EXT_LIBS=launcher.dll LINUX_EXT_LIBS=liblauncher.so SOLARIS_EXT_LIBS=liblauncher.so HPUX_EXT_LIBS=liblauncher.sl AIX_EXT_LIBS=liblauncher.so LibCfgFunc=launchext_configure UtilityExt=two_way_comm ExtMessageQueue=0 SecurityMode=On ExtCmdLine= -HB_TIMEOUT 120000 ExtCmdLine= -HB_TIMEOUT_LIMIT 5

Note:

The "-HB_TIMEOUT 120000" option sets the time-out to send heartbeat messages every 120 seconds.

The "=-HB_TIMEOUT_LIMIT 5" option sets the number of times to resend heartbeat messages to 5.

4. Repeat steps 2-3 for the mdrv.dat file in <LoadRunner>\launch_service\dat directory of the Load Generator machine.

Note:
If you change the heartbeat to be every 2 minutes, the Controller will be able to establish a connection, however since the hosts still send heartbeat messages it cause the UI to get stuck every 2 minutes for half minute.

Make sure that there are sufficient memory available

To check available memory on a machine:
Right-click the status bar, and select Task Manager. Select the Performance tab to check the physical memory available. Select the Processes tab to check which processes have high memory consumption in the CPU column.  

To free up memory:

• Close any unnecessary processes running on the machine, and try running the scenario again 
• Restart your computer. 
• If the problem persists, reduce the number of virtual users that you are running on the same machine

To enlarge the size of your virtual memory:
1. Click Start -> Settings -> Control Panel -> System.
    a. For Windows 2000, select the Advanced tab, and click Performance Options.
    b. For Windows NT, select the Performance tab.
2. In the Virtual memory section, click Change.
3. In the Drive list, click the drive that contains the paging file you want to change.
Under Paging file size for selected drive, type a new paging file size in megabytes in the Maximum size (MB) box, and then click Set.

To boost performance, and allow more Virtual Users to run on the load generator machine:

·        On Windows 2000 machines, select Start -> Settings -> Control Panel -> System -> Advanced -> Performance Options, and select the Background Services option.

·         On Windows NT machines, select Start -> Settings -> Control Panel -> System Properties > Application Performance. Set Performance boost to "None."

- Contributed by Muthukumaran K