STARLIMS Performance Triage

When STARLIMS feels slow, finding the cause is a joint effort. You know your users, your network and your servers; we know how STARLIMS behaves and what it needs from the platform underneath it. This article is the shared checklist we work through together with you, so that neither side has to guess.

We'll guide you through it in a ticket and tell you which chapters apply to your situation; you do not have to do everything at once. Most performance tickets are solved with the information in the first three phases. If yours needs our database specialists, phase 4 makes sure they get what they need in one go instead of a second round of questions.

The four phases:

1. Understanding the problem — what is slow, for whom, since when and what changed. You answer; we turn it into a precise problem statement and confirm it back to you.
2. Checking the requirements — what STARLIMS needs from hardware, software and configuration, with the reason behind each requirement. Many findings here can be corrected by your own IT team straight away, and we review the rest with you.
3. Collecting data — measurements and read-only scripts that give both of us facts about your servers, database and network.
4. Advanced data collection — additional information our database administrator may ask for in complex cases.

A few things make this go faster for everyone:

• Please answer in the ticket rather than in separate e-mails, in English, and include dates and times wherever you can; we match your answers against log files.
• Tell us who your IT and database contacts are, so we can discuss technical details with them directly or address them in the conversation.
• Measure while the problem is happening, not after a restart.
• Nothing in this article changes your system. All scripts are read-only. Any change we recommend comes later as a separate, explicit request, so it can follow your own change procedure.
• If your STARLIMS environment is hosted by STARLIMS, chapter 2 and most of chapter 3 are our responsibility; your engineer will tell you which parts still apply to you (usually the client and network checks).

The goal of this chapter is to understand scope, timing, and history and not to jump into technical details yet.

• Is this happening in one specific screen/module/report, or everywhere in STARLIMS?

• Does it happen for one user, a specific group/site, a location or everyone?

• Does it happen on all client machines, or only some? (Same issue on a fresh/different PC?)

• Is it slow logging in, opening a folder/sample, running a report, saving data, or something else specific?

• Is it slow all day, or only at certain times (e.g. first thing in the morning, around shift change, end of (work)day)?

• Did it start suddenly on a specific date, or has it been gradually getting worse?

• Is it worse under heavier load (more users logged in, batch jobs running, day-, week- or month-end reporting)?

• Any recent STARLIMS updates/patches, hotfixes, or custom code deployments?

• Any recent infrastructure changes: server moves, VM resizing, network changes, antivirus updates, Windows/SQL patching, storage changes?

• Any recent integration or volume changes (new instrument feeds, new interface, data migration, larger data sets)?

• Are servers rebooted / IIS reset / SQL Server restarted?
  ⚠️This is not an invitation to do so, we only want information at this point.

• Has the local IT team already looked at CPU, memory, or storage on any server?

• Has it been reproduced on a different client PC, on the STARLIMS application server (using localhost) or off-network (e.g. VPN vs. LAN)?

• Is there an existing support ticket history for similar complaints?

Outcome of Step 1: it should now be possible to describe the problem in one paragraph or sentence, e.g. "Since early spring, STARLIMS has been getting gradually slower for all users, in both the browser and the App. Loading the login page takes up to 50 seconds in the morning, Edit Batch takes 20–30 seconds to open, and result entry times out often enough that users get locked out and calculations fail. It is worst in the afternoon and in the last week of the month; nothing changed in STARLIMS, but IT moved the servers to a new VM cluster in February and a BI extract was added in March. Restarting IIS helps for about an hour." 
or "Since the June 3 patch, opening the Edit Folder screen takes 20+ seconds for all users on-site, but is fine over VPN." 
If you can't yet, keep digging — don't move to Step 2 with a vague problem statement.

2. Intake Questions; Checking the Requirements

Performance tuning only works on a system that meets the basics. This chapter lists what STARLIMS needs from your workstations, servers and database, and why. Please go through each item with your IT and database colleagues, correct what you can, and report back using the table at the end of the chapter. Where you cannot or do not want to change something, tell us; the reason is often just as useful to us as the change.

Requirements below are the minimums for an out-of-the-box STARLIMS system with up to 100 concurrent users (users, instruments and batch processes together). Customized systems, interfaces and more users need more. The authoritative figures are in the System Requirements page of the Technology Platform guide or Release Notes for your version; the values quoted here are from Technology Platform 12.10.

If your STARLIMS environment is hosted by STARLIMS, only section 2.1 (client workstations) applies to you. The rest is ours.

2.1 Client workstations

2.2 Application server

2.3 STARLIMS configuration on the application server

2.4 Manager Service (batch), SDMS and Search servers

2.5 Database server (hardware and platform)

Read more details about this subject: Quick Checklist for SQL Server installations for STARLIMS

2.6 SQL Server configuration

Read more details about this subject: Quick Checklist for SQL Server installations for STARLIMS

2.7 Oracle configuration

2.8 Supported versions and lifecycle

Windows Server, SQL Server or Oracle versions that are out of extended support, and STARLIMS Technology Platform versions older than a couple of years, are not a performance problem in themselves, but they limit what we can recommend and fix. Please report the exact versions (the STARLIMS About screen has a Copy to clipboard button) and we will tell you where you stand.

For more details on supported versions see our Product Release and Support Matrix (Compatibility Matrix) and / or What's new in the latest Technology Platform (T.P.)?

3. Intake Questions; Getting some basic Data

This chapter collects quick facts and measurements that do not need special tools: screenshots, a few copy-and-paste queries and some timings. Most of it can be done by your STARLIMS administrator together with a Windows administrator in an afternoon. The results tell us where to look: at the servers, at the database, at the network or at the client. Please add the date and time to every measurement and, where possible, take them while the system is slow.

If your STARLIMS environment is hosted by STARLIMS, sections 3.5 to 3.7 (speed matrix, ping and trace route, download and HAR) and maybe 3.9 (other load on the database) are the ones that apply to you; we take care of the rest.

3.1 About screen

3.2 Configuration files

3.3 Load on the servers

On each server (application, database, Manager Service/batch, SDMS if present) open Task Manager and fill in the table below. Take the values at a moment users report slowness; if you have a monitoring tool (Zabbix, SCOM, PRTG, vCenter…), a graph of CPU, memory and disk over the last 7 days is even better than a snapshot: attach it.

3.4 Manager Service (batch processor)

The Manager Service runs the background work of STARLIMS: reports, interfaces, scheduled scripts and everything submitted to the batch queue. When it struggles, users see it as slow printing, late results or time-outs.

SELECT SETTINGNAME, VALUE
FROM LIMSENTERPRISESETTINGS
WHERE SETTINGNAME LIKE 'BATCH%'
ORDER BY SETTINGNAME

SELECT STATUS, COUNT(*) FROM LIMSBATCHQUEUE GROUP BY STATUS

SELECT CAST(SCRIPT AS VARCHAR(200)) AS SCRIPT,
       CAST(ERROR_MESSAGE AS VARCHAR(1500)) AS ERROR_MESSAGE,
       COUNT(*) AS OCCURRENCES, MAX(QUEUE_DATE) AS LAST_SEEN
FROM LIMSBATCHQUEUE
WHERE STATUS = 2
GROUP BY CAST(SCRIPT AS VARCHAR(200)), CAST(ERROR_MESSAGE AS VARCHAR(1500))
ORDER BY 3 DESC

3.5 Speed matrix

This is the single most useful measurement in this chapter. Time the same actions from different places, three times each, with a stopwatch. Before the first run on each machine, close all STARLIMS windows, and for XFD STARLIMS, clear the client cache with ccc.exe (Utilities on the STARLIMS landing page).
Note the date and time of each series.

Actions (adjust the last two (or more) to applications your users complain about):

1. Open <your STARLIMS URL> until the login page is fully shown.
2. Log in until the dashboard is fully shown.
3. Open Edit Folder / Edit Batch empty.
4. In Edit Folder / Edit Batch, load one month of data.
5. Open a second frequently used application and load its data.
6. Open a third frequently used application and load its data.

Locations: on the application server itself (http://localhost/<site>/), from a workstation in the office network, from a workstation at the site or network segment where users complain (factory, other building, remote site), and over VPN if users work that way. For XFD systems, run the series once in the browser and once in the STARLIMS App. If you can, repeat the whole matrix at a quiet moment and at a busy one.

3.6 Ping and trace route

3.7 Download test and browser trace

3.8 Log files

STARLIMS writes system logs, user logs and many temporary files. Their size and content tell us about the health of the system, and too many of them slow the server down. On the application server (and the batch server if separate) and SDMS server report for each folder below: total size, number of files, oldest file, and the five largest files.

• <application path>\Log and <application path>\Log\Users
• SDMS: <SDMS site>\Logfiles

3.9 Other load on the database and customizations

STARLIMS is often not the only user of its own database, and not the only code running in it. Please answer each point with yes/no and, where yes, what, how often and when (time of day).

Outcome of chapter 3: together we should now know whether the slowness lives on the server/database, in the network or on the client, and whether background work (batch queue, interfaces, external readers) plays a part. We confirm this to you in the ticket and tell you which parts of chapter 4 we need.

4. Intake Questions; Getting some Data

5. Narrow down the Cause (by Support)

6. Intake Questions; Getting even more Data

7. Narrow down the Cause (by Support & DBA's)

8. Additional information (appendix)