CSV Import Fails Intermittently in a Load-Balanced STARLIMS Environment

Summary

A CSV import failed intermittently on a two-node load-balanced STARLIMS installation. The file existed, permissions were correct, and antivirus was ruled out, yet the import failed most of the time. The root cause was that the import file sat on a local drive, while the batch command could be picked up by any node in the farm. Sharing the work path resolved it.

Symptom

• The CSV import failed with an error referencing the file path.
• Behaviour was inconsistent. It occasionally succeeded, but failed most of the time.
• The file genuinely existed at the path reported in the error.
• The same path read successfully using ReadText.
• Running the parser script (ImportParsers.ValidateText) directly from the Designer with the default path set completed with no errors.
• Running the script with ExecFunction instead of SubmitToBatch worked reliably.

Environment

Two servers behind a network load balancer. The import file was located on a local drive (D:\Temp\Temp), not on a shared network path.

Steps Already Attempted

• Changed the application pool identity to a user with full administrative privileges.
• Disabled antivirus on the application server. The issue persisted.
• Tested the import directly on a single node. The issue still occurred.

Investigation Path

• Antivirus / antimalware. Intermittent failures at different points in a process are a classic signature of dynamic interference, and "learning" antivirus products fit that pattern well. Note that a disabled antivirus can still interfere, and in some cases another security tool takes over when one is disabled. To rule it out properly, the antivirus must be temporarily removed and the server rebooted, not merely disabled. Process Explorer can be used to identify which process is holding the file at the moment STARLIMS tries to access it.
• Path and permissions. Confirm that the folder and file referenced in the error genuinely exist, especially when the error cites the default value of the sServerPath parameter. Grant full access to Everyone on the folder and its child objects, then recycle the STARLIMS application pool. A simple SSL script calling ReadText(<path>) is a quick way to verify server-side access to the location.
• Intermittency handling. Where a failure is genuinely transient, a retry pattern is appropriate: on error, wait a few seconds and try again. It is also worth questioning whether LimsNetConnect is needed at all when ReadText can be used directly.

Root Cause

Commands submitted through SubmitToBatch can be picked up and processed by any application server or batch server in the farm. The import file was stored on a local drive (D:) rather than a shared network path, so it was not visible to the other node. Whichever node happened to pick up the batch command determined whether the import succeeded, which is exactly why the failure looked random.

This also explains the two diagnostic clues: ExecFunction worked because it executes on the node handling the request, whereas SubmitToBatch could be routed elsewhere.

Resolution

The work path was shared so that it is accessible to all nodes. The import then worked consistently.

Key Points

• In a load-balanced or multi-node STARLIMS environment, any file consumed by a SubmitToBatch operation must live on a shared network path, never a local drive.
• If ExecFunction works but SubmitToBatch does not, suspect node-to-node file visibility before suspecting file locking or antivirus.
• "Sometimes works, mostly fails" in a farm usually points to node routing, not to a dynamic blocker.
• Disabling antivirus is not the same as removing it. Remove and reboot if you need to genuinely rule it out.
• An alternative to a custom local folder is the STARLIMS temp location, which the application already reads and writes.