The bench works, the script works, and then the engineer who wrote it leaves. Six months later a limit has to change from 4.9 V to 4.85 V, and nobody knows which of the 3,000 lines holds it, or which test results were produced before and after the last edit. Most test software does not fail because the measurement is wrong. It fails because it cannot be changed safely.
This guide shows how to grow a test script into a structured, maintainable test program, with the separation of concerns, the data handling and the error strategy that make it survive years of production use.
1. The problem with the “one big script”
A first script usually mixes everything: instrument commands, the order of tests, the limits, the pass/fail logic and the report formatting, all in one file. It is fast to write and works well for a lab prototype. It becomes a problem when:
- the same test must run on several bench variants with different instruments;
- limits change with each product revision;
- an operator, not the author, runs the bench every day;
- a customer or quality team asks which software and which limits produced a given result.
Example of a fragile line: if dmm.read() > 4.9: fail() hard-codes the instrument, the limit and the decision in one place. Changing any one of them means editing and re-validating code.
2. Principle: four layers kept apart
2.1 Described architecture diagram
Picture four horizontal layers, top to bottom:
- Sequence — the ordered list of test steps: setup, measure, evaluate, cleanup. It says what to test and in which order, nothing about how each instrument works.
- Limits and configuration — an external, versioned file (CSV, JSON, YAML or a database table) with each step’s low limit, high limit, unit and parameters.
- Instrument drivers — one module per instrument type, exposing simple functions (
measure_voltage(),set_output()), hiding the communication protocol (SCPI over USB/Ethernet, vendor API, serial). - Results and reports — a single service that receives every measurement with its context and writes it to the report and the data files.
Arrows go only downward from the sequence to drivers, and from every layer to the results service. No layer reaches around another.
2.2 Why it pays off
| Change requested | One big script | Layered program |
|---|---|---|
| New limit | Edit code, re-test | Edit limits file, version it |
| Replace a DMM by another model | Search all commands | Replace one driver |
| New product variant | Copy the script | New sequence, same drivers |
| Audit a past result | Guess | Report states software and limit versions |
3. Choosing the tool: Python, TestStand, LabVIEW
These are the three tools we see most often on test benches. Our own test software is built on a Python base, with TestStand or LabVIEW integration on request.
3.1 Python
- Strengths: free, large ecosystem, easy to version with Git, good for data processing, strong libraries for instrument communication.
- Watch out for: you build the sequencer framework, operator UI and report format yourself, or choose a framework. Test frameworks such as pytest can structure steps, but production sequencing needs extra work (operator prompts, serial numbers, station data).
3.2 NI TestStand
- Strengths: a dedicated test sequencer with built-in step types, limits, reports, user management and parallel testing. See NI’s own description in What is TestStand.
- Watch out for: licences per station, and code modules still to write (in Python, LabVIEW, C or .NET).
3.3 LabVIEW
- Strengths: graphical programming, close integration with NI hardware, fast for instrument control and UIs.
- Watch out for: graphical code is harder to diff and review in version control; discipline is needed for large projects.
3.4 Selection criteria
- What the customer’s teams already maintain (the best tool is the one they can support).
- Number of stations and licence budget.
- Hardware used (vendor drivers available).
- Need for parallel testing of several units.
- Traceability requirements (who ran what, with which version).
- Data-processing needs after the test.
4. Limits: data, not code
4.1 What a limit record contains
For each step: step ID, description, low limit, high limit, unit, comparison type (between, greater than, equal), and optionally the measurement parameters (range, integration time, settling delay).
Example record (illustrative values):
STEP_120 ; Supply current, standby ; 0 ; 15 ; mA ; GELE
(GELE = greater-or-equal / less-or-equal, TestStand notation.)
4.2 Version control
- Store limit files in the same version control as the code.
- Each release gets a version identifier written into every report.
- Changes go through review: who changed what, why, approved by whom.
4.3 Worked example: guard bands
Example, illustrative values. The product specification allows 5.00 V ± 0.10 V. The measurement uncertainty of the bench at that point is estimated at ± 0.02 V. To reduce the risk of passing a part that is actually out of tolerance, the acceptance limits can be moved inside the specification by a guard band, here equal to the measurement uncertainty: 4.92 V to 5.08 V. The price is a higher risk of rejecting good parts close to the limits. The guard band is usually based on the expanded uncertainty (k = 2), and decision rules of this kind are described in ISO 14253-1 and JCGM 106, used here as references. Whether and how to apply a guard band is a decision agreed with the customer and written in the test specification.
5. Results and reports
5.1 What each result must carry
- Unit serial number, date and time, station ID, operator.
- Software version and limit-file version.
- Step ID, measured value, unit, limits, pass/fail status.
- Instrument identifiers where traceability to calibration is needed.
5.2 Formats
- Human report (PDF or HTML) for the operator and the customer.
- Data export for analysis: CSV for universal use, TDMS when working in the NI ecosystem with large waveform data.
- Timestamps in one documented time zone, and filenames that include serial and timestamp so nothing is overwritten.
6. Instrument errors and recovery
6.1 Errors to expect
- Instrument not responding (cable, power, address).
- Timeout on a long measurement.
- Value out of instrument range.
- Unexpected state left by a previous aborted run.
6.2 Strategy
- Distinguish test failure from test error. A part out of limits is a FAIL. An instrument that doesn’t answer is an ERROR, and the part status is unknown, not bad.
- Retry with a limit. One or two retries on communication errors, logged, never silent. Never retry a measurement that failed its limits.
- Always run cleanup. Whatever happens, outputs are switched off and loads disconnected. In code, this belongs in a
finallyblock or the sequencer’s cleanup group. - Start from a known state. Each run begins by resetting instruments and checking their identity.
- Log everything with enough context to reproduce the problem.
6.3 Timing note
The PC sequencer is not a timing reference. When a step depends on precise timing, delegate it to hardware-timed acquisition and let the software read the result. See our guide on acquisition timing and jitter.
7. Common mistakes
- Limits written in code.
- Instrument commands scattered through the sequence.
- No software version in the reports.
- Retrying until it passes.
- Treating an instrument error as a product failure (or worse, as a pass).
- No cleanup after an abort, leaving outputs live.
- Reports that overwrite each other because filenames are not unique.
8. Checklist for a maintainable test program
- [ ] Sequence, drivers, limits and reporting are in separate modules.
- [ ] Limits live in an external, versioned file.
- [ ] Each report states software and limit versions.
- [ ] PASS, FAIL and ERROR are distinct statuses.
- [ ] Cleanup always runs, including after an abort.
- [ ] Instrument identity checked at start-up.
- [ ] Data exported in an analysis-friendly format with unique filenames.
- [ ] A change procedure exists: review, test on reference units, release.
- [ ] The software was validated on known good and known bad units before production.
9. Going further
Software is half of a validated bench; the other half is proving the bench measures correctly, covered in qualifying a test bench before production (FAT/SAT). Sequencers also drive HIL benches and fault-injection matrices. For how test procedures relate to acceptance, read ATP vs QTP and our ATP methodology.
RAYNOV writes test software as part of its custom validation systems and can support your team through Validation as a Service.
Want a second look at your test software architecture? Contact our engineers.
