Quick answer: Display digits and setting resolution are not accuracy. Evaluate setting/readback accuracy, range, temperature coefficient, regulation, noise and the measurement setup against the permitted error at the DUT.
On this page
Accuracy is an error specification
A datasheet may combine a percent-of-reading or percent-of-range term with an offset. Use the stated range, temperature and warm-up conditions to calculate the permitted error at the intended setpoint. Setting accuracy and readback accuracy are separate specifications.
Resolution is an increment
Resolution defines selectable setting steps or measurement granularity. A display can show many digits without guaranteeing that every digit is accurate. Compare the setting/readback error against the DUT requirement rather than selecting only by screen format.
Build a DUT-side error budget
- Supply setting and readback specifications under the operating conditions.
- Lead/contact drop and the chosen sensing point.
- Ripple, noise and transient deviations relevant to the test.
- Independent measurement instrument error and connection method.
Programming and settling
Confirm supported interfaces and the timing needed after a setpoint change. Step/list functions, readback rates and communication protocols vary by instrument. A test must wait for the relevant electrical condition and use an appropriate measurement interval.
Calibration and acceptance
Use the documented warm-up, calibration and environmental requirements. Check representative points using suitable traceable equipment when the application requires it. High precision alone does not imply medical approval, bidirectional operation or a particular command set.
Practical checks
| Check | What to verify |
|---|---|
| Accuracy | Use the stated percent-of-reading/range terms and conditions. |
| Resolution | Check setpoint steps and readback granularity separately. |
| DUT measurement | Include cable drop, instrument error and connections. |
| Automation | Verify settling, logging and available commands. |
Updated