Reply To: Resolution error

  • Encyclios

    May 3, 2023 at 3:14 PM

    Reading analog instruments

    The resolution error in reading an analog instrument depends on:

    • the value of the format unit;
    • the length of the graduation;
    • the size of the index;
    • the reading ability of the operator.

    Example: For a pressure gauge with a unit size of 1 bar, a distance between two graduations of 5 mm and an index thickness of 1 mm, the theoretical limit of resolution is 0.2 bar.

    In practice, despite the use of optical aids (lenses, microscopes), the reading resolution of an analog instrument is severely limited by the operator’s ability to discern fractions of graduation, as well as by the parallax error that can be generated (index and graduated scale are not on the same plane). Although trained personnel can discern 1-2 microns when reading a micrometer equipped with a centesimal nonius (10-micron units), a general operator is unlikely to be able to reliably distinguish 1-mm shifts when the index is far from the scale markings.

    To adjust for the subjectivity of these considerations, some activity standards or for instruments (e.g., manometers or dynamometers) specifically define how to calculate the reading resolution. In all other cases, common sense and the principle of prudence apply: when in doubt, an operator should at least be able to discern to which notch the index is closest, in which case the reading resolution becomes equal to the unit of format.

    Exceptions are the cases:

    • when the index is larger than the scale division, in which case the reading resolution is a multiple of the format unit;
    • when measurements are made in conditions where there is a coincidence between index and notch, in which case the reading resolution will be equal to the theoretical limit.