Significance and Use
5.1 Various levels of visual acuity are essential when remotely operating robots in unstructured and often hazardous environments. Missions typically include establishing situational awareness, finding available paths, maneuvering through obstacles, identifying objects of interest, and performing detailed inspections. This test method measures robot system far-field and near-field visual acuity which are applicable to virtually every mission. These quantitative measures of performance provide a common language that allows robot users to better understand and express their own requirements and improve the way visual sensing capabilities are specified.
5.2 Multiple cameras could be incorporated into remotely operated robotic systems since a single camera is unlikely to be effective for all aspects of a mission. For example, cameras with zoom lenses are often used for far-field tasks. Cameras with close focus capabilities are often used for near-field tasks. Wide-angle lenses are often used for driving and obstacle avoidance. This test method characterizes each onboard camera to understand overall system capabilities.
5.3 This test method provides a way to unambiguously specify robot requirements in terms of the related measures of visual acuity and field of view. This helps quantify the trade-offs and general usefulness of optical versus digital zoom cameras and fixed versus variable focus lenses. The visual acuity charts can also help provide quantitative measures of performance within other test methods and training scenarios. See Figs. 2-4 for illustrations.
FIG. 2 This Baseline Image is Used for Purposes of Comparisons Below
FIG. 3 Three Images of the Same Scene with the Same Image Resolution. Top Row Shows Field of View Increasing from Left to Right (the image “zooms out”) While Bottom Row Shows Acuity Decreasing (features of the same size become harder to clearly observe)
FIG. 4 Three Images of the Same Scene with the Same Field of View. The Top Row Shows the Field of View is Unchanged While Bottom Row Shows Both Resolution and Acuity Increasing (features become clearer)
5.4 This test method helps evaluate the effect of illumination on visual acuity. In dark environments, robots typically need to illuminate the scene to be effective. Far-field objects downrange require much greater light intensity than near-field objects close to the robot. Variable illumination helps ensure the scene is neither too dark nor overwhelmingly lighted so as to thwart the camera’s ability to discern visual details (so-called “washout” of the image). Variable illumination is especially important when quickly transitioning from far-field to near-field and back again.
5.5 Key features of response robots are that they are remotely operated from safe standoff distances, deployable at operational tempos, capable of operating in complex environments, sufficiently hardened against harsh environments, reliable and field serviceable, durable or cost-effectively disposable, and equipped with operational safeguards. As such, a major advantage of using robots in response operations is to enhance the safety and effectiveness of responders or soldiers.
5.6 This test method aligns user expectations with actual capabilities to understand the inherent trade-offs in deployable systems at any given cost. For example, an increase in image resolution typically results in improved field of view or acuity, but not necessarily both. An increase in both may not be possible for robots of a desired weight, endurance, or cost. Appropriate levels of understanding can help ensure that requirement specifications are articulated within the limit of current capabilities.
5.7 This test method provides a tangible representation of essential robot capabilities with quantifiable measures of performance. When considered with other related test methods in the suite, it facilitates communication among communities of robot users and manufacturers. As such, this test method can be used to:
5.7.1 Inspire technical innovation and guide manufacturers toward implementing combinations of capabilities necessary to perform essential mission tasks.
5.7.2 Measure and compare essential robot capabilities. This test method can establish the reliability of the system to perform specified tasks, highlight break-through capabilities, and encourage hardening of developmental systems.
5.7.3 Inform purchasing decisions, conduct acceptance testing, and align deployment objectives with statistically significant robot capabilities data captured through repeated testing and comparison of quantitative results.
5.7.4 Focus operator training and measure proficiency as a repeatable practice task that exercises actuators, sensors, and operator interfaces. The test method can be embedded into training scenarios to capture and compare quantitative scores even within uncontrolled environmental variables. This can help develop, maintain, measure, and track very perishable skills over time and enable comparisons across squads, regions, or national averages.
5.8 Although this test method was developed as part of a suite of sensing tests for response robots, it may be applicable to other domains. Different user communities can set their own thresholds of acceptable performance within the test method for various mission requirements.
5.9 It is recommended that users of this test method consider their particular robot requirements when interpreting the test results. The capability evaluated in this test method alone shall be interpreted according to the scope of this test method and shall not be considered as an overall indication of the capability of the robot’s mobility subsystem nor of the entire robotic system. A single test method only captures the specified single aspect of a robot’s capabilities. A more complete characterization of a robot’s capabilities requires test results from a wider set of test methods.
Scope
1.1 The purpose of this test method is to specify the apparatuses, procedures, and performance metrics necessary to quantitatively measure a robot’s visual acuity as displayed to a remote operator or vision algorithm. The primary performance metric for this test method shall be a robot’s possession of such a capability with a specified statistical significance level.
1.2 Secondary performance metrics are the robot’s field of view and aspect ratio.
1.3 This test method can also be used to measure the operator proficiency in performing the specified task. The corresponding performance metric may be the number of completed task repetitions per minute over an assigned time period ranging from 10 to 30 minutes.
1.4 This test method is a part of the sensing suite of response robot test methods, but this test method is stand-alone and complete. This test method applies to systems operated remotely from a standoff distance appropriate for the intended mission. The system includes a remote operator in control of all functionality and any assistive features or autonomous behaviors that improve the effectiveness or efficiency of the overall system.
1.5 The apparatus, specified in Section 6, can only test a limited range of a robot’s capabilities. When the robot has been tested through the limit or limits of the apparatus, a note shall be associated with the results indicating that the robot’s actual capability may be outside of the limit or limits imposed by the test apparatus. For example, the robot could exceed the capabilities of the printing process used to create the charts used in the apparatus.
1.6 Performing Location—This test method may be performed anywhere the specified apparatuses and environmental conditions can be implemented.
1.7 Units—The values stated in either SI units or inch-pound units are to be regarded separately as standard. The values stated in each system may not be exact equivalents; therefore, each system shall be used independently of the other. Combining values from the two systems may result in nonconformance with the standard. Both units are referenced to facilitate acquisition of materials internationally and minimize fabrication costs.
1.8 This standard does not purport to address all of the safety concerns, if any, associated with its use. It is the responsibility of the user of this standard to establish appropriate safety, health, and environmental practices and determine the applicability of regulatory limitations prior to use.
1.9 This international standard was developed in accordance with internationally recognized principles on standardization established in the Decision on Principles for the Development of International Standards, Guides and Recommendations issued by the World Trade Organization Technical Barriers to Trade (TBT) Committee.
