Performance Evaluation and Benchmarking of Robotic and Automation Systems: Committee Information
IEEE Robotics and Automation's Technical Committee
on:
Performance Evaluation and Benchmarking of Robotic and Automation Systems
Scope

The Technical Committee on Performance Evaluation and Benchmarking for Robotic and Automation Systems (TC-PEBRAS) will serve as a forum to address performance evaluation and benchmarking for robotic and automation systems, in general. Currently, there is no accepted standard for quantitatively measuring the performance of intelligent systems – particularly robotic or automation systems - against user-defined requirements; and furthermore, there is no consensus on what objective evaluation procedures need to be followed to deduce the performance of these systems. The lack of reproducible and repeatable test methods has precluded researchers working towards a common goal from exchanging and communicating results, inter-comparing robot performance, and leveraging previous work that could otherwise avoid duplication and expedite technology transfer. This lack of cohesion in the community hinders progress in many domains, such as manufacturing, service, healthcare, and security. Quantifying performance via benchmarking and standardization will improve the utility of robotic and automation systems in already established application areas. Having agreed-upon measures of performance and merit is a critical pre-requisite to wider acceptance and proliferation of emerging technologies. It is our hope that the efforts of this technical committee will bring together what is currently an amorphous community to provide a baseline for comparison and mechanisms for targeting specific aspects of a system thus allowing researchers and practitioners to assess the performance of various systems in different scenarios and environmental conditions. The end-user communities can use this forum to monitor progress in emerging technologies and to provide input regarding their needs and requirements. Benchmark and challenge problems in specific technology areas can be submitted to this TC to stimulate progress in concrete and directed ways.
Even though the performance evaluation and benchmarking activities might be carried out under individual TCs pertaining to their specific area, it is our belief that TC-PEBRAS will serve all the existing TCs by providing a common forum to address this issue jointly and benefit from lessons learned. This will lessen the limitations currently faced by individual TCs and enable steps required to speed up the achievement of meaningful results by cross-germination and cooperation. TC-PEBRAS will concentrate on performance aspects of intelligent systems including software and other influencing factors. We expect to learn from the experiences and insights of the members of the existing TCs gained both through their theoretical development and practical implementation of intelligent systems in a variety of diverse application domains.
We welcome you to join this effort and contribute in any way possible!

Activities
  • Benchmarking Intelligent (Multi-)Robot Systems   Workshop    August 20, 2010 - Lisbon, Portugal

    The aim of the workshop is to focus the attention of researchers in Artificial Intelligence and Robotics on the need for evaluation and benchmarking methods that can provide quantitative reliable measurements of the ability of robots to perform complex tasks, that often cannot be achieved with simple reactive behaviors. A substantial basis to tackle this issue is given by robotic competitions and RoboCup in particular. While RoboCup competitions provide a first significant step and experience in benchmarking intelligent robot behaviors, we are still far from having a well-established performance evaluation methodology that can be widely accepted and generalized to other domains and tasks, and that is not restricted to robot subsystems, but rather tackles the full integrated system. Specifically, we believe that a significant contribution to this topic should be brought by the Artificial Intelligence community, both within competitions, such as RoboCup, and in other scientific endeavors. The Workshop programme will include 2 invited talks and presentations of submitted contributions (the targeted number is 12, but this will be adjusted based on the number and quality of the submissions). A final panel discussion will summarize the contributions presented at the workshop. We plan to propose a special issue of ECCAI Communications to publish selected contributions from the workshop.

  • TAB TC Chairs Meeting   meeting    May 3, 2010 - Prince William - Anchorage Hilton Hotel, Anchorage, Alaska
    Contact Information: Raj Madhavan (raj dot madhavan at ieee dot org)

    The Technical Committee on Performance Evaluation and Benchmarking of Robotic and Automation Systems (TC-PEBRAS) is planning a meeting of TC Chairs at ICRA in Anchorage to better understand performance evaluation issues within different TCs and how to collaborate, and jointly take steps required to speed up the achievement of meaningful results by cross-pollination and cooperation.

    The main objectives of this meeting are to facilitate discussions and to understand the following from other TC Chairs:
    1) what previous work (if any) the TCs have done with respect to performance evaluation/benchmarking/standardization
    2) how to learn from particular experiences/lessons/mistakes and extrapolate these to other domains
    3) how TC-PEBRAS can help other TCs
    4) what joint future activities can be undertaken

  • Autonomous Robots Journal, Special Issue (R. Madhavan, C. Scrapper, and A. Kleiner (Eds.))   Special Issue    November 1, 2009 - New York
    Contact Information: Raj Madhavan (raj dot madhavan at ieee dot org)

    The primary focus of this special issue is to quantify performance characteristics of various approaches to mobile robot localization and mapping in a variety of domains. The nine accepted articles detail the capabilities and limitations of several approaches by the inter-comparison of experimental results, development of schemes for ground truth generation as well as the underlying mechanisms used to formulate these solutions.

  • Performance Evaluation and Benchmarking for Next Intelligent Robots and Systems   Workshop    October 15, 2009 - St. Louis, MO, USA
    Contact Information: Raj Madhavan (raj dot madhavan at ieee dot org)

    2009 IROS Full-day Workshop, St. Louis, MO, USA, Organizers: Angel del Pobil, Raj Madhavan and Fabio Bonsignorio.

  • 2009 Performance Metrics for Intelligent Systems (PerMIS’09) Workshop   Workshop    September 21, 2009 - USA
    Contact Information: Raj Madhavan (raj dot madhavan at ieee dot org)

    General Chair: Elena Messina
    Program Chair: Raj Madhavan
    The Performance Metrics for Intelligent Systems workshop is the only one of its kind dedicated to defining measures and methodologies of evaluating performance of intelligent systems. Started in 2000, the PerMIS series focuses on applications of performance measures to practical problems in commercial, industrial, homeland security, and military applications. It has proved to be an excellent forum for discussions and partnerships, dissemination of ideas, and future collaborations between researchers, graduate students, and practitioners from industry, academia, and government agencies.
    The main theme of the ninth iteration of the workshop, PerMIS’09, seeks to address the question: Does performance measurement accelerate the pace of advancement for intelligent systems? In addition to the main theme, as in previous years, the workshop will focus on applications of performance measures to practical problems in commercial, industrial, homeland security, and military applications.
    Please refer to the workshop website at http://www.isd.mel.nist.gov/PerMIS_2009/ for additional information on the workshop. Templates, deadlines, and other pertinent information relating to submission are available from: http://www.isd.mel.nist.gov/PerMIS_2009/submission.htm/.

  • Performance Evaluation and Benchmarking of Intelligent Systems   Book    September 10, 2009 - NY, USA
    Contact Information: Raj Madhavan (raj dot madhavan at ieee dot org)

    Edited Book Volume, Raj Madhavan, Edward Tunstel and Elena Messina (eds.), Springer, ISBN: 978-1-4419-0491-1, September 2009.

Background
  • URL:http://tab.ieee-ras.org/committeeinfo.php?tcid=35
  • Founding Date: May 17, 2009
  • Member Count: 90
  • Fall 2010 Committee Chairs:
    • Madhavan, Raj (raj dot madhavan at ieee dot org)
      • ORNL/NIST
      • corresponding chair - send email to join committee
    • Messina, Elena (elena dot messina at nist dot gov)
      • National Institute of Standard & Technology
    • del Pobil, Angel (pobil at icc dot uji dot es)
      • University Jaume I
      • corresponding chair - send email to join committee
  • Committee Chairs Emeritus:
  • Recent Innovations Summary:click here to download