Performance Evaluation and Benchmarking of Robotic and Automation Systems: Committee Information
IEEE Robotics and Automation's Technical Committee
on:
Performance Evaluation and Benchmarking of Robotic and Automation Systems
Scope

The Technical Committee on Performance Evaluation and Benchmarking for Robotic and Automation Systems (TC-PEBRAS) will serve as a forum to address performance evaluation and benchmarking for robotic and automation systems, in general. Currently, there is no accepted standard for quantitatively measuring the performance of intelligent systems – particularly robotic or automation systems - against user-defined requirements; and furthermore, there is no consensus on what objective evaluation procedures need to be followed to deduce the performance of these systems. The lack of reproducible and repeatable test methods has precluded researchers working towards a common goal from exchanging and communicating results, inter-comparing robot performance, and leveraging previous work that could otherwise avoid duplication and expedite technology transfer. This lack of cohesion in the community hinders progress in many domains, such as manufacturing, service, healthcare, and security. Quantifying performance via benchmarking and standardization will improve the utility of robotic and automation systems in already established application areas. Having agreed-upon measures of performance and merit is a critical pre-requisite to wider acceptance and proliferation of emerging technologies. It is our hope that the efforts of this technical committee will bring together what is currently an amorphous community to provide a baseline for comparison and mechanisms for targeting specific aspects of a system thus allowing researchers and practitioners to assess the performance of various systems in different scenarios and environmental conditions. The end-user communities can use this forum to monitor progress in emerging technologies and to provide input regarding their needs and requirements. Benchmark and challenge problems in specific technology areas can be submitted to this TC to stimulate progress in concrete and directed ways.
Even though the performance evaluation and benchmarking activities might be carried out under individual TCs pertaining to their specific area, it is our belief that TC-PEBRAS will serve all the existing TCs by providing a common forum to address this issue jointly and benefit from lessons learned. This will lessen the limitations currently faced by individual TCs and enable steps required to speed up the achievement of meaningful results by cross-germination and cooperation. TC-PEBRAS will concentrate on performance aspects of intelligent systems including software and other influencing factors. We expect to learn from the experiences and insights of the members of the existing TCs gained both through their theoretical development and practical implementation of intelligent systems in a variety of diverse application domains.
We welcome you to join this effort and contribute in any way possible!

Activities
  • Autonomous Robots Journal, Special Issue (R. Madhavan, C. Scrapper, and A. Kleiner (Eds.))   Special Issue    November 1, 2009 - New York
    Contact Information: Raj Madhavan (raj dot madhavan at ieee dot org)

    The primary focus of this special issue is to quantify performance characteristics of various approaches to mobile robot localization and mapping in a variety of domains. The nine accepted articles detail the capabilities and limitations of several approaches by the inter-comparison of experimental results, development of schemes for ground truth generation as well as the underlying mechanisms used to formulate these solutions.

  • Performance Evaluation and Benchmarking for Next Intelligent Robots and Systems   Workshop    October 15, 2009 - St. Louis, MO, USA
    Contact Information: Raj Madhavan (raj dot madhavan at ieee dot org)

    2009 IROS Full-day Workshop, St. Louis, MO, USA, Organizers: Angel del Pobil, Raj Madhavan and Fabio Bonsignorio.

  • 2009 Performance Metrics for Intelligent Systems (PerMIS’09) Workshop   Workshop    September 21, 2009 - USA
    Contact Information: Raj Madhavan (raj dot madhavan at ieee dot org)

    General Chair: Elena Messina
    Program Chair: Raj Madhavan
    The Performance Metrics for Intelligent Systems workshop is the only one of its kind dedicated to defining measures and methodologies of evaluating performance of intelligent systems. Started in 2000, the PerMIS series focuses on applications of performance measures to practical problems in commercial, industrial, homeland security, and military applications. It has proved to be an excellent forum for discussions and partnerships, dissemination of ideas, and future collaborations between researchers, graduate students, and practitioners from industry, academia, and government agencies.
    The main theme of the ninth iteration of the workshop, PerMIS’09, seeks to address the question: Does performance measurement accelerate the pace of advancement for intelligent systems? In addition to the main theme, as in previous years, the workshop will focus on applications of performance measures to practical problems in commercial, industrial, homeland security, and military applications.
    Please refer to the workshop website at http://www.isd.mel.nist.gov/PerMIS_2009/ for additional information on the workshop. Templates, deadlines, and other pertinent information relating to submission are available from: http://www.isd.mel.nist.gov/PerMIS_2009/submission.htm/.

  • Performance Evaluation and Benchmarking of Intelligent Systems   Book    September 10, 2009 - NY, USA
    Contact Information: Raj Madhavan (raj dot madhavan at ieee dot org)

    Edited Book Volume, Raj Madhavan, Edward Tunstel and Elena Messina (eds.), Springer, ISBN: 978-1-4419-0491-1, September 2009.

Background
  • URL:http://tab.ieee-ras.org/committeeinfo.php?tcid=35
  • Founding Date: May 17, 2009
  • Member Count: 90
  • Spring 2010 Committee Chairs:
    • Madhavan, Raj (raj dot madhavan at ieee dot org)
      • ORNL/NIST
      • corresponding chair - send email to join committee
    • Messina, Elena (elena dot messina at nist dot gov)
      • National Institute of Standard & Technology
    • del Pobil, Angel (pobil at icc dot uji dot es)
      • University Jaume I
      • corresponding chair - send email to join committee
  • Committee Chairs Emeritus:
  • Recent Innovations Summary:click here to download