OpenGENESIS WG

The openGENESIS Working Group is now closed. We thank everyone in the community for their collaboration

The openGENESIS collaboration platform leverages knowledge among its members, enabling them to cooperate efficiently and share research results in an open access domain. This will establish a strong global exchange between industry, research and regulators to develop common criteria for the quality of AI.

Charter and Participation Agreement

The openGENESIS Charter is available for review. The openGENESIS Participation Agreement is also available.

Please contact ralph dot mueller at eclipse minus foundation dot org for more information.

Mission

openGENESIS is a collaborative platform with the mission to provide knowledge, methods and tools for the assessment of artificial intelligence (AI) that is used within autonomous driving applications. Before deployment onto public roads, learning algorithms must be proven safe and roadworthy. However, our current understanding of AI’s complex functionality is limited, especially in relation to machine learning algorithms. openGENESIS will provide both public and regulatory authorities with approaches to help them deal with the challenges of AI approval and certification. Providing a “TÜV for AI”.

Scope

openGENESIS drives and supports the investigation of an understandable, verifiable and certifiable AI. This includes the adoption of functional safety approaches to symbolic and sub-symbolic AI, as well as the development of new methods and metrics for the verification of neuronal networks. Approaches regarding the assessment of performance, robustness and comprehensibility will be compiled. In addition, an open data set is planned to be shared to enable research based on realistic cases and safety relevant challenges. Finally, social concerns should also be considered relating to the certification of learned algorithms.

Structure

openGENESIS provides a convenient and legal framework with lean and efficient structures. It is ensured by the Eclipse Foundation, which hosts the working group of openGENESIS. The collaborative platform also provides a flexible setting for openGENESIS spotlight projects (GSLP) to solve specific challenges in the community.

These projects can be initiated by one or more participant, taking a new aspect or improving approaches within the scope of openGENESIS. Partners retain their intellectual property, but are required to share their results and methodologies within the community. If the spotlight partners agree, spotlight projects can provide data to other projects or to the public. A steering committee manages the openGENESIS platform, defining and maintaining the scope and road map. It also decides on the acceptance and integration of new spotlight projects, and acts as a first contact.

Steering Committee

Steering Committee Chair

Matthis Eicher, TÜV Süd

Members of the Steering Committee

• TÜV SÜD [TÜV SÜD]

Members

• DFKI (German Research Center for Artificial Intelligence) [DFKI]
• Incenda AI GmbH [Incenda AI]
• TÜV SÜD [TÜV SÜD]

Interested parties

• Mathworks [Mathworks]
• Phantasma Labs [Phantasma Labs]
• Bosch [Bosch]
• Eteration [Eteration]
• iMotion [iMotion]
• Intel Labs [Intel Labs]

Participation

The openGENESIS working group is open at any time to all new members who meet these conditions. There are different classes of openGENESIS working group membership - Driver Member, Development Member and Guest Member. Each of these classes is described in detail below:

Driver Members

Driver Members are organizations that view the openGENESIS activities as strategic to their organization. These members will invest resources (see openGENESIS Charta for more details) to sustain and shape the activities of this working group as well as maintain the scope of openGENESIS. An openGENESIS WG Driver Member must be at least an Solutions Member of the Eclipse Foundation ([1]) and must execute the openGENESIS Participation Agreement. They can also initiate and participate in spotlight projects, and are additionally part of the steering committee.

Development Members

Development Members are organizations that are interested to benefit from the openGENESIS community. They will lead or participate in "openGENESIS Spotlight Projects" ("GSLP") and investigate the technical questions. Development membership is usually without an additional fee. Development members will invest efforts to the openGENESIS Spotlight Projects ("GSLP") as defined in the proposal. An openGENESIS WG Driver Member must be at least an Solutions Member of the Eclipse Foundation ([2]) and must execute the openGENESIS Participation Agreement. They are also able to initiate and to participate in spotlight projects under the umbrella of openGENESIS.

Guest Members

Guest members are organizations who have been invited for one year by the Steering Committee of openGENESIS WG. Typical Guest Members include academic entities or non-profit organizations, who want to have a closer look before deciding on their strategy. Guest Members are not allowed to become a Steering Committee Member and they have no right to vote concerning the matters of the working group. Guest Members must be at least an Associate Member of the Eclipse Foundation and are required to sign the openGENESIS Participation Agreement. Guest Members are allowed to participate in the openGENESIS WG within the scope of their invitation.

Start a new openGENESIS spotlight project (GSLP)

Note: the planning and publication of a GSLP proposal does not necessarily require the execution. It can also be used to share the idea and find cooperation partners.

The steering committee is pleased to discuss possible ideas for new GSLP's, to assist in the search for a cooperation partner and to provide assistance in the process of initiating the spotlight project. Depending on the project type, different approaches are possible.

Generally conceivable project types

The necessary procedure is agreed for each project individually together with the Eclipse Foundation and the openGENSIS Steering Committee.

Classical Software development projects

This kind of project is marked by a long lifecycle, maintanance activities, updates and adjustments to new environments. Eclipse Foundation is requested to provide necessary infrastructure like GIT, an building-environment, etc.. Complete Eclipse Foundation community is invited to contribute to the project. Examples for this kind of projects are a Neuronal Network Debugger, an AI Metrics calculation tool, or a simulation environment to provide training and testing data for AI.

Knowledge building projects

This kind of project is marked by a relatively short lifecycle, self-contained project and no need for maintanance or updates. The Eclipse Foundation is only requested to provide lilltle resources like an Wiki or Tuleap or similar. Examples for this kind of projects are the descriptions of a data labelling process for machine learning, or the elicitation of requirements regarding an AI evaluation process, or fundamental mathematical considerations regarding properties of AI.

Provide data-sets

This kind of project is marked by high demands on storage capacities and traffic. In addition, concerns with regard to GDPR and similar must often also be taken into account.

General process for establishing a new spotlight project

• Informal description* of the proposed GSLP is available
• Synchronization about the GSLP proposal with the steering committee
• Formal description (dependent on the project type) of the proposed GSLP
• Official steering committee approval of the GSLP
• Depending on the project type, an EMO approval is passed by the Eclipse Foundation.
• Start of the new openGENESIS spotlight project

For an informal description of the proposed GSLP, the following aspects should be considered:

• (Tentative) title of spotlight project
• Your contact information
• Participating entities and project leader
• Scope of the spotlight project proposal
• Description of the spotlight project
• Planned deliverable
• Planned licensing
• Known legal issues
• Project schedule and resources
• Future work
• Necessary infrastructure/tools
• Further interested parties

Examples for potential openGENESIS spotlight projects (GSLP)

The participants of a spotlight project define their own development or research object. In order to give a better idea of possible goals within openGENESIS, the following examples of presumably relevant considerations are listed:

• Tool to check the datasets for suitability and representation of the ODD and compliance with the labeling specification
• Establishment of an open and public available training and validation data set
• Methods for goal-oriented generation of synthetic data of behavior of vulnerable road users
• Identification of the necessary properties of synthetic data for an adequate real data representation
• Advances in understandable and explainable AI
• Find and define metrics for quantitative statements about performance and robustness
• Discuss possible adjustments of safety standards to fit for machine learning
• Platform based AI abstraction for better algorithm verification

Established openGENESIS spotlight projects (GSLP)

A reliable AI data labeling process

Scope: A process for annotating data and generating corresponding ground-truth information to train and test Artificial Intelligence (especially for Machine Learning) shall be described, investigated and potential weaknesses identified.

Description: Machine learning is based on training networks from data and also evaluating them on such data. This requires both the data itself and the associated ground truth information of this data. This ground-truth information is created in an "annotation/labeling process" partly automatically, partly manually and basically has the potential to be erroneous. If the data contain erroneous ground truth information, the function of the network can be learned only to a limited extent or even incorrectly, and secondly, an evaluation of the AI function on this data cannot be trusted. Therefore, it is absolutely necessary that the ground truth information is correct. This is already anchored in the annotation process, which is why this should be the focus of consideration of this project.

As deliverable a whitepaper is planned, which describes the process, the potential weaknesses and possible mitigations.

To reach the deliverable, the following steps were performed:

• creation of a ML meta development process diagram
• creation of an labeling process diagram
• description of labeling process
• analysis of labeling process and weakness identification
• discussion of potential mitigations
• creation of the whitepaper

Participants:

• Incenda AI GmbH [Incenda AI]
• TÜV SÜD [TÜV SÜD]

Launch: April 2019

Links & Publications:

[WhitePaper - Process considerations: A reliable AI data labeling process] (V 1.3 - August 2020 / CC-BY 4.0)

Abstract:

Due to its wide success in recent years, Artificial Intelligence (AI) is being used in more and more systems. As established Software Engineering practices, including development processes, fail to capture the complexity and additional challenges of developing AI systems, many Software Developers struggle using AI, especially in safety critical areas like healthcare or automotive.

One of the AI methods with the highest impact so far is supervised Machine Learning (ML). The performance of supervised ML is determined to a large extent by the data used to train and evaluate the developed models and the application of established Software Engineering practices. Common issues include data and label quality, immature frameworks and processes for supervised ML development, a lack of traceability of requirements to implementation and limited transparency of some models.

The contribution of this whitepaper is the discussion and establishment of a sound supervised ML lifecycle, with a focus on data quality, from the intent of developing a system using supervised ML to the decommissioning of the developed system. The different steps of the lifecycle are detailed and a deep-dive into the labeling steps is provided by defining a labeling process. The discussion includes activities that are recommended to be performed in order to create high quality labels and raises typical issues during labeling.

Regular Meetings

Schedule

• Face to face meetings - twice a year
• Online meetings - once a month

Communication

• Steering committee meetings are open for openGENESIS members
• Meeting minutes are shared over [mailing list] and this wiki

Past Events

Steering committee meeting of the openGENESIS Working Group, online

Date: Tue, July 08, 2020, 14:00 PM - 15:00 PM CEST

Steering committee meeting of the openGENESIS Working group.

More information:

• Steering Committee Meeting Minutes

Steering committee meeting of the openGENESIS Working Group, online

Date: Tue, Nov 05, 2019, 10:00 AM - 11:00 AM CET

Regular steering committee meeting of the openGENESIS Working group.

More information:

• Steering Committee Meeting Minutes

Steering committee meeting of the openGENESIS Working Group, online

Date: Wed, Sep 25, 2019, 09:00 AM - 10:00 AM CEST

Regular steering committee meeting of the openGENESIS Working group.

More information:

• Steering Committee Meeting Minutes

Steering committee meeting of the openGENESIS Working Group, online

Date: Thu, Aug 20, 2019, 09:00 AM - 10:00 AM CEST

Regular steering committee meeting of the openGENESIS Working group.

More information:

• Steering Committee Meeting Minutes

Foundation meeting of the openGENESIS Working Group, online

Date: Thu, July 18, 2019, 09:00 AM - 11:00 AM CEST

This meeting was the official foundation of the openGENSIS working group. Various bodies were set up and elected, a working mode for the steering committee was set up and requirements for the approval process of spotlight projects were discussed.

More information:

• Foundation Meeting Minutes

First meeting of the openGENESIS Working Group, TÜV Süd, Munich

Date: Tue, April 30, 2019, 9:00 AM – 4:30 PM CEST

From the invitation letter:

This event will bring all interested parties together for information about the Eclipse Working Group idea. We will also discuss technical aspects which seem interesting for closer examination and start an open exchange.

More information:

• Agenda

Presentations from the event:

• The GAP Between AI Development and Automotive Safety Criteria, C. Müller (DFKI)
• Eclipse Working Groups, R. Mueller (EF)
• Eclipse SUMO, R. Hilbrich (DLR)
• Proposal Spotlight Projects, Lucy Song (Imotion)

Interfaces to other initiatives

Lernende Systeme – Plattform für Künstliche Intelligenz

Link: [https://www.plattform-lernende-systeme.de/home.htm]

Project type: German national funded project - by BMBF

Coordinator: acatech - Deutsche Akademie der Technikwissenschaften e.V.

Scope: Scope is to leverage knowledge and to point out perspectives in the area of artificial intelligence to position Germany internationally as a technology leader for learning systems. This should be reached by combination of the expertise from science, business and society.

Interface to openGENESIS:

Safety aspects – e.g. in the following groups

• AG 3 (IT Security, Privacy, Law and Ethic)
• AG 5 (Mobility and intelligent Infrastructure)

KI-Plattform - Konzipierung einer KI-Datenplattform zum Entwickeln und Testen autonomer Fahrzeuge

Link: [https://www.elektronikforschung.de/projekte/ki-plattform]

Project type: German national funded project - by BMBF

Coordinator: Volkswagen AG

Scope:

KI-Plattform will be used to design a common database for learning and test data, which will be available to industrial companies, research institutes and testing organisations as well. The project will design uniform and flexible data structures that can be used for current and future sensors. Beside real data, synthetic generated or modified data will also be considered. In addition, legal framework conditions and data protection aspects are examined and suitable operator concepts are developed.

Interface to openGENESIS:

AI Training and Testing Database is one of the major aspects of openGENESIS. Therefore, a close cooperation and coordination should be established and maintained.

VDA Leitinitiative - KI Absicherung

Link: tbd

Project type: German national funded

Coordinator: tbd.

Scope: tbd.

Interface to openGENESIS: Yes - tbd.