Rating Policies
Rating components of high complexity is no easy task, and a limited list of criteria may not reflect all fancy details of a product. We are aware of this general limitation and you should be aware, too, when trying to interpret the ratings (hopefully, they won’t need much of that).
We also know that products will change over time. The typical release cycle for simulation software is two releases per year (one major, one minor). Therefore, we do our best to follow-up on new releases, provided that the information necessary for the rating is made available in due time.
If you are a product owner and you have a new version either coming up shortly or already released, please drop us a line at contact@simcert.org and let us know. The more information you provide about your product, the more precise our assessment will have a chance to become.
Rating Methods
We provide two methods of rating:
- General
- we use publicly available information (websites, brochures etc.)
- we perform a general rating based on a series of “obvious” criteria for a given component
- we enable users to provide their own ratings based on the same set of criteria
- we enable users to leave their comments on our ratings and the ones of other users
- Advanced
- we use information provided by the owner of a product or a qualified, named user
- we perform an in-depth rating within and beyond the initial set of rating criteria
- depending on individual arrangement, the report will either be published on our website or it will made available to an agreed set of parties only
Our free service will focus on General product ratings with some Advanced flavor where applicable (eg, in case of free or open source products).
Protecting Intellectual Property:
We do not intend to interfere with any party’s IP. Therefore, we reserve the right to perform Advanced ratings only per request by a legitimate owner of a product or by a licensee. Publication of an advanced rating is an option but not a must. We also reserve the right to request a reimbursement for the extra effort implied in providing an advanced rating report.
Rating Criteria per Product Category
Each product category comes with its own rating criteria. Some of them are quite common (e.g. documentation, usability) and may be applied across a broad range of products, others are more specific. Please find hereafter the criteria for each category, including some information about requirements for achieving a high rating.
Note that this list of criteria represents just the top levels of a rating system that looks at more than 1000 aspects in detail for the most complex products.
For easier navigation, use the following links to jump directly to one of the categories:
- Common Criteria (used in more than one category)
- Integrated Solutions
- Frameworks
- Scenario Simulation
- Traffic Simulation
- Sensor Simulation
- Environment Databases
Common Criteria
Some criteria may be applied across various product categories. These are listed here. Be aware that the meaning of a criterion might need to be adapted slightly when applied to a specific category; if applicable, we will indicate any deviation from the common definition in the criteria list of the respective product category
- Usability
- writing a script in a text file is a powerful option for configuring a complex system; but hardly any user will want to start with this step
- good usability means that the user is guided by an intuitive user interface throughout all stages of setting up and executing a simulation; the workflow has to be clear
- a fragmentation of user interfaces or an explicit switch from one application to another for setting up a simulation is not considered a good option
- the meaning and naming of components, parameters etc. has to be clear
- data storage and retrieval has to be easy
- installation and setup shall work out-of-the-box without any need for a support call
- Standards Compliance
- the request for standards refers to data communication, interface and storage formats; others may be added in the future
- a standard will typically be specific to the industry where it is applied; it may either be an offical standard issued by a recognized institution or a de-facto standard supported by a substantial fraction of players in the market
- standards have been created for being used – at least this is what one hopes for; “standards compliance” does not so much rate how many standards are supported as how well relevant standards are supported and to which degree they are part of the product
- the following standards are considered mandatory or “nice-to-have” (ie, relevant) for the different product categories:
- Documentation
- a well documented product is of great value; the user will know what to expect and how to operate the system
- a user manual is the bare minimum expected in terms of documentation
- a tutorial, preferably interactive or by videos, is a big plus to a getting familiar with a complex product
- meaningful documentation that is available up-front, i.e. while a potential user is still in the selection process, will increase a product’s documentation rating
- context-sensitive help within a product (tool-tips, links to documentation) are considered beneficial
- release notes provide a quick overview of improvements and extensions a product may have undergone with its latest version
- Support
- questions are meaningless without answers; and a (novice) user will typically have a lot of questions when it comes to using a product
- the key factors for achieving a high rating for the support of a product are
- availability
- responsiveness
- correctness of answers / information
- follow-up on “hard” issues
- it is quite common in industry that support questions have to be addressed within one business day
- products are expected to come with a good basic level of support; an option for professional services will increase the rating since it may allow the user to better tailor the extent of support to his needs
- Licensing
- not everything that comes for free is good; but it is good if something comes for free
- a product that comes as open source or with little restrictions on use by a broad community has a strong point to make
- hybrid products that come with some part of little restriction on use and “high-fidelity” extensions are a good compromise
- fully commercial products may achieve good ratings by providing flexible licensing terms
- Completeness
- more features do not necessarily mean that something is getting closer to completeness
- we will rate this criterion according to the applications covered vs. the requirements of the market and how a product’s feature set may help reducing the overall need for the user to perform an integration of their own or 3rd party tools.
- Openness
- hardly any product will be used standalone, only communicating with itself
- a high rating for the openness criterion can be achieved by making sure that the user has transparent access to all data flows and data formats (including documentation) that are used within the product
- Extensibility
- no matter how highly integrated a product may be, there will always be something else that the user needs in addition or in exchange for a given component
- it is crucial for a high rating of this criterion that a product provide a means to extend functionality either by recommended extensions of the solution provider or by well documented interfaces
- bi-directional data communication interfaces are considered a plus in the rating of this criterion
- a well-documented API for implementing custom functionality within the solution will lead to a higher rating
- plug-in concepts are highly appreciated
- Scalability
- one running simulation is good – many are better; if you want your system-under-test (SuT) to be “challenged” by as many test cases in as short a time as possible then you have two options: first, increase the number of tests that run in parallel, second, run each test faster (and “third”, do both at the same time)
- the higher the execution speed of a single simulation run that a solution can achieve without any decrease of accuracy, the higher its rating will be
- in terms of increasing the number simulation runs, we will not only rate how many instances can be executed in parallel but, most of all, how good the management of the respective deployment and of the test cases is; this includes the selection or generation of test cases, the execution itself and the storage and retrieval of simulation results
- Versatility
- a product needs to adapt to the use case, not vice versa
- development and testing of ADAS and AD systems by means of environment simulation solutions is performed in various operating environments:
- Model / Software-in-the-Loop (MiL / SiL)
- Driver-in-the-Loop (DiL)
- Vehicle-in-the-Loop (ViL)
- Hardware-in-the-Loop (HiL)
- etc.
- a high rating in versatility shall be achieved if the user is able to deploy the product to different operating environments (X-in-the-loop) with little configuration effort and with seamless exchange of data between these environments
- Cloud Deployment
- when it comes to managing hardware costs, cloud deployment is an attractive option (you pay what you use)
- the rating of this criterion shall take into account whether the product can be operated on a cloud provider’s hardware and what the effort is to get it onto the cloud
- we shall give higher ratings to products that come as ready-to-go cloud deployments, either as simulation-as-a-service implementation or just by an easy way of getting started on virtual hardware
- partnerships of a product with established cloud providers are considered a benefit
- Determinism
- the most annoying thing in assessing the system-under-test are “effects” that come-and-go with each new round of testing
- determinism is a criterion which is important to achieve for a product since it means that with identical input (also in terms of timing) the results – even randomized parts – will be exactly the same.
- the highest rating is reserved for products that provide determinism for the overall operation with all components being active
- Real-time Capabilities
- a product will typically claim that it can cover a broad range of use cases (see Versatility); one of the key uses of environment simulation software is the so-called hardware-in-the-loop (HiL) testing. But hardware hardly knows anything but real-time operation. Therefore, real-time capability is a strong point in the overall rating of a product (BTW: also SiL testing may require real-time operation)
- real-time may be achieved in a couple of ways – by adapting speed or accuracy. Depending on whether a product is on the faster or slower side under normal operation, the strategy for real-time may be different and will be rated accordingly
- how real-time is achieved, and whether accuracy has to be compromised in order to achieve this goal will have great influence on the rating
- how “hard” the real-time is that a solution can achieve is also part of the rating; are there any frame drops over a given measurement period? Is there a mitigation concept for frame losses?
- flexibility in triggering and synchronization when it comes to real-time operation will play a certain role in the overall rating
So much for the common rating criteria. Hereafter you will find the specific criteria for certain product categories.
Integrated Solutions
Integrated solutions are among the most complex products to rate. They comprise various elements that may also be rated within other categories – provided they can be identified as individual components and may be operated as such. For integrated solutions, the focus of our rating will remain on a quite high level.
- Usability
- see Common Criteria
- Completeness
- for the baseline, see Common Criteria
- at a minimum, we expect an evironment model, a traffic model, and a scenario simulation capability
- editors for an in-depth adaptation of the previously mentioned components will lead to a higher rating
- ready-to-go content that comes with the product itself and which is of value to the average user will improve the overall rating
- Standard Compliance
- see Common Criteria
- Openness
- see Common Criteria
- Extensibility
- see Common Criteria
- Scalability
- see Common Criteria
- Versatility
- see Common Criteria
- Cloud Deployment
- see Common Criteria
- Determinism
- see Common Criteria
- Real-time Capabilities
- see Common Criteria
- Scenario Modeling
- traffic, 3d environment and sensors provide the main components of an environment simulation solution; now it comes to parameterizing these components so that a large set of scenarios can be experienced and covered
- scenario modeling is strongly inter-connected with the aforementioned components; to some extent, it can be considered as a layer on top of them; the scenario modeling may control the selection of parameters for executing individual simulation runs;
- but the scenario modeling may also describe traffic maneuvers, environmental conditions (e.g. weather) in great detail; it is, then, the task of other components of the integrated solution to execute the respective commands
- the criterion scenario modeling will achieve high ratings if scenarios can be configured and parameterized to a large extent
- ease of re-using existing scenarios and/or libraries will increase the rating
- Traffic Model
- one is hardly ever alone on the road; there are many other participants, and we consider the modeling of their behavior and their reaction to actions of the system-under-test the key factor of the traffic simulation aspect of an integrated solution
- high ratings shall be achieved by providing a large variety of other actors that the SuT can interfere with (vehicle classes, pedestrians, mobility devices, infrastructure elements etc.)
- it is crucial that the user doesn’t have to script every single detail of the surrounding traffic’s behavior; therefore, solutions that provide a high level of “autonomous” behavior of other participants will see a good rating
- at the same time, a detailed scripting of individual participants or groups of participants is a key aspect of traffic simulation; a good rating also depends on fulfilling this aspect
- traffic models which are scalable – from individual participants to entire traffic flows within larger environments (e.g. city models) will be beneficial to users who start small but want to expose their systems to more complex environments over time
- Sensor Models
- without our senses, we would not know that there is a world around us – the same applies to the system-under-test in connection with environment simulation
- sensor simulation is a broad field with many aspects, and it is also treated in our ecosystem as a dedicated application; but an integrated solution wouldn’t be integrated if it did not provide sensor simulation capability
- the types of available sensors will strongly influence the rating
- the number of sensors that can be instantiated in parallel will mark another key factor in the rating
- flexibility of placing sensors on various actors, on infrastructure etc. will increase the rating
- the interaction of each sensor model with the environment in terms of geometry and materials is important; the better a sensor can make use of available data, the higher the rating of the sensor models will be
- ground truth lets you know what your sensor should have seen; it is crucial that the integrated solution provides a good sensor model along an idea of the “perfect” result of sensing
- realism of the sensor models, i.e. a behavior close to actual sensors will be a crucial factor in rating this criterion
- Vehicle Dynamics
- typically, the system-under-test (SuT) will be mounted on and/or control a dedicated vehicle; therefore, the level of fidelity of the vehicle dynamics is of interest in order to provide sufficient motion cues to the SuT; but also the dynamics of other vehicles in the environment simulation is of interest, especially in terms of them being sensed
- an integrated solution shall come with at least a five-mass (body + 4 wheels) dynamics simulation for a typical passenger car, bus, van or rigid-body truck.
- more parameters of the vehicle dynamics will provide a better configurability and shall lead to a higher rating (but beware that Usability has to align with this requirement)
- ready-to-go parameter sets for the configuration of vehicles shall be beneficial to the product rating, especially in cases where the vehicle dynamcs component provides a rich set of parameters
- 3d Environment Modeling
- the 3d modeling aspects of an integrated solution are primarily focused on the creation of 3d environments, not so much on the modeling of individual objects like vehicles, pedestrians, infrastructure, vegetation etc.
- a high rating under the criterion 3d modeling can be achieved by providing means to create 3d environments from scratch in an intuitive and interactive way
- further beneficial features will be import/export capabilities for existing 3d environments and/or libraries of objects that can be placed in the 3d scene
- a feature of procedural database generation shall clearly be an indicator of a highly rated product
- extensive libraries of objects for the population of 3d environments shall be a clear plus
- the creation of consistent 3d content and accompanying logical descriptions (road, objects etc.) is a must
- Ready-to-go Content
- environment simulation needs environments – and sensors – and vehicles etc.
- a good integrated solution shall provide a relevant set of content so that a user can actually perform relevant simulation runs out-of-the-box
- the content that shall be considered beneficial to the user (and the product rating) includes but is not limited to
- 3d environments (graphics, materials, logics)
- participants (vehicles, pedestrians etc.)
- template / sample scenarios
- vehicle dynamics data sets
- models of commercial sensors
- Documentation
- see Common Criteria
- Support
- see Common Criteria
- Licensing
- see Common Criteria
Frameworks
Frameworks are a bit like integrated solutions with an emphasis on openness and extensibility, but without a need for proprietary functional modules (although they may be provided as an option). They serve as the backbone of an actual simulator installation.
- Usability
- see Common Criteria
- Completeness
- see Common Criteria
- Standard Compliance
- see Common Criteria
- established standards that shall be supported by frameworks are
- Openness
- see Common Criteria
- this is THE prerequisite for a Framework
- Extensibility
- for the baseline, see Common Criteria
- it is, of course, extremely important for a Framework product to provide extensibility; the standard framework shall consist of a communication backbone with APIs and plug-in interfaces
- Scalability
- see Common Criteria
- Versatility
- see Common Criteria
- Cloud Deployment
- see Common Criteria
- Determinism
- for the baseline, see Common Criteria
- determinism in a framework depends on the way that triggering and frame-synchronization of various connected components are handled
- Real-time Capabilities
- see Common Criteria
- Documentation
- see Common Criteria
- Support
- see Common Criteria
- Licensing
- see Common Criteria
Scenario Simulation
Scenario Simulation enables the user to expose the system-under-test to certain situations. It will typcially use a traffic or maneuver simulation component as well as some means to influence the environment (weather etc.). Scalable scenario simulation is the key to covering many test cases with reasonable effort.
- Usability
- for the baseline,see Common Criteria
- setting up scenarios may be a tedious job for a test engineer; therefore, the usability rating of a scenario simulation component depends to a large degree on the productivity that can be achieved, based on the intuitiveness and “smartness” of the user interface
- scenarios also tend to come with parameters and the variation of those; for a high rating in terms of usability, the definition of parameter ranges, distributions and dependencies needs to be done in an intuitive and efficient way
- Standards Compliance
- for the baseline,see Common Criteria
- initial standards for describing and storing scenarios are already available (e.g. ASAM OpenSCENARIO) and additional ones are currently under development or have been proposed as underlying concepts (e.g. M-SDL);
- a high rating for Standards Compliance shall be given to products that support the latest released standards and which are open to further developments
- a manufacturer with active participation in one of the scenario standardization activities will typically increase confidence in its tool being state-of-the-art in terms of supporting standards
- Extensibility
- for the baseline, see Common Criteria
- scenarios may contain, among other, complex actions that are best described in scripts
- products that support scripting, give access to plug-ins, come with a well-documented API etc. shall receive a higher rating
- Scalability
- for the baseline,see Common Criteria
- scenarios may need to be run in thousands of variants in order to cover the so-called event space; any technology that will lead to an intelligent way of scanning many variants of relevant events shall lead to a higher rating
- Cloud Deployment
- see Common Criteria
- Determinism
- for the baseline, see Common Criteria
- determinism is crucial for Scenario Simulation, especially if you want to re-run the one within the thousands of variants that resulted in a failure of the system-under-test
- Ready-to-go Content
- Scenario Simulation can hardly work without underlying scenario templates
- a good product shall provide a relevant set of content so that a user can perform relevant simulation runs out-of-the-box
- the content that shall be considered beneficial to the user (and the product rating) includes but is not limited to
- road networks
- maneuver catalogs
- parameter sets
- Documentation
- see Common Criteria
- Support
- see Common Criteria
- Licensing
- see Common Criteria
Traffic Simulation
Traffic Simulation and scenario simulation are pretty interlinked. Traffic simulation is part of the execution engine of scenario simulation but may also be a unique feature and the sole purpose of a product. Therefore, it comes with its own criteria, although many are shared with other product categories.
- Usability
- for the baseline,see Common Criteria
- setting up traffic maneuvers and actions may be a tedious job for a test engineer; the usability rating of a traffic simulation component shall depend to a large degree on the productivity that can be achieved, based on the intuitiveness and “smartness” of the user interface
- assigning actions to individual participants of any kind (vehicles, pedestrians etc.) and groups of the same with an intuitive interface is key to a high rating
- Completeness
- see Common Criteria
- Standard Compliance
- for the baseline,see Common Criteria
- road networks are usually laid out in one of the industry standards (ASAM OpenDRIVE and OpenCRG being among the most common); a high rating for standard compliance will be given to products that support the latest released standards and which are open to further developments
- the active contribution of a manufacturer in the applicable standardization committees shall be beneficial to the rating
- Extensibility
- for the baseline, see Common Criteria
- products that support scripting, give access to plug-ins, come with a well-documented API etc. shall receive a higher rating
- Scalability
- for the baseline, see Common Criteria
- traffic models may need to be run in thousands of variants in order to cover the so-called event space; the control will typically be exerted by a scenario simulation component; how well the traffic simulation scales with scenario simulation shall be rated with this criterion
- Cloud Deployment
- see Common Criteria
- Determinism
- for the baseline, see Common Criteria
- determinism is crucial, so that a re-run of a simulation with a different system-under-test that does not influence the behavior of the traffic (provided that the inputs to the traffic module remain identical)
- Traffic Model
- on the road (and aside) there are many other participants, and we consider the modeling of their behavior and their reaction to actions of the system-under-test the key factor
- high ratings shall be achieved by providing a large variety of other actors that the system-under-test can interact with (vehicle classes, pedestrians, mobility devices, infrastructure elements etc.)
- it is crucial that the user doesn’t have to script every single detail of the surrounding traffic’s behavior; therefore, products that provide a high level of “autonomous” behavior of other participants shall see a good rating
- at the same time, a detailed scripting of individual participants or groups of participants is a key aspect of traffic simulation; a good rating also depends on fulfilling this aspect
- traffic models which are scalable – from individual participants to entire traffic flows within larger environments (e.g. city models) will provide a good scalability of the simulation task for the user and shall, thus, achieve a higher rating
- traffic models that have been validated against real-world data are a clear plus to the user; this validation may come in terms of composition of traffic as well as the strength of traffic flows, routing etc.
- Ready-to-go Content
- traffic simulation can hardly work without underlying traffic and road network templates
- a good product shall provide a relevant set of content so that a user can perform relevant simulation runs out-of-the-box
- the content that will be considered beneficial to the user (and the product rating) includes but is not limited to
- road networks
- template traffic scenarios
- Real-time Capabilities
- see Common Criteria
- Documentation
- see Common Criteria
- Support
- see Common Criteria
- Licensing
- see Common Criteria
Sensor Simulation
Sensor simulatio is a wide field in itself and it is hard to come up with a reasonable set of criteria that fit under this single category and still address the majority of features.
We opt for having a few common criteria for all sensor technologies and then rate the implementation per technology according to similar criteria for each of them.
Here are the common criteria across various sensor technologies:
- Usability
- for the baseline,see Common Criteria
- setting up a sensor means setting up its location and parameters (i.e. extrinsic and intrinsic parameters). The easier and more intuitive this can be done the higher the rating shall be.
- Extensibility
- for the baseline, see Common Criteria
- users will often want to implement their own sensors in a sensor simulation framework; or they may opt for running a basic sensor that comes with the product and feed its output into their more detailed model; whatever the strategy, it is crucial that the user can extend the functionality that comes with the product
- the user shall also have a means to use the Sensor Simulation for querying the underlying 3d environment and traffic module for specific properties (objects, materials, weather conditions etc.)
- Determinism
- for the baseline, see Common Criteria
- determinism is crucial, so that a re-run of a simulation with a different system-under-test yields identical output of the sensors
- Ready-to-use Models
- setting up sensor models by yourself might be a nice task but what might be even nicer is a selection of models of commercially available sensors
- Real-time Capabilities
- see Common Criteria
- Documentation
- see Common Criteria
- Support
- see Common Criteria
- Licensing
- see Common Criteria
The following sensor technologies are rated individually, but based on common criteria:
- Object Sensor
- Camera Sensor
- LiDAR Sensor
- RADAR Sensor
- Ultrasonic Sensor
- GPS Sensor
- IMU
In order to achieve a high rating within any of these technologies, the respective sensor model shall provide
- realistic emitter model (for active sensors only)
- realistic receiver model
- modeling of signal propagation (active sensors) including reflection
- configurability of extrinsic and intrinsic parameters
- lens model (for optical sensors)
- material model
Environment Databases
The citeria for rating of an Environment Database are the following:
- Relevance
- a highly relevant database is one that covers a broad range of use cases
- a database of a race circuit, for example, shall be rated lower than a database of a downtown area where many users will share an interest
- the rating of a proving ground database is influenced by the relevance of its real-world installation
- Data Quality
- lots of data may still mean little – or even misleading – information
- for a high rating of data quality, an environment database must, first of all, provide information about its quality (accuracy indication, date of recording etc.)
- we will perform random verification of the indicated quality indices and this will influence our rating of its property data quality
- note: environment databases that provide no or little information about their data quality will be rated in the lower ranges
- Logics Model
- road, infrastructure and the surrounding’s logics in an environment database have to cover a broad range of elements; we will rate primarily the extent to which they are available in a database in a way that they can be retrieved for an interpretation by simulation models
- for roads, an extensive set of logics features is required: lanes (including type indication and other attributes), lane merges, lane splits, complex road marks, junctions, crossings, bridges, tunnels, underpasses, traffic signs (aside and on-road), traffic lights (different types), bus stops, tram lines, railroad crossings, country-specific features; several objects which are within the actual driving area are considered part of the road itself and are rated with it: potholes, covers, speed bumps etc.
- for the infrastructure, all elements which are not an inherent part of the road logics model are considered in the rating: barriers, parking garages, traffic control systems, lighting, gas stations, charging points etc.
- for the surrounding, the rating depends on the variety of elements which are covered: trees, buildings, landmarks, various objects (from trash can to advertising screens)
- Road Surface Model
- the road surface may have a strong influence on the performance of an entity moving on it (ranging from vehicles to pedestrians)
- a good road surface model shall provide at least information about a road’s roughness and friction
- the spatial resolution of a road surface model and the model’s correspondence with reality are important
- road surface models which provide repeated generic data of limited extent are considered inferior
- Material Model
- the material model will influence the interaction of participants with the environment, either in terms of the road contact or, to an even wider extent, in terms of sensing capabilities
- the material model of an environment database shall at minimum provide material assignments per surface category (e.g. lane, roadmark, sidewalk, building, vegetation)
- a detailed material model shall provide “per pixel” accuracy, i.e. multiple material assigments shall be available within a surface category (e.g. “brick vs mortar” in a wall or “metal vs concrete” in a manhole cover)
- an extensive material model shall provide material properties across the electromagnetic spectrum, so that sensors ranging from infrared to radar can perform proper service
- material models adhering to state-of-the-art best-practices or standards shall be rated higher than custom models
- 3d Model
- the 3d model represents the underlying shape of all surfaces in the environment model; it provides the basic triangulation on top of which further properties are assigned to individual surfaces (see Material Model)
- a good 3d model shall be detailed to a degree that allows for sufficiently precise modeling of environment and sensor effects; the provision of realistic surface normals is a key aspect, as well as the modeling of significant edges
- the content of a realistic 3d model shall comply with its real-world location as closely as possible. Procedurally generated 3d models shall contain sufficient detail according to the environment type they claim to resemble.
- Data Formats
- it is hard to list all data formats that are potentially available when it comes to storing an environment database
- a good database shall be available in the most common state-of-the-art database formats; of course, these may change over time with new simulation software emerging – our rating will be based on the favoured formats at the time of the rating
- any standard formats which are supported by an environment database shall lead to a higher rating
- Documentation
- for the rating baseline see Common Criteria
- a highly rated documentation of an environment database shall include in addition:
- information about a database’s location, its creation date, the original purpose of creating the database
- information about the data that can be retrieved from interacting with the database (e.g. by knowing what sensor data can be collected
- a the list of supported platforms and applications
- information about database hierarchy, materials, object retrieving methods etc.
- Support
- see Common Criteria
- Licensing
- see Common Criteria