The road to safe perception is no bed of roses. Having enough coverage, getting enough data with sufficient quality, training your model, making sure that it performs as it should… and if not, which is often the case, resolving the issues that appear in the way. That is why having a product that precisely covers and helps solve those problems can make a big difference in the daily job of technical teams.
The question here is how can Kognic’s software help to solve these problems? Well, it’s not a superhero complex, in our case it’s the two Es: experience and expertise. We have worked with technical teams of automotive companies for a long time now, and we have learnt many lessons that we have applied when developing our offering. How do we make it possible for teams to maximize velocity and ensure safety, while developing models for safe perception? Keep reading to find out!
A matter of time
Time matters! We all want to spend our time at work efficiently. Engineering teams have many tasks to do everyday and limited time. At Kognic, we know this very well because that’s our reality, too! That’s why we believe it is paramount to provide teams with tools to reduce the time it takes to both identify and resolve model performance issues.
How do we make this a reality? Let us do a quick review of some of our software modules.
Ground-Truth Refinement
With our Ground-Truth Refinement module, which can be found on our Ground-Truth Platform, we empower your team with a solution where to improve model performance by fixing obvious errors in ground-truth.
How do we make this possible? With an intuitive and user-centered dashboard, which is regularly enhanced with new features, Kognic’s Ground-Truth Platform allows technical teams to control what’s important to them and their function by browsing data and utilizing model in the loop techniques such as ranking annotations based on its contribution to loss function. This “model & human-in-the-loop” workflow combined with effortless access to annotations means your teams can quickly improve the quality of their dataset.
Dataset Coverage Improvements
Another way Kognic can help you maximize velocity is by selecting the data that actually matters for your safety case.
How does our software help here? By leveraging all your ground truth data with existing and additional metadata, we help you to identify and communicate about gaps in coverage that need to be filled. These gaps can be specifically visualized, analyzed and connected to your domain specific ODD, enabling data collection prioritization to target scenarios that impact automated driving performance and safety the most.
By highlighting coverage gaps and making them visible to you and your teams, we ensure that everyone understands the importance of certain data. And by continuously monitoring your data distributions, we ensure that teams get the data they actually need. Meaning that, once you have identified the gaps, you can make use of our tools that utilize state-of-the-art active learning methods tailored for integrating humans and machines in the data selection process.
We know that it is of utmost importance that you can fill your coverage gaps, and we have seen that combining humans and machines is the best approach in order to reach the highest impact on the data you obtain.
It’s about optimizing time… but also about ensuring safety
In the end, your goal is to launch a safe product. Therefore, to only focus on velocity would be useless if you don’t ensure safety along the way. You need to know that your end product is reliable and performs as expected. Which Kognic modular software products can help you here?
Dataset Quality Analytics
Let’s say that you have received new perception data, ideally procured via our scalable and cost-efficient Ground-Truth Platform. How confident are you that your data captures your desired system behavior? How do you know whether there are errors you need to solve? Do you need to spend more time (and probably more 💰 ) to acquire more data? If you do this, will the data improve your perception performance?
Unfortunately, it doesn't happen every day that datasets come without any errors... if it ever happens 🦄! Ambiguities in the data will always be a factor for most tasks which introduces non negligible variability. Consequently, you need objective dataset quality metrics to be able to argue if your dataset is good enough for your use case. That’s when our Dataset Quality Analytics product enters the stage to perform its starring role.
How does it work? More importantly, how will it save you time while ensuring safety? Our solution compares the difference between your model performance goals with the estimated reliability of your ground-truth dataset. In this way, dataset quality issues are found quickly, indicating whether better data would improve perception performance or not. In cases where the performance of your model or safety requirements exceed the ground-truth quality, we provide the opportunity to request better quality data.
Finally, we believe that in order to move fast, you need to be able to iterate with data just like code. You start with a baseline and if you find out that you need more data or that some aspects of your data needs to be redefined, you can use our tools that have been created to help you!
Perception Performance Analytics
As a final step to ensure safety, our fully integrated performance comparison solution, Perception Performance Analytics, includes reporting and traceability features. This helps you understand how perception performance is connected to previous steps in the data pipeline. In this way, you will be able to identify the origin of model performance issues and to work to improve them.
Kognic’s last piece of advice on how to optimize your road to safe perception
It goes without saying that if you want to optimize how your teams work you should start by defining clear and relevant KPIs for your use case. No matter whether you start your quest to Ithaca with our Ground-Truth product or our whole suite of modular software products, measuring your KPIs will help facilitate the understanding of the performance of your model. With support from our tooling, once you understand them, you will be able to identify performance issues and understand their origin. In short, our software will let you understand where your model is not performing well, and you will be able to use that knowledge to make informed decisions on where to focus your development efforts.
Stay with us - there is more to come!
As part of our ongoing work to develop and refine our software, we are constantly working to improve our tooling - and extend it. If you want to be informed about our updates first, we recommend that you follow us here and on LinkedIn.
Not a customer, but want to know more about how you can optimize your models, and make your technical teams’ life easier while always ensuring safety? Feel free to contact us at contact@kognic.com, schedule a demo (button at the top of the webpage) or reach out directly to any of our Account Executives on LinkedIn. 🤗🚀
