Making cars pass their vision test is a vital task. That is why at Kognic we’re driven by performance, and safety, while always aiming to be cost-efficient. In everything we do.
Indeed, that is something evident in our Ground-Truth Platform. Knowing that perception performance is crucial to the distribution of safety analytics, it couldn’t be any other way! But there’s another important factor that makes our work stand out: our experts, or as we also call it, human judgement. As a player in the automotive industry, you shouldn’t just rely on simulation. Therefore, we trust our dedicated teams to ensure that there are no errors, as part of our quality assurance. As our colleague Andreas greatly put it in his article “Why (even random) annotation errors are problematic for perception”, if you care about achieving good predictions, not only in terms of accuracy but also reliable uncertainties, you should make sure you eliminate any annotation errors. Quality is of the utmost importance if we want autonomous vehicles to become safe and, hence, universally accepted!
In this article, we put the focus on the Kognic teams who directly take part in each step of the quality assurance process, and the key role that continuous communication plays. With that, we hope that you can get a better picture of what you can expect from us during this process (high quality and reliable training data for you to develop your safe perception system), and why it works so smoothly. The key is to keep a strong collaboration and an ongoing dialogue between all parties at each step of the way. 🤗
Needless to say, the journey starts with you as a customer and your needs. After you’ve met with our talented Sales team and we’ve set an account team for you, your requirements as a customer are handed over to our Data Delivery team, who create a project execution plan based on these requirements. To enable successful deliveries to clients, this team selects an LQM (Lead Quality Manager)for quality assurance; and the most skilled and experienced BPO for annotation.
Meera Ranganathan, Data Delivery Manager, has talked to us about the challenge of having everything set in place to ensure good quality in our projects. In her words, “regular and right communication makes such a huge difference in working with our teams, who are located in very different parts of the world, especially for them to understand our quality expectations”. Despite the difficulty of the job, she describes it “as an absolute delight” to integrate and interact with them and work towards improving efficiencies both-ways.
After that, it’s time for the crucial part to begin. Yes, once again we’re going to talk about guidelines! We have written a lot about them, and you can always re-visit our Perception expert Tommy Johansson’s text “Creating data labeling guidelines for safe self-driving vehicles”, where he explains the importance of creating consistent guidelines, and how we do it at Kognic. Speaking of perception experts, what’s their role during this phase? Before we start onboarding annotators, their main task is to understand our clients use case and all its implications, starting out with guiding our clients with what kind of annotation would fit their use case the most and would give them most value for their investment in building a dataset. After this, the work with the labeling guideline starts. This process is called the Guideline Agreement Process. Here, our experts support customers in how to formulate the instructions, in order to reduce the possibility of misinterpretation, identify ambiguities and what might be missing before the project even starts.
Together with the LQMs, they follow up and remove ambiguity in the annotation guidelines as early as possible in the project, setting the base for a fruitful collaboration. By understanding and identifying with the customer which errors the client’s use case is sensitive to, our Perception Experts and the LQMs set the correct quality and cost balance. During this stage, communication between Perception experts, LQMs and the customer is continuous, taking the shape of slack conversations or meetings. Involving customers is not only crucial for staying connected or understanding their requirements, but also for establishing quantitative quality expectations so that both we at Kognic and the customer can evaluate the project during later phases in the most efficient way.
After all stakeholders involved have agreed on the guideline, the next step is ensuring the knowledge is effectively transferred to the annotators. Annotating or labeling images is not an easy task. The fact that humans make different interpretations about the same item can lead to inconsistency. To make a perception system reliable and, as a result, autonomous vehicles trustworthy, the workforce needs to be aligned on requirements when labeling data. The question is: how to ensure thousands of annotators label in a consistent manner?
In our post called “How to efficiently onboard thousands of annotators to ensure consistent quality annotations” we explained how we make this happen. In essence, we provide training to the annotation teams, and the LQMs and QMs support the annotators with uncertainties, edge cases, and questions. This ensures that the whole workforce fully understands the guidelines, and that during the production phase everything works according to the expectations. The quality monitoring process helps in determining who would need additional training, and the good performers. Because only the qualified annotators are moved to the production phase.
Anna Khmura, an experienced LQM, explains this to us a bit further. “Before every project starts off, brief meetings with each BPO are held to explain how to work on tasks, the expectations and goals. Each mistake found in the task needs to be fixed according to the guidelines and the quality of reviewed tasks (or quality assured tasks) should always be Golden. So until I see that a specific user annotates without errors, I can’t approve that the annotator starts working on annotation tasks”. Yep, we leave nothing to chance!
How do we ensure our communication workflow operates as it should during this stage? Apart from the information sharing and training sessions, where experienced LQMs and QMs explain in detail what is expected from annotators, these managers have continuous conversations with annotators who need help. At Kognic, being transparent is in our DNA. This way of working enables us to help the rest of the annotators who participate in the training and prepare for the annotation examination tasks. Another example of this is the fact that LQMs and QMs gather all general doubts in a shared feedback document, so that everyone can benefit from that.
As one of our QMs narrates, everything works thanks to good communication. Giving regular, positive and constructive feedback, and setting meetings with the annotators to give one-on-one coaching is key. This helps us determine where the person struggles, why it is so and provide the appropriate support to avoid structure errors. That is why at Kognic we always encourage our workforce to ask questions when doing their tasks.
During the production phase, Perception experts review quality in datasets, being the goal of it to get an understanding of which error types are hidden and how severe these are for the customer use case. With the help of the integrated quality reports and data quality deep dives, Perception Experts can spot and get rid of structured errors in the dataset. Keeping close contact with LQMs to understand their challenges and doubts and trying to solve them play a key part. As Laura Wörns puts it, “We need to ensure the annotation guidelines and quality expectations are understood from our LQMs, since they are correcting and teaching up our workforce”.
What about our LQMs and QMs? LQMs work closely with QMs and annotators to identify and fix errors (may be repetitive errors, edge cases, new clarifications from a client, or similar) and ensure everyone is on the same page. With the guidelines as a starting point, they all make sure their work is consistent according to these rules. In this process, our LQMs pay special attention to false negative, false positive, the size of the boxes (both 2d and 3d), the assigned properties, the assigned classes, and the understanding of the rules as stated in the guidelines.
We aim for good quality and reliable training data to ensure safe perception. To reach this goal, clear and continuous communication is thus paramount. As Meera Ranganathan expressed it, it’s about consistency and safety, of course. But also about an added human touch to everything our platform displays, and about learning and improving together. “We receive such amazing ideas, feedback and positive comments which motivates us further to provide better support and create better workflows”.
Communication, collaboration, action, iteration. It all comes down to setting the right expectations from all parts from the start, keeping clear and constructive communication, paying attention to details and, when needed, iterating, to deliver on our promises. No matter the steps and number of Kognic teams involved, our communication processes ensure a positive customer experience. Our software does the rest. 🚀⚡