In data annotation, the balance between speed and quality is crucial. Companies need large volumes of high-quality data delivered efficiently. This balance isn't just about meeting quotas - it's about creating reliable, accurate datasets that can effectively train machine learning models.
Introducing Annotation Points (AP)
To measure and improve productivity, we're experimenting with an innovative point-based system called Annotation Points (AP). Every annotation task carries a specific point value, making it easier to track and measure productivity across different types of annotations. Here's an example on how points can be awarded:
| Annotation Type | Points (AP) |
|---|---|
| 2D bounding box | 5 AP |
| 3D Cuboid | 10 AP |
| Dynamic property | 0.5 AP |
| Completed task | 20 AP |
This system transforms complex productivity tracking into a simple, understandable metric. Instead of saying "You've created 6 polygons, 2 3D cubes, and 11 2D boxes," we can now simply say "You've produced 100 AP." This streamlined approach makes it easier for both annotators and managers to track progress and set achievable goals. This system provides stability over time since AP values are calibrated to account for varying task distributions - whether a task contains sparse or dense annotations, the points system ensures fair and comparable measurements across different scenarios.
Performance metrics are available in two key locations within the system: annotators can view their metrics in Campus under Productivity → Team Productivity, while managers access team performance data through Team Management → Team Overview. The following example shows the Productivity metric, with each line representing an individual annotator.
Additionally, we provide annotators with real-time feedback, adding performance KPIs directly in the projects. This creates incredibly fast feedback loops, allowing annotators to experiment with their technique and see within minutes whether changes lead to improvements. This includes:
- Current AP count for the active task
- Daily target progress (updated every 10 minutes)
- Comparison with team performance
- Detailed time allocation statistics
Quality Matters
While productivity is important, quality remains crucial. Each task undergoes a thorough review process, where errors are tracked and counted against the total number of annotated objects. This creates a clear quality score that helps maintain high standards while pushing for efficiency.
For example, if an annotator creates 200 objects across two tasks, and 4 errors are found in a review of 80 objects, their quality score would be 95%. This means we can expect 95% of their objects to meet acceptable quality standards.
Keys to Success
To achieve optimal results, we've identified several critical success factors:
- Clear Instructions and Expectations
- Detailed guidelines for each project
- Well-defined quality standards
- Regular updates on performance metrics
- Proper Equipment and Training
- Access to necessary tools and resources
- Comprehensive initial training
- Ongoing skill development opportunities
- Open Communication
- Regular feedback sessions
- Clear channels for questions and concerns
- Supportive environment for learning
Continuous Improvement
When productivity targets seem challenging, we emphasize that targets should be reached through skill development, not through stress or long hours. The fastest way to improve skill is to get direct support from proven experts who can share their techniques and best practices. We encourage annotators to:
- Seek guidance from quality managers on improving efficiency
- Analyze their workflow for potential bottlenecks
- Learn from high-performing team members
- Focus on steady improvement rather than dramatic changes
Looking Forward
Through this balanced approach to productivity and quality, we're creating a more efficient and accurate annotation process. While we typically see a significant spread in user performance (up to 3x difference), our target-based system acts as an equalizer. Top performers may stabilize once reaching targets, while those below targets work to improve, leading to more consistent team performance over 1-3 weeks.
The combination of clear metrics, supportive infrastructure, and continuous improvement focus helps ensure high-quality data delivery while maintaining reasonable workloads. We achieve this through two key approaches: either setting ambitious targets upfront with clear timelines, or gradually increasing targets toward the ultimate goal. This flexibility, combined with comprehensive support and skill-based improvement strategies, ensures sustainable progress.
Remember: Quality and productivity aren't opposing forces - they're complementary aspects of successful annotation work. By providing the right tools, support, and environment, including demonstrable proof of achievable targets, we enable our teams to excel in both areas through skill development rather than increased work hours.
