Notes from phase I of our pilot of our pilot with Khabar Lahariya

It's been over a year since I first introduced Turkbox, our mission, and initial progress. Since then, we've been working towards finding answers to questions that are crucial to setting up our ecosystem and building a platform where readers can pay for their news by labeling data used to train machine learning models.

Support from the Google News Initiative (GNI) has been instrumental in our progress and, in particular, has allowed us to focus on studying an important component: how readers respond to our widgets. Readers are important stakeholders in our ecosystem, and understanding their willingness to label data along with gathering insight on how accurately readers on the internet can label data is critical for scaling our two-sided marketplace and determining its overall feasibility.

GNI also provided us with the opportunity to work with Khabar Lahariya, who were also recipients of GNI’s Asia Pacific Innovation Funding from India last year. We began our pilot with them as our first publisher last September, as part of which we deployed the first iteration of our widgets on select articles across their website. In the first phase of our pilot, we recorded 11,000+ visits to articles with our widgets.

Partnering with Khabar Lahariya

KL reporters interviewing a local politician

Building widgets for Khabar Lahariya has been an incredible learning experience for us. Khabar Lahariya (KL) is a rural women-run news organization that is focused on telling unheard stories from remote, rural parts of India. What makes Khabar Lahariya unique is not just its all-women team of journalists from Dalit, tribal, Muslim, and disadvantaged communities, but also their unique approach to reporting and covering local issues.

We were quick to realize that as a team of engineers and designers who’ve spent most of their lives in urban settings, we lacked the intuition and empathy required to build a product that would be largely used by a rural audience. To bridge this gap in understanding, we traveled to Banda, one of KL’s regional offices in Uttar Pradesh, to shadow their reporters, interview readers, and test prototypes of our widgets. We were thrilled to witness Khabar Lahariya’s style of reporting firsthand as we visited the district president’s office where the KL reporter asked forthright questions while recording answers in a live video. Their work continues to inspire us towards our vision in which Turkbox widgets can sustainably support such grassroots journalism.

User experience testing with KL readers on the outskirts of Banda, Uttar Pradesh

Conducting extensive research on KL’s audience in rural areas gave us a lot of direction for designing our widgets, especially from a user experience perspective. For instance, a lot of readers we interviewed were familiar with reCAPTCHA widgets, which are frequent fixtures in the Indian Railways ticket reservation app. Therefore, for the initial phase of the pilot, we curated a dataset of images taken from a car dashboard and created a widget where readers have to identify squares containing cars, similar to reCAPTCHA tasks. Our interviewing in Banda also equipped us with a better understanding of what on-screen instructions were easy to follow, which was particularly useful when we translated our instructions for widgets that would be deployed on Hindi articles.

We translated our image segmentation tasks for widgets deployed on Hindi articles

Milestones & Results from Phase I

In March, we achieved the first milestone of our engagements relating to the GNI Innovation Challenge. This involved extensive work on building the underlying infrastructure to support our widgets, pipelines to provide tasks and validate reader responses, and dashboards to provide us as well as our partnering publishers (such as Khabar Lahariya) with tools to analyze reader behavior over time. Our next and final milestone is focused on iteratively improving our widgets through A/B testing and comparing their performance against traditional monetization methods (subscriptions, micropayments, advertisements) using metrics such as user retention and revenue received.

To this end, we’ve already worked with Khabar Lahariya on building the engine that supports their subscription product — Sound, Fury & 4G. We’ve also developed modules that interface their subscription product with existing Turkbox infrastructure, which will be useful for future analytics. Furthermore, having already deployed our first iteration of widgets on the KL website, we’ve already identified a few performance indicators on which we are looking to improve based on results from the first phase of the pilot.

So far, articles containing our widgets have received a total of 11,295 visits. From these visits, we have recorded 726 successful data classifications. With a conversion rate of 6.4%, there’s a lot of work still to be done in improving our widgets to make tasks easier to understand and complete. Ever since completing the first milestone, we've gone back to the drawing board, understanding the data we have gathered so far on user behavior from our widgets, including device configurations, internet speeds, and how often users view detailed instructions. We will use this data to drive our strategy for improving widgets and reader experience.

In the coming weeks, we will also participate in the Digital Product Launch track of the GNI Design Accelerator, through which we will receive training and mentorship across numerous stages in our product development lifecycle, which in turn, will help us improve conversion rates in widgets as well as other components of our platform.

The Turkbox Publisher dashboard visualizes insight from widgets in real-time

Join our pilot

Our immediate priority for now is to consolidate our widget platform, analytics dashboard and related tools into a comprehensive product for publishers that can be easily integrated with their news website. If you are interested in joining our pilot program and deploying our widgets for revenue on your website, please fill out this form and we will get back to you soon.

Simultaneously, we're looking for partners that need their data classified to pilot our data annotations platform with. Currently, we support image segmentation, image classification, handwriting recognition, but we're open to building widgets capable of supporting different types of tasks as well. If you’re interested in using Turkbox to label or classify data, please fill out this form.

I’m happy to answer any questions about Turkbox. Feel free to reach out to me on Twitter, or at [email protected].

Image by Black Ticket Films, courtesy Khabar Lahariya