Measuring Impact of Inaccurate ETAs for an On-Demand Ride/Delivery Organization – Effective ETAs 102

Welcome to part 2 of the Effective ETAs series! Check out part 1 if you want to understand why ETAs are such an integral part of an on-demand ride or delivery service: On-Demand Deliveries and Rides: Effective ETA Strategy.

Creating an ETA strategy can prove to be tricky, especially when you’re not well versed with the parameters one should keep an eye on while evaluating ETAs. Here’s a quick refresher from our last article on which metric we generally focus on:

When analyzing ETAs at NextBillion.ai, we focus on a key metric called ‘promise kept’ in addition to traditional statistical metrics like RMSE and MAE.

But what exactly do we mean by ‘promise kept’? Simply put, whenever the time taken to provide the service (deliver an order/ complete a trip) to a customer is met successfully, the promise made has been kept! Here’s what this metric looks like:

Delivery time = Promised ETA ± Threshold time for the territory

The threshold time is generally a function of the prevailing socio-cultural practices of the geography  — we’ve noticed that the ETA accuracy of the local public transport is a good measurement of this! This helps us understand consumer expectations when it comes to an acceptable waiting time.

Based on this, we get:

Promise kept metric = (Number of accurate orders / total orders of the day) X 100%*

*where delivery time was within the threshold

Why use promise kept as a metric?

As we discussed in Part 1, employing a metric like ‘promise kept’ is extremely effective because of:

1. It is a better indicator of user experience.

For a customer, not all ETA inaccuracies are created equal.

• For a promised ETA of 5 mins, reaching in 7 minutes may be acceptable to some customers, but reaching in 9 mins may not be.
• Similarly, for food to arrive in 35 mins instead of the promised 30 mins may be okay, but 45 mins may not be.
• On the other hand, an actual arrival time of 15 minutes for a promised ETA of 30 also drives a bad user experience and reduces platform efficiency.

2. Traditional statistical metrics are less effective in a multi-team use case:

• Statistical metrics like RMSE are symmetric, so they don’t differentiate between an order arriving 10 minutes early (which is acceptable) vs 10 minutes late.
• While this lack of distinction might be acceptable for the operations team, the analytics team will look at both as a negative result because they fail to meet the promise made to the customer.
• These statistics are extremely difficult to communicate — your RSME showing an improvement from, say, 531.25 to 424.78, holds much less value than saying that you’re meeting your promise on 20% more rides or deliveries.

As we see, a metric like ‘promise kept’ is simple to understand and communicate  — yet it also allows the specificity of a mathematical value  — giving you a usable and effective measurement. This means you can use it to not only track customer experience and other areas of value (like order/ride cancellations, missed prices, etc.), but also bring your various stakeholders on the same page with ease.

Understanding the role played by an accurate ETA for these stakeholders is the next crucial piece of the puzzle.

So how can you use ETAs to track the ROI of a mapping solution?

There are three lenses to optimize the ‘promise kept’ metric through:

1. Customer Experience

For an on-demand delivery or ride-hailing organization, this means striking a balance between the needs of the end customer and the delivery agent/driver to ensure that both these stakeholders have a satisfying experience with the product.

Customer POV:

• For the end-consumer, the role played by an ETA begins even before the service request is placed. It’s what your dispatch team uses to determine whether the customer is serviceable in the first place.
• It’s also crucial for estimating the promise time, which sets the benchmark for a delightful customer experience. This is even more important for use cases where the consumer waiting threshold is only a few minutes.
• And finally, since the customer is often charged based on the trip time estimate, it ensures he/she is being charged fairly for the service provided.

Driver POV:

• For the delivery agent/driver, an accurate ETA sets a service promise that needs to be met. It’s integral for this to be accurate  — because an unachievable estimate can push the delivery agent/driver to deny the request from the customer, or cancel it after accepting
• In addition, routing for these professionals is a different ball game altogether. Since they are already familiar with navigating the locale, it becomes essential to provide routes and ETAs that take their driving behavior into consideration.

2. Platform efficiency

Time-based allocation strategies are often better than distance-based ones since they account for variables that include traffic, weather, and road conditions  — and should include serviceable routes and modifications based on local contexts.

The order allocation process is a key piece of the puzzle when it comes to platform efficiency. Better allocation leads to faster arrival times, better utilization of resources, and lower cancellations.

Because of this, time-based allocation strategies are often better than distance-based ones since they account for variables that include traffic, weather, and road conditions  ±and should include serviceable routes and modifications based on local contexts.

And of course, better ETAs directly translate to improvements in the platform’s allocation efficiency. Here’s how:

• An optimized allocation process is ‘big picture material’ — it might not offer the fastest delivery/drop-off time for a specific order, but smoothen operations across orders to ensure minimum ‘wastage’.
• For example, overestimating an ETA can mean the allocated delivery agent actually arrives early at the restaurant and then has to wait for the food to be prepared, reducing the number of deliveries that the agent can make.
• Similarly, ETA takes center stage when it comes to order batching and shared mobility (for example, a single ride serving multiple customers like the UberPool service). More accurate ETAs mean higher match rates and lesser detours. This helps your team serve a larger number of customers using fewer agents and vehicles  – especially during peak hours.
• Using ETAs during allocation typically requires APIs that can handle very high throughput and work at low latencies. In most cases, this need can only be met by custom APIs designed specifically for these scenarios.

3. Revenue and Profitability

Pricing is a crucial component of determining revenue  — and better ETAs directly translate to better pricing, driving revenue growth.

• Similar to allocation, time-based pricing/delivery fee strategies are often better than distance-based ones as they’re a more accurate reflection of the real-world conditions faced by your drivers/delivery partners.
• Inaccurate ETAs lead to suboptimal pricing, where overpriced orders can give rise to/increase customer attrition, and under-priced rides/orders can cause a direct loss of revenue.
• A great metric to track the impact of ETAs on pricing is the % of orders/rides that are priced accurately. You can arrive at this value by comparing the predicted estimates against the actual time taken.

ETAs also indirectly impact your revenue through an improvement in the first two factors (i.e. CX and platform efficiency) even though the degree of the impact itself may vary.

• For example, the more efficient your platform is, the more effectively you can utilize available resources to tackle more orders. So in the case of food delivery, better order batching means you can serve more customers over the same period of time while also reducing fuel costs.
• When it comes to ride-hailing, an accurate ETA means the cab arrives in time  — leading to better CX, lower cancellations, and higher revenue.

Questions to think about:

Once you have laid down the foundation of your strategy, the next step is to identify and isolate any key areas that are specific to your business case and may have a dramatic impact on your ETA. Here’s a quick summary of some of the parameters you should consider:

• What are the different types of vehicles used by your agents? How are they impacting your routing and ETAs?
• What does your last yard routing look like for locations like malls, parks, apartment complexes, and airports?
• Can your mapping provider support your scalability needs like latency and throughput when it comes to order allocation, batching, or ride pooling? How easily can you build those on top of your mapping solution?
• How do you account for road restrictions and other regulations that are region-specific — such as planned closures, construction activity, local events, and non-serviceable areas?

Looking for a better mapping solution?

At NextBillion.ai, we empower organizations to have their own custom map stack. Backed by leading investors like M12-Microsoft, Lightspeed Venture Partners and Falcon Edge Capital, we work with numerous on-demand transportation and delivery businesses across the globe, serving > 100M API calls every day.

Our platform helps customers build better map stacks that offer:

• Better Performance  — By delivering better (and at the very least comparable) accuracy than other players with up to 6x higher latency, and 10x higher throughput.
• Ability to handle your unique use case  — By allowing vehicle type customizations, setting up road restrictions, and utilizing other use case-specific needs
• Greater Customization: By leveraging your historical data through AI to generate ETAs and maps based on your drivers’ behavior and routes.
• Superior Control: Through DIY tools that allow you to update local nuances in real-time like closed/open roads, serviceability conditions, and other local markers.
• Lower Costs: By providing APIs and on-premise solutions that are better priced and easy to replace your current mapping tool with.

If this sounds like the solution for you, reach out for a demo today!