March 3rd, 2021
Defensible Machine Learning
Ankur Goyal, Alana Anderson | 12 minute read
B2B machine learning (ML) companies are an enigma: they have the opportunity to revolutionize how we do business, but they look & feel quite different from their traditional SaaS counterparts and have proven difficult to scale. In this post, we aim to demystify the challenges of building an enduring ML company by providing a simple framework for the three ways to do so. For each, we outline common attributes of successful companies and walk through potential pitfalls. These learnings are based on our experience building Impira and evaluating companies at Base Case Capital.
For aspiring entrepreneurs thinking about problems in this area, we hope this gives you a starting point to understand the landscape, frame your idea, and avoid the challenges that may arise.
Three types of ML companies
Enterprises who seek to solve problems with machine learning have two options: (1) develop ML models in-house or (2) purchase software that has embedded ML. Startups can play a significant role in both scenarios. The first option involves providing in-house users, traditionally referred to as ML practitioners, with ML systems that enable them to work better, faster, and smarter to develop and operate models. The second option is usually packaged as a Powered-by-ML solution, which solves a specific business challenge with the help of machine learning.
There is an emerging third group, AutoML solutions, which allow non-ML practitioners to develop their own models. This is somewhat a hybrid of the first two, and correspondingly represents a massive opportunity, but with significant product and technical risks. In fact, many of the common pitfalls for both ML systems and Powered-by-ML solutions can be overcome with AutoML. We’ll discuss this in depth below.
ML systems
ML systems make ML scientists and engineers more productive. Companies of this type provide tools that enable the development workflow, deployment, monitoring, and testing of ML. This market is obviously limited to organizations that have in-house ML practitioners; however, it is expanding rapidly. In fact, the number of AI specialist jobs has grown on average 74% each year for the past 4 years.
ML systems: key considerations
| Target customer | Organizations with in-house ML developers |
| Target buyer | ML leadership |
| Target user | ML practitioner (scientist or engineer)" |
| Founding team | Systems builders + empathy for ML practitioners’ workflow" |
| Pitfalls | Narrow market, easy to build niche product" |
| Interesting cos | Aquarium Learning, Scale AI, Weights & Biases, Databricks" |
ML systems: building for success
ML systems companies are typically founded by engineers who worked at technology companies developing high scale machine learning systems in-house. These founders often worked in the infrastructure group supporting internal machine learning efforts and now want to bring this technology to others. They should have deep empathy for ML practitioners in order to build a need-to-have product.
Aquarium Learning1, for example, is a machine learning data management system. The product makes it easy for machine learning practitioners to find labeling errors and model failures, then helps them curate the dataset to fix these problems and optimize model performance. The founders, Peter and Quinn, were early employees at Cruise, where they saw first-hand that the best way to improve model performance was to improve the data it was trained on. With this insight, they built Aquarium to provide this tooling to machine learnings teams across a variety of different industries.
ML systems: avoiding pitfalls
For talented engineers, building an ML system is a dream come true: you get to build tools for other engineers, in a space with tons of greenfield that is developing rapidly. The flip side is that the market is still very early. In 2019, a report from Tencent estimated there were 300k ML practitioners. In comparison, the number of software developers worldwide in 2019 was 19M, more than 60x greater.
While this market is growing rapidly, it’s important that companies in this space figure out a pricing structure that works. For example, Weights & Biases charges an impressive $200/user/month for their Teams offering. A $50/user/month product simply won’t cut it. Similarly, when companies choose a specific vertical (e.g. financial services) or functional area (e.g. computer vision), they are unlikely to target enough users to justify a user-based pricing model. Scale AI, which initially targeted the automotive industry, charges a massive fee for their service based on the amount of data they label rather than per-user. Other products, like Sagemaker, charge for resource consumption. Since machine learning is so data-intensive, both pricing models will result in a high price per customer and a correspondingly large TAM.
There are a couple other tricky market segmentation challenges. First, products that target production ML use cases (monitoring, deployment, and scale) have struggled to gain traction. The field is very early, and there are almost an order of magnitude more ML projects in development than in production. Second, there is a disconnect between big software spenders (traditional enterprises) and companies with ML in production (Uber, Facebook, etc.) who often prefer to build technology in-house. For this reason, a lot more users can be reached with products that enable ML development (Aquarium, Scale, W&B, Snorkel, etc.) than those that require a model to be deployed in production (e.g. feature store products). To broaden their value proposition beyond ML practitioners, some ML systems are adding AutoML capabilities. For example, Scale now offers full-service document processing and Databricks launched an AutoML platform.
Powered-by-ML
Powered-by-ML products are solution-oriented and target specific business problems or workflows that benefit from automation or insight. They often solve problems within specific industries or departments that are consistent from customer-to-customer. For most of these products, machine learning is an invisible “magic” that runs behind the scenes and boosts the value proposition.
Powered-by-ML: key considerations
| Target customer | Organizations with a specific problem that can be measured with data |
| Target buyer | Executive |
| Target user | Business user |
| Founding team | Industry experts + ML practitioners |
| Pitfalls | Certain problems require 100% accuracy or per-customer training |
| Interesting cos | Vise, Abnormal Security, Sisu, Gong, Cresta |
Powered-by-ML: building for success
Powered-by-ML products generally target a key problem that occurs consistently across companies in a specific industry or department. For example, Gong provides common coaching tips for salespeople by recording and analyzing videos of their conversations. Abnormal Security analyzes the content of emails in an enterprise to predict whether they are indicative of a cyber attack. When they work, the ROI of these products is astounding since by definition you can measure how they impact the key problem. Companies like Cresta leverage this to price in terms of ROI which leads to very high ACVs.
The winning combination for a Powered-by-ML company is a founding team that includes domain and ML expertise. Ideally, the domain expert is so familiar with the problem that there is a marvelous “ML model” that solves the key problem locked away somewhere inside their brain. For example, Amit from Gong worked previously as a go-to-market executive in technology companies and considers Gong the ultimate user of their own software. Of course, there are exceptions too: Evan and Sanjay of Abnormal Security reframed their expertise in ad tech personalization to the problem of email behavior in the enterprise.
Powered-by-ML: avoiding pitfalls
Powered-by-ML companies are highly controversial, and much has been written about how and why they are difficult to scale (some of our favorites include: The New Business of AI, Taming the Tail: Adventures in Improving AI Economics, Why AI/ML Fails). We agree with these ideas, but want to go one step further to explain how to avoid these pitfalls.
With Powered-by-ML solutions, one must be careful not to over-promise. Problems with a low margin for error are poor targets for these solutions. For example, if a product tries to automatically generate user-facing content, it’s likely to have errors that reflect poorly on the brand. Similarly, if it tries to make medical recommendations, it’s likely to make some unsafe mistakes. On the other hand, a product like Abnormal Security provides substantial value by finding phishing emails that would have otherwise never been caught. It’s okay that it doesn’t find every bad email, and it’s okay if it finds a few false positives, as long as customers are catching more phishing attempts with the product than without it. Beyond that, most problems require significant customer-specific data to provide high enough performance, so be prepared to work around poor results in the early days. If you’re an entrepreneur building a Powered-by-ML solution, it’s important to frame the problem you’re solving so that an imperfect ML model can still provide significant customer value.
Powered-by-ML companies targeting the right problems are highly repeatable. They can train a model in terms of a large corpus of data, spanning multiple companies, in a way that allows the next several customers to benefit almost immediately. That said, problems are rarely this neat & clean, and the solution to most problems varies from customer to customer. Imagine a fictional product that predicts which marketing campaigns will succeed based on a number of factors (e.g. product category, creative, pricing, target user, etc.). These factors vary significantly from company to company and likely require a different model in each case. If you plan to build a Powered-by-ML company, you’re best suited to pick problems that have a consistent set of data attributes (schema) and underlying drivers from customer to customer. A smoke test for this is whether the domain expert on your founding team can manually solve the problem for your entire customer base. Communication best-practices in a sales call and phishing emails are highly consistent from customer to customer. Marketing strategy and supply chain processes are not.
Some suggest leveraging professional services, which can be used to customize models manually, across customers. While this is viable, it’s a difficult model to scale while keeping attractive gross margins. For some subset of these problems, a much more technically ambitious approach is possible: AutoML.
AutoML
AutoML technology automates the machine learning training and development process, enabling non-ML practitioners to train customized models to solve problems. Instead of pre-training for every possible scenario as in Powered-by-ML, AutoML products can learn on the fly from customer to customer. And unlike in-house ML efforts, they do not require ML practitioners and instead engage a much broader audience (today: IT and ops, tomorrow: business users). AutoML has been successfully applied to a number of problems — including document extraction, image labeling, time series forecasting, and entity recognition — where data varies significantly from customer to customer.
AutoML: key considerations
| Target customer | Organizations with custom or multiple problems that can be measured with data |
| Target buyer | IT Executives, business stakeholders |
| Target user | IT + Ops |
| Founding team | Systems builders + ML practitioners |
| Pitfalls | Stepping on ML users’ toes, require training data |
| Interesting cos | Impira2, Abacus AI, DataRobot |
AutoML: building for success
AutoML products use a fixed set of machine learning techniques that train on an individual customer’s data to produce a tailor-made model that solves their problem. By constraining the machine learning techniques used, they do not need an ML practitioner to manually devise a new model for each new customer or use case. DataRobot uses AutoML to automatically model trends in customer data. Impira** uses AutoML to automatically build custom text extraction models that understand a customer’s unique document formats.
Building an AutoML product is no easy task. The founding team needs ML expertise to build highly general models that can train on small volumes of data as well as systems expertise to automate training/deployment/execution of the generated models. Beyond that, the average user is not used to training machine learning models, so the product needs to have a user experience that simplifies these concepts. For example, Impira strives to build an experience that is as simple as a spreadsheet.
AutoML: avoiding pitfalls
While AutoML solutions enable everyday users to train their own models, the data science population has mixed reactions. Technically speaking, data scientists can often produce much better solutions than AutoML. In fact, companies like DataRobot have developed an adversarial relationship with data scientists, which can create internal struggles for buyers who employ them. A simple way to avoid this is to frame AutoML solutions in terms of the problems they solve — document extraction (Impira), business intelligence (DataRobot), and personalization (Abacus) — instead of the ML techniques used to solve them. This positioning will more likely lead to the customers actually experiencing these problems, who tend not to be data scientists.
Unlike Powered-by-ML solutions, which can reuse a model trained on one customer’s data for the next customer, AutoML solutions must be trained for each new customer. This requires an initial training data set and some time/effort from the user. It’s important to pick problems that can be trained quickly and effectively with small amounts of data and a quick feedback loop. For example, a problem that requires optimizing large or complex deep neural nets is a poor candidate for AutoML because you need massive volumes of data and compute power to train and cross-validate them. One trick is to break a problem down into smaller pieces, some of which can be pre-trained over a massive dataset and others which can be learned for a particular customer. For example, Impira uses pre-trained OCR models which feed into per-customer AutoML text extraction models. Another emerging technique is transfer learning, which reduces the volume of data required to specialize a pre-trained neural net to a particular problem.
Conclusion
The market for ML technology is massive. ML systems provide tools for a new group of users, Powered-by-ML solutions provide automation around previously manual tasks, and AutoML solutions have the ability to disrupt existing categories because they target everyday users. At Base Case, we believe there are many great companies to be built across ML systems, Powered-by-ML, and AutoML. The winning combination is a founding team with a skill set that matches the problem they’re out to solve. In machine learning, problem framing is everything. If you’re a founder with an idea in the ML space, or even one who is passionate about ML, but trying to find the right kind of company to start, we would love to chat.
Thanks to Saam Motamedi, Grant Shirk, Richard Stebbing, Tom Swartz, and Julie Pang for their feedback on this post.
Share on Twitter, Facebook, or LinkedIn.