In this phase, you will need to assess the benefits of the work that’s been done, and setup a pilot so you can deploy the work in a controlled way before broadening the work to a full enterprise or ecosystem of users. In phase 4, you scored the model in the sandbox, and phase 6 represents the first time that most analytics approach deploying the new analytical methods or models in a production environment. Rather than deploying this on a wide scale basis, we recommend that you do a small scope, pilot deployment first. Taking this approach will allow you to limit the amount of risk relative to a full, enterprise deployment and learn about the performance and related constraints on a small scale and make fine tune adjustments before a full deployment.
As you scope this effort, consider running the model in a product environment for a discrete set of single products, or a single line of business, which will test your model in a live setting. This will allow you learn from the deployment, and make any needed adjustments before launching across the enterprise. Keep in mind that this phase can bring in a new set of team members – namely those engineers who are responsible for the production environment, who have a new set of issues and concerns. They want to ensure that running the model fits smoothly into the production environment and the model can be integrated into downstream processes. While executing the model in the production environment, look to detect anomalies on inputs before they are fed to the model. Assess run times and gauge competition for resources with other processes in the production environment.
After deploying the model, conduct follow up to reevaluate the model after it has been in production for a period of time. Assess whether the model is meeting goals and expectations, and if desired changes (such as increase in revenue, reduction in churn) are actually occurring. If these outcomes are not occurring, determine if this is due to a model inaccuracy, or if its predictions are not being acted on appropriately. If needed, automate the retraining/updating of the model. In any case, you will need ongoing monitoring of model accuracy, and if accuracy degrades, you will need to retrain the model. If feasible, design alerts for when model is operating “out-of-bounds”. This includes situations when the inputs are far beyond the range that the model was trained on, which will cause the outputs of the model to be inaccurate. If this begins to happen regularly, retraining is called for. Many times analytical projects yield new insights about a business, a problem, or an idea that people may have taken at face value or thought was impossible to big into.
- If appropriate, hold a post-mortem with your analytic team to discuss what about the process or project that you would change if you had to do it over again.