Key Performance Indicators (KPIs)
Introduction
Key Performance Indicators (KPIs) are used measure, track and report how the organization is performing in relation to its objectives. They can be used both for post-hoc analysis and reporting and for setting goals for the organization. It is important to design relevant and well-understood KPIs.
Challenges
It is important to balance the work that goes into creating, maintaining and reporting on the key performance indicators against the value created. More time should be spent on acting on the metrics than on collecting and reporting them.
Another challenge is choosing, developing and defining what to measure. What data does the organization really need to understand and improve the health of the network automation and operations? It is easy to get carried away and measure/track everything that can be measured!
To measure improvements in the network a baseline needs to be defined and since many networks are manually operated it may require a large effort to get a useful baseline measurement.
Develop KPIs for your organization
As described in the section about objectives, the S.M.A.R.T approach is highlighted and recommended to be used. The "M" stands for Measurable and gives valuable input for how the KPIs could be defined. The KPI is not necessarily a direct reflection of an objective, rather the KPIs could be a subset of several objectives or focus on daily operations that is not part of a specific objective.
It is important to reflect on what is critical to follow up and to develop an efficient process to identify new KPIs and to measure and report on the defined KPIs. Implementing KPI and report processes take time, focus and patience but when working well, it will give very valuable insight to the organization and the performance of the network automation and Automation group.
Examples of technical KPIs suitable for network automation
| What to measure | Why | How to measure |
|---|---|---|
| Number of transactions in the automation system | Understand the scale of the solution | From logs in the automation system |
| Number of events in the automation system | Understand the scale of the solution | From logs in the automation system |
| Number of port configurations | Understand how much use the system sees | From logs in the automation system |
| Duration of port configuration | Understand what the benefits are of the system | Calculated from automation logs |
| Number of post check for-sub interface | Understand how much use the system sees | From logs in the automation system |
| Duration of sub-interface post checks | Understand what the benefits are of the system | Calculated from automation logs |
| Number of updates to the automation software | Understand the change velocity of the automation group | From the CI/CD system |
Examples of business/organizational KPIs suitable for network automation
| What to measure | Why | How to measure |
|---|---|---|
| Time it takes to run a use case through Define to Deployment | Understand the Automation group efficiency | Agree upon when define state begin and when the use case is deployed |
| Time saved on port configuration | Understand a benefit of automation | Comparing baseline measurements to measurements and counts from the automation system. |
| Time to launch new services | Understand the Automation efficiency | Agree on entry and exit criteria in the use-case process. Sub-measurements are of interest |
Reporting format and cadence
When deciding on reporting cadence one should consider how many KPIs that are being tracked and what the effort is to get them in presentable state. If the KPIs gets automatically tracked and displayed on a realtime dashboard the cadence is a non-problem. On the other side, if it requires someone to do manual work which takes x minutes per KPI to generate and analyze, one should try to keep the reporting cadence more distributed over time. It basically ends up in a trade-off discussion where valuable information is on one side and time spent on the other side. As the organization gets more mature, the reporting cadence and the number of detailed and valuable KPIs should increase. It is suggested to use the strategy of: starting small and incrementally refine.
Best practices
- Define and agree on KPIs aligned with the network automation strategy
- Report on agreed KPIs to management and stakeholders
- Optimize and review KPIs
Checklist
- Long term success has been defined and which KPIs to track the team’s success are agreed
- Established routines for follow up and reporting on defined KPIs