Welcome!

Cognitive Computing Authors: Yeshim Deniz, Pat Romanski, Liz McMillan, Jason Bloomberg, Elizabeth White

Related Topics: @DevOpsSummit, Microservices Expo, Agile Computing, @CloudExpo

@DevOpsSummit: Article

Fifteen Metrics for DevOps Success | @DevOpsSummit #DevOps #AI #Serverless

The word DevOps means different things to different people

Fifteen Metrics for DevOps Success
By Matt Watson

How is DevOps going within your organization? If you need some help measuring just how well it is going, we have prepared a list of some key DevOps metrics to track. These metrics can help you understand how your team is doing over time.

Define what DevOps means to your organization
The word DevOps means different things to different people. Some say it a culture and every vendor in the industry claims that their tools help with DevOps. Depending on how you define DevOps, some of these metrics may matter more or less to you and your team.

I define DevOps as everything that relates to deploying and monitoring your applications. In many ways, this bleeds over to site reliability engineering. At Stackify, we don't even have an operations team to collaborate with. Our developers deploy directly to the cloud and we operate in more of a "NoOps" style. Check out my blog post to learn more about my views on what DevOps is.

Identify your DevOps challenges
Before you figure out what DevOps metrics to track, you need to identify what challenges your organization has and what problems you are trying to solve. At Stackify, our biggest issue has been not deploying often enough and lowering our defect escape rate. Our executive team is putting a big focus on those specific metrics for 2018.

Types of DevOps metrics
DevOps is all about continuous delivery and shipping code as fast as possible. You want to move fast and not break things. By tracking these DevOps metrics, you can evaluate just how fast you can move before you start breaking things.

  1. Deployment frequency
  2. Change volume
  3. Deployment time
  4. Lead time
  5. Customer tickets
  6. Automated test pass %
  7. Defect escape rate
  8. Availability
  9. Service level agreements
  10. Failed deployments
  11. Error rates
  12. Application usage and traffic
  13. Application performance
  14. Mean time to detection (MTTD)
  15. Mean time to recovery (MTTR)

Goals of DevOps: Velocity, Quality, Performance
The main goals of DevOps are velocity, quality and application performance.

You want to ship code as fast and often as possible. How fast you can do this will vary wildly based on your type of product, team, and risk tolerance.

Even if you don't track any DevOps metrics around your velocity, you should at least measure how you are doing on quality. Perhaps you try to ship when you can, and you don't really care how fast exactly. However, you always care about quality. The last thing you want is to be chasing production fires all the time.

The third piece of the equation is performance. You could argue that it is also at odds with your goals of high velocity and quality. Performance is also related to quality, but perhaps a little different.

Deployment size
Tracking how many stories, feature requests, and bug fixes are being deployed is another good DevOps metric. Depending on how large your individual work items are, their counts could vary wildly. You could also track how many story points or days' worth of development work are being deployed.

Deployment frequency
Tracking how often you do deployments is a good DevOps metric. Ultimately, the goal is to do more smaller deployments as often as possible. Reducing the size of deployments makes it easier to test and release.

I would suggest counting both production and non-production deployments separately. How often you deploy to QA or pre-production environments is also important. You need to deploy early and often in QA to ensure time for testing. Finding bugs in QA is important to keep your defect escape rate down.

Deployment time
This might seem like a weird one, but tracking how long it takes to do an actual deployment is another good metric. One of our applications at Stackify is deployed with Azure worker roles and it takes about an hour to deploy. It is a nightmare. Tracking such things could help identify potential problems. It is much easier to deploy more often when the task of actually doing it is quick.

Lead time
If the goal is shipping code quickly, this is a really key DevOps metric. I would define lead time as the amount of time that occurs between starting on a work item until it is deployed. This helps you know that if you started on a new work item today, how long would it take on average until it gets to production. This is also a good metric to help with BizDevOps.

Customer tickets
The best and worst indicator of application problems is customer support tickets and feedback. The last thing you want is for your users to find bugs or have problems with your software. Because of this, they also make a good indicator of application quality and performance problems.

Automated tests pass %
To increase velocity, it is highly recommended that your team makes extensive usage of unit and functional testing. Since DevOps relies heavily on automation, tracking how well your automated tests work is a good DevOps metrics. It is good to know how often code changes are causing your tests to break.

Defect escape rate
Do you know how many software defects are being found in production versus QA? If you want to ship code fast, you need to have confidence that you can find software defects before they get to production. Your defect escape rate is a great DevOps metric to track how often those defects make it to production.

Availability
The last thing you ever want is for your application to be down. Depending on your type of application and how you deploy it, you may have a little downtime as part of scheduled maintenance. I would suggest tracking that and all unplanned outages.

Service level agreements
Most companies have some service level agreement (SLA) that they operate with. It is also important that you track your compliance with your SLAs. Even if there are no formal SLA, there probably are application requirements or expectations to be achieved.

Failed deployments
We all hope this never happens, but how often do your deployments cause an outage or major issues for your users? Reversing a failed deployment is something we never want to do, but it is something you should always plan for. If you have issues with failed deployments, be sure to track this metric over time. This could also be seen as tracking mean time to failure (MTTF).

Error rates
Tracking error rates within your application is super important. Not only are they an indicator of quality problems, but also ongoing performance and uptime related issues. Good exception handling best practices are critical for good software.

  • Bugs - Identify new exceptions being thrown in your code after a deployment
  • Production issues - Capture issues with database connections, query timeouts, and other related issues.

Errors are a fact of life for most applications. At Stackify, we process millions of messages an hour across a couple hundred servers and over a thousand SQL databases. A few errors here and there are just part of the noise of a busy system. It is important that you keep a pulse on your error rates and look for spikes.

Application usage & traffic
After a deployment, you want to see if the amount of transactions or users accessing your system looks normal. If you suddenly have no traffic or a giant spike in traffic, something could be wrong.

The last thing you ever want to see is no traffic at all. You could also see a spike in traffic if you are using microservices and one of your applications is causing a lot more traffic all of a sudden.

Application performance
Before you even do a deployment, you should use a tool like Retrace to look for performance problems, hidden errors, and other issues. During and after the deployment, you should also look for any changes in overall application performance.

It might be common after a deployment to see major changes in the usage of specific SQL queries, web service calls, and other application dependencies. Tools like Retrace can provide valuable visualizations like this one below that helps make it easy to spot problems.

Mean time to detection (MTTD)
When problems do happen, it is important that you identify them quickly. The last thing you want is to have a major partial or broad system outage and not know about it. Having robust application monitoring and good coverage in place will help you detect issues quickly. Once you detect them, you also have to fix them quickly!

Mean time to recovery (MTTR)
This metric helps you track how long it takes to recover from failures. A key metric for the business is keeping failures to a minimum and being able to recover from them quickly. It is typically measured in hours and may refer to business hours, not clock hours.

Having good application monitoring tools in place to quickly identify issues and quickly deploy the fix is important to reducing your MTTR.

Application metrics
Beyond the DevOps metrics listed above, there are dozens of other metrics you can track that are specific to your applications. Most of them are not necessarily relevant to DevOps in regards to deploying your application. However, they are very critical for monitoring the usage and performance of your applications in production.

For example, at Stackify, we use custom metrics to track how many log messages are received via our API per minute. This is a critical metric that helps us understand the volume of data flowing through our system. Depending on your application, you may have similar custom metrics that are critical to your application.

Read more:

After a deployment, you will want to keep an eye on all of your critical application metrics to ensure that everything still looks normal.

Summary

If you want to take DevOps to the next level, I'm sure that our list of DevOps metrics will help give you some ideas of what to track and improve. The goal of DevOps is collaboration and getting developers more involved in the deployment process and application monitoring. If you need some help with monitoring your applications, then be sure to check out our product Retrace.

Related articles:

The post 15 Metrics for DevOps Success appeared first on Stackify.


DXWorldEXPO LLC, the producer of the world's most influential technology conferences and trade shows has announced the conference tracks for CloudEXPO | DXWorldEXPO 2018 New York.

DXWordEXPO New York 2018, colocated with CloudEXPO New York 2018 will be held November 11-13, 2018, in New York City.

Digital Transformation (DX) is a major focus with the introduction of DXWorldEXPO within the program. Successful transformation requires a laser focus on being data-driven and on using all the tools available that enable transformation if they plan to survive over the long term.

A total of 88% of Fortune 500 companies from a generation ago are now out of business. Only 12% still survive. Similar percentages are found throughout enterprises of all sizes.

Register for Full Conference "Gold Pass" ▸ Here (Expo Hall ▸ Here)

Sponsorship Opportunities Here

Speaking Opportunities Here

Sponsorship and Speaking Inquiries: [email protected].

2018 Conference Agenda, Keynotes and 10 Conference Tracks

DXWordEXPO New York 2018 and Cloud Expo New York 2018 agenda present 222 rockstar faculty members, 200 sessions and 22 keynotes and general sessions in 10 distinct conference tracks.

  • Cloud-Native | Serverless
  • DevOpsSummit
  • FinTechEXPO - New York Blockchain Event
  • CloudEXPO - Enterprise Cloud
  • DXWorldEXPO - Digital Transformation (DX)
  • Smart Cities | IoT | IIoT
  • AI | Machine Learning | Cognitive Computing
  • BigData | Analytics
  • The API Enterprise | Mobility | Security
  • Hot Topics | FinTech | WebRTC

Register for Full Conference "Gold Pass" ▸ Here (Expo Hall ▸ Here)

DXWorldEXPO | CloudEXPO 2018 New York cover all of these tools, with the most comprehensive program and with 222 rockstar speakers throughout our industry presenting 22 Keynotes and General Sessions, 200 Breakout Sessions along 10 Tracks, as well as our signature Power Panels. Our Expo Floor brings together the world's leading companies throughout the world of Cloud Computing, DevOps, FinTech, Digital Transformation, and all they entail.

As your enterprise creates a vision and strategy that enables you to create your own unique, long-term success, learning about all the technologies involved is essential. Companies today not only form multi-cloud and hybrid cloud architectures, but create them with built-in cognitive capabilities.

Cloud-Native thinking is now the norm in financial services, manufacturing, telco, healthcare, transportation, energy, media, entertainment, retail and other consumer industries, as well as the public sector.

CloudEXPO is the world's most influential technology event where Cloud Computing was coined over a decade ago and where technology buyers and vendors meet to experience and discuss the big picture of Digital Transformation and all of the strategies, tactics, and tools they need to realize their goals.

FinTech Is Now Part of the DXWorldEXPO | CloudEXPO Program!

Financial enterprises in New York City, London, Singapore, and other world financial capitals are embracing a new generation of smart, automated FinTech that eliminates many cumbersome, slow, and expensive intermediate processes from their businesses.

Accordingly, attendees at the upcoming 22nd CloudEXPO | DXWorldEXPO November 11-13, 2018 in New York City will find fresh new content in two new tracks called:

  • FinTechEXPO
  • New York Blockchain Event

which will incorporate FinTech and Blockchain, as well as machine learning, artificial intelligence and deep learning in these two distinct tracks.

Register for Full Conference "Gold Pass" ▸ Here (Expo Hall ▸ Here)

Sponsorship Opportunities Here

Speaking Opportunities Here

Sponsorship and Speaking Inquiries: [email protected].

FinTech brings efficiency as well as the ability to deliver new services and a much improved customer experience throughout the global financial services industry. FinTech is a natural fit with cloud computing, as new services are quickly developed, deployed, and scaled on public, private, and hybrid clouds.

More than US$20 billion in venture capital is being invested in FinTech this year. DXWorldEXPOCloudEXPO are pleased to bring you the latest FinTech developments as an integral part of our program.

DXWorldEXPO | CloudEXPO are accepting speaking submissions for this new track, so please visit Cloud Computing Expo for the latest information or contact us at [email protected]

Register for Full Conference "Gold Pass" ▸ Here (Expo Hall ▸ Here)

Sponsorship Opportunities Here

Speaking Opportunities Here

Sponsorship and Speaking Inquiries: [email protected].

Download Slide Deck ▸ Here

Only DXWorldEXPO | CloudEXPO bring together all this in a single location:

Attend DXWorldEXPO | CloudEXPO. Build your own custom experience. Learn about the world's latest technologies and chart your course to Digital Transformation.

22nd International DXWorldEXPO | CloudEXPO, taking place November 11-13, 2018, in New York City, will feature technical sessions from a rock star conference faculty and the leading industry players in the world.

Register for Full Conference "Gold Pass" ▸ Here (Expo Hall ▸ Here)

Sponsorship Opportunities Here

Speaking Opportunities Here

Sponsorship and Speaking Inquiries: [email protected].

Download Slide Deck: ▸ Here

Cloud computing is now being embraced by a majority of enterprises of all sizes. Yesterday's debate about public vs. private has transformed into the reality of hybrid cloud: a recent survey shows that 74% of enterprises have a hybrid cloud strategy. Meanwhile, 94% of enterprises are using some form of XaaS - software, platform, and infrastructure as a service.

With major technology companies and startups seriously embracing Cloud strategies, now is the perfect time to attend and learn what is going on, contribute to the discussions, and ensure that your enterprise is on the right path to Digital Transformation.

Every Global 2000 enterprise in the world is now integrating cloud computing in some form into its IT development and operations. Midsize and small businesses are also migrating to the cloud in increasing numbers.

Register for Full Conference "Gold Pass" ▸ Here (Expo Hall ▸ Here)

Sponsorship Opportunities Here

Speaking Opportunities Here

Sponsorship and Speaking Inquiries: [email protected].

Download Slide Deck: ▸ Here

Companies are each developing their unique mix of cloud technologies and services, forming multi-cloud and hybrid cloud architectures and deployments across all major industries. Cloud-driven thinking has become the norm in financial services, manufacturing, telco, healthcare, transportation, energy, media, entertainment, retail and other consumer industries, and the public sector.

Sponsorship Opportunities

DXWorldEXPO | CloudEXPO are the single show where technology buyers and vendors can meet to experience and discus cloud computing and all that it entails. Sponsors of DXWorldEXPO | CloudEXPO will benefit from unmatched branding, profile building and lead generation opportunities through:

  • Featured on-site presentation and ongoing on-demand webcast exposure to a captive audience of industry decision-makers.
  • Showcase exhibition during our new extended dedicated expo hours
  • Breakout Session Priority scheduling for Sponsors that have been guaranteed a 35-minute technical session
  • Online advertising on 4,5 million article pages in SYS-CON's i-Technology Publications
  • Capitalize on our Comprehensive Marketing efforts leading up to the show with print mailings, e-newsletters and extensive online media coverage.
  • Unprecedented PR Coverage: Unmatched editorial coverage on Cloud Computing Journal.
  • Tweetup to over 100,000 plus Twitter followers
  • Press releases sent on major wire services to over 500 industry analysts.

Secrets of Our Most Popular Sponsors and Exhibitors ▸ Here

For more information on sponsorship, exhibit, and keynote opportunities, contact [email protected].

Sponsorship Opportunities Here

Download Slide Deck:Here

Speaking Opportunities

The upcoming 22nd International DXWorldEXPO | CloudEXPO November 11-13, 2018 in New York City, NY announces that its Call For Papers for speaking opportunities is now open.

Secrets of Our Most Popular Faculty Members ▸ Here

Submit your speaking proposal Here or by email [email protected].

Download Slide Deck: ▸ Here

About DXWorldEXPO LLC

DXWorldEXPO LLC is a Lighthouse Point, Florida-based trade show company and the creator of DXWorldEXPODigital Transformation Conference & Expo. The company produces and presents CloudEXPO, DevOpsSummitFinTechEXPO Blockchain Event, the world's most influential conferences and trade shows.

More Stories By Stackify Blog

Stackify offers the only developers-friendly solution that fully integrates error and log management with application performance monitoring and management. Allowing you to easily isolate issues, identify what needs to be fixed quicker and focus your efforts – Support less, Code more. Stackify provides software developers, operations and support managers with an innovative cloud based solution that gives them DevOps insight and allows them to monitor, detect and resolve application issues before they affect the business to ensure a better end user experience. Start your free trial now stackify.com

@ThingsExpo Stories
The best way to leverage your CloudEXPO | DXWorldEXPO presence as a sponsor and exhibitor is to plan your news announcements around our events. The press covering CloudEXPO | DXWorldEXPO will have access to these releases and will amplify your news announcements. More than two dozen Cloud companies either set deals at our shows or have announced their mergers and acquisitions at CloudEXPO. Product announcements during our show provide your company with the most reach through our targeted audienc...
JETRO showcased Japan Digital Transformation Pavilion at SYS-CON's 21st International Cloud Expo® at the Santa Clara Convention Center in Santa Clara, CA. The Japan External Trade Organization (JETRO) is a non-profit organization that provides business support services to companies expanding to Japan. With the support of JETRO's dedicated staff, clients can incorporate their business; receive visa, immigration, and HR support; find dedicated office space; identify local government subsidies; get...
DXWorldEXPO LLC announced today that All in Mobile, a mobile app development company from Poland, will exhibit at the 22nd International CloudEXPO | DXWorldEXPO. All In Mobile is a mobile app development company from Poland. Since 2014, they maintain passion for developing mobile applications for enterprises and startups worldwide.
@DevOpsSummit at Cloud Expo, taking place November 12-13 in New York City, NY, is co-located with 22nd international CloudEXPO | first international DXWorldEXPO and will feature technical sessions from a rock star conference faculty and the leading industry players in the world.
"Akvelon is a software development company and we also provide consultancy services to folks who are looking to scale or accelerate their engineering roadmaps," explained Jeremiah Mothersell, Marketing Manager at Akvelon, in this SYS-CON.tv interview at 21st Cloud Expo, held Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA.
As data explodes in quantity, importance and from new sources, the need for managing and protecting data residing across physical, virtual, and cloud environments grow with it. Managing data includes protecting it, indexing and classifying it for true, long-term management, compliance and E-Discovery. Commvault can ensure this with a single pane of glass solution – whether in a private cloud, a Service Provider delivered public cloud or a hybrid cloud environment – across the heterogeneous enter...
DXWorldEXPO LLC announced today that ICC-USA, a computer systems integrator and server manufacturing company focused on developing products and product appliances, will exhibit at the 22nd International CloudEXPO | DXWorldEXPO. DXWordEXPO New York 2018, colocated with CloudEXPO New York 2018 will be held November 11-13, 2018, in New York City. ICC is a computer systems integrator and server manufacturing company focused on developing products and product appliances to meet a wide range of ...
More and more brands have jumped on the IoT bandwagon. We have an excess of wearables – activity trackers, smartwatches, smart glasses and sneakers, and more that track seemingly endless datapoints. However, most consumers have no idea what “IoT” means. Creating more wearables that track data shouldn't be the aim of brands; delivering meaningful, tangible relevance to their users should be. We're in a period in which the IoT pendulum is still swinging. Initially, it swung toward "smart for smart...
Headquartered in Plainsboro, NJ, Synametrics Technologies has provided IT professionals and computer systems developers since 1997. Based on the success of their initial product offerings (WinSQL and DeltaCopy), the company continues to create and hone innovative products that help its customers get more from their computer applications, databases and infrastructure. To date, over one million users around the world have chosen Synametrics solutions to help power their accelerated business or per...
Dion Hinchcliffe is an internationally recognized digital expert, bestselling book author, frequent keynote speaker, analyst, futurist, and transformation expert based in Washington, DC. He is currently Chief Strategy Officer at the industry-leading digital strategy and online community solutions firm, 7Summits.
In an era of historic innovation fueled by unprecedented access to data and technology, the low cost and risk of entering new markets has leveled the playing field for business. Today, any ambitious innovator can easily introduce a new application or product that can reinvent business models and transform the client experience. In their Day 2 Keynote at 19th Cloud Expo, Mercer Rowe, IBM Vice President of Strategic Alliances, and Raejeanne Skillern, Intel Vice President of Data Center Group and ...
Founded in 2000, Chetu Inc. is a global provider of customized software development solutions and IT staff augmentation services for software technology providers. By providing clients with unparalleled niche technology expertise and industry experience, Chetu has become the premiere long-term, back-end software development partner for start-ups, SMBs, and Fortune 500 companies. Chetu is headquartered in Plantation, Florida, with thirteen offices throughout the U.S. and abroad.
Bill Schmarzo, author of "Big Data: Understanding How Data Powers Big Business" and "Big Data MBA: Driving Business Strategies with Data Science," is responsible for setting the strategy and defining the Big Data service offerings and capabilities for EMC Global Services Big Data Practice. As the CTO for the Big Data Practice, he is responsible for working with organizations to help them identify where and how to start their big data journeys. He's written several white papers, is an avid blogge...
"We are a well-established player in the application life cycle management market and we also have a very strong version control product," stated Flint Brenton, CEO of CollabNet,, in this SYS-CON.tv interview at 18th Cloud Expo at the Javits Center in New York City, NY.
It is of utmost importance for the future success of WebRTC to ensure that interoperability is operational between web browsers and any WebRTC-compliant client. To be guaranteed as operational and effective, interoperability must be tested extensively by establishing WebRTC data and media connections between different web browsers running on different devices and operating systems. In his session at WebRTC Summit at @ThingsExpo, Dr. Alex Gouaillard, CEO and Founder of CoSMo Software, presented ...
Most people haven’t heard the word, “gamification,” even though they probably, and perhaps unwittingly, participate in it every day. Gamification is “the process of adding games or game-like elements to something (as a task) so as to encourage participation.” Further, gamification is about bringing game mechanics – rules, constructs, processes, and methods – into the real world in an effort to engage people. In his session at @ThingsExpo, Robert Endo, owner and engagement manager of Intrepid D...
Recently, WebRTC has a lot of eyes from market. The use cases of WebRTC are expanding - video chat, online education, online health care etc. Not only for human-to-human communication, but also IoT use cases such as machine to human use cases can be seen recently. One of the typical use-case is remote camera monitoring. With WebRTC, people can have interoperability and flexibility for deploying monitoring service. However, the benefit of WebRTC for IoT is not only its convenience and interopera...
Michael Maximilien, better known as max or Dr. Max, is a computer scientist with IBM. At IBM Research Triangle Park, he was a principal engineer for the worldwide industry point-of-sale standard: JavaPOS. At IBM Research, some highlights include pioneering research on semantic Web services, mashups, and cloud computing, and platform-as-a-service. He joined the IBM Cloud Labs in 2014 and works closely with Pivotal Inc., to help make the Cloud Found the best PaaS.
Everything run by electricity will eventually be connected to the Internet. Get ahead of the Internet of Things revolution. In his session at @ThingsExpo, Akvelon expert and IoT industry leader Sergey Grebnov provided an educational dive into the world of managing your home, workplace and all the devices they contain with the power of machine-based AI and intelligent Bot services for a completely streamlined experience.
Cloud-enabled transformation has evolved from cost saving measure to business innovation strategy -- one that combines the cloud with cognitive capabilities to drive market disruption. Learn how you can achieve the insight and agility you need to gain a competitive advantage. Industry-acclaimed CTO and cloud expert, Shankar Kalyana presents. Only the most exceptional IBMers are appointed with the rare distinction of IBM Fellow, the highest technical honor in the company. Shankar has also receive...