Partner Eco System

Devops Capability Assessment

Digital transformation often requires skills across multiple roles and departments. This DevOps assessment questionnaire will walk you through a series of questions to evaluate your company’s readiness to adopt continuous processes throughout.

Please take a moment to complete the questionnaire

Describe how your team plans, prioritizes, and schedules work.

An Agile development team typically operates from a backlog of requirements, often referred to as user stories. The backlog is prioritized to ensure that the most critical user stories are at the top. The product owner is responsible for managing the backlog, which includes adding, modifying, and reordering user stories based on customer needs. One of the most significant impediments to an Agile team's productivity is an ill-defined backlog. To consistently deliver high-quality software each sprint, it's essential to have well-defined requirements. <br> By completing features in accordance with the requirement specifications*

Within time-boxes, such as sprints or iterations lasting 1-4 weeks

Our goal is to achieve a continuous, one-piece flow of product backlog items, prioritized by their stack rank

We formulate measurable experiments based on hypotheses and utilize production data to evaluate the outcomes

We use other tools for planning

None of the above

Describe the methodology is used to estimate work.

Estimating in Agile software development differs from estimating in traditional IT delivery. Agile teams estimate the workload for the upcoming sprint and, potentially, a portion of the backlog for future sprints.In contrast, sequential IT delivery teams seek to provide precise estimates for the entire project, which may span many months or even years.*

T-shirt sizing

Planning poker

Dot voting

Story points

Story Counting

Swim Lane Sizing

None of the above

Can you describe how much work - equivalent to the number of sprints - is present on your backlog or to-do list?

During sprint planning, the team selects backlog items to include in the upcoming sprint based on priority and their estimation of what can be completed during that sprint. Breaking down these items into smaller tasks helps the sprint team better prioritize their work.*

Undefined, or unclear

A couple of sprints

More than a couple of sprints but less than three month's worth of work

Three month's worth of work or more but no more than six months

Up to a year's worth of work

More than a year's worth of work

Can you describe how many products your team is currently working on simultaneously?

Managing too many projects simultaneously can result in decreased morale among lower-level teams. When resources are stretched thin, they spend more time 'context switching' between projects, which significantly impacts their productivity.*

A couple of sprints

More than a couple of sprints but less than three month's worth of work

Three month's worth of work or more but no more than six months

Up to a year's worth of work

More than a year's worth of work

Can you describe how many products your team is currently working on simultaneously?

We work exclusively on one product.

We have a few things we are working on.

We work on dozens of products.

There are too many to count.

How do members of the team collaborate?

In your DevOps journey, inadequate communication can result in unfavorable outcomes. It is essential to maintain regular communication and collaboration within your team. Consider conducting daily stand-up meetings and leveraging tools such as work items, ChatOps, and other mechanisms for effective collaboration.*

We hold daily stand-up meetings and collaborate using work items, ChatOps, and other mechanisms.

We use e-mail or instant messaging for communication as needed.

We opt for weekly calls to ensure that we stay updated on our project.

Our teams operate in silos, and communication primarily occurs when issues arise.

None of the above

How does the team gather and measure feedback from customers and users?

Collecting feedback from your end-users and incorporating it into your development process is crucial for providing an excellent customer experience. A team adds value to the customer when they receive feedback and then adjust their backlog according to that feedback.*

Manual testing, user acceptance testing, or general acceptance criteria.

Direct email, phone, or web contact forms, web meetings, or chat.

In-product qualitative feedback (e.g., Send-a-Smile) and surveys such as Net Promoter Score (NPS).

We maintain ongoing engagement with our stakeholders, capturing and responding to feedback, as well as transforming and implementing new ideas into functionalities within our applications.

None of the above

How do coordinate or split work up across teams?

Agile assumes that each team possesses all the necessary resources to deliver an increment of work independently. This involves creating cross-functional teams equipped with the knowledge and permissions required to build, test, and deploy a product from start to finish with minimal external interaction, which may not be achievable by a single team alone.*

We have a Project Management Office that handles it.

We utilize a "Scrum of Scrums" approach.

We conduct demos for other teams working on components that interface with our system.

We maintain a Portfolio Board.

We have an open office space, ensuring that everyone is kept in the loop.

We regularly share our backlog with other teams to keep them informed.

None of the above

Describe how your team collaborates, shares risks, innovates, and learns.

In Agile, there are several effective methods for sharing project information with your team. Not having a structured way to share information with the team can result in poor collaboration, increased project risks, and reduced innovattion. Encouraging collaboration and consistently practicing it tends to lead to the generation of beter ideas, greater buy-in, and improved outcomes.*

We isolate ourselves to avoid being held responsible for failures, and risk-taking is discouraged.

There is some communication. When mistakes occur, the responsible person is held accountable.

Communication and collaboration are strong among teams, and innovation is actively encouraged.

There is high cooperation among teams. Risks are shared, and failure leads to learning and inquiry to help the team improve. Innovation is not only encouraged but also implemented.

None of the above

Describe how your team handles risk with a project.

Risk is defined as uncertainty in a project. Agile projects tend to have fewer quality risks compared to traditional projects due to iterative cycles that aim to mitigate risks continuously. In an Agile team, anyone can identify risks and propose mitigation strategies.*

Prepare a clearly detailed project scope.

Define requirement specifications and acceptance criteria.

Obtain sign-off from stakeholders on the scope agreement.

Establish and implement a change control process that stakeholders are aware of and agree to.

Develop and implement a change control process that the stakeholders are aware of and agree to

Monitor project progress against the defined scope. For example: By holding regular review meetings. (based on the needs and priorities of the team)

None of the above

Describe how your team regularly comes together to share and evaluate new ideas.

In an agile team, sharing ideas is an integral part of the iterative process. By making ideas known to the team, it fosters an environment of experimentation and innovation.*

We document the ideas and revisit them every few sprints to determine which ones to pursue.

We try to allocate some time in the upcoming sprint to explore certain ideas that have been suggested.

We can engage in discussions with the team, and for promising ideas, we follow up promptly. This is why we have dedicated slack time in our schedule.

None of the above

Describe how your team manages code?

As teams scale, the number of stakeholders who depend on and contribute to codebases can quickly increase. Without a strategy in place to manage changes to their source code, teams expose themselves to significant risks, including ongoing confusion, errors, and decreased productivity*

Use a common code repository (for example: Git based repo) for source files

Use a common code repository (for example: Git based repo) for configuration files

Have a defined branching structure

Have a defined branching policy that is enforced daily

Manage branch and repository permissions

Utilize artificial intelligence solutions (such as GitHub Copilot) to boost Developer productivity

Enforce check-in comments and workflows for code commits

None of the above

Describe what files do you keep under source control?

A version control system tracks changes to a file or a set of files over me, allowing you to recall specific versions later. You should aim to place all application-related files under source control.*

Application source code

Application source code and tests

Application source code, tests and configuration files

All files used to build, test, configure, deploy, run and document our service

None of the above

Describe how your team merges changes into a shared branch?

In Git, each developer saves changes to their own local code repository. As a result, there can be many different changes in a single commit. Git provides tools for isolating changes and later merging them back together. Git repositories otien have one or more shared branches where developers collaborate and eventually contribute their changes. The top level shared branch is usually called "main".*

Use Pull Requests to pull code changes back into the parent branch

Directly merge changes into the parent branch without a pull request

Directly change source code in the shared branch

None of the above

Describe the branch strategy in place to control pull/push requests?

In Git, branches, which serve as lightweight pointers to work in progress, help manage this separation. Once the work created in a branch is completed, it can be merged back into the team's main (or trunk) branch. While there are various valid methods, it is essential to have a clearly defined strategy in place and ensure that it is understood.*

We keep new work in separate branches and merge it only when finished, usually within weeks or months

We have no branches and develop only in the trunk or master

We use branches that live for a few days at most and commit all changes to the master or trunk at least daily

We maintain traceability on code changes but treat branches as temporary by using topic branches that we squash-merge to master when the work is complete

None of the above

Describe how the team monitors and reviews code changes?

All code changes should be peer-reviewed and approved, regardless of the scope of the change, as even one-line code updates frequently introduce bugs to a system. Pick the option that represent the most common practices for your team or organization.*

We do not have a policy; however, some code changes may undergo review, and at times, the reviewer may also be the requestor

We review code changes through live peer reviews or pair programming

Some code is reviewed, and we capture the review history using a pull request workflow or its equivalent

We ensure that all code changes are peer-reviewed before being commited to the master or trunk through a pull request workflow, branch policies, or their equivalent, and we enforce branch policies

None of the above

Describe how your team deals with cross-team code dependencies?

Do you store compiled code in version control? Do you duplicate components? Do you have inefficient processes for managing dependencies? If the answer is 'yes,' you should consider sharing and reusing code across the organization using packages.*

We track work items in a spreadsheet or document

We use a work item tracking system

The dependencies are published as packages to a repository, so we can track the dependencies in our code. We publish our own artifacts as well so other teams that depend on us can see the new artifacts.

We have full access to the source code of other teams and can build our own version of the dependencies if necessary (inner source)

We have an open-source model, so anyone can contribute and use our code as well as we can use other team's code

We use an ad-hoc or manual process for communicating dependencies.

None of the above

Describe how your team performs static code analysis?

Static analysis tools enable you to enforce coding standards, identify common errors, and address failures to follow best practices. Many static analysis tools can also be extended to implement custom rules and enforce your own policies. The following products are examples of static code analysis tools: SonarQube, SonarCloud, ESLint, FindBugs*

Static code analysis is performed early in development, before software testing begins.

Static code analysis is performed later in development, after software testing begins.

Static code analysis is performed after development, before deployment begins.

None of the above

Continuous Integration and Continuous Delivery (CI/CD)

Describe how your team gets notified about the build status and failures?

Sometimes failures happen, so it is important to ensure you have visibility into your pipeline. Receive a notification from the build system. Notifications may be delivered via a communication app such as Teams, SMS, or e-mail*

Receive alerts through IT service management tools

Receive alerts from DevOps build system directly

Teams review the status manually on a regular basis

None of the above

Continuous Integration and Continuous Delivery (CI/CD)

Describe how frequently your team refactors code or addresses technical debt?

Code refactoring is defined as the process of restructuring computer code without changing or adding to its external behavior and functionality. The primary benefit of refactoring is to enhance the quality of code and reduce technical debt.*

We do this when we have time

We plan to do this once every quarter

We use stabilization sprints to perform this kind of work

We continually do this

We do this on a set weekly interval

We do not refactor code in our team / A separate team handles it

Continuous Integration and Continuous Delivery (CI/CD)

Describe how your team release/deployment process works?

The entire end-to-end CI/CD deployment process should be understood, and ideally, as much of the deployment process as possible should be automated.*

Use environments for development and testing stages that simulate the production environmentMaintaining production-like environments for your pre-production stages means that if issues occur in production, you are far more capable of developing, testing, and releasing hotfixes/updates quickly.

Manage and actively resolve any technical debt in your environments technical debt includes anything that increases the complexity of the database workload or code, usually incurred due to delivery pressures.

There is a systematic approach to the development and release process The use of source code control systems, such as Azure Repos or GitHub, and build and release systems, such as Azure Pipelines or GitHub Actions, should be understood, including the corresponding processes for accessing, reviewing, and approving changes.

The application can be deployed automatically from scratch without any manual operations Manual deployment steps introduce significant risks, where human error is concerned and also increase overall deployment times.

There is a documented process for any portions of the deployment that require manual intervention Without detailed release process documentation, there is a much higher risk of an operator improperly configuring settings for the application.

N-1 or N+1 versions can be deployed via automated pipelines, where N is the current deployment version in production N-1 and N+1 refer to roll-back and roll-forward. Automated deployment pipelines should allow for quick roll-forward and roll-back deployments to address critical bugs and code updates outside of the normal deployment lifecycle.

The application deployment process leverages blue-green deployments and/or canary release Blue/green or canary deployments are ways to gradually release new features or changes without impacting all users at once.

Releases to production are gated by having it successfully deployed and tested in other environments Deploying to other environments and verifying changes before going into production can prevent bugs from reaching end-users.

Feature flags are used to test features before rolling them out to everyone To test new features quickly and with lower risk, feature flags are a technique to help frequently integrate code into a shared repository, even if incomplete. It allows for deployment decisions to be separated from release decisions.

None of the above

Continuous Integration and Continuous Delivery (CI/CD)

Choose the deployment strategy that your team uses. If you use several, choose the one that is most frequently used.

To ensure that the software connects with the operations system effectively, business goals must align with the DevOps deployment. By merging software development and IT operations, an effective DevOps deployment approach can achieve this.*

Blue/Green- Version B is released alongside version A, then the traffic is switched to version B.

Canary - The rollout takes place in a set of adoption "rings" or "groups".

A/B testing - The amount of users seeing version B is increased over time until the rollout is complete.

Rolling - Version B is slowly rolled out and replacing version A.

Recreate - Version A is terminated then version B is rolled out.

Shadow - Version B receives real-world traffic alongside version A and doesn’t impact the response.

None of the above

Continuous Integration and Continuous Delivery (CI/CD)

Describe the approval process for production deployments?

The quality signals used to approve deployments should be constantly tracked every day. While it's not critical for every service to always be available, teams should approach their DevOps delivery and operation stages with the mindset that they can and should deploy new versions without having to take them down at any time. Please choose the option that most closely reflects the most common method used in your team or organization.*

The approval process is infrequent or manual

The decision from the program/project office serves as the approval for deployment

We hold meetings before the deployment that establish our approval

We utilize gates and team approvals to manage our deployment

None of the above

Continuous Integration and Continuous Delivery (CI/CD)

Describe how your team rolls out new features to users?

Feature flagging allows DevOps teams to make new capacity available to only a small segment of application users as organizations roll out new features. Assuming all goes well, the DevOps teams can confidently make that functionality available to all users.*

We deploy when the appropriate manager approves

We run a full test suite in a production-realistic environment and proceed with deployment after obtaining any necessary approvals

We use progressive deployment with feature flags or exposure control and monitor telemetry to determine whether to continue the rollout

All changes get deployed continuously but we use feature flags to control which users have access to new features in production

None of the above

Continuous Integration and Continuous Delivery (CI/CD)

Describe how your team is using feature flags to test features before rolling them out to all users?

"Feature flags" (or "feature toggles") comprise another technique used for experimentation and "dark launches." Feature flags can toggle features on or off for different end-users based on their identity and group membership. Feature flags assist developers with code branch management, testing in production, and selective activation.*

Developers and testers test new features and functionality

Use for creative purposes, such as A/B testing

Using for progressive experimentation

Beta Feedback and Feature Targeting

Hypothesis-Driven Development

None of the above / Feature Flags not implemented

Continuous Integration and Continuous Delivery (CI/CD)

Describe your team's standard for testing?

If your team is atempting to use DevOps to deliver sotiware frequently into production, continuous testing is vital. Continuous testing, when done correctly, provides rapid and continuous insight into the health of the most recent build of your application.*

Testing that validates conformance to requirements and addresses identified bugs

Testing that is both fast and thorough enough to provide feedback in the continuous integration and continuous delivery flow

Testing that fosters continuous improvement in user satisfaction, quality of service, and performance on business objectives

Testing that covers the most used functionality, data, code paths, and configurations, with coverage measured against them

None of the above

Continuous Integration and Continuous Delivery (CI/CD)

Describe how your team runs tests on a built code package?

While most testing should occur within the testing and staging environments, it's beneficial to run a subset of tests against the production system.*

Perform regular disaster recovery (DR) and business continuity drills. Are you testing your high availability (HA) and DR processes to ensure your workload is available in the case of a disaster?

Automate testing for code and environment configuration deployments. Automating tests is the best way to ensure that they are repeatable, comprehensive, and exhaustive.

We use manual testing when necessary. Some processes can't be automated. How do you ensure that those manual processes are still adequately tested?

We create unit tests during development and run them as part of our build process.

None of the above

Continuous Integration and Continuous Delivery (CI/CD)

Describe how your team is testing and validating your applications?

Regular testing should be conducted as a part of every major change and, if possible, on a recurring basis to validate existing thresholds, targets, and assumptions. This also ensures the validity of the health model, capacity model, and operational procedures.*

The application is tested for performance, scalability, and resiliency

At least a subset of tests is also performed in the production environment

Tests for performance, scalability, and resiliency are conducted as part of each major change

Fault injection tests are being utilized

Smoke tests are performed during application deployments

Unit and integration testing is part of the application deployment process

All these tests are automated and performed periodically

Failing tests at least temporarily block a deployment and lead to a deeper analysis of the issue

Business Continuity 'fire drills' are executed to test regional failover scenarios

Regular security and penetration testing is performed

Tests are regularly updated and validated to reflect any necessary changesTests and test data should be reviewed and updated after each major application change, update, or outage.

Mocks and stubs are used to test external dependencies in non-production environmentsMocks and stubs can help validate external dependencies and increase test coverage when accessing those external dependencies is not possible, for example, due to IP restrictions.

None of the above

Continuous Integration and Continuous Delivery (CI/CD)

Describe how your team uses tests in environments other than production?

Software testing is not a new concept, but testing in a traditional environment differs significantly from testing in a DevOps environment*

We test all environments with unit, acceptance, functional, and integration tests

We use a continuous and automated process that enables ongoing testing of all our environments

We leverage test automation as much as possible

We continually revise the testing plan with an up-to-date automated testing scope

None of the above

Continuous Integration and Continuous Delivery (CI/CD)

Describe the code scanning tools used as part of the continuous integration (CI) process?

Make sure that static code scanning tools are integrated into the continuous integration (CI) process.*

Credential/Secret Scanner

Dependency Scanning

Source code analysis toolssuch as: OWASP scanners or CodeQL

Vulnerability Scanning Tools

SAST Tools

None of the above

Continuous Integration and Continuous Delivery (CI/CD)

Describe when your team conducts security testing.

Establish a minimum-security baseline that takes into account both security and compliance controls. Ensure that these controls are integrated into the DevOps process and pipeline.*

As part of the pull request and code review flow before code is committed.

Security tests are run In the continuous integration and continuous delivery pipeline.

Before any production update or Before a major release.

None of the above

How extensively is security testing conducted, and what is your team's approach to incident response in the event of a breach?

Conducting security and penetration tests, like checking for open ports or potential vulnerabilities, is crucial to ensure the overall safety and smooth operation of the system. In case prevention measures fall short, responding promptly and mitigating the security breach can help reduce harm and limit the impact within a confined area.*

Penetration testing is conducted either internally or by a third-party entity to assess the existing security defenses of this workload and ensure their validation.

Regular simulated atracks, including phishing campaigns, are executed against users of this workload to enhance preparedness and resilience.

A comprehensive security training program is established and consistently updated to keep the security personnel associated with this workload well-informed and equipped with the necessary skills.

Incident response playbooks are crafted to facilitate a swiftunderstanding of the workload and its components, enabling incident responders to efficiently mitigate atacks and conduct investigations.

None of the above.

What code scanning tools are utilized in the continuous integration (CI) workflow?

Make sure that static code scanning tools are seamlessly integrated into the continuous integration (CI) process.*

Credential Scanner

Dependency Scanner

OWASP source code analysis tools

Vulnerability Scanning Tools

SAST Tools

None of the above

How does your team make sure that there is enough of a focus on security while writing code?

Secure coding involves the creation of software that mitigates security-related vulnerabilities by adhering to industry best practices and security standards, thereby avoiding potential flaws in the development process.*

Robust authentication and authorization measures are implemented to ensure data security against unauthorized access. Examples of this are: enforcing strong password policies and incorporating twofactor authentication.

To prevent data leakage during the transmission process between sources, secure channels (HTTPS) are implemented, along with the encryption of data in transit.

Sensitive data is encrypted at rest.

Access is restricted to authenticated users only and authentication is implemented at every layer of the workload.

All communication channels are encrypted to safeguard authentication tokens.

All keys, passwords, and certificates are properly stored and protected. Proper storage and protection of keys, passwords, and certificates are imperative to maintain the integrity of the security measures in place.

Tools that support secure development (such as GitHub Advanced Security and GitHub Copilot) are used throughout the inner and outer development loops.

None of the above

How frequently does your team conduct security assessments on the code?

Security holds utmost importance in all applications; thus, it is essential to incorporate frequent security assessment activities into your development lifecycle.*

Security reviews on code changes occur sporadically.

Certain code undergoes security scrutiny, and we document the review history through a pull request workflow or a comparable method.

We ensure that all modifications to the code undergo security peer-review before being commited to the master or trunk, utilizing a pull request workflow, branch policies, or equivalent mechanisms.

None of the above

How does your team handle the management of passwords and other credentials?

It is essential to perform scans for credentials and other sensitive information in source files before commiting, as this practice minimizes the risk of unintentionally including such confidential data in your team's CI/CD process.*

Uncover and categorize all forms of passwords, keys, and other confidential information throughout your entire IT infrastructure, centralizing their management. Continuously identify and integrate new secrets into the centralized system as they are generated.

Remove hardcoded or embedded secrets from configurations in DevOps tools, build scripts, code files, test builds, production builds, and applications.

Enforce best practices for password security, covering aspects like password length, complexity, uniqueness, expiration, rotation, and more, across all categories of passwords.

Implement privileged session monitoring to record, audit, and supervise all privileged sessions, including accounts, users, scripts, and automation tools, enhancing oversight and accountability.

Expand secrets management to include third-party entities, ensuring that partners and vendors adhere to best practices when using and handling secrets.

Utilize threat analytics to continually assess the usage of secrets, detecting anomalies and potential threats.

None of the above

Explain the approach your team takes in conducting threat analysis.

Threat modeling, as an engineering approach, aids in recognizing potential threats, atacks, vulnerabilities, and corresponding countermeasures that might impact an application. The steps involved in threat analysis encompass defining security requirements, recognizing threats, implementing measures to mitigate threats, and validating the effectiveness of threat mitigation. These elements collectively contribute to ensuring comprehensive security for a workload, covering both proactive prevention and reactive responses.*

The organization follows threat modeling processes wherein identified threats are prioritized according to their impact on the organization. These threats are then associated with corresponding mitigations and communicated to relevant stakeholders.Threat modeling, as an engineering technique, serves to identify potential threats, atacks, vulnerabilities, and countermeasures that might impact an application. The process involves defining security requirements, recognizing threats, implementing threat mitigations, and validating the effectiveness of these measures. Microsoft employs the STRIDE model for threat modeling, and tools like the Microsoft Threat Modeling Tool can be valuable in this context.

There is a systematic procedure for monitoring, prioritizing, and resolving security threats throughout the application development lifecycle.A threat modeling tool generates a comprehensive report listing all identified threats. Typically, this report is uploaded into a tracking tool or work items, which developers then validate and address. Cybersecurity teams leverage the report to identify atack vectors during penetration tests. As the solution evolves with new features, the threat model should be regularly updated and seamlessly integrated into the code management process. In the event of discovering a security issue, a structured process is in place to triage the problem either into the next release cycle or a expedited release, depending on its severity

Establishing timelines and procedures is essential for deploying security fixes to address identified threats.Promptly addressing identified vulnerabilities is crucial for maintaining security and preventing the emergence of additional attack vectors.

Clear security requirements are outlined for all applications.Applications failing to meet the specified security requirements during service enablement should be restricted.

Comprehensive threat protection measures have been implemented for all applications.Enterprise applications face numerous threats that can compromise confidentiality, availability, or integrity, necessitating the use of advanced security solutions for protection.

Business-critical workloads are identified and categorized appropriately. This is essential as their compromise or unavailability could significantly impact operations.Enterprise organizations commonly possess an extensive application portfolio. Have key business applications been recognized and categorized? This encompasses applications with substantial business implications if disrupted, such as those involving critical data, regulated information, or essential availability. Additionally, these applications may encompass those particularly vulnerable to atacks, such as pivotal public-facing websites crucial to the organization's success.

None of the above

What factors related to compliance and governance are taken into account when building apps?

Organizations face limitations in terms of resources, including finances, personnel, and time. To ensure a meaningful return on investment (ROI), organizations must prioritize their investments wisely. This constraint also applies to the implementation of security measures throughout the organization. To atain a suitable ROI on security, it is crucial for the organization to comprehend and clearly define its security priorities.*

Regulatory and governance requirements are well understood and implemented.Operational data, such as application logs and metrics, may be required by regulatory mandates to remain within specific geopolitical regions. This imposes clear implications on the operationalization approach for the application.

The Landing Zone concept is applied to workloads.This concept entails predefined components, like a network topology with the Hub/Spoke model, already in place before workload deployment. The purpose of the Landing Zone is to ensure that necessary infrastructure is established upon the workload's arrival in Azure, enhancing agility and compliance with enterprise security and governance standards. It is essential to hand over a Landing Zone to workload owners with deployed security guardrails

Security and organizational standards are enforced and controlled through policies.

A Root Management Group is utilized, and any modifications applied using this group undergo careful consideration.

Systematic monitoring and maintenance of compliance for applications are conducted, with regular compliance atestations.

Periodic external or internal audits of this workload are conducted.

A security plan for this workload was formulated and is actively maintained.

Proactive review and application of best practices and guidelines, based on industry recommendations, are undertaken.

When implementing defenses, considerations are given to the costs associated for atackers and defenders. Preventing accessible and inexpensive attack methods is always prioritized.

None of the above

Describe how your team monitors applications?

In order to successfully maintain applications, it's important to 'turn the lights on' and have clear visibility of important metrics both in real-time and historically.*

An Application Performance Management (APM) tool like Azure Application Insights is used to collect application level logsIs an Application Performance Management (APM) tool used to collect application-level logs?

Application logs are collected from different application environments. Are application logs collected from different application environments?

Log messages are captured in a structured format and can be indexed and searched Are log messages captured in a structured format?

Application events are correlated across all application components Events coming from different application components or different component tiers of the application should be correlated to build end-to-end transaction flows. For instance, this is often achieved by using consistent correlation IDs transferred between components within a transaction.

It is possible to evaluate critical application performance targets and non-functional requirements based on application logs and metrics Is it possible to evaluate critical application performance targets and non-functional requirements (NFRs)?

End-to-end performance of critical system flows is monitored. Is the end-to-end performance of critical system flows monitored?

Black-box monitoring is used to measure platform services and the resulting customer experience. Blackbox monitoring tests externally visible application behavior without the knowledge of the internals of the system. This is a common approach to measuring customer-centric SLIs/SLOs/SLAs.

None of the above

Describe how your team visualizes application data and then alert relevant teams when issues occur?

Dashboarding tools should be used to visualize metrics and events collected at the application and resource levels to illustrate the health model and the current operational state of the workload.*

Application and resource level logs are either aggregated in a single data sink, or it is possible to crossquery events at both levels Are application and resource level logs aggregated in a single data sink, or is it possible to cross-query events at both levels?

Application level events are automatically correlated with resource-level metrics to quantify the current application stateAre application-level events automatically correlated with resource-level metrics to quantify the current application state?

A health model is used to qualify what 'healthy' and 'unhealthy' states represent for the workloadIs a health model used to qualify what 'healthy' and 'unhealthy' states represent for the application?

Critical system flows are used to inform the health modelAre critical system flows used to inform the health model?

The health model can distinguish between transient and non-transient faults Can the health model distinguish between transient and non-transient faults?

Long-term trends are analyzed to predict operational issues before they occur Are long-term trends analyzed to predict operational issues before they occur?

Retention times for logs and metrics have been defined and with housekeeping mechanisms configured Have retention times for logs and metrics been defined and with housekeeping mechanisms configured?

None of the above

Describe how does your team get insights into metrics of production systems?

We don’t, Ops will contact us when something is wrong*

We receive a monthly report detailing availability and average response times and trends

We receive a daily report detailing workload with availability, response times, numbers of requests and load on the systems

Ops has published a live metrics dashboard for us and sends us a roll up report with day-to-day comparison

We have full self service access to live and aggregated data through a portal

None of the above

Describe the instrumentation your team uses to monitor applications running in production?

Availability monitoring via pinging or synthetic transactions*

Performance monitoring of the servers and infrastructure

Real user monitoring

Multi-tier views of end-to-end transactions including processing and network combined

None of the above

Describe how your team interprets the collected data to inform relevant teams about application health?

Log aggregation technologies should be used to collate logs and metrics across all application components for later evaluation. Resources that logs are captured for may include Azure IaaS and PaaS services as well as 3rd-party appliances such as firewalls or anti-malware solutions used in the application.*

A log aggregation technology, such as Azure Log Analytics or Splunk, is used to collect logs and metrics from Azure resourcesWhich log aggregation technology is used to collect logs and metrics from Azure resources?

Azure Activity Logs are collected within the log aggregation tool Are you collecting Azure Activity Logs within the log aggregation tool?

Resource-level monitoring is enforced throughout the application Is resource-level monitoring enforced throughout the application?

Logs and metrics are available for critical internal dependencies Are logs and metrics available for critical internal dependencies?

Log levels are used to capture different types of application events.Different log levels, such as INFO, WARNING, ERROR, and DEBUG should be pre-configured and applied within relevant environments. The approach to change log levels should be simple configuration change to support operational scenarios where it is necessary to elevate the log level within an environment.

Critical external dependencies are monitored It's common for applications to depend on other services or libraries. Despite these are external, it is sell possible to monitor their health and availability using probes

There are no known gaps in application observability that led to missed incidents and/or false positives. What you cannot see, you cannot measure. What you cannot measure, you cannot improve.

The application is instrumented to measure customer experience. Effective instrumentation is vital to detecting and resolving performance anomalies that can impact customer experience and application availability.

None of the above

Describe how your team uses tools to be notified and alerted about application health?

Alerts can proactively notify or respond to operational states that deviate from the norm. Alerts can also enable cost-awareness by watching budgets and limits and helping workload teams to scale appropriately*

A tool such as Azure Monitor or Grafana is used to visualize the application health model and encompassed logs and metricsWhat technology in used to visualize the application health model and encompassed logs and metrics?

Dashboards are tailored to a specific audience such as developers, security or networking teams

A tool such as Azure Monitor or Splunk is used for alerting

Specific owners and processes are defined for each alert type

Operational events are prioritized based on business impact

Alerting is integrated with an IT Service Management (ITSM) system such as Service Now

Push notifications are used to inform responsible parties of alerts in real time

None of the above

Describe how your team's applications are engineered towards providing information for operations? (logging, instrumentation)

Effective application instrumentation is an important practice to help identify and understand possible issues. Efficient instrumentation of your environment and applications enable engineering and operations teams to identify and resolve potential issues more efficiently.*

We implement custom counters in our production code that stream service health, up-time, and code invocation and execution to a central repository

We poll our production system at regular intervals with a monitoring solution such as IpSwitch, SolarWinds, Quest, etc... and log the data to a central repository

We use AGENTLESS technology to conduct PING tests and verification of services. We log this data to a central repository that allows us to build and visualize trends over time

We use AGENT technology to conduct PING tests and verification of services. We log this data to a central repository that allows us to build and visualize trends over time

None of the above

Which of these DevOps practices are commonplace in your organization or team?

The merging of "Dev" and "Ops" entails the integration of previously isolated Development and Operations functions, resulting in the formation of cross-functional teams that collaborate using common and effective practices and tools. While a genuine DevOps approach assigns operational responsibilities to developers, some organizations choose not to fully adopt DevOps, opting instead to maintain a certain level of team segregation between operations and development. This separation may serve to uphold a clear division of duties in regulated environments or to treat operations as a distinct business function.*

Our change management process involves stakeholders from all aspects of the software development lifecycle.

Stakeholders in different roles working on the same product communicate frequently and efficiently.

The organization operates platforms and services for cross functional teams to consume.

When we build a workload or application we understand that any new product we create will be a service that needs long-term maintenance and investment.

We seek a diversity of views when it comes to IT delivery

Our mission statement is aligned across all teams that participate in the software development lifecycle. This also includes traditional "business"-functions.

None of the above

Describe how frequently does your team tries new ideas?

New Ideas breed innovation and trust among teams. Look to drive experimentation into your team's culture to improve on new ideas.*

All the time

Frequently

Occasionally

Periodically

Not very much. / Not at all.

Explain how your team establishes and maintains a unified and clear vision and direction

For DevOps to thrive, it's essential for both technical and non-technical teams to synchronize. Nontechnical stakeholders must actively engage, working alongside engineering teams to define realistic KPIs and SLAs. Meanwhile, technical teams should develop systems and processes that enhance software delivery efficiency and offer effective tools for conveying the value that DevOps brings to the organization. Choose the option that most closely applies to your organization.*

The overall direction is somewhat apparent but could benefit from further clarification or a clearer vision statement.

There may be a vision statement, but it may not be very broadly publicized. Not all team members are always aware of how their work contributes to a greater goal.

While there is a single vision statement, we have some more alignment work to do.

While a vision statement exists, it's not always clear how all work aligns to it.

Our organization strongly supports and aligns with a single clear vision.

Individuals can express what the mission is and how their work contributes to the vision.

There is little or no overall direction/vision.

How does your organization perceive culture change, considering it as either an ongoing, continuous effort or as a series of event-based initiatives?*

Consistently and openly monitoring your DevOps process enables teams to pinpoint potential areas for improvement. Choose the option that most closely aligns to your team's or organization's situation.

Our organization embraces DevOps throughout and is seeing benefits from having adopted a DevOps model.

While we have implemented many DevOps culture changes, we still have more work to do.

We embraced some DevOps culture changes but we have a long way to go and further changes to make.

Our organization has not embraced DevOps culture changes and our culture needs to change.

Explain how your team communicates and shares risks with other departments within the organization

DevOps involves not only experimentation but also fostering a culture of transparency, sharing, and collaboration. Inherent to experimentation are risks, and sharing these risks ensures that everyone is actively involved. This shared risk-taking fosters a deeper understanding between the business and IT, aligning their goals and facilitating collaboration.*

Our team jointly bears risks with various business teams.

Our team bears certain risks alongside different business teams.

Our team refrains from sharing risks with different business teams.

Does your team allocate time for innovation and learning activities?

Learning and innovation are interconnected. Without both happening, the organization becomes stagnant and vulnerable to competition and employee turnover.*

We promote a mindset of continuous growth.

We embrace failure as an opportunity for learning and integrate insights into our processes.

Employees have time to develop experimental projects to foster innovation.

Team members take a data-driven development approach.

None of the above

How seamless is the transition for your team to embrace the next innovation after facing a setback with a previous one?

The desire to learn can serve as a powerful motivator for many individuals. One of the key factors driving the team to embrace a DevOps mindset is their curiosity to consistently learn, adapt, and enhance their skills and knowledge through experiences, including failures. Review the below scenarios and pick the one that is most common in your team or organization.*

Consequences ensued, prompting us to return to the program.An example of this might be layoffs that are directly connected to a momentary setback.

We encounter significant resistance and are reminded of previous mishaps.

We face some resistance and are advised to improve in future attempts.

Presenting lessons learned allows us to receive the benefit of the doubt.

We are motivated to explore the impact through non-fatal experiments.

Partner Eco System

Schedule a consultation

Feel free to shoot us a message anytime!

Oh j uh s v ee

USA OFFICE

INDIA OFFICE