IT:AD:Continuous Delivery:SAD:02 Solution Summary

The scope of this solution includes and excludes the following:

The scope of this project includes:

• The investment into using Visual Studio Team Services as the Organisation's default ALM Service.
  • This choice implies the stopping of the use of Testing Services JIRA for new green-field PaaS based services.
• The development of a custom Build Step Extension hosted in the Visual Studio Marketplace to provide ministry projects a cost effective way of implementing Automated Quality Testing, and produce a Continuous Accreditation Report of use to Accreditation Services.

At this point in time the scope of the Solution described within the rest of this Section and the rest of the Solution Architecture Description (SAD) does are the following Goals, Services or Systems.

Excluded Goals: * Reengineer current and legacy applications to be manageable via an ALM Service. Specifically, the following actions are deemed to be too costly to be of benefit with any current legacy applications:

• Reengineer to be hostable on cloud based PaaS infrastructure, or – less drastically – move to cloud based IaaS infrastructure, while
  • updating the systems to be secure, independent of any on-premise network boundary security they may have relied upon previously,
  • uncouple any cross-system tight coupling – eg: ETL operations – that are non best practices for cloud hosted development.
• Migrate to an ALM Service, including:
  • building an automated development pipeline for legacy applications
  • automating all manual test scripts,
  • migrating management artefacts (requirements, work items tracking, issue tracking, reports etc.)

Key Goals of the Solution, based on Assessments that are listed later in this section, are listed below:

• Faster Responsiveness: automation of quality testing can ensure artefacts are always kept deployment ready.
• Better informed Planning:
• Minimize Non-Productive Work:

Supporting Goals are directly related to the above Goals:

• Iterative Delivery: providing tools for managing Agile iterative deliveries allows planned effort to be regularly reassessed and optionally re-prioritized to deliver value to stakeholders.
• Improve Communication: an Agile communication system groups can replace time consuming processes, consensus and scheduling meetings.
• Improve Automation: automation can be used to replace time consuming building, testing, deployment and documentation manual operations.
• Improve Quality: automation can be used to test more, regularly.
• Empower Stakeholders: provides better tools, more transparency, more options
• Appropriate Security: providing a more flexible environment to host services transparently without sacrificing security when appropriate.

The above supporting Goals can be summarized as:

• Use Cloud Services for Infrastructure
• Implement DevOps Culture, Processes, Tooling
• Automating Auditing Processes appropriate to Cloud Infrastructure and DevOps

Key Concerns (internal and external Drivers relevant to Stakeholders) addressed by the above Goals are listed below.

As per most Organisations, the following Concerns are of importance to key Stakeholders:

• Reputation:

  • According to the World Quality Report 2015, “Corporate image is the number 1 executive concern when it comes to quality, demanding protection from software glitches that hit the news”¹⁵⁾¹⁶⁾

  * Customer Satisfaction: “Customer Experience or Customer Value is a key objective, and this is determined by speed, fit for purpose solutions and ease of use.”¹⁷⁾¹⁸⁾. “61% of organisations rate time to market as a very important part of their corporate strategy”¹⁹⁾.

• Prepareness and Adaptability: change is constant. Speed of change is not – it's increasing. Being ready, and able to adapt is a key organisational concern.
• Better Value for Money: the organisation, although a monopoly, does not have infinite resources. Delivering solutions more rapidly, at less cost is a Driver for all within the organisation.
• Strategic Alignment: strategies have been developed to meet the above concerns and are expected to be aligned to.

The above primary key Concerns encompass several secondary Concerns:

• IT Reputation:

  • improve the IT Group's reputation to being perceived by business services as trusted advisors who enable – rather than costly deployment blockers that cannot be worked around.
  • changing from manual time and cost inefficient operations to automated services where IT Service resources are Trusted Advisers on impact and impact improves the reputation of IT Services.

  * Cloud First: alignment with the Government's stated Cloud First objectives.

• Digital Strategy: alignment with the long term goals of the sector by the organisation, its departments and various projects is a driver that minimizes the cost to achieve agreed strategic goals.
  • The Digital Strategy states: “Smart tools and common IT systems make delivering service improvements and implementing policy changes simpler and less expensive than they used to be, freeing up investment to improve outcomes for students and educators.”
• Software as a Differentiator: it is nearly a decade since it was accepted that the use of software is a key service differentiator for most businesses. The use of more services, delivered initially more rapidly to stakeholders, and iteratively re-released with more functionality is a key strategic driver.
• Minimize Cost: Although savings can come from buying new solutions, savings start by not wasting time and expenses on cumbersome low value processes.
• Deliver Value: essential to the key concern of improving Reputation is delivering what was requested
• Deliver Quality: to be valuable, what is delivered must be tested to ensure it is not be defective.
• Innovate to Meet New Opportunities: the organisation, although a monopoly, is mandated to assess and take advantage of new opportunities, while remaining Principled.

Key Stakeholders impacted or driven by the above Concerns are:

• Deputy Secretary
  • Chief Financial Offier (CFO)
  • Chief Financial Performance Officer (CFPO)
  • Chief Information Offier (CIO)
  • Chief Security and Privacy Officer (CSPO)
  • Business Owner(s)
    • Product Owner(s)

A Stakeholder is anyone with a Concern in this Solution (eg: Development, Delivery, Change Management, Operations, Users, etc.) designed to address the Concerns of the previously defined Stakeholders.

• IT Group:
  • ICT Strategy, Planning and Architecture (SPA) Senior Manager
  • ICT Project Services Senior Manager
  • Chief Information Security Officer (CISO)
  • Customer Services Manager
    • Service Desk Manager
    • Training Services Manager
  • Operations Infrastructure Services Senior Manager
    • Infrastructure Services Manager
    • Application Support Services Manager
  • Web and Application Services Senior Manager
    • Web Services Manager
    • Development Services Manager
    • Testing Services Manager

The Goals listed above were developed in order to address SWOT Assessments that affected key Concerns of Stakeholders.

These Assessments are listed below.

An Assessment of current Weaknesses is as follows:

• Testing Services processes could be improved to deliver more value more rapidly:
  • Testing remains costly in terms of budget and time, as it is repeated – mostly manually – for each deployment. This cost and delay is a direct contributor to the reason project managers prefer to roll out new features on such large intervals.
  • Manual Functional Testing is commissioned from internal Testing Services near to the scheduled delivery date. The unexpected failure to satisfactorily pass manual UAT testing leads to subsequent negotiations, and potentially costly re-scheduling, resource engagement extensions, and subsequent re-testing.
  • According to the CapGemini report “DevOps with Quality”, “automation of the quality activities is not only required but it is the core enabler of increasing throughput and velocity”.
  • 49% of organisations complain that still largely manual testing phases are a bottleneck in speeding up the development cycle time [2015 World Quality Report].
  • According to Cap Gemini's “DevOps with Quality” report “Without maximizing automation, the speed at which a team can deploy features is limited by the speed at which the quality activities can be successfully completed. In other words, testing activities, done traditionally will become the constraining factor or the alternative is testing will not be done adequately, therefore putting the organisation reputation at risk.”
  • “After accelerating other aspects of the delivery pipeline, organisations typically find that traditional Testing and overall Quality and Accreditation processes remain problematic, preventing organisations from achieving the benefits of their expected SDLC acceleration initiative.”²⁰⁾. Reasons given by other Organisations include:
    • Initiatives to automate GUI testing have proved problematic, due to requiring frequent adjustments.
    • Traditional test script based testing can be developed without an understanding of what stakeholders considers to be an acceptable level of risk, leading to release candidates that pass all Testing Services, but stakeholders do not consider ready for release due to failing stakeholder's tolerance for risks.
    • Traditional test scripts often focus on business functionality requirements, relegating around security, performance, reliability, and compliance testing to out of band, and costly, specialists.
    • Even if testing is automated and effectively measures the level of stakeholder risk, teams without a coordinated end-to-end quality process tend to have trouble satisfying stakeholder expectations within today's compressed delivery cycles. Trying to remove risks at the end of each delivery iteration has been shown to be significantly slower and more resource-intensive than building quality into the product through defect prevention strategies such as development testing and subsequent continuous delivery pipeline testing²¹⁾.
• Infrastructure Services processes could be improved to deliver more value more rapidly:
  • Releases to production incorporating stakeholder feedback are few and far between, often in year-plus intervals, rather than fortnightly (see NSI).

    • Due to several factors (meetings, scarcity of resources, communication issues) commissioning of a new environment to host solutions can take weeks. This delay is largely repeated for each environment needed (ST, UAT, TEST, PROD).
    • Deployment operations are performed manually, and take on average a week due to meetings to schedule deployments, manual step documentation, manual deployment. This lengthy process is repeated for each deployment environment (ST, UAT, TRAIN, PROD).
      • Note: in partial defence of Infrastructure Services several of the cumbersome manual processes were put in place partially to compensate for the quality of the developed artefacts received. But note that that although code quality is a concern outside the control of Infrastructure Services, the solution should not have been manual.

    * Lengthy manual deployments (eg: 2 days) are disruptive to end users, or must be done for more cost after hours.

• Accreditation Services processes could be improved:
  • Accreditation Performance and Security Testing is commissioned from external resources near to the scheduled delivery date. The unexpected failure to satisfactorily pass these external reviews leads to costly re-scheduling, resource engagement extensions, and repeated costly re-testing. To my knowledge, to date there has never been a project that has passed these tests on their first pass.
  • Existing security practices are insufficient to control access to Confidential data by contractors. There is never a Use Case for development resources to access production data – yet there are several cases of whole databases of Confidential data being taken off site and even off country by developers, with no controls as to their subsequent disposal.
• Development Services and Development quality could be improved to deliver more value more quickly:
  • Code quality has impacted downstream. Several of the slow, manual processes implemented by Infrastructure Services were put in place to compensate for the quality of the artefacts received.

    • The maintainability of the code base the organisation must support – whether developed in house, or received from vendors – creates hiring difficulties.
    • Development Services have no peer or automated review of deliverables. Outmoded and inappropriate development patterns continue to be used and accepted causing long term accessibility, availability, security, modularity, infrastructure, support costs born by both IT Services and business users. Associated resource lock-in maintaining outmoded and inappropriate artefacts is another real liability to the organisation.
• General Tooling could be improved:
  • The current Issue Management System (JIRA) used by Test Services is IaaS based on internal infrastructure, and configured for use only by the Test Services (change requests for firewall rules are needed to allow temporary unmonitored remote access by vendors).
  • The Organisation does not have a common and continuous method of managing project work items, artefacts, deployments, testing and support, from inception through to decommissioning. Instead, artificial and management, process and physical environment boundaries have been put up between development, testing, and support. These boundaries add complexity and cost, while decreasing transparency and ability to respond to end user requests for value.
  • Without a common communication and resource planning service shared by all stakeholders, Business Project Managers decommission functionality without understanding the negative impact on other stakeholders Deployment, Configuration, Deployment, Monitoring, Support, Maintenance functions.
  • The organisation does not have have in place an automation platform that can be used both internally and by vendors over the duration of a project (often a decade).

    • The organisation does not yet use an iterative approach to delivering value – even when 61% of organisations rate time to market as a very important part of their corporate strategy.
    • contrary to Organisational Behaviours (We work together for maximum impact “Ka mahi ngātahi mo te tukinga nui tonu”) the ministry uses several different – silo-ed – systems to manage the development of User Stories, Acceptance Tests, Test Scripts, Issues and Documentation between groups and vendors. Transparency remains problematic:
    • Even when more current development patterns (eg: Service based UIs) and related test automation practices are added to existing test process, decision makers still lack adequate insight into the level of risk associated with releasing an application at any given point in time.
      

    The above issues in turn contribute to project scheduling and cost issues:

• Missed delivery dates cause costly negotiation meetings, re-planning, rescheduling, communication changes.
• Delivery expectations versus available resources and cost lead to compromises on security, privacy, features, performance, availability, usability and/or modularity(see Helios Project).
• Compromises on features, availability, usability and modularity at the cost expended negatively affect the reputation of the organisation.
• The cost of individual projects remains non-measured and unoptimised as costs of infrastructure to support applications is not directly attributable on a per project basis.
• Due to the gated and slow nature of the current delivery process, small errors cascade to large disruptions, scheduling, resource needs and associated costs to individual projects, the IT department and the Organisation as a whole.

The above issues in turn contribute to strategic issues:

• The slow process employed to request and get approval – followed with the delay and norm compliance required to spin up new servers – is so onerous that it is very rarely employed beyond absolutely necessity. The process stifle investigation of new products and development techniques, innovation and potential efficiency gains.

The following are a mostly set of Assessments of Opportunities obtained by other Organisations moving to a more efficient SDLC process

• ITIL is insufficient:

  • 77% respondents to Noel Bruton's 2004 survey either agreed or strongly agreed that “ITIL does not have all the answers”.
• DevOps provides higher value to stakeholders:
  • 88% of Organisations already have or are planning to adopt DevOps in the near future²²⁾, because

    • DevOps delivers 18% faster time to market²³⁾,
    • 19% better quality and performance²⁴⁾, and
    • addresses the issue that 49% of organisations complain about: that largely manual testing phases remain the bottleneck to speeding up development cycle times²⁵⁾.
    • https://www.rackspace.co.uk/sites/default/files/devops-automation-report.pdf

    Note:
    Many more Assessments of DevOps are available in the Appendices.

Although other organisations have reported benefits from moving to DevOps, there are several key points worth rememebering when planning to to move to using it: * Companies do not migrate their whole organisation to it in one go (less that 3% have applied all processes to all legacy and future apps). Most companies have taken a reduced risk approach, and deployed DevOps in a staged manner²⁶⁾, exactly as we are suggesting – using DevOps for new green field projects. it is important to note that it was reported that Automation of Testing needs careful design:

• 31% of organisations have difficulty to determine the right coverage of quality validation checks [2015 World Quality Report].

Beyond focusing on the Assessments that directly support the development and operations of Services which support the stated Goals, some additional recommendations can be made.

• Limit investment in legacy system infrastructure and tools to manage legacy processes and legacy system infrastructure.
• Document some of this organisations most costly current processes (see Appendix) in order to make more people aware of the poor practices we currently employ, in order that we don't repeat them in the future – or design new infrastructure environments so that similar poor practices become the norm again.

The Project's Goals are constrained by real and decided limitations.

Key Constraints and Decisions are listed below.

• Cloud First: it is a both Government and Ministry strategy that new systems be hosted in the Cloud.
• Being IaaS based, and not in alignment with either Government or Ministry strategy, no further CAPEX investment should be made in the current Test Service's JIRA service, or extensions thereof.
• Bimodal Deployment: the cost of rehosting existing applications in cloud infrastructure is too onerous to achieved successfully while meeting addressing specified Concerns and stated Goals.

The Project set Goals in a Principled manner, reflecting both the Organisation's Core Values the Project Desired Qualities.

The following requirements could be mapped from the stated Goals within the Motivation Content above:

The above key Requirements defined Services to meet the stated Goals.

These Services required to meet these requirements are: * Cloud Platform Services, including:

• PaaS Hosting Services: green field applications commissioned by this organisation should be commissioned to take advantage of the cost savings of PaaS infrastructure²⁷⁾.
• IaaS Hosting Services: to host existing 3rd party products that were not designed to take advantage of PaaS' cost savings, an IaaS infrastructure service is required.

* A SaaS based Agile/DevOps capable Work Item Management Service: an all-organisation, cross-team, service available from within the organisation, as well as external services (3rd party consultant, development, support services, etc).

• note: The system must specifically be capable of managing Acceptance Test Definitions as distinct work Items.

* Automated Artefact Build Service: an automated build service that can automate builds, and invoke the Automated Testing Service defined below.

• note: the Build Service is in charge of running the tests that are the basis of any automated accreditation report.

* Automated Delivery Service: an automated delivery service that can be triggered – either automatically or manually – to deploy artefacts to target environments. * Automated Testing Service: a service invoked by the Automation Build Service to automated the running of static tests, deploy to build environments for further dynamic testing, develop a report, and accept/reject code integration.

• Test Script Management Service: an management system for managing manual test scripts that cannot yet be automated (to be used as sparingly as feasible).

The Services defined above are provided by the following System.

• Enterprise-grade DevOps appropriate ALM Service providing all of the services defined earlier.
  * Azure Cloud Services.

The 3rd Party Services chosen to deliver the required Services are all market leaders in their respective fields.

Microsoft's Azure is a leader in the PaaS space:

Microsoft's Azure is a leader in the Cloud IaaS space:

Microsoft's Visual Studio Team Services is a leader in the ALM Service space:

The assessment of the ALM Service being a market leader remains consistent whether it is developed formally (Gardner) or based on a crowd assessment ²⁸⁾.

Of specific note considering historical choices by this organisation is Gartner's rating of the service as being more valuable than Atlassian's (developers of JIRA and Confluence) offerings in its Ability to Execute and Completeness of Vision.

In addition – as per findings outlined in the Accreditation View of this document – VSTS has achieved stringent Certifications whereas the Atlassian products have not, as well as failed multiple internal security audits.

The solution's defined systems and services rely heavily on DevOps being used to manage them.

This implies a basic understanding of what DevOps is, and is not.

Some evidence as to what DevOps can provide was given in the Summary and Assessments, and even more information has been provided in the Appendices, but a very small description is provided below.

DevOps is the union of people, Agile processes, and tools to enable continuous delivery of value to end users, by removing barriers between Development, Operations and Quality Assurance, emphasizing communication, collaboration, and continuous automated integration, quality assurance and delivery.

A primary goal of DevOps is to establish an environment where more reliable evolving applications can be released more frequently.

DevOps is an Enterprise reaction to the documented benefits of Agile delivery, and extending it beyond the development phase to the whole application lifecycle.

Of specific note is that DevOps is not: it is not DevOnly, replacing Ops (NoOps). It is all groups working at delivering the benefits of DevOps on an equal footing.