State Data Center

Independent Review

Texas State Data Center

November 1, 2004

DIR 05-101

INTRODUCTION

All state agencies are required by statute to consider moving operations to the Texas State Data Center (TSDC) in San Angelo, Texas before expending appropriations for data center operations, including server or mainframe investments. Although statutory mandate governs consideration of the TSDC, there is no clear process in place to accomplish this mandate. In practice, the system today is characterized by an unstructured approach that causes friction for agencies and the operational service provider.

Currently, each state agency negotiates and executes a contract (supplemental agreement) with the TSDC’s service provider. Each state agency agreement differs with regard to services, pricing, measurement of services and conditions surrounding breach of the services. In some cases, provisions of a supplemental agreement entail a complete rewrite of the master contract.

The objective of reviewing the current TSDC was to identify problems with the existing processes for analyzing the cost effectiveness of acquiring data center services and contracting for those services. To accomplish this, DIR convened a review team, composed of staff from other state agencies. This team conducted an independent review of the TSDC. The review focused on an evaluation of:

• Current contract terms and management procedures, including roles and responsibilities of DIR, the service provider and the participating agency.
• Contracting opportunities (present and future), including possibilities of amendment(s) to existing master and supplemental contract(s).
• A new proposal analysis and contracting process for data center service delivery from the TSDC, which integrates and complements the existing waiver review process at the Legislative Budget Board (LBB).

The following are the findings and recommendations from the review team.

Finding 1

Lack of a comprehensive enterprise approach to outsourcing data center services encouraged agencies to buy these services individually. This resulted in solutions that did not optimize integration and therefore some economy-of-scale opportunities have not been achieved.

Recommendation 1.1:

Lessons learned from the current contracts and approach should be documented and incorporated into the planned Sourcing Strategy that will guide future contract negotiations for data center outsourced services.

Recommendation 1.2:

The Department of Information Resources (DIR) should immediately take a more active role in identifying and coordinating data center outsourcing activities among agencies. In particular, DIR must take the lead to define the appropriate policies, processes and time frames for such activities like such as preparation of the statement of work (SOW), cost comparison between agency and service provider, and data center transition. DIR should consult with the State Auditor’s Office (SAO) and Legislative Budget Board (LBB) during development of the policies and processes to realize maximum cost savings and operational efficiencies.

Recommendation 1.3:

Legislation or Rules are required to facilitate DIR’s efforts to produce and implement an enterprise strategy (plan) to provide consolidated data center services to the State.

Finding 2

Attempts to provide guidance to agencies in defining the data center outsource process have not been sufficient in defining the data center outsource process This includes items such as roles, responsibilities, guiding principles, Statement of Work (SOW) development, consistent terms and conditions for agency contracts, templates for performance metrics and monitoring , and the LBB waiver process , etc. Inconsistencies are costly and negate the vision to manage the service provider and process at an enterprise level.

Recommendation 2.1:

DIR, in cooperation with the SAO and LBB should develop and issue a comprehensive Management Guide¹ that ensures standardized structure and consistency to the current contracting and administrative processes. It also will serve as the foundation for a new contract that will be forthcoming from the Sourcing Strategy and follow-on procurement.

Finding 3

The Statement of Work (SOW) documents prepared by agencies are not consistent and many times not comprehensive , resulting in multiple iterations of revisions, clarifications and updates. This causes additional expense for both the agency and the service provider. The LBB has developed and recently implemented a new Summary Statement of Work (SSOW)² document. The LBB is requiring all agencies and the service provider to use this tool document to facilitate the cost comparison of outsourcing an IT service or retaining it in-house.

Recommendation 3.1:

The SSOW, and related process for completing the SSOW , should be incorporated into the Management Guide. In addition, the Management Guide must provide agencies the necessary structured methodology explaining the entire process when considering or using data center outsourced services.

Finding 4

Many differing views exist on the degree of agency satisfaction for the current outsourced data center services. Unfortunately the service provider has conducted no recent customer satisfaction surveys.

Recommendation 4.1:

DIR and the service provider should jointly develop a new customer satisfaction survey and plan for a timely release to each participating agency.

Finding 5

Lack of consistent enterprise performance measurement requirements and differing language in individual agency contracts makes it difficult, at best, to compare the service provider’s performance among participating agencies.

Recommendation 5.1:

It is recommended that no changes be made to the current performance metrics or the method of calculation within the existing agency contracts because as it will be disruptive and add little value through the remaining 36 months of the contract. The Management Guide needs to define minimum standards for all new agency contracts.

Finding 6

Service Level requirements are defined differently in most agency contracts even though the State Data Center Advisory Board Sub-Committee once established standards to be used for all agencies to use.

Recommendation 6.1:

Any new contracts executed by agencies should use, as a minimum, the performance standards developed and approved by the State Data Center Advisory Board. Agencies needing more stringent performance standards should be granted an exception if they can properly justify such a requirement to DIR. The Management Guide should contain the minimum performance standards and the process to justify higher performance standards.

Finding 7

Liquidated damages for failure to meet performance standards under the Service Level Agreement (SLA) are referred to as Service Level Credits (SLCs). Agencies have flexibility within their specific SLA to determine how to enforce SLC ’s. In some SLAs, agencies can arbitrarily forgive SLC ’s or accept some other remedy of comparable value.

Recommendation 7.1:

The Management Guide should require agencies to identify and recover all (SLCs), as defined in their specific agency contract. If alternate remedies are accepted, then the agency must document the remedy for audit purposes and ensure that the value of the alternate remedy is comparable to the value of the assessed financial credit. Agencies receiving services at the State Data Centers should be instructed in the Management Guide to report all Service Level Credits SLCs that are due and collected, to DIR in order to track overall service provider performance. The service provider should provide the SLC data to the state entity and DIR.

Recommendation 7.2:

Future contracts should provide that the collection of SLC ’s is automatic with no provision for “in-kind” remedy.

Finding 8

Agency supplemental agreements were found to have contract language that was in conflict with DIR’s Master Services Agreement. In those cases, agency supplemental contract language takes precedence over the Master Services Agreement, in accordance with the terms of the Master Services Agreement, further deteriorating the efforts to manage the service provider in a consistent way.

Recommendation 8.1:

It is recommended that DIR not significantly renegotiate the existing Master Services Agreement or related agency contracts at this time.

Recommendation 8.2:

It is recommended that new contract requirements, terms , and conditions , etc. should be forward looking and not affect current contractual agreements. The Management Guide must define baseline terms and conditions that all future agency contracts must embody.

Finding 9

The current process allows agencies seeking proposals from the service provider to request higher levels of service that may be unsubstantiated or unrealistic within their current operating environment, or to minimize their current costs in order to help justify a waiver from LBB.

Recommendation 9.1:

Agencies should not specify performance requirements to replace existing services unless there is documented proof the agency is abiding by the same or similar performance requirement.

Recommendation 9.2:

Adopting the recommendation in 3.1 will more effectively facilitate the full disclosure of agency costs for comparative analysis.

Finding 10

There are no standards or criteria to define when the State should accept the service provider’s proposal rather than retain the service in-house. For example, how large of a difference (percentage and/or dollar amount) should exist between outsource and in-house costs to justify use of outsourced services? Also, what value should be given to qualitative benefits offered by the service provider , such as greater redundancy, 7x24 24x7 operations , and better security, to help minimize future operational risk?

Recommendation 10.1:

The Management Guide must provide consistent quantitative and qualitative criteria for all agencies to use during the proposal evaluation process.

Recommendation 10.2:

DIR needs to work closely with the LBB, State Auditor SAO , and state agencies in the development of the criteria and processes needed to validate current and proposed agency costs.

Finding 11

There have not been periodic downward adjustments to operational data center charges reflecting economies-of-scale as additional agencies participate. The service provider has made no program wide downward adjustments since the inception of the Master Services Agreement.

Recommendation 11.1:

An independent benchmarking should be conducted to define price/performance metrics for the TSDC. The bench marker must use proven tools and techniques that yield valid results. The metrics coming out of the benchmarking can then be compared to the current outsourcing market to determine whether current pricing is competitive. There should be agreement between DIR and the service provider before any each benchmarking occurs , specifying the remedy if the TSDC is measurably worse than comparable outsourced peers. It is important the agencies appropriately share in any overall reductions in cost.

Finding 12

Some service provider proposal responses to agency SOWs were viewed as not complete or lacked quality. In turn, agencies lost confidence in the service provider’s understanding of their SOW and certainly the proposed solution. The service provider has an obligation to tender high quality proposals to agency requests for bids. The obvious benefit is accurate and well-defined, understandable proposals that agencies and others can better evaluate. The less obvious, but important benefit, is the agency perception of the service provider. Customers equate poor proposals with expected poor service provider performance.

Recommendation 12.1:

The service provider needs to immediately address and correct the reasons some service provider proposals are deficient. DIR should be the focal point for this effort and work collaboratively with both the service provider and agencies to determine when the quality of future proposals meets the expected quality threshold.

Finding 13

The TSDC has had several security reviews, to include penetration tests as well as and physical security audits. However, the TSDC has never undergone a comprehensive audit (commonly known as an SAS70) to judge the adequacy of internal controls, including change management. Similarly the TSDC has not had a recent comprehensive security risk assessment to judge the level of vulnerability from a physical and infrastructure (e.g. telecommunications) perspective.

Recommendation 13.1:

DIR and the service provider should plan and implement a SAS70 audit as soon as possible and then routinely repeat thereafter. A comprehensive risk assessment should be conducted either as part of the SAS70 or separately. This should provide the new Executive Director of DIR with the needed information to properly judge the quality of internal controls, change management and level of security in place at the TSDC. Depending on the results of the audit , further actions may be required of DIR, agencies or the service provider.

Finding 14

The existing waiver process can be enhanced to more effectively achieve the goals envisioned by Senate Bill 1701 (78th R). Senate Bill 1701 states that no funds appropriated by the legislature may be expended for entering into or renewing contracts or issuing purchase orders for data center operations, disaster recovery plan testing services , or disaster recovery services without first obtaining a waiver from the LBB. Currently waivers have a one-year life cycle. Agencies that secure a waiver approval often enter into lease /purchase contracts that last two or more years. This requires an agency to secure additional waivers on solutions already in place.

Recommendation 14.1:

DIR should work closely with the LBB, State Auditor SAO and state agencies to agencies to determine whether legislative changes are required, and if so, collectively seek such changes. This would include an examination of the current annual waiver review requirement.

The following are examples of revisions to consider under the existing Master Services Agreement:

• The one-year time frame of a waiver approval should be revised. The waiver approval time frame should be tied to the two-year budget cycle, the initial contract period , or the life cycle of the hardware/software solution, whichever is determined to be best during the review process.
• The trigger point for requiring the submission of a waiver request should correspond to the $25,000 threshold specified in the Information Technology Detail tool that is managed by the LBB.

SUMMARY

Based on the aforementioned findings and recommendations, DIR and the review team developed a new Data Center Service Delivery Guide that addresses the recommendations.

ACKNOWLEDGMENTS

Independent Review Team Members

Tim Kennedy, DIR (Chair)
Terry Casparis, HHSC
Bob Covington, TDCJ
Mike Fernandez, TLC
Ruth Hooks, DIR
Donna Kahn, TDCJ
Dale Kreuger, TEA
Martin Zelinsky, DIR

Editors

Genice Mancini
Vidhya Sriram

END NOTES

¹ Foundation for Change Report – State Data Center Initiative referred to a “Management Guide.” This document is now referred to as the Service Delivery Guide for the Texas State Data Center.

² The summary statement of work (SSOW) was developed by the LBB to facilitate their analysis of agency waiver requests.

Return to 2004 Biennial Performance Report homepage.