## IT Planning For Large Scale ERP Rollouts

### Christopher J. Wood

Forsythe Solutions Group
7440 N. Long Ave
Skokie, IL 60077
Phone: (847) 982-3361
E-mail: cwood@forysthemca.com

Table Of Contents

I. INTRODUCTION

II. IT-FOR-ERP OVERVIEW

III. MAJOR ERP ARCHITECTURES

IV. CRITICAL IT-FOR-ERP COMPONENTS

A. Backup/Restore

B. AVAILABILITY & RECOVERY

C. STORAGE SYSTEMS

D. NETWORK

E. END-TO-END TECHNOLOGY MANAGEMENT

F. IT SERVICE MANAGEMENT

V. GUIDELINES

Appendix A: NRP for ERP Planning Book

I. INTRODUCTION

Even now IT-for-ERP infrastructure planning is more art than science. The intent of this document is to focus on fundamental policies, procedures, and concerns that will have a direct impact on whether or not you succeed with an on-time, on-budget, and adequately performing (from an IT point of view) ERP implementation.

The urge to cram a maximum amount of acronyms and discourses about trendy technologies that have recently emerged from laboratories into this paper is hopefully contained. This may be disappointing if your ERP solution is already running on top of a rock-solid and proven IT foundation and you want to find out how to go from leading to bleeding edge.

Those of you in this category are clearly not the intended audience. The intended audience are those of us regular folk who are preparing for an ERP implementation, or have already implemented and still have a ways to go in establishing the aforementioned rock-solid and proven IT practices.

Figure 1: A sampling of the wide array of tier 1 and 2 ERP vendors.

II. IT-FOR-ERP OVERVIEW

A. Your IT Landscape: Before and After ERP . . .

1. A Double Whammy. Whatever your IT landscape looks like now, it will likely look a lot different after implementing an ERP solution. Why? Most ERP products are intense business applications that happen to use a client/server (typically 3-tier) architecture.

So here we have, on one hand, an ERP product that is probably a lot more complex (and more powerful) compared to your existing business systems, and on the other hand is a client/server application. Client/server computing itself is much more complex, from an IT point of view, than the good old integrated, monolithic host/terminal legacy systems.

2. Critical Vs. Mission Critical. An ERP initiative is possibly the first foray into truly mission critical client/server computing for your company. Even if you have experience in this area, this may be the first time you are dealing with a client/server application that is literally running your core business! The client/server model is daunting because it is comprised of many layers, and many components within each layer. In other words, there is a lot more that can go wrong, and frequently does, with client/server systems. Having said this it should be pointed out that client/server is clearly here to stay and is absolutely required to better align IT with your core business objectives.

Figure 2, Client/Server Complexities: the lack of tight
inter- and intra-layer integration presents critical challenges
to ERP deployment.

It is the inherent client/server complexities that are usually underestimated. Nothing reveals and exacerbates these complexities as intensely as an ERP application.

B. Who Does What?

1. Outside Implementation Partners. Many companies decide to engage an outside consultant to lead their ERP implementation effort. Examples of Tier 1 implementation partners are the Big 6 (or is it 5? Or 4?) consulting companies such as Andersen Consulting, Price Waterhouse, Deloitte & Touche, etc., and many of the large systems manufacturers—Hewlett-Packard, Digital Equipment Corporation, IBM, etc. There is also a galaxy of Tier 2 and Tier 3, or boutique, implementation partners. These companies usually differ from their Tier 1 brethren in one or more of the following: size, breadth, geographic presence, average experience levels, and fees.

2. Common Or Advisable Staff Ratios. Having outside consultants make up about thirty to forty percent of your ERP implementation staff seems to be a healthy range. Anything much less may not provide you with enough expertise and knowledge transfer; anything much more runs the risk of generating excessive costs and poses management control problems.

3. ERP Cross-section. In very general terms, there are three fundamental ERP implementation layers. In order of executive-level visibility and awareness they are:

1. High-level Business Consulting (e.g., BPRs, consolidations, re-alignments, etc.).
2. Functional Application Configuration Consulting (e.g., actually configuring your companies’ new sales order entry system within the selected ERP package).
3. IT-for-ERP Planning and Implementation (e.g., high availability, backup/restore, network issues—more on these topics later!).

4. Resource Alert. Most implementation partners focus primarily on the first two categories. Some attempt to provide "one-stop shopping" by addressing all three—and go even further by trying to provide the underlying hardware platform.

Heads-Up! Although it would clearly be the optimal theoretical solution, alarms should sound whenever an implementation partner attempts to be the primary provider for all three categories. All partners, regardless of focus, have resources that are likely stretched very thin. This is true for any one of the categories; it becomes even more pronounced if they attempt to take on two or all three.

C. Iceberg Alert.

What is almost always the case—and the real topic of this paper—is that insufficient or delayed attention and resources are invested in the IT-for-ERP Planning and Implementation category.

There are many possible reasons for this. Perhaps the most common is that a companies’ decision to implement an ERP solution is almost always driven by one or more core business needs. It is natural to expect that those areas closest to the bottom line need top priority—sales order entry, MRP, accounts payable/receivable, etc. These are the areas where you see the most up-front and generous investments in business and application configuration consulting—either external or internal.

It is not all that rare to find a company intentionally excluding its own IT group from the ERP application evaluation and decision making process. This is not necessarily a faulty strategy. What is at fault is, for whatever reason, IT not getting to work soon enough or seriously enough on IT-for-ERP planning.

Figure 3: The vast majority of ERP planning resources are spent above the water line.

III. MAJOR ERP ARCHITECTURES

A. N-Tier Client/Server Architectures

Most major ERP products use a two- or three-tier client/server architecture. Additional client/server "glue" is required to connect one tier to the next, and will be discussed later in this document. The following diagram shows a somewhat simplified diagram of a 3-tier approach.

Figure 4: Sample 3-Tier ERP Hardware Architecture

As complex as the above sample may appear, in reality it most likely represents a simplified version of everything that is truly required to make a three tier client/server ERP application work. On the other hand, as simplified as this diagram may be, there is no way it could be seen as near as simple as an old-fashioned (but maybe making coming back!) two-tier host/terminal architecture—where you have a centralized mainframe, or mainframe-class, host with relatively "dumb" character-based terminals.

IV. CRITICAL IT-FOR-ERP COMPONENTS

1. Backup/Restore
2. Availability & Recovery
3. Storage Systems
4. Network
5. End-to-End Technology Management
6. IT Service Management

The above box contains a minimum set of ten IT-for-ERP components that should be scrutinized. At first glance there may not appear to be anything ERP-specific about them—that they are simply standard requirements for any client/server IT infrastructure. Despite this possible first impression, you will hopefully derive two key benefits:

Number 1, you will either be acquainted or re-acquainted with many or most of the major client/server disciplines—which is certainly advisable given that you may be new to client/server, or have not had to build a truly mission-critical client/server environment.

Number 2, each category will include ERP-specific commentary and examples, which will further drive home the issues under discussion.

A. Backup/Restore
Key Questions: The system running your ERP application is most likely running your business. It is possible that you will not be able to take down the system to do a backup, or will be able to take it down only for a short period of time, e.g., Monday through Saturday from midnight to 2:00am; Sunday from midnight to 6:00am. What is your available backup window? Note: this parameter should be addressed as a service level metric. What is the amount of data that needs to be backed-up? What is the projected growth rate of the data you need to backup? Does all database (data, redo logs, control files), file systems, applications have the same backup/restore requirements (hint: not likely)? How fast will restores need to occur? Note: this parameter should also be addressed as a service level metric. Do you have a departmental- or enterprise-wide backup/restore solution? Should you? Is there a backup product that can integrate directly with the ERP solution you are using?
Recommendations: A careful up-front analysis of your backup/restore requirements along with formal or informal service level agreements should yield a fairly obvious solution—in terms of required hardware, software, and procedures. Offline Backups? During an off-line backup the whole database, or parts of it, are unavailable for use. The advantage of such a full backup is consistency. Using this approach, a restored database could be opened without needing to do a recovery. On-line Backup? A full or partial on-line backup is more appropriate for large databases. The backup includes all or some of the data files and a control file. The off-line redo log files generated while the backup was done need to be available as well in order to use the backup. The advantage is that the database is available for normal use while the backup is taken. Two disadvantages to on-line backup are 1) performance deterioration, and 2) risk of inconsistency (backed up data files are inconsistent without the redo log files). Consider 3rd Party backup/restore products that support a direct application interface to your ERP product (e.g., SCH’s dbBRZ, HP’s OmniBack, Legato’s NetWorker, CA’s ARCserve). Carefully study capacity, throughput, and reliability features of tape devices. Consider triple-mirror backups. Determine whether off-line backups, on-line backups, or a combination will be used. Stagger backups at the tablespace level. Increase frequency of partial backups. Increase backup frequency of heavily used tablespaces. Do backups in parallel. Adjust backup frequency by carrying out tests. Skip index tablespaces to reduce backup downtime. Back up to disk first. Use hardware compression with backup. Verify that your backup tapes are readable.

B. AVAILABILITY & RECOVERY
1. Key Questions: Your ERP application will need to be available for use most—if not all—of the time. After all it is running your business. The extent of this availability should ultimately be determined by answers to basic business questions: What will it cost your company in terms of lost revenue and profits and in terms of user and customer pain and anguish if your ERP system is unavailable? What are the viable high availability levels, and how much will they cost? Do you have a disaster recovery plan? Do you need one? How much will it cost?
2. Recommendations: Review Support Contracts. Make sure your various hardware and software support contracts provide a level of support consistent with your availability goals and commitments. Many vendors provide some kind of "business critical" support option designed with ERP in mind. Proactive Monitoring. Keeping tabs on the various client/server layers (see Figure 2, Client/Server Complexities) is critical. Detecting, identifying, and resolving issues before they cause problems is arguably the most effective high availability strategy. Reduce Planned Downtime. Planned downtime is certainly better than unplanned downtime. Nevertheless, minimizing or eliminating it is, or will be, a requirement. Three tasks that have traditionally required planned downtime include backups, tuning, and upgrades (hardware and software). Minimizing or eliminating downtime caused by backups (online Vs. offline, triple mirror backups) is discussed in Section A. Clustering solutions provide a means of not only ensuring your systems are fault resilient, but also allow you to perform a certain amount of non-disruptive upgrades. For example, a cluster member (say, a server) can be disassociated from the cluster, upgraded and tuned, and re-associated with the cluster with no resultant downtime. Basic Hardware Redundancies. Meticulously check each hardware component (hubs, routers, NICs, power supplies, fans, I/O channels, disks, memory) to see if and how making them redundant makes sense. ERP-specific clustering support. Be aware of ERP-specific cluster support, e.g., is Microsoft’s Cluster Server a supported or certified solution for SAP? Does HP’s MC/ServiceGuard work with QAD? Remote Mirroring. E.g., EMC's Symmetrix Remote Data Facility (SRDF), MTI's Business Critical Remote Mirroring (BCRM), Hitachi Data Services’ Remote Copy/Open.

Figure 3: 99.95% uptime means a little more than 4 hours of downtime per year;
99.999% means about 5 minutes of downtime per year.

C. STORAGE SYSTEMS
1. Key Questions Host Consolidation. Do you require, or would you benefit from, an enterprise storage solution that can connect to multiple hosts (various UNIX, NT, AS/400, MVS, etc.)? Availability. What are the required availability features and functions, e.g., RAID, triple mirroring, remote mirroring, automated fail-over, automated recovery. Scale-ability Performance
2. Recommendations: Death & Taxes. There are two truisms for ERP storage: 1) you will always need more storage capacity, and 2) performance will always be a concern.

D. NETWORK
1. Key Questions: ERP applications are becoming ever more network-centric. Understanding their impact on your LAN and WAN is critical. A clear "red alert" situation is exemplified by degraded, or erratic, sales order entry response times due to excessive network latencies. Network bandwidth is a finite resource. Knowing exactly what resources are required for your ERP application calls for a lot more than estimating or using rules of thumb. What is the capacity of your network? What is your projected growth rate? How will changes (e.g., modifying your ERP application, moving or consolidating servers, adding a new branch office, expansion due to business growth) affect your network? How resilient is your network? Who is affected--and how--by network failures? How long will it take you to pinpoint and resolve network bottlenecks and/or failures caused by your ERP application? What is the optimal network topology for your ERP application?
2. Recommendations: Network Monitoring. At the very least, you should be to view your network. Consider implementing a network monitoring product (e.g., HP’s Network Node Manager). If you already have one, extend it to cover all ERP-related network elements; if you will be implementing a network management product for the first time, using it for your entire enterprise probably makes sense. Baseline Analysis. This entails determining ERP-imposed network traffic pattern trends and detailed utilization statistics. This information is used to learn about your network behavior. Having done this, you will be able to make subtle configuration changes to optimize your networks’ cost-performance, and proactively detect vulnerabilities. An in-depth baseline analysis is a crucial step before planning a successful network redesign or expansion. Benefits include long and short-term cost savings from better utilization of existing network systems, proper installation and migration strategies for new technologies, and improved systems management capabilities. Capacity Planning. At the very least, knowing where and when to add bandwidth or expanded routing or switching capability will be required for bare-bones capacity planning decisions. Service Levels. Network service levels should at a minimum address performance, availability, and recoverability. Modeling & Simulation. There are now a number of computer-aided design, simulation, and modeling tools available that will help you put together the optimum network for your specific ERP application. These tools allow you to model ERP applications’ transactions and simulate the resultant network load. Network Resource Planning (NRP). This is the ultimately the best approach you could use for network planning—whether or not you are using an ERP product. As previously mentioned, it is important—if not critical—to design or modify your network infrastructure around your ERP solution. Again, whatever your network infrastructure looks like before ERP, it will likely look a lot different after ERP. Using a seat-of-the-pants approach will not likely result in the best possible network environment. An analytical approach needs to be used to design and size your network. See Appendix A--NRP For ERP Planning Book.

E. END-TO-END TECHNOLOGY MANAGEMENT
1. Key Questions: How much bandwidth does your IT support staff have? Are you able to monitor and manage each and every IT infrastructure link? That is, can you "see" everything that touches your ERP application—hubs, routers, storage, servers, printers, etc.? Do you practice formal or informal capacity planning? Are you able to correlate service levels with underlying technology performance and capacity data?
2. Recommendations Few companies have, on a large scale, achieved a true proactive, correlated, end-to-end IT management capability. Their number has recently, however, begun to grow substantially (hopefully you will soon join their ranks!). There are a couple of reasons that explain this trend: 1. ERP is truly mission critical. Anything and everything that has a potential impact on it needs to fall under a management umbrella. Maintaining high availability depends on this discipline more than any other hardware or software product. 2. Qualified IT staffers are in extreme short supply. You need to do two things: 1) leverage their resources by embracing increased productivity measures, and 2) augment their expertise as much as possible by using tools that help make predictions and correlate cause-and-effect data. 3. ERP solutions have a voracious appetite. You constantly need to feed them ever-increasing amounts of memory, network bandwidth, storage space, etc. Getting a handle on the magnitude and timing of these upgrades is critical for budget planning and service level reasons. Identify. Make sure you identify all pertinent IT entities that need to be monitored and managed. Trying to track too many entities is as counter-productive as tracking too little. Examples: CPU, network, memory, and storage utilization; user response time, etc. End-to-end. You should strive to monitor and measure round trip response times, account for the influence of all client/server layers and components into account (see Figure 2). 3rd Party Tools. A plethora of products from numerous companies are available for enterprise systems and network management (e.g., IBM Tivoli, CA UniCenter, HP OpenView). Products in this market are starting to mature but, by themselves, are usually not 100% complete. You will need to do a lot of homework to see where the gaps are and how to address them—add other 3rd party modules, fill the gap by building the functionality yourself, or ignore the gap). Note: a very common problem you may face when trying to tie together many nodes that are spread-out over large distances is network bandwidth and performance.

F. IT SERVICE MANAGEMENT
1. Key Questions: Are your users happy with the performance and availability of your ERP application? Are you able to dissect the root cause of ERP performance and availability problems? Are you able to provide reports that convey the level of service you provide to your end-users?
2. Recommendations: Establish and prioritize business-driven service level metrics. Enter into a formal or informal agreement with your end-user organization. Generate periodic or demand-driven reports that show how well the service level agreements are being met. Integrate, as much as possible, service monitoring and technology management. For example, correlate a particular transaction for a specific user to CPU utilization, I/O, etc. Consider 3rd Party tools: HP OpenView IT Service Manager, Luminate Software Corporation, Envive Corporation, Empirical Software

V. GUIDELINES

1. Start Preparations Now
2. Perform IT Health Check Assessment
3. Take Formal Sizing & Capacity Planning Seriously
4. Stress Test
5. Establish Technology Management Framework
6. Use Service Levels
7. Study NT, UNIX, and NT/UNIX Integration
8. Prepare Your Staff
9. Test Procedures

Appendix A: NRP for ERP Planning Book

(from: http://www.makesystems.com/nrp_book/nrp_book.html)

> Network Resource Planning For SAP R/3, BAAN IV, and PEOPLESOFT:
> A Guide to Planning Enterprise Applications

Annette Clewett, et al.
Paperback / Published 1998

This "how to" book with real-life examples is a guide for network planners, technical staff and managers who are responsible for ensuring a wide area network infrastructure can support mission-critical enterprise application deployments. Learn about Network Resource Planning methodologies and how to assure existing applications and planned deployments achieve performance and cost objectives.

Authored by Annette Clewett, senior network consultant at Make Systems who has helped define and deliver Network Resource Planning.

This book will help you understand how to do the following:

- Set focused, achievable objectives for a Network Resource Planning project.

- Collect relevant, valid data on current network performance, as well as understand limitations of that data to a current baseline for network planning purposes.

- Collect valid data on traffic demands made by specific client/server applications and how to develop valid assumptions about how the application will be used when deployed.

- Determine criteria for evaluating network performance under various capacity planning ("what-if") scenarios.

- Set up capacity planning scenarios, run models or simulations, and evaluate results in light of both performance criteria and assumptions.

- Use results of capacity planning activities to redesign a network to meet performance objectives.