Finding the right partner

The complexity of delivering full system products means that more and more companies are looking to outsource some activities to concentrate engineering resources on areas that enable greater product differentiation. For small and medium-sized companies (SMEs) in particular, outsourcing provides access to resources and skills that allow them to undertake projects that would be otherwise beyond their capacity.

Flexible labor pools allow SMEs to complete one-time projects without having to ramp-up internally. They can also focus more on their core business units, and get to market faster with new products and services.

However, outsourcing, and especially offshore outsourcing, does entail risk. Companies must have a clear process that allows them to determine which activities are to be passed to partners, as well as their own internal readiness to outsource work in such a way that it will be completed efficiently.

The article describes how parallel processing specialist ClearSpeed Technology approached its decision to outsource software test activities.

The company

ClearSpeed Technology specializes in advanced parallel processing for applications ranging from commerce to science to security. Its product range includes chips, accelerator cards, rack modules, software and support services. These are designed to work alongside industry-standard x86-based systems. The chips themselves are programmed in Cn (C with extensions for parallel processing), and ClearSpeed offers the customer a complete integrated development environment (IDE) with all standard software tools. ClearSpeed has just released the latest boards (the e710 and e720) based on its CSX700 chip.

The challenge

Verification of ClearSpeed products is extremely complex. This article concentrates on test for the software aspects of one product. This process can mainly be broken down into two tasks: test of the software’s tool chain and IDE; and test of the software’s runtime aspects, including checks on its compatibility with host platforms and accelerated host applications.

ClearSpeed uses an automated build and test system to perform most of this work. The automated tests are mainly targeted at unit and integration level functions with some performance testing. However, some system and requirements level testing is harder to achieve within the automated system. Developing and running such tests are very labor-intensive tasks, so ClearSpeed decided to outsource these activities in mid-2006. Specifically, the company outsourced the development of some tests targeted at the automated test system, the development and execution of system and requirements level testing, and testing from a customer viewpoint.

The decision had mixed results, but overall the benefits have far outstripped the downside. This paper describes ClearSpeed’s experience with outsourcing software test, highlighting the lessons learned. In the simplest terms, the entire process was encapsulated within the following stages:

1. Define the strategic goals behind your decision to outsource.
2. Decide what to outsource.
3. Review your readiness to outsource.
4. Select your outsource partner.
5. Begin working with that partner.
6. Review the success of the outsourcing.

This article focuses on steps one through four.

Strategic goals

ClearSpeed set the following strategic (prioritized) goals for its decision to outsource software testing:

• improve company focus;
• improve company resource flexibility and by doing so, reduce time-to-market;
• improve quality;
• address cost variables; and
• reduce cost.

These goals were agreed upon across the appropriate senior staff: the software manager, the testing manager, the engineering manager and the chief financial officer. The outsourcing needed the support of a cross-functional group within ClearSpeed to ensure that there was clarity across the company regarding its objectives. The goals also needed to be a source of guidance for important decisions related to the outsourcing and a set of benchmarks against which its success could be measured.

At the same time, it was recognized that outsourcing could deliver benefits above and beyond the five main goals (e.g., access to more specialized staff and their skills). However, these specific targets were also connected because they had company-wide relevance and could thereby be most broadly prioritized.

What to outsource

Some of the decisions here were specifically dictated by test and verification activities that we had already decided to focus upon, but there were others of a more general nature worth noting for any project of this type. These included:

• Intellectual property (IP): how much information are you prepared to divulge?
• For software tests, we found that we only needed to deliver executables to allow our outsource partners to perform their work. However, to develop new tests, we did need to divulge details from the specification that would typically be hidden from customers.
• Hardware verification (e.g., simulation, property checking) can require that you give third-parties access to source code. That code can be obfuscated and companies tend to be more cautious on hardware verification than with software verification.
• Proximity to product development.
• Unit-level testing requires closer proximity to the development than does system or acceptance testing. ClearSpeed therefore found it easier to outsource the latter activities.
• Test development vs. test execution.
• Outsourcing execution requires less knowledge about the product than test development. ClearSpeed, however, decided to outsource both activities because of the company’s strategic goals.

Figure 1 (p. 18) shows the types of testing activities that ClearSpeed is outsourcing.

Readiness for outsourcing

The next step was an audit of the processes earmarked for outsourcing to determine how suitable and how ready they are. Does the development process actually lend itself to outsourcing? There are some formal ways to measure this and other relevant metrics.

Capability Maturity Model Integration (CMMI) was developed during the late 1980s at Carnegie Mellon University in Pittsburgh. It was originally a tool used to assess whether government contractors had the right processes to fulfill a federal software contract. Subsequently, it has been deployed more widely as a means for measuring the process capability maturity of organizations in diverse areas (more information on its use is available online at www.sei.cmu.edu/cmmi).

If, after running the relevant CMMI-framed assessment, you find you are at Level 1 (Initial–“Processes are usually not documented and change based on the user or event”), then you are unlikely to succeed at outsourcing. Simply sharing your chaos with other organizations is unlikely to improve the situation. CMMI Level 2 is Repeatable–“The software development processes are sometimes repeatable, possibly with consistent results and may not repeat for all the projects in the organization.” Level 3 is Defined–Your processes can be considered repeatable.” Obviously that third level is what you aim for.

The Test Maturity Model Integration (TMMi) was developed to complement the CMMI framework, by providing a structured presentation of maturity levels, specifically for test process measurement method (more information is available at www.tmmifoundation.org).

You may regard these two frameworks as too formal. Certainly, you may not be able to wait until your company attains CMMI Level 3 before beginning an outsourcing program. Indeed, ClearSpeed did not initially use such formal measures. Instead, it used PACE, a phase-based product development process. It is described in Michael E. McGrath’s book, Setting the PACE in Product Development, A Guide to Product and Cycle-time Excellence (Butterworth-Heinemann, 1996).

The development process tended to follow a ‘V-model’. This meant we had documented requirements at the start and functional specifications that were written during the development process. Such documents are a must for outsourcing. You should also have the following in place:

• a change management process so that whenever a requirement or specification changes, your outsourcing partner is informed as soon as possible (requirements tracing also allows the impact of such changes to be more easily assessed);
• good project management processes (particularly plans that track the deliverables exchanged between you and the outsourcing partner);
• good risk identification and management; and
• incoming acceptance reviews with your outsource partner (you especially need these early on but can cut them back as you build more trust in your outsourcing partner).

ClearSpeed also looked at the maturity of the vendor management process. Again, there are many formal models available to help here, such as the Robbins-Gioia Vendor Management Maturity Model (a.k.a. VM³).

Outsource partner selection

Before selecting the partner (or partners), there were still some strategic issues to consider. For example,

• Which commercial model would be used?
  • Project-based
  • Fixed price
  • T&M
  • Team-based
  • Use a team of engineers from a supplier
  • Have our own offshore development center
• Which location model?
  • Onsite
  • Near-shore
  • Off-shore
• Which country?

Each of these options had its own set of variations. As mentioned already, your strategic goals should guide you through the decision process. ClearSpeed settled on an off-shore model using India. We chose India because it has a stable political and economic base, a good cultural fit with the UK and a mature outsourcing industry.

The next step is to define the partner selection criteria. Once again, the strategic goals should guide you here. Figure 2 shows the criteria used by ClearSpeed. Our partner selection process then followed four basic steps.

1. Identify potential partners.

We did this by talking to other companies that had already outsourced similar functions, by Internet-based research and by attending appropriate conferences and similar events (e.g., Outsource World). This process generated a list of 10 potential partners.

2. Assess potential partners against selection criteria.

We asked potential partners to provide information that could be benchmarked against the criteria in Figure 2. We also asked them to bid for a real project. This meant we had much more detailed information and a chance to see their expectations for a typical project execution. We used the resulting information to produce a five-company shortlist.

3. Visit potential partners to undertake a more detailed analysis.

These visits provided invaluable and detailed insights into the shortlisted candidates and allowed us to bring the final choice down to two companies.

4. Have potential partners undertake a pilot project.

ClearSpeed had identified small pilot projects that allowed us to assess the partners’ performance during the execution of a real project.

We finally decided to sign up with two of the shortlisted companies as multi-sourcing partners. On one level, this does mean that we have to commit more time and effort to supporting our outsourcing program, but, on another, it has established a wider pool of resources and skills to access. Also, competitive tendering is now built into the process permanently, allowing for better cost control.

The above process took ClearSpeed into the execution and governance phase of the outsourcing relationship (i.e., the areas identified as process steps five and six earlier in the article). ClearSpeed has developed a set of procedures for managing the outsourcing relationship and regularly measures how well it is delivering against its strategic objectives. Overall, the results are very positive but a more detailed consideration of this must wait on a subsequent article.