Finding the right partner

By Mike Bartley |  No Comments  |  Posted: December 1, 2008
Topics/Categories: Embedded - Integration & Debug  |  Tags: ,

The complexity of delivering full system products means that more and more companies are looking to outsource some activities to concentrate engineering resources on areas that enable greater product differentiation. For small and medium-sized companies (SMEs) in particular, outsourcing provides access to resources and skills that allow them to undertake projects that would be otherwise beyond their capacity.

Flexible labor pools allow SMEs to complete one-time projects without having to ramp-up internally. They can also focus more on their core business units, and get to market faster with new products and services.

However, outsourcing, and especially offshore outsourcing, does entail risk. Companies must have a clear process that allows them to determine which activities are to be passed to partners, as well as their own internal readiness to outsource work in such a way that it will be completed efficiently.

The article describes how parallel processing specialist ClearSpeed Technology approached its decision to outsource software test activities.

The company

ClearSpeed Technology specializes in advanced parallel processing for applications ranging from commerce to science to security. Its product range includes chips, accelerator cards, rack modules, software and support services. These are designed to work alongside industry-standard x86-based systems. The chips themselves are programmed in Cn (C with extensions for parallel processing), and ClearSpeed offers the customer a complete integrated development environment (IDE) with all standard software tools. ClearSpeed has just released the latest boards (the e710 and e720) based on its CSX700 chip.

The challenge

Verification of ClearSpeed products is extremely complex. This article concentrates on test for the software aspects of one product. This process can mainly be broken down into two tasks: test of the software’s tool chain and IDE; and test of the software’s runtime aspects, including checks on its compatibility with host platforms and accelerated host applications.

ClearSpeed uses an automated build and test system to perform most of this work. The automated tests are mainly targeted at unit and integration level functions with some performance testing. However, some system and requirements level testing is harder to achieve within the automated system. Developing and running such tests are very labor-intensive tasks, so ClearSpeed decided to outsource these activities in mid-2006. Specifically, the company outsourced the development of some tests targeted at the automated test system, the development and execution of system and requirements level testing, and testing from a customer viewpoint.

The decision had mixed results, but overall the benefits have far outstripped the downside. This paper describes ClearSpeed’s experience with outsourcing software test, highlighting the lessons learned. In the simplest terms, the entire process was encapsulated within the following stages:

  1. Define the strategic goals behind your decision to outsource.
  2. Decide what to outsource.
  3. Review your readiness to outsource.
  4. Select your outsource partner.
  5. Begin working with that partner.
  6. Review the success of the outsourcing.

This article focuses on steps one through four.

Strategic goals

ClearSpeed set the following strategic (prioritized) goals for its decision to outsource software testing:

  • improve company focus;
  • improve company resource flexibility and by doing so, reduce time-to-market;
  • improve quality;
  • address cost variables; and
  • reduce cost.

These goals were agreed upon across the appropriate senior staff: the software manager, the testing manager, the engineering manager and the chief financial officer. The outsourcing needed the support of a cross-functional group within ClearSpeed to ensure that there was clarity across the company regarding its objectives. The goals also needed to be a source of guidance for important decisions related to the outsourcing and a set of benchmarks against which its success could be measured.

At the same time, it was recognized that outsourcing could deliver benefits above and beyond the five main goals (e.g., access to more specialized staff and their skills). However, these specific targets were also connected because they had company-wide relevance and could thereby be most broadly prioritized.

What to outsource

Some of the decisions here were specifically dictated by test and verification activities that we had already decided to focus upon, but there were others of a more general nature worth noting for any project of this type. These included:

  • Intellectual property (IP): how much information are you prepared to divulge?
  • For software tests, we found that we only needed to deliver executables to allow our outsource partners to perform their work. However, to develop new tests, we did need to divulge details from the specification that would typically be hidden from customers.
  • Hardware verification (e.g., simulation, property checking) can require that you give third-parties access to source code. That code can be obfuscated and companies tend to be more cautious on hardware verification than with software verification.
  • Proximity to product development.
  • Unit-level testing requires closer proximity to the development than does system or acceptance testing. ClearSpeed therefore found it easier to outsource the latter activities.
  • Test development vs. test execution.
  • Outsourcing execution requires less knowledge about the product than test development. ClearSpeed, however, decided to outsource both activities because of the company’s strategic goals.

Figure 1 (p. 18) shows the types of testing activities that ClearSpeed is outsourcing.

Readiness for outsourcing

The next step was an audit of the processes earmarked for outsourcing to determine how suitable and how ready they are. Does the development process actually lend itself to outsourcing? There are some formal ways to measure this and other relevant metrics.

Capability Maturity Model Integration (CMMI) was developed during the late 1980s at Carnegie Mellon University in Pittsburgh. It was originally a tool used to assess whether government contractors had the right processes to fulfill a federal software contract. Subsequently, it has been deployed more widely as a means for measuring the process capability maturity of organizations in diverse areas (more information on its use is available online at www.sei.cmu.edu/cmmi).

If, after running the relevant CMMI-framed assessment, you find you are at Level 1 (Initial–“Processes are usually not documented and change based on the user or event”), then you are unlikely to succeed at outsourcing. Simply sharing your chaos with other organizations is unlikely to improve the situation. CMMI Level 2 is Repeatable–“The software development processes are sometimes repeatable, possibly with consistent results and may not repeat for all the projects in the organization.” Level 3 is Defined–Your processes can be considered repeatable.” Obviously that third level is what you aim for.

The Test Maturity Model Integration (TMMi) was developed to complement the CMMI framework, by providing a structured presentation of maturity levels, specifically for test process measurement method (more information is available at www.tmmifoundation.org).

You may regard these two frameworks as too formal. Certainly, you may not be able to wait until your company attains CMMI Level 3 before beginning an outsourcing program. Indeed, ClearSpeed did not initially use such formal measures. Instead, it used PACE, a phase-based product development process. It is described in Michael E. McGrath’s book, Setting the PACE in Product Development, A Guide to Product and Cycle-time Excellence (Butterworth-Heinemann, 1996).

The development process tended to follow a ‘V-model’. This meant we had documented requirements at the start and functional specifications that were written during the development process. Such documents are a must for outsourcing. You should also have the following in place:

  • a change management process so that whenever a requirement or specification changes, your outsourcing partner is informed as soon as possible (requirements tracing also allows the impact of such changes to be more easily assessed);
  • good project management processes (particularly plans that track the deliverables exchanged between you and the outsourcing partner);
  • good risk identification and management; and
  • incoming acceptance reviews with your outsource partner (you especially need these early on but can cut them back as you build more trust in your outsourcing partner).

ClearSpeed also looked at the maturity of the vendor management process. Again, there are many formal models available to help here, such as the Robbins-Gioia Vendor Management Maturity Model (a.k.a. VM³).

Outsource partner selection

Before selecting the partner (or partners), there were still some strategic issues to consider. For example,

  • Which commercial model would be used?
    • Project-based
    • Fixed price
    • T&M
    • Team-based
    • Use a team of engineers from a supplier
    • Have our own offshore development center
  • Which location model?
    • Onsite
    • Near-shore
    • Off-shore
  • Which country?

Each of these options had its own set of variations. As mentioned already, your strategic goals should guide you through the decision process. ClearSpeed settled on an off-shore model using India. We chose India because it has a stable political and economic base, a good cultural fit with the UK and a mature outsourcing industry.

The next step is to define the partner selection criteria. Once again, the strategic goals should guide you here. Figure 2 shows the criteria used by ClearSpeed. Our partner selection process then followed four basic steps.

1. Identify potential partners.

We did this by talking to other companies that had already outsourced similar functions, by Internet-based research and by attending appropriate conferences and similar events (e.g., Outsource World). This process generated a list of 10 potential partners.

2. Assess potential partners against selection criteria.

We asked potential partners to provide information that could be benchmarked against the criteria in Figure 2. We also asked them to bid for a real project. This meant we had much more detailed information and a chance to see their expectations for a typical project execution. We used the resulting information to produce a five-company shortlist.

3. Visit potential partners to undertake a more detailed analysis.

These visits provided invaluable and detailed insights into the shortlisted candidates and allowed us to bring the final choice down to two companies.

4. Have potential partners undertake a pilot project.

ClearSpeed had identified small pilot projects that allowed us to assess the partners’ performance during the execution of a real project.

We finally decided to sign up with two of the shortlisted companies as multi-sourcing partners. On one level, this does mean that we have to commit more time and effort to supporting our outsourcing program, but, on another, it has established a wider pool of resources and skills to access. Also, competitive tendering is now built into the process permanently, allowing for better cost control.

The above process took ClearSpeed into the execution and governance phase of the outsourcing relationship (i.e., the areas identified as process steps five and six earlier in the article). ClearSpeed has developed a set of procedures for managing the outsourcing relationship and regularly measures how well it is delivering against its strategic objectives. Overall, the results are very positive but a more detailed consideration of this must wait on a subsequent article.

ClearSpeed Technology

Software testing

Test development

Development of tests for the ClearSpeed SDK, runtime libraries and applications.
A clear test specification is required for such test development.

Running tests

Running tests during development and signoff of candidate software releases. Once sufficient knowledge about ClearSpeed products is achieved, then this could be extended to debug of failing tests.

Quality analysis

Help in measurement of the quality of the ClearSpeed software verification.

3rd party application compatibility testing

Development of tests using 3rd party applications (e.g. The Mathworks’ MATLAB, Wolfram’s Mathematica) to show that the ClearSpeed acceleration libraries are compatible. That is, they are functionally correct and provide performance acceleration.

Hardware compatibility testing

Run ClearSpeed testing across a number of platforms (Operating system + chipset) in order to verify that a candidate software release works across all supported platforms

Hardware verification

Hardware component verification

Development of hardware verification test benches (stimulus, coverage and assertions). These would (probably) be written in “e” which is a specialist language1. A clear test specification will be given for such test bench development.

System verification

HW+SW system test development

Development of system level tests to demonstrate correct HW and SW integration.
A clear test specification will be given for such test development.

Reverse engineering of existing HW+SW system tests

ClearSpeed has a number of system tests where the coverage achieved by the tests is no longer known and they require reverse engineering to recover a specification for each test.

1 See http://www.cadence.com/partners/industry_initiatives/e_page/index.aspx

FIGURE 1 Outsourced test activities

ClearSpeed Technology

Area

Criteria

Measurement

People

Skills

Academic background of staff

Staff training statistics

Retention

Staff attrition rates

Process

Process maturity

CMM and other process maturity measures attained

Quality

Quality processes employed

Methodology

What development processes and methodologies are employed

Project management

What project management processes do they follow?

How do they report progress?

What project management processes do they follow?

How do they report progress?

How do they manage risks?

Change management

What is the ECO management process?

Company

References

Obtain references from existing customers

Financial

Five-year historic financial data (to allow us to determine that they are well-funded and profitable)

Key points of the 3-year financial plan

Size

Size of company over past 5 years

Profile of people over past 5 years

Strategy

Main strategic goals for the company

Goal for relationship

What are their goals for working with ClearSpeed?

Technology

Domain knowledge

What are the top five company knowledge domains

Tools

What tools do they have experience of

Cost

Cost structures

What are the cost structures used on projects (e.g. fixed price bid, time and materials)?

How are change proposals managed from a cost point-of-view?

Cost history

Provide changes in costs over the last 5 years

FIGURE 2 Partner selection criteria

Mike Bartley (mike@tandvsolns.co.uk) is an independent software test and hardware verification consultant with specialist knowledge in outsourcing. More information on ClearSpeed is available at www.clearspeed.com.

Comments are closed.

PLATINUM SPONSORS

Synopsys Cadence Design Systems Mentor - A Siemens Business
View All Sponsors