Background

This document is the result of being at various tech teams throughout my career — NASA, Oracle, NewsCorp, Grab, MongoDB, dbt Labs. At each place, there is always the challenge of how to mix incubation of new ideas and innovation on projects with the standard flow of the software development business — which by its very nature also has innovation as a core element.

Overview and purpose

The purpose of this document is to lay out how Product and Engineering can approach the incubation of some projects. Of course, most new innovation occurs within already-formed teams. However, some things are not on the clear backlog of any team, and we want to have the ability to experiment (including failure and cancellation) with new ideas.

Variations on the process described here have been used at multiple groups and companies I have worked at previously. This document formulates what I’ve learned from those projects into a repeatable process for turning disruptive ideas into shipping products.

Phases of incubation

Creating a new product or major new feature consists of research, ideation, incubation, and productization. There is a decision process at the end of each phase about whether to proceed to the next phase, including allocation of resources and responsibilities.

Four phases of incubation: Research → Ideation → Incubation → Productization, with required inputs at each gate: Idea selection, Light Initiative Brief, Full Initiative Brief + Plan

Roles and responsibilities

Roles involved in incubation include the Incubation Manager, Researcher, Receiving Manager, Lead Engineer, Product Manager, Software Engineer, and Technical Program Manager (TPM). It is important to note that the same person can be assigned to multiple roles, especially while ramping up.

The Incubation Manager is responsible for leading and managing the end-to-end incubation process across multiple incubation streams, assuming different roles along the way for different projects if those roles don’t have owners. This includes sourcing ideas from across the company, directing researchers, and collaborating with product and engineering leaders to ensure status and priorities for incubation are clearly communicated. Anyone wanting to start an incubation should work with the Incubation Manager to get their ideas into this process. The Incubation Manager is responsible for communicating the status of all incubation projects to everyone at the company who is interested.

Researchers are individuals who have been sponsored by leadership (typically the CTO, though sometimes the CEO has ideas) to identify areas to incubate. They collect ideas from different sources — within the company, the industry, and the research community. Their findings serve as input for the Incubation Manager and leadership to decide which area should move to ideate next. They are responsible for research and may be involved with ideation and some or all of incubation.

The Receiving Manager is a manager (Director or above) who eventually assumes sole ownership of a given incubation project. The lead engineer for the new team will report to the receiving manager when the project transfers out of incubation.

The Product Owners — comprising a Product Manager, Lead Engineer, and Technical Program Manager — are responsible for a specific incubation project, building an engineering and product team on behalf of a receiving manager, and establishing proper work and team culture within the newly built team. Together, they are responsible for driving a specific incubation project end to end. They work closely with all stakeholders to ensure proper product scoping, architecture, and smooth project transition from one phase to another. If you adopt a distributed Product model, they eventually move along with the entire team to the receiving manager. If you don’t, they stay with their team.

Software Engineers are added during incubation (more may be added during productization) with the expectation — but not strict requirement — that they will transfer along with the product to the receiving manager. They are expected to stay with the product through to launch, and most would remain part of the team long term. However, this expectation is within the boundaries of standard employment and transfer policy. Together the members of the team must possess the skills and experience necessary to take the product to production.

Technical Program Management resources will be assigned to each team at Ideation. This may be part of a person, multiple people, etc., as appropriate. The TPM commitment will ramp up as an incubation team grows.

Regular review and communication

The Incubation Manager meets with leadership at least every six weeks to review the status of active incubations, approve progression of projects to ideation and incubation, and cancel incubation projects that do not look promising.

Each active incubation project prepares an update every six weeks. As well as informing the leadership review, these updates are shared with anyone at the company via a public email group.

Innovation steps

Research phase

During the research phase, the Incubation Manager is responsible for identifying a set of directions to ideate. Anyone can submit ideas, and the Incubation Manager owns responding to them and bringing ideas to leadership (typically head of product, CTO, etc.) for prioritization.

The Incubation Manager assigns researchers to projects, who produce a set of artifacts including docs, working code, use cases, etc. as inputs to subsequent phases. The results of this research are published internally and discoverable. The Incubation Manager and researchers actively seek to engage anyone at the company who may be interested in an incubation project.

Decision 1: Idea selection

During regular review, ideas are selected for further exploration. These are ideas that are not being addressed by current roadmaps and plans, but might be really important for the company to succeed.

Ideation phase

(Required input: Idea Selection)

At the beginning of the ideation phase, the product owners are assigned (transferred or hired if needed) to drive the entire project through all phases of incubation. The product owners work with the Incubation Manager and assigned researchers to refine the problem space. This includes sketching end-to-end scenarios and making them work in a proof of concept.

Assuming the project becomes more promising, details are regularly shared with stakeholders across Engineering and Product. Collectively they build a set of key scenarios, think about product–market fit, etc. They also start identifying a receiving manager and the leaders who will end up taking on the staff. The initial headcount typically comes out of the Incubation Manager’s current budget and headcount, and they may have to borrow resources from outside their org.

Decision 2: Incubation approval

(Required input: Light Initiative Brief and Opportunity Documents)

If the leaders involved believe that the project has a significant chance of being important and successful, they create an Initiative Brief to get it approved. The approver list should be kept small — preferably the CPO, the CTO, and EVPs or their delegates — so effort isn’t wasted at this early stage on building broad consensus.

At this stage, the Initiative Brief may be light on details in some sections if that information is not required for a decision on incubation. While the main part of the document focuses on the benefit to the company and a lightweight sketch of the proposed work to be done (similar to pitching an idea for a startup), there are other parts as well. For example, as part of that document, a destination for the project team will be chosen for all disciplines. It may be the case that the incubation team has dedicated engineering, product, and TPM (and potentially other disciplines). The receiving manager for all of those must be agreed at this point. These decisions can be reviewed as the project evolves, and any agreed changes will be notified to all stakeholders.

In addition, the initial proposed and likely needed headcount and budget for each phase of the project needs to be included in the Brief. It is expected that the understanding of headcount needs will evolve along with the project, and changes to the requested budget will be reviewed at each subsequent decision point.

Incubation phase

Once committed, the Incubation Manager and product owners build their team. During the early phases of the project, startup-like processes can be followed so that the team can iterate fast, fail fast, and succeed fast. Outcome: clarity around MVP and/or development plan, clarity around the expected UI / UX / API, potentially a demo.

“Startup-like processes” encourages incubation teams to move quickly during the early stages of a project that may not be approved for productization. It doesn’t mean incubation teams can arbitrarily choose whatever processes and tools they like. Using a subset of the company’s processes may be acceptable during incubation, but choosing solutions that conflict with the company’s standard processes is discouraged. For example, teams might choose to keep documentation in markup files alongside source code rather than writing scopes and PDs for review.

The roadmap that is the input to Decision 3 must capture any work required to fully transition projects to standard processes.

Decision 3: Roadmap to launch

The team prepares an updated, complete Initiative Brief and an Initiative Plan covering at least the work required to launch the initial version of the product or feature, including a timeline. As part of ratifying the plan, additional resources are approved. The project can be canceled at this point by rejecting the Initiative Plan, in which case the team members would be redistributed. Until the plan is approved or rejected, the project remains in incubation.

Productization

During productization, the team takes on normal procedures including all standard SDLC processes and tools. During this phase the receiving manager becomes increasingly involved in managing the project.

The model is for the initial incubation team to stay with the product all the way to being shipped and used by customers. The team may grow along the way, but knowledge stays with the code. This is crucial to avoid stalls between phases, to deliver on the initial vision, and to ensure the receiving manager is equipped to extend, maintain, and support the new product. In addition, the team must stabilize the product in production, so that no engineers are insulated from the reality of shipping production-quality software.

Transfer

At some point during productization, the team(s) — including engineering leaders and product managers — and responsibility for the product transfer to the receiving manager. From then on out, the receiving manager and teams own the product.

Additional notes

  • Incubation managers and product owners must be seasoned leaders who have built teams and/or orgs before. The nuances of an Incubation Manager’s role don’t lend well to being a new or junior leader.

  • Ideas are sourced from across the company by regular communication between the Incubation Manager and Product Managers, by having the Incubation Manager engaged in review processes across teams, and by having open channels (Slack, email, and in person) for ideas to be shared — including the results of skunkworks projects.

  • Resourcing for the project must be carved out early (and earmarked, even if not all hired or spent immediately), which is why headcount must be included in the Initiative Brief from Decision 2 onwards. It also must report “high” in the org, preferably to a C-level. For example, on one project I was involved with, a cross-platform initiative was sponsored by the CTO, who formed a team by transferring some experienced engineers to incubate the idea, which then grew to take the work to production. This is because in order to give the project a chance, it must not be put in the situation of competing for resources with other important and urgent things at each phase transition. The investment commitment, barring a directional change, is to carry the project through.

  • Generally, the preferred model is for the team created during incubation, including engineering leaders and product managers, to stay with the project and transition to the receiving manager. That said, all employee movement (or staying) must abide by standard policies for transfers, required headcount at the destination, backfills approved, etc.

    • The reason the team moves with the project is to create all the right incentives for producing great-quality software, and also not having any kind of “elitist” element to the incubation team.
    • In addition, if the project is truly this important, it cannot take the stall of having one team leave the project and another one be spun up.

  • To the degree possible, the incubation teams should have dedicated resources, and it should be considered whether they report directly to the team lead. For example, it may make sense for a product manager or TPM to report directly to the lead engineer. However, it may also be the case that an arrangement of “secondment” is agreed to. If you don’t believe they can succeed with secondment, cancel the project. If the product or TPM receiving managers don’t believe they can succeed with dedicated resources reporting to the lead engineer, cancel the project.

  • Given the diversity of aspects and inherent unpredictability of innovation, further analysis and design may yield little additional value on the margin. In this case, the next step is to leave observance behind and test these ideas in the breach. One approach:

    • Agree upon an objective way to determine if the innovation incubation process is working (in both qualitative and quantitative terms).
    • Pilot the best first guess at a specific process implementation by taking the pro forma process template described here and reducing it to practice — adding named individuals, identifying a specific first innovation area, and allocating a preliminary budget.
    • Assess how well the process is working and determine if adjustments are indicated. Perform this exercise about every six months to assure as little avoidable friction as possible — innovation is hard enough without it.

References