A Practical Tool and Process for Long Term Use
For information contact: Jonathan A. Morell jmorell@syntek.org
Syntek’s R&D Planning and Outcome Monitoring tool (POM) is a method by which outcomes of R&D projects can be determined and tracked over time, on a routine basis. The POM is:
- applicable at all stages of the R&D process, from basic research to commercialization,
- able to assess research projects, groups of projects, and entire R&D portfolios
- scalable across those levels of aggregation, and
- able to detect changes and trends over time.
It does all this while imposing a minimal burden on the task of data collection.
Other functions of the POM are that it identifies:
- stakeholders and evolving changes in their interests, and
- critical incidents that explain R&D outcomes.
Because of these features, the POM can address questions such as:
- Which types of technologies that we fund provide the most desirable outcomes?
- What were the most common reasons why prototypes did not function as planned?
- What are the most common reasons why markers of R&D success fall behind schedule?
- At which technology readiness level are we most successful at funding successful projects?
- What is the relationship between a project being late, and its success with respect to its desired technology outcomes?
- Are the success rates different for projects depending on the time needed to get from project start to the first success deliverable?
Design Approach
POM is designed to provide a record that avoids many common data quality problems (e.g. inconsistent data, or gaps in data recording). By providing an accurate record over time, POM buffers against problems of hindsight bias, confirmation bias, and the planning fallacy. POM does this by adhering to the principle that the best way to evaluate R&D is to collect a small amount of carefully chosen data on a routine basis.
Routine collection of small amounts of data is superior to trying to collect a lot of data frequently because so doing imposes a burden that will result in poor quality information. It is superior to infrequent data collection because with infrequent data collection, trends and inflection points cannot be detected.
Operating within this design philosophy, POM is structured so that:
- Data collection is determined by calendar, not by event.
- Prospective and retrospective views are paired.
- Users never have to enter more than a small amount of information.
- Projects that require a “deep dive” evaluation, separate from routine assessment, are flagged.
POM Structure
POM is structured by four linked spreadsheets.
Project description: foundational information, e.g. name of project, project manager, organizational home, and so on.
Project Description Screen Shot
Stakeholders: identifies who the stakeholders are, and the reasons for their interest in the work.
Stakeholders Screen Shot
Success indicators: identifies three classes of success: 1) R&D technical activities, 2) knowledge transfer and sharing, and 3) technology dissemination and commercialization. Specifics are identified within each category, e.g. “demonstrate proof of concept at a professional meeting”, “publish an article”, or “complete a market ready prototype”.
Success Indicators Screen Shot
Critical factors: identifies important issues that may facilitate or inhibit progress, e.g. “expert technical consultant may not be available”, “computer models may be too sensitive to initial conditions”, “promised commercial partners may choose to back out”, and so on.
Critical Factors Screen Shot
Critical factor explanation: allows explanation and elaboration about critical factors
Critical Factor Explanation Screen Shot
Each worksheet has a prospective and a retrospective view:
- What will happen?
- What has happened?”
POM worksheets are customized for each setting, using drop-down menus to simplify the task of data entry.
POM Process
In-depth interviews are conducted with relevant personnel to set up the initial worksheets. After that, brief interviews are conducted every three months to update the data. Interviews can be conducted either by an organization’s personnel, or by consultants. What matters is that because of the POM’s structure, the ongoing data collection and data entry can be carried out by mid-level personnel, with a minimum of demand on leaders’ time. Data from each collection period are stored in a fashion that allows easy perusal of the data history. POM worksheets are linked in order to assure the objective of: “Enter once, use many times”.
Do you have any questions about how to go about determining the consequences of your R&D programs? Ask our experts!