A Compliance Matrix in under 10 minutes – Myth or Reality?

During the last 3 years, we at VisibleThread saw more than our fair share of Compliance Matrices.

And we continue to work closely with proposal managers who manually create compliance matrices. Depending on size, they spend anything from several hours to days; creating it, honing it, and getting it right.

Of course, no one reading this post will be too surprised at the time sales proposal managers spend creating a strong compliance matrix. After all, if your proposal is non-compliant, you may as well pack up and head home. From the perspective of government agencies and commercial entities, a non-compliant response can be a great excuse for culling the field.

So we had some questions

• Can we automate some or all of the process? If so which parts?
• Can we bring the initial creation of the compliance matrix, down from many hours to minutes, (well actually under 5 minutes to be precise)?
• If we automate the process can we keep it simple? Is it possible with no ‘prep work’?
• If we automate the process, is the output readable?
• Can we ‘shred’ both PDF and word formatted solicitations?

We wanted to answer these questions with real-world data and started down this path over 3 years ago. We are fortunate; we work with some of the biggest federal contractors (>$10b/annum revenue), right down to small businesses. Roughly speaking about 75% are government contractors and the remainder are commercial. So we had a good testbed.

We have come a long way on our journey, so I wanted to blog about our findings and answer the questions posed above. We have learned a lot. You may find it useful.

What do I mean by compliance?

Shipley defines compliance like this; ‘you have agreed to meet all requirements as asked in the bid request’. I like this definition. It is clear; it leaves no room for vagaries. It also aligns closely with the view held by the close to 100 organizations we have worked with over the last year. So, let’s go with this definition.

I should also say that compliance is not responsiveness, that is; compliance is not about communicating your differentiation. Larry Newman of Shipley wrote about this topic here; Compliance vs. Responsiveness. If you haven’t read it, you should.

So, it’s compliance with requirements that we’re interested in. Can we save time with automation?

General Findings

To help answer our questions, we took a sample set of data.

Who was in the sample?

1. Our sample included 28 organizations
2. 72% (20) of these are federal contractors
3. 26% (8) are primarily chasing commercial business
4. All 28 are North American-based.

Who’s using a compliance matrix?

1. Of our total sample, 79% (22) use a compliance matrix in their proposal development process.
2. All 20 of the federal contractors use a compliance matrix.
3. Of the 8 commercial vendors, only 2 use a compliance matrix and it depends on the bid size.

Government / Federal Findings

For the 20 federal contractors, is the process manual or automated?

1. 17 of the 20 federal contractors create the matrix manually and internally.
2. Of the remaining 3, 2 use automation in some form. Both have internally developed a set of MS Word Macros. These macros identify ‘shall’, ‘will’, and ‘must’… and create a first pass matrix in excel.
3. 1 of the 20 federal contractors outsources the creation of the compliance matrix to a 3^rd party for a fee based on the number of pages in the solicitation.
4. 100% of federal contractors create the compliance matrix either as the very first or second step of their proposal development process.

How do people create a compliance matrix?

If we consider the 20 federal contractors, we see a similar approach.

It begins with a formatted grid. In most cases, this is a Microsoft Excel spreadsheet. The grid typically contains these columns:

• From the solicitation doc; Col 1: ‘Page no’, Col 2: ‘Para’, Col 3: ‘text extract’.

• From the response; Col 4: ‘Page no / Para no’, Col 5: ‘volume’. Sometimes we have seen additional headings such as: ‘Compliant (yes/No)’ and ‘Person responsible’.

These are the common steps we see when creating the matrix:

1. If the solicitation is in PDF format, convert it to MS Word.
2. In MS Word, using search, identify all occurrences of ‘shall’, ‘will’, ‘must’, and any other indicators. In some cases, this is restricted to only looking at certain sections, for example, L & M.
3. For each occurrence found, populate a row in the compliance spreadsheet. This will include the page or paragraph number and the text of the compliance statement.
4. Once all occurrences are in the spreadsheet, then for each row, fill in the details relating to where in the response we meet the requirement.

So, can we create a first-pass matrix in under 5 minutes?

It would be naive to suggest that tools could magically crank out a “fully-baked” compliance matrix.

But… could we cut out a lot of the drudge work? cut several hours down to a few minutes? Well, the good news is yes, read on.

Here are the original questions we asked, along with some answers:

Can we automate some or all of the process? If so, which parts?

We have shown that steps 2 and 3 are good candidates for automation.

From our original sample of 20 federal contractors, 13 now regularly use VisibleThread for performing steps 2 and 3. 1 is actively comparing the results of their own internal word macros with VisibleThread outputs and considering a change.

Can we bring the initial creation of the compliance matrix, down from many hours to minutes, (well actually under 10 minutes to be precise)?

We have seen during the course of our studies that with automation, the total time for steps 2 and 3 above is between 3 and 5 minutes. This consists of uploading a PDF or MS Word solicitation, selecting the ‘dictionary’ of terms to scan with (e.g. ‘shall’, ‘will’), and generating a first-pass excel spreadsheet.

If we automate the process can we keep it simple? Is it possible with no ‘prep work’?

One of the key risks when automating any parsing process is this; users often need to ‘prep’ the automation. This often involves some kind of special massaging or formatting of the solicitation content, so that ‘tags’ or some other delimiter helps the parse algorithms.

Often this results in nothing more than automation getting in the way of the process and such products can become ‘shelfware‘. So, complexity is the enemy.

We prioritized simplicity over sophistication. For instance we only output page no, not paragraph number, but this means the output is very accurate and extremely fast. We were able to generate a compliance matrix in 3 clicks/steps.

• Step 1: Upload & analyze the PDF or Word doc (Time required 30-90 secs, hits 90 secs for lengthy docs in excess of 200 pages)
• Step 2: Run specific Compliance scan (Time: 10-20 secs, depending on doc size)
• Step 3: Export to Excel (Time: 5-10 secs, depending on doc size)

If we automate the process, is the output readable?

This was a big concern. If you automate an initial document parse, how will the output show? Will it be readable?

We went through 4 significant algorithmic tweaks to get this right. Our first tests back in early 2011 resulted in some pretty scrappy outputs. In the early days, probably the biggest mistake we made was how we ‘chunked’ (parsed) the text. We didn’t provide enough context text in our outputs. We got some really great feedback from a few major beltway integrators who tried it. This helped us tweak the algorithm, so these days the outputs strike the right balance in a majority of cases.

To give a flavor, I took an arbitrary PWS (Performance Work Statement) issued on FedBizOpps just last week (Feb 22^nd), uploaded it, and did a quick shred. (Total time: 90 secs)

Here is a screenshot of the resulting first-pass compliance matrix:

Can we ‘shred’ both PDF and word formatted solicitations?

On this front, we wanted to make sure that we could handle PDF and Word docs equally well, with sensible outputs. Technically speaking, while the parsing mechanics are slightly different, our results show very readable exports in both cases.

Summary Takeaways

I will leave you with some summary observations:

• Certain of your peers already have automation internally developed that helps efficiency.

• Tools are not the panacea, but the right tool definitely can save you a lot of time.

• We have evolved one way to do the task that is performing well ‘under fire’, and in short timeframes of under 5 minutes for quite large docs.
• You don’t always need a compliance matrix. For example, commercial proposals generally don’t require it. This is due to the small size and fixed format of responses.

To see how this all works watch this short video.