Shall statement extraction from an RFP

What a shall statement is

A shall statement is a sentence that creates an obligation on a named party, most often a bidder or the successful contractor. In procurement writing the convention is that shall and must are binding, should is desirable, and may is permission. A compliance matrix needs each of them as its own row, because the bid manager has to answer every shall and has to be able to show that none was missed.

The convention is not universal. Some documents define the words in a glossary and some use them loosely. TenderMatrix applies the common convention to every document and shows the word it relied on, so a reviewer can overrule a row in seconds.

The words that decide a row's type

These are the cue lists the converter applies. The Signal column in your matrix shows the cue that decided each row.

Mandatory

Desirable

Informational

Question

Notes on the cue lists

• Should is not read as Desirable when it is conditional: Should the Contractor fail to deliver, ..., if the system should fail and it should be noted are not preferences.
• May counts as Informational inside numbered clauses, list items and tables, and in a sentence that opens with a bidder-type subject (Bidders may...). A buyer right such as The Authority may terminate produces no row at all.
• Will followed by an action verb is Mandatory. The action verbs are provide, deliver, supply, ensure, maintain, submit, comply, perform, support, implement, meet, demonstrate, include, offer, furnish, notify, report, host, train, install, configure, guarantee, warrant, certify and a few more of the same kind.
• Mandatory as a word counts, but non-mandatory and not mandatory do not.

Tags and table flags

Some RFPs mark the type themselves. A tag at the start or end of a sentence overrides every wording cue, and a flag column in a requirement table decides the row.

A table row is typed in this order: its flag, then its own wording read like any sentence, then the heading above the table. If none of those decides it, a column headed Question gives Question and a column headed Criteria gives Informational. A row that is still undecided is listed as Mandatory, with a note on the row and a warning on the desk, because an unflagged row in a requirement table is more often binding than not. Check those rows first.

The heading rule also applies outside tables. List items and short clauses with no modal verb that sit under a heading such as Mandatory requirements, Desirable, Optional or Questions for bidders take that heading's type, with the signal heading: mandatory (or desirable, informational, question) and a note on the desk.

When a sentence has more than one cue

The first match in this order decides the type.

1. An explicit (M), (D) or (O) tag.
2. The buyer as the sentence's subject with shall, must, will or should: Informational. A buyer may produces no row.
3. A sentence that ends with a question mark: Question.
4. A mandatory cue.
5. Will followed by an action verb: Mandatory.
6. A desirable cue.
7. An informational cue.
8. A question opener such as Describe or Explain: Question.

So The vendor shall provide training and should offer a manual is Mandatory, and Describe how the Contractor shall manage risk is Mandatory too, with the row also marked as asking for a written answer. A cue beats a question opener, but not a question mark.

A worked example

This clause group goes in:

## 4. Service levels

4.1 The Contractor shall maintain a service desk available 24 hours a day.

4.2 The Contractor should offer a named account manager.

4.3 The Authority shall provide access to the site within 5 working days of award.

4.4 Bidders may propose a variant bid.

4.5 Describe how the Contractor will escalate a failed fix.

4.6 Response time: 4 hours for priority 1 incidents.

This is what comes out. The table is produced by the same code the converter runs.

Clause 4.6 is a real requirement and it produces no row. That is the main way the method fails, and it is why the Mandatory rows are a first pass to read against the RFP, not a substitute for reading it.

From PDF to typed rows

1. Mark the pages If the converted text carries page markers, rows get a page number. Running headers and page footers that repeat on every page are removed first.
2. Set aside contents pages Contents entries, dot-leader lines and footer lines are skipped and counted, so they never become requirements.
3. Read the structure Headings, clause numbers (3.2.1, C.3.1), sub-items ((a), (1), (i)) and the document's own IDs (REQ-101, Q4) give every sentence a section.
4. Read the tables A table with a Req ID, Requirement or M/D/O column becomes one row per table row, keeping the table's own IDs and flags. Other tables, such as price schedules, are left alone.
5. Classify each sentence Each sentence is checked against the cue lists on this page, in the order shown below, and the first match decides the type. The matched word is kept in the Signal column. A row with no cue takes the type of the heading above it, when that heading names one.
6. Expand lists When a clause with a modal verb introduces a list, each list item becomes its own row, typed by the lead-in.
7. Merge, trim and count Identical wording that repeats is merged into one row with a note. Text longer than 600 characters is cut with a visible note. Totals by type and by section are counted.

Where the rules go wrong

• No modal verb, no row. Specifications written as facts (Response time: 4 hours, Delivery within 30 days) are missed unless they sit in a requirement table, under a lead-in with a modal verb, or under a heading that names a type.
• Conditions stay in the text. If the Contractor subcontracts, it shall notify the Authority is one Mandatory row. The condition is not modelled, so read it before you mark the row not applicable.
• Cross-references are not followed. The Contractor shall comply with Schedule 4 is one row. Schedule 4's own obligations become rows only if Schedule 4 is part of the PDF you uploaded.
• Buyer duties that do not open the sentence. Following award, the Authority will provide access can be typed Mandatory, because the buyer check looks at the start of the sentence.
• Two obligations in one sentence stay in one row. Split them in Excel if your owners differ.
• Notices that use should. Bidders should note the closing date is typed Desirable, because the word should is there. Delete such rows in Excel.
• Defined terms. A glossary that redefines shall or should is not read.
• Headings can mislead. A row typed from its heading is only as good as the heading. The Notes column and the desk's notes say when it happened.
• English only. The cue lists are English.
• Page numbers exist only when the converted text carries page markers.
• Long clauses are cut at 600 characters with a note and the source line.
• Scans and multi-column layouts. Reading order errors can join sentences from two columns, and a misread word can hide a shall. Check scanned documents row by row.

Questions bid teams ask