Evaluator Guide
This page is the accompanying documentation for the prototype. Print it (or save it as PDF) to produce the attachment referenced in the proposal.
A twenty-minute walkthrough of this prototype, organised by the six areas the technical approach is scored on, plus the bilingual standard Amendment 2 added. Every section names what to click and what it demonstrates.
Synthetic data only
Everything in this system is invented for evaluation. No real patient, provider, or Government information is present, and none should be entered. Reports filed here are never sent to VAERS.
Before you start
| Administration console | /admin |
|---|---|
| Demo credentials | Username cdc-evaluator. The password is given in the proposal attachment that accompanies this prototype. |
| Published form schema | version 1 |
| Browsers tested | Edge and Chrome on Windows, Safari on macOS and iOS, Chrome on Android, at widths from 320 to 1440 pixels. |
What is real and what is simulated
| Area | State | Detail |
|---|---|---|
| Form, branching, validation, completeness | Real | Driven by a versioned schema and a documented condition language. |
| Persistence of reports, drafts, surveys, events | Real | PostgreSQL through a data-access layer; Azure SQL Managed Instance is the documented production alternative. |
| VAERS-compatible output and exports | Real | Generated from each submission using the public data-set headers. |
| Transmission to VAERS | Simulated | Posted to a mock intake endpoint with retry and a receipt. The real interface specification is furnished at kickoff; only configuration changes. |
| Uploads | Real | Stored in Azure Blob Storage with server-generated names. Malware scanning is a documented seam; Microsoft Defender for Storage is the production path. |
| AI assistant | Real | Live model behind a provider adapter, grounded in published content, with guardrails, logging and a kill switch. It runs on Azure OpenAI, the same service CDC operates: a GPT-5.4-mini deployment in a Sweent Azure subscription. Moving it to a CDC subscription, or to EDAV, changes three environment variables and no code. |
| Symptom coding (VAERSSYMPTOMS) | Out of scope | Requires MedDRA licensing and coding; documented rather than built. |
| Single sign-on for administrators | Simulated | Demo credentials for evaluation. Microsoft Entra ID is the production path. |
| All data in the system | Simulated | Synthetic only. No real patient, provider, or Government information is present. |
1. Understanding of the problem and objectives about 3 minutes
- Open the home page. Report, FAQ, and Data are each one tap away, and the emergency notice is above the fold.
- Select Report. The first screen asks who is reporting, which is the distinction the solicitation introduces.
- Choose "a patient, parent, caregiver" and continue. Note the plain-language wording.
- Go back and choose "a healthcare professional". The same items now use clinical wording and the facility step appears.
- Switch the language to Español in the header. The whole site, including form labels and help, changes language.
What this shows. PRS#3 access paths, plain-language and clinical wording, the existing follow-up tool preserved as a link.
2. Soundness and feasibility about 4 minutes
- Complete a healthcare-professional report with two vaccines on the same date.
- Sign in to the admin console with the credentials above.
- Open Submissions, then the report you just filed.
- Read the VAERSDATA and VAERSVAX rows, then download either as CSV. The headers are the public data-set headers.
- Check the transmission status: the report was posted to the intake adapter and acknowledged with a receipt.
What this shows. PRS#6 VAERS-compatible output, a working data layer, and a real deployment on the Azure services CDC already operates.
| Capability | In this prototype | In the CDC environment |
|---|---|---|
| Web application (Next.js, Node 20) | Azure App Service (Linux) | Azure Web Apps |
| Relational data | Azure Database for PostgreSQL Flexible Server | Azure SQL Managed Instance |
| Uploaded documents | Azure Blob Storage (private container) | Azure Blob Storage |
| Secrets | Azure Key Vault via managed identity | Azure Key Vault |
| Logs, metrics, uptime | Azure Monitor and Application Insights | Azure Monitor |
| Language model | Azure OpenAI (GPT-5.4-mini deployment) | Azure OpenAI in EDAV |
| VAERS intake | Mock endpoint behind an adapter interface | The real intake interface, furnished at kickoff |
3. Innovation and originality about 3 minutes
- Open the assistant from any page and ask "Where do I download data?". It answers with a citation and a link.
- Ask "What is a lot number?". The answer is grounded in the published form instructions.
- Ask "Should I take my child to the ER?". The assistant refuses, redirects to a clinician or 911, and the refusal happens before any model call.
- On the review step, watch the essential-items meter and the list of what is still missing.
- On a healthcare-professional report with a hospitalisation, note the suggested documents.
What this shows. Grounded assistance with guardrails and a deterministic fallback, the completeness meter, and rule-driven document suggestions.
4. Technical risks and mitigation about 3 minutes
- Read the risk register below. Each row names where the mitigation is visible in the running system.
- Open Admin → Feature flags and turn the AI assistant off.
- Reload a public page and open the assistant: it now serves FAQ search results instead, with no error.
- Turn the flag back on.
What this shows. Risks are mitigated by design decisions you can see and toggle, not only by narrative.
| Risk | Mitigation | Where you can see it |
|---|---|---|
| The real VAERS intake interface specification is not available until kickoff | All transmission goes through one adapter interface with retry and status; the mock endpoint implements it today | Admin → Submissions → transmission status |
| Environment and access approvals can lag the schedule | The application runs on the same Azure services CDC operates and is deployable into a CDC subscription without code changes; configuration is entirely environment variables and Key Vault | This page, Architecture section |
| An AI assistant could give unsafe or ungrounded answers | The model runs on Azure OpenAI rather than a third-party service. Answers are grounded in published FAQ and form instructions only, medical advice and causation questions are refused before any model call, prompts are injection-filtered, exchanges are logged without form values, and an admin kill switch disables the feature instantly | Assistant panel; Admin → Feature flags |
| Accessibility regressions appear late | Automated accessibility checks run against every route in the test suite, and manual assistive-technology passes are recorded before each release | Accessibility section below |
| No baseline exists for abandonment or completion on the current form | The same first-party instrumentation used here can be deployed against the legacy form at kickoff to establish the baseline | Admin → Metrics |
| Paperwork Reduction Act clearance may delay survey deployment | Survey instruments are configuration and can be edited or disabled without a release | Admin → Surveys, Admin → Feature flags |
| Phase 2 image uploads arrive after launch | The accepted file types are an allow-list behind a feature flag; enabling images is a configuration change | Admin → Feature flags |
| Spanish wording needs professional review | Every string, including form labels and help, lives in editable catalogues and the form schema, so a reviewed translation replaces the draft without a release | Admin → Content, Admin → Form schema |
5. Alignment with performance requirements about 4 minutes
- Read the requirements table below and follow any row to the feature it names.
- Run the four low-code scenarios in the next section. None of them requires a developer or a deployment.
What this shows. Every performance requirement in scope has a place to click, with its status stated plainly.
| Requirement | What it asks for | Where | Status |
|---|---|---|---|
| PRS#1 | Branching logic for public and healthcare-professional reporters, and for vaccine administration errors Branching is data in the form schema, not code. Choosing "vaccine administration error" removes items 5, 18, 19, 20, 21 and 23 and requires the error narrative instead. | Report → first screen; then the vaccine-error branch | Demonstrated |
| PRS#2 | Intelligent validation, tooltips, and informational pop-ups Field and cross-field rules with plain-language messages, an error summary with anchors, tooltips on every item, and keyword nudges that never block. | Every step; try an onset date before the vaccination date | Demonstrated |
| PRS#3 | Redesigned navigation: report, FAQ, and data within two taps All three paths are one tap from any page. Emergency notice sits above the fold. | Landing page and header | Demonstrated |
| PRS#4 | Performance and analytics instrumentation First-party events only. Median submission time, abandonment, step entries, validation errors by field, and Core Web Vitals at the 75th percentile. | Admin → Metrics | Demonstrated |
| PRS#5 | Completeness of essential items The meter counts only the essential items visible in the current branch, so the vaccine-error branch is scored against its own shorter set. | Review step: the essential-items meter | Demonstrated |
| PRS#6 | VAERS-compatible data capture and transmission Exact public data-set headers for VAERSDATA and VAERSVAX. Transmission is simulated against a mock intake endpoint with retry and a receipt; the real interface is a configuration change. | Admin → Submissions → a report → CSV and JSON downloads | Demonstrated |
| PRS#7 | Customer satisfaction surveys Both instruments are editable in the admin console. Results are summarised under Admin → Surveys. | Footer survey and the survey shown after submitting | Demonstrated |
| PRS#8 | Low-code updates to content and to form fields and branching Four scripted scenarios below change wording, a field’s visibility, a vaccine brand, and a suggestion rule with no code and no deployment. | Admin → Content, Form schema, Suggestion rules, Vaccines | Demonstrated |
| PRS#9 | Medical record upload PDF, DOC, DOCX and TXT up to 25 MB. Images are refused with the Phase 2 explanation; a feature flag turns them on. Files are stored under server-generated names. | Report → Review step → Documents | Demonstrated |
| PRS#10 | Supplemental-document suggestions Deterministic rules over workflow data only, never over free text. Rules are editable in the admin console. | Report → Review step, healthcare-professional reports | Demonstrated |
| PRS#11 | Free-text supplemental information Maps to the continuation page of the VAERS 2.0 form and is carried in the canonical JSON. | Report → last step | Demonstrated |
| PRS#12 | Section 3 tasks (help desk, correspondence) Out of scope for the prototype; addressed in the technical volume. | — | Documented only |
| PRS#13 | Security and privacy controls HTTPS with HSTS, a content security policy with no third-party origins, CSRF protection, rate limiting, an audit log, and secrets in Key Vault. FedRAMP boundary work and ATO artefacts are post-award. | This page, Security section; response headers | Partially demonstrated |
| PRS#14 | Section 508 / WCAG 2.1 AA Automated axe checks run on every route in both languages as part of the test suite, with zero critical or serious findings. Manual assistive-technology results are listed below. | Every page; Accessibility section below | Demonstrated |
| PRS#19 | Bilingual support: English and Spanish for the report form, the satisfaction surveys, the landing page and navigation (PWS 1.13, added by Amendment 2) Every label, help text, option, validation message, nudge, step title, survey question and FAQ entry has an English and a Spanish text, and the same schema rules drive presentation and suppression in both languages. Switching language in the middle of a report keeps the draft. The Spanish text is complete and machine-drafted; a professional linguistic review is scheduled at kickoff. | The Español button in the header and footer of every page; the note under the home-page heading; walkthrough 7 below; Admin → Dashboard, Spanish coverage | Demonstrated |
Low-code scenarios
Each of these is a change a programme office would make on its own. None needs a developer, a release, or a restart.
L1. Add a vaccine brand
- Admin → Vaccines.
- Under Influenza (FLU4), add brand "Example Vax (brand)" with manufacturer "Example Biologics Inc.".
- Open a new report, reach the vaccine step, and type "Example" in the brand field.
Expected result. The new brand appears in the type-ahead and is exported as VAX_NAME. Adding it published a new schema version.
L2. Change the public wording of item 18 in both languages
- Admin → Form schema → the field ae.description.
- Change the public English label and the public Spanish label, and edit the tooltip.
- Publish the new version, then open a new report as a member of the public.
Expected result. The public sees the new wording in both languages, the clinical wording is unchanged, and a report already in progress keeps the old version.
L3. Hide item 24 (race) from public reporters
- Admin → Form schema → the field patient.race.
- Under "Show this field when", add the condition: ctx.audience is hcp.
- Publish, then open a new report as a member of the public.
Expected result. Race no longer appears for public reporters but still appears for healthcare professionals, and it is absent from the exported record for public reports.
L4. Add a suggestion rule for disability
- Admin → Suggestion rules → Add a rule.
- Set the document type, the label in both languages, and the condition: ae.outcomes.disability is answered / checked.
- File a healthcare-professional report that selects "Disability or permanent damage".
Expected result. The new document appears in the suggested-documents checklist on the review step.
6. Scalability and extensibility about 3 minutes
- Open Admin → Form schema. Every field, label, help text, visibility rule and requiredness rule is data, versioned on publish.
- Note the version history and the ability to roll back.
- Open Admin → Suggestion rules and read a condition in the builder, then switch it to JSON.
- A report already in progress keeps the schema version it started on, so publishing never disturbs a reporter mid-form.
- Read the capacity table below. It sets the volumes CDC stated in Amendment 2 (Q116) against how this prototype is sized and how the production deployment scales.
What this shows. Versioned schema, a documented condition language, rules in tables, feature flags, locale catalogues, and a stateless application tier sized against stated volumes.
| Measure | CDC, Q116 | This prototype | Production, proposed and not yet load-tested |
|---|---|---|---|
| Site visits | 2,256,501 a year (2025), about 6,183 a day | One App Service P1v3 instance serving server-rendered pages; no third-party requests | Autoscale to 2–4 instances on CPU; Azure Front Door with WAF in front |
| Report submissions | 22,016 a year, about 60 a day | PostgreSQL Flexible Server B2s; each submission is one transaction and one VAERS ID from a database sequence | General Purpose tier with high availability; the same data layer |
| Concurrent users | About 64 on average (15-minute sessions); real peak "several times higher", so we plan for five times | Rate limits are enforced in the application process, per instance | Rate limiting moves to the edge or a shared store, because per-instance limits do not add up across instances |
| Existing performance baseline | Not available | Admin → Metrics records page load, time to submit and abandonment from the first visit | The same instrumentation establishes the PWS 1.10 baseline at kickoff; the 30 percent abandonment reduction is measured against it |
7. Bilingual support (PRS#19, PWS 1.13) about 2 minutes
- On the home page, select Español in the header (or in the footer). The page, the navigation and the note under the heading are now in Spanish, and the note offers English.
- Select Report as a patient and complete the first two steps in Spanish. Every label, hint, option and error is Spanish; the items shown and hidden are the same ones the English form shows and hides.
- Now select English in the header. You are on the same step of the same report, with your answers intact: the language control carries the draft with it.
- Switch back to Español, finish the report and submit. The confirmation and the satisfaction survey that follows are in Spanish, and the survey response is stored with its language.
- Open Admin → Form schema and any field. English and Spanish sit side by side for the public wording, the clinical wording and the help text, and publishing changes both at once.
- Open Admin → Dashboard. The Spanish coverage panel counts every string in the schema, the site content, the FAQ, the surveys and the interface, and shows how many lack a Spanish text.
What this shows. PRS#19 asks for correct field presentation and suppression in both languages for both submitter types. The same schema and rules produce both languages, so the branching scenarios run in Spanish exactly as they do in English, and the test suite runs them that way.
Accessibility
The prototype targets WCAG 2.1 AA. Automated checks with axe run against every route in both languages on desktop and mobile viewports as part of the test suite, and the build fails on any critical or serious finding. Automated checking is not sufficient on its own. A keyboard-only submission of a complete report, a 320 pixel reflow check with no horizontal scrolling, a 200 percent zoom check, and a simulated screen-reader pass (a specification-driven virtual reader, not VoiceOver or NVDA, whose transcripts are committed with the tests) are part of the automated suite. A human VoiceOver pass in Safari and an NVDA pass in Firefox on Windows are scheduled and their outcome is recorded in the scenario record; neither has been completed at the time of writing, and this guide does not claim otherwise.
Specific choices worth noting: a skip link, one label per control, fieldsets and legends for grouped questions, an error summary that takes focus and links to each field, live-region announcements on step changes, focus moved to the step heading, focus trapped in dialogs and returned to the opener on close, no reliance on colour alone, and reduced-motion support.
Security and privacy
- HTTPS only, TLS 1.2 or higher, HTTP Strict Transport Security in production.
- A content security policy with no third-party script origins. There is no third-party analytics, advertising, or tracking anywhere in the application.
- Cross-site request forgery protection on every state-changing request, and rate limits on drafts, submissions, uploads, surveys, and the assistant.
- Secrets in Azure Key Vault, read through a managed identity. No secret is present in the code or in logs.
- Uploads are stored under server-generated names; the file name a person chose never becomes a storage path.
- Administrative changes are written to an audit log with the previous and new values.
- The assistant never receives anything typed into the form, and assistant logs contain no form values.
What changes after kickoff
- The VAERS intake interface specification replaces the mock adapter.
- The Government data dictionary and business rules replace the seeded schema and rule tables, using the same editors.
- Azure OpenAI in EDAV replaces the prototype model provider by configuration.
- Microsoft Entra ID replaces demo credentials for administrators.
- A professional linguistic review of the Spanish text, which is complete across the site, the form, the surveys and the FAQ (PWS 1.13, PRS#19).
- Paperwork Reduction Act clearance governs when the surveys are enabled.