MAUDEScope method, version 0.1-live
Every figure on the MAUDEScope scorecard is computed by one of the definitions below, from the openFDA request printed under it. Nothing on the scorecard comes from a model, an average of anything, or a number typed into a template.
1. What "method 0.1-live" means
The specification defines a method version 0.1 computed by an offline pipeline over the bulk openFDA download. This build computes a different, smaller set of figures live in the browser, so it carries a different version string — 0.1-live — and every source line on the scorecard prints it. Two figures produced by two methods must never be able to wear the same version number; that is the entire purpose of versioning a method.
2. Populations: exact, and sampled
Two populations appear on the scorecard and the difference is stated on every figure.
- Exact, over every matching report. Counts, the quarterly series, the report source mix, the reporter mix, the event types, field completeness, identifier shares and the problem-code tables. Each comes from an openFDA aggregate query that covers the whole result set, however large.
- Sampled, over the 300 most recently received reports. The three delay intervals only. They need both dates off the same record, which no aggregate query can give, so they are computed from records — and the sample is the newest slice of the file rather than a random draw. That is the right sample for "how long is it taking now" and the wrong sample for "how long did it take in 2014", and the page says so.
3. Bounded counts, and why a range is printed instead of a number
openFDA returns at most 999 groups to an anonymous caller on a count= query, sorted by size descending. When the smallest group returned holds a single record, every group beyond it must also hold exactly one — it cannot hold more, because the list is sorted, and it cannot hold zero. The distinct count then falls out arithmetically and is exact: groups + (total − covered).
When the smallest group returned holds more than one, the records beyond the returned groups could be spread over anywhere between covered ÷ tail and covered further groups. The honest answer is that range, and the scorecard prints the range with the reason beside it. It never prints the midpoint, because a midpoint is indistinguishable from a measurement.
4. What this build does not compute, and why
Three metrics the specification defines are not on the scorecard. Each renders as a labelled absence rather than as a zero, because a zero would be read as a finding.
Estimated duplicate events, and the duplicate rate
The specification estimates distinct real-world events by clustering reports in two passes: an exact pass grouping on event_key, and a fuzzy pass using narrative similarity within a product code. Neither is available here.
openFDA serves event_key as the empty string on every record in device/event — all 26,136,889 of them.
That is reproducible in one request: device/event.json?count=event_key.exact&limit=5 returns a single group, term "", holding the entire endpoint. So the exact pass has nothing to group on. The fuzzy pass needs every narrative in the product code, which is the bulk download rather than the API, and it needs the hand-labelled validation set the specification asks for before its precision could be published at all. Publishing a duplicate rate without that validation set would be publishing an error bar nobody measured.
What the scorecard reports instead is distinct report numbers, which is exact and is a smaller claim: it counts submissions filed against the same MDR number, not separate reports describing the same event.
Summary-reporting exposure
Whether a product code appears in FDA's 2019 Alternative Summary Reporting release, or on the Voluntary Malfunction Summary Reporting eligible list, changes what its counts mean across a program boundary. Neither list is in openFDA, and neither source address has been verified for this build. The exposure is therefore left unmeasured rather than reported as absent: "not exposed" is a claim, and nobody here made it.
Revision behaviour
How often an already-published report is silently changed is measured by diffing weekly snapshots keyed on mdr_report_key. That needs at least two snapshots held over time, and a query made in a browser has one. It needs the offline pipeline.
5. Language and display rules
- No figure on any surface is a rate. There is no denominator in MAUDE to make one from.
- No manufacturer, brand or model is named on any surface in this version. The brand metric is a count of distinct strings; the strings themselves never leave the query layer, and the recall record type does not carry the recalling firm's name at all.
- Two product codes are never put in an order, and no difference between them is computed.
- No MDR narrative text is displayed anywhere. Narratives carry patient detail and manufacturer language, and neither belongs on an ungated page.
- Every figure states its population, its n, and the count of anything excluded from it. A figure whose exclusions are not counted does not ship.
6. Metric definitions
| Metric | Definition | Population | openFDA query |
|---|---|---|---|
| Reports in MAUDE | Count of MDR records carrying the product code in any device.device_report_product_code. | Every matching report | search=device.device_report_product_code:"CODE"&limit=1 → meta.results.total |
| Reports received, trailing 365 days | Sum of the daily date_received buckets falling in the 365 days ending on openFDA’s own meta.last_updated. Not "the last four quarters": openFDA cuts the file mid-quarter, so a four-quarter window is a different length every week. | Every matching report in the window | search=…&count=date_received, summed over the window |
| Report source mix | Share of reports by report_source_code — manufacturer, user facility, distributor, voluntary. Single-valued, so the shares add to 100%. | Every matching report | search=…&count=report_source_code.exact |
| Reporter category mix | Share of reports carrying each source_type value. The field is a list, so one report can appear in several rows and the column does not add to 100%. Values differing only by case or punctuation are folded together and the number folded is reported. | Every matching report | search=…&count=source_type.exact |
| Reporting delay (three intervals) | date_manufacturer_received − date_of_event; date_received − date_manufacturer_received; date_received − date_of_event. Reported as p50 / p90 / p95 in calendar days by linear interpolation between the two closest ranks — one method for all three percentiles. A report missing or carrying an impossible value in either date is excluded from that interval and counted as an exclusion. A negative interval is kept as published, counted, and named on the page. | The 300 most recently received reports | search=…&limit=100&skip=N&sort=date_received:desc, computed per record |
| Publication lag | Calendar days between the newest date_received in the sample and openFDA’s own meta.last_updated. How far behind the published file runs, measured rather than asserted. | Two dates, both from the data | derived from the sample and meta.last_updated |
| Field completeness | Share of reports where the field is present and not the empty string. For a field on the device array, a report counts when at least one device on it carries a value. openFDA represents "no value" two ways in the same endpoint — an absent key and an empty string — so the query tests both, and a query testing only one is wrong in a way that looks right. | Every matching report, one query per field | search=… AND _exists_:FIELD AND NOT FIELD.exact:"" &limit=1 → meta.results.total |
| Identifier joinability | Share of reports carrying a harmonised 510(k) number, and share carrying a UDI device identifier. These decide whether a MAUDE report can be joined to the clearance that put the device on the market, or to its GUDID record. | Every matching report | the completeness query above, on device.openfda.k_number and device.udi_di |
| Distinct brand strings | Number of distinct device.brand_name values across the product code, and the number remaining once case and punctuation are folded together. A proxy for how dirty brand naming is. The strings themselves are never displayed — see section 5. | Every matching report, bounded when openFDA caps the groups | search=…&count=device.brand_name.exact&limit=999 |
| Distinct report numbers | Number of distinct report_number values. Reports sharing a number are submissions filed against the same MDR. This is NOT an estimate of duplicate reports of the same event — see section 4. | Every matching report, bounded when openFDA caps the groups | search=…&count=report_number.exact&limit=999 |
| Quarters that stand out | Daily date_received counts rolled up to calendar quarters, with quarters holding no reports materialised as zero rather than left absent. A quarter is marked when it exceeds the median of the preceding eight quarters by more than three median absolute deviations AND holds at least 20 reports. The absolute floor is not optional: the MAD of a quiet product code is frequently zero, and without it a two-report quarter would be marked. | Every matching report | search=…&count=date_received |
| Recall annotation | A marked quarter is annotated with any FDA device recall for the same product code initiated in that quarter or in either of the two before it. The window is directional: a recall that starts after a jump cannot explain reports already filed, so a later recall is not attached. A marked quarter with no recall in the window is shown as unannotated, which is not the same as having no cause. No litigation or media annotation, in any version. | Every device/recall record for the product code | device/recall.json?search=product_code:"CODE"&sort=event_date_initiated:desc |
| Reports received in the last 30 days of the file | Reports with date_received in the 30 days ending on openFDA’s meta.last_updated — the file’s own date, never the reader’s clock, which would silently shorten the window by however far behind openFDA runs. Structured fields only; no narrative text is displayed on any surface in this version. | Every matching report in the window, listed to 50 rows | search=… AND date_received:[FROM TO TO]&sort=date_received:desc |
7. The completeness fields, and what each absence costs
| Field | openFDA path | What its absence costs |
|---|---|---|
| Date the event occurred | date_of_event | Without it a report can only be placed by when FDA received it, not by when it happened. |
| Date the manufacturer became aware | date_manufacturer_received | Without it the reporting delay cannot be split into its two halves. |
| Brand name | device.brand_name | Without it a report cannot be tied to a specific marketed device. |
| Model number | device.model_number | Without it reports cannot be grouped by model within a product code. |
| Catalogue number | device.catalog_number | Without it a report cannot be matched to a purchasing or inventory record. |
| Lot number | device.lot_number | Without it a report cannot be traced to a manufacturing lot, so a lot-specific pattern is invisible. |
| UDI device identifier | device.udi_di | Without it a report cannot be joined to the GUDID record for the device. |
| 510(k) number (harmonised) | device.openfda.k_number | Without it a report cannot be joined to the clearance that put the device on the market. |
| Device problem codes | product_problems | Without them the report can only be read as free text, so it cannot be counted by problem. |
| Patient problem codes | patient.patient_problems | Without them the consequence to the patient cannot be counted at all. |
| Narrative text | mdr_text | Without it the only account of what happened is the coded fields. |
8. Method version history
| Version | Change | Affects |
|---|---|---|
| 0.1-live | First published method. Browser-direct computation over the openFDA API, with the three metrics in section 4 left unmeasured rather than approximated. | Every figure on the scorecard |
A change to any definition above bumps this version and adds a row to this table. Figures computed by two methods never wear the same version string, and no definition is ever changed in place.