A P R O P O S A L F O R I N C R E A S I N G S P E E D O F VA L I D AT I N G S A R S - C O V-2 D I A G N O S T I C T E S T S

summary Rapid access to diagnostics for emerging high priority pathogens is of great importance from a clinical, public health and economic point of view. Currently in the United States there are several technological, regulatory and organisational improvements that might be adopted for the COVID-19 and future pandemics; In this article, we suggest a proposal for increasing the speed of validating SARS-CoV-2 diagnostic tests, suggestions include: a structured EUA data submission process, distributed FDA directed CLIA led validation

1 the opportunity and goals 1.1 Diagnostic tests are powerful tools, we need rapid access to them Well characterised diagnostic tests for diseases are powerful tools to help clinicians, public health and policy makers treat patients, and lessen the burden of communicable diseases on communities.In situations of newly emerging serious infectious diseases, rapid access to diagnostics tests is of utmost importance.Diagnostic tests allow clinicians to better treat their patients.Population screening allows public health officials to help trace and break chains of infection and find asymptomatic individuals.In conjunction with policy makers providing support, this allows for test, trace and isolate public health programs to reduce or eliminate community transmission.[4] 1.2 We need trustworthy knowledge of diagnostics performance As important as the test itself is clear and trustworthy knowledge of the tests performance characteristics.Although we do need tests "that work", it's important that we know how well tests work, which includes the uncertainty of those metrics describing the test's performance.
One of the objectives of the VITAL Act (Verified Innovative Testing in American Laboratories) of 2020 is to "ensure that laboratory-developed testing procedures are accurate, precise, clinicallyrelevant". [22] [21] In other words, it is necessary for an initial validation to be performed for each new diagnostic test and for that initial validation to be well characterised and consistent to allow comparison between different tests.

Emergency Use Authorisation, United States Code
The United States Food and Drug Administration (FDA) oversees the Emergency Use Authorisation (EUA) process, including that for CoViD-19 related medical devices.[5] The criteria for the issuance of EUAs are stated in Act 21 U.S.C. § § 360bbb-3(c)(1-5) [29].Included is the legal and binding duty to ensure: " " the known and potential benefits of the product, when used to diagnose, prevent, or treat such disease or condition, outweigh the known and potential risks of the product, taking into consideration the material threat posed and that

" "
there is no adequate, approved, and available alternative to the product for diagnosing, preventing, or treating such disease or condition This thought piece does not disagree with these objectives and criteria; it raises the question of whether and how we might fulfil these requirements but do so more quickly.

Initial response time
Crucial to the early access to diagnostics was the timely and accurate publication of the SARS-CoV-2 genome.[32] This enabled high complexity CLIA labs and commercial manufacturers to respond quickly; both in the production of the tests and in the quality control material necessary to begin reproducible measurement and validation of test performance.
Many elements of our molecular pathology diagnostic response to this pandemic have been exceptional.The rapid speed at which diagnostics were made ready by molecular pathology laboratories was on par with the 2009 H1N1 pandemic.[17][21] However the FDA's EUA process initially significantly slowed efforts to bring these tests to market.Without these tests, samples were sent to central public health labs which limited the scale of testing and slowed down the test to result time.Both testing capacity and responsiveness are crucial factors in getting ahead of outbreaks and containing them through public health counter measures.This lack of testing and responsiveness also limited clinicians and policy makers in treating and providing scientific, data driven leadership.[16] The FDA responded to these challenges and updated their guidance to accommodate more risk.(todo: find and cite press release about updated guidance on diagnostics) This was a crucial leadership decision that will surely be reflected upon and copied for future infectious disease outbreaks.

EUA backlog delays tests coming to market
Over the following months, almost 1 new diagnostic test was given EUA status per day.[14] The FDA simultaneously grew their diagnostics review team from 25 to 100 (as of mid October; [9]) with many more support staff.Despite these admirable efforts there remains a backlog of new diagnostic tests awaiting FDA review for EUA.[10][9] This backlog delays companies from bringing their test to market.This delay has a negative effect on our collective ability to leverage all supply chains and technologies as quickly as possible and bring them to bear against this new infectious disease.In hindsight this backlog and associate delay, along with many other shortcomings outside of the FDA's EUA process [7][20] may be shown to have had a significant negative effect on the ability of the United States to effectively respond to the pandemic.

Heterogeneous quality of test performance data
As previously stated, rapid access to tests with trustworthy evaluation of test performance is a fundamental requirement.Although many tests have now attained EUA, the quality of the documentation and validation of these tests is heterogeneous.[27] This characteristic has persisted since the beginning of the outbreak, is a large source of uncertainty for laboratory staff and managers making decisions about which test to purchase, and increases the burden needed to validate a new diagnostics test.The information heterogeneity shows itself in many forms, the first of which being information in EUAs.Some EUAs are thorough and concise with all the necessary information present.On the other extreme are EUAs that lack crucial information, have errors, or reference data that does not exist.The clearest example of this are primer sequences which although requested by the FDA are only present in 34% of submissions dropping to 5.8% of the most frequently used tests.[27] This is important information given the range of mutations which have been found to negatively impact test performance.[12,31,2] Performance degradation might have occurred in other tests and is unlikely to have been detected in all cases.Inclusion of primer / probe sequence information might increase the ability of laboratory, public health and others to be more proactive in the face of new mutant viral strains.[28]

No guarantee of accurate test performance data
Finally the current EUA process does not ensure the validation results reported are reproducible.In several instances [13][3] widely used diagnostic test were assessed and a significant difference was found between the limit of detection (LOD) reported in the EUA and that which the investigators measured using their own reagents and protocol.These are not isolated incidences and the sensitivity of diagnostics given by its LOD is important for effective public health Test Trace Isolate (TTI) programs.[1] As mentioned previously, in several cases the EUA process has not guaranteed disclosure of some essential attributes of diagnostic test characteristics.Similar to the previously cited example where LOD reported in EUAs was not reproducible, several EUAs also report LOD with non SI units of TCID50 ml -1 .[27] This prohibits investigations to reproduce the claimed performance characteristics.
The lack of information or guarantees about the performance of diagnostics tests has transferred the cost of validation to the labs adopting the tests.[21][19] Various open and closed information networks have taken up the role of sharing information.As suggested by others it would be powerful if there was an open, structured data repository of diagnostic test performance data.Similar to Wikipedia, lab staff could contribute validation data of diagnostic test performance; benefiting other labs, in return benefiting from validation data submitted by other labs and collectively benefitting the patients they serve.

Quality control and reference materials
Viral positive control materials were produced early on and are now widely available.Providers include a range of commercial and government sources.[6] Viral material includes positive clinical specimens, naked RNA, virus like particles, live virus and inactivated virus.

possible extensions to current validation process
These proposals revolve around 3 main themes.Firstly, the merits of defining a minimum set of required data for a diagnostics EUA and doing so in a structured form.Secondly, utilising the validation work already done by some of the 200,000 CLIA laboratories around the United States.Thirdly, combining existing quality control materials to provide a representative and extensible panel of synthetic clinical specimen challenges for tests to pass.We hope these suggestions aid further discussion around new models of validating tests, potentially helping bring tests to market sooner, both during this pandemic and any future pandemics.

EUA minimum structured data
The normal submission process for a new diagnostic tests accumulates thousands of pages and takes years.[25] The EUA process instead requires only tens of pages and should take days to complete.This might enable labs and manufacturers to legally distribute their test within days and meet the urgent need for diagnostics to contain and aid treatment of SARS-CoV-2.
Some tests awaiting review and approval by the FDA have taken over a hundred days and counting.This has largely been due to the volume of testing required which has stimulated a large market response.The correspondingly large volume of submissions has saturated the FDA's capacity to review new submissions in a timely fashion.
As mentioned previously the FDA has been exemplary in their continual adaptation and responsiveness.Expanding their team by many factors, [9] reducing and refining the minimum set of information required for EUAs and providing multiple template EUA documents.[11] These templates have aided manufacturers, laboratories and others in their EUA submission process by helping to ensure the minimum information is included.And secondly may have aided the FDA reviewers by providing a more standardised form of EUA submission which makes comprehending and reviewing the EUAs a faster process.
For the same EUA submission type, for example "Molecular Diagnostic Template for Commercial Manufacturers", each EUA submission should contain the same types of data.However despite the templates provided there is still significant heterogeneity in the data.[27] Adopting a tool to enforce structured data upon submission for an EUA might aid both the companies in ensuring all the necessary fields are completed, and the reviewers in checking the data.There are many providers of such tools such as TypeForm, Microsoft Forms, etc.These off the shelf solutions are cheap, secure, fast to configure and reliable.The structured data they take in could then be provided as is or additionally used to automatically populate the templates mentioned earlier.The applications for structured data are many but at a minimum might allow laboratories looking for a new test to more easily compare and search across the available EUA data.The structured data might also aid the FDA's internal processes, allowing for better prioritisation, communication with and tracking of applicants.

FDA directed CLIA led validation
The requirement of an EUA for a Laboratory Developed Test (LDT) has recently been lifted by the FDA.[15] Whilst there may have been a component of political pressure, this in part is due to the depth of expertise of the nation's CLIA laboratory staff.The LDT EUA rescission has allowed many hundreds of laboratories to legally begin immediate use of their LDTs for diagnosis.In conjunction with the LDTs they have developed and validated, many laboratories have and will continue to increase their testing capacity by evaluating and purchasing commercial test manufacturers equipment and reagents.It has already been highlighted that expert validation work of newly purchased diagnostic tests is being conducted by the purchasing labs, [21] as is required by CLIA regulations.[26] Taken together this validation work may collectively be being conducted with a much broader range of acceptable sample types.This is important as it has the potential to contain a sufficient number of specimens to be representative of the population of specimens and avoid being a statistically underpowered validation of a diagnostic test.Secondly this work is inherently aligned to the interests of patients, society, laboratories, regulators and test manufacturers by avoiding the potential unconscious bias of diagnostic manufacturers as they validate their own test.As previously suggested, [19] there is an obvious opportunity to leverage this expert validation work being conducted by CLIA labs.What follows is a potential model the FDA could design, implement and coordinate in this and future pandemics: 1.A prospective supplier seeking an EUA sends a number of copies of the machine (a minimum of 3, preferably more) and sufficient reagents (e.g. for 1000+ tests each) to the FDA along with their submission of structured data describing the test, its performance characteristics as assessed by the manufacturer, claimed capabilities, etc.

2.
Separately, for free or a fee, laboratories make themselves available to the FDA to review the EUA and validate machines.This is a task all have already had to do during this pandemic and seems prudent to harness this significant capacity of distributed talented work

3.
The FDA can conduct their own evaluation of the machine whilst also sending the draft EUA, a machine, and reagents to a random subset of available laboratories.Optionally a reference panel (see "Synthetic clinical specimen panel challenge " below) can be sent as well.Amongst other potential benefits a reference panel would enable harmonising results between laboratories

Current strategy: waiting for infections to get specimens to validate against
When a new pathogen emerges the availability of clinical samples to test with is very limited.This limitation slowed down the effective development and validation of much needed new diagnostics for COVID-19 pandemic.(todo: find and cite paper reiterating benefits of international sharing of specimens) Our current strategy is to wait for sufficient new infections to get sufficient samples needed for multiple labs, commercial manufacturers and government agencies to all develop and validate their diagnostics tests.If not addressed, this strategy will slow down the response to the next pathogen or strain that emerges.
Secondly the threshold for which diagnostics tests must overcome to show clinical efficacy was 20 and is now set at 30 positive patient samples.During the early stages of a pandemic, when testing is most important to control spread, there are insufficient samples which makes this barrier insurmountable.

New strains are harder to counter
In later stages of a pandemic that is under control there will also be insufficient samples.If there are many new infections, such as the United States and other countries like the United Kingdom are experiencing coming into the winter of 2020, there is a different problem, namely the increased infections, increases the chance of mutations rendering existing diagnostics less effective.[12,31,2] This is an identical, if not harder, scenario to contend with than at the outset of a pandemic.It is harder because there will be many infections from existing strains which makes identifying appropriate samples for testing against more challenging.Simultaneously, in the early phase of a new strain spreading, there will be a very low absolute and relative number of specimens for which to validate tests against.

Underpowered validation: Clinical specimen edge cases
Finally, the validation of new diagnostic tests is conducted using a random but small set of samples from clinical specimens.This is likely to be statistically underpowered whilst also likely not including many or any of the significant edge cases a diagnostic test would preferably be able to correctly deal with.Edge cases such as a concurrent infection with pus, elevated mucin levels, blood, or for saliva samples, the presence of food, or acidity from orange juice etc.Some EUAs, often submitted during periods of relatively low disease prevalence, used contrived samples based on repeating tests on the same clinical sample, further limiting the range of samples the test is validated for.Other EUAs have been given based on samples of RNA in solutions of stabilizing buffer RNA Diluent P which although likely justifiable given the circumstances, would seem to have significant room for improvement to meet US Federal Regulations calling for samples used (in proficiency testing) to "mimic actual patient specimens when possible".[30]

Routes to characterised viral material for reproducible validation of diagnostics
In the absence of a national or international standard of SARS-CoV-2 material the FDA has made available a reference panel of samples.[8] This material is an important step and one called for previously to enable the standardisation of diagnostic test sensitivity.[19] In the context of the current pandemic, wider and early access to such a control material would enable a more robust characterisation of diagnostic tests between different laboratories, machines, and batches of reagents and consumables.
As this material is still restricted to the diagnostics tests which have received EUA an alternative approach is the use of commercially available control materials.[6] As these are not yet benchmarked a harmonisation study being prototyped by JIMB is underway.[18] The results from this important work should allow the use of commercial control materials to yield commutable results of diagnostics test sensitivity.This study is predicated on the quality control material being homogenous and consistent with one batch.

Synthetic patient specimen panel
With the FDA reference panel, or an open source panel constructed from harmonised commercial viral material it would then be possible for multiple testing labs, and or by the FDA or another organisation like FINDDX [23] to conduct independent and comparable validation of diagnostic sensitivity.
If an open source list of ingredients for each of the reference panel specimens were made available, then over time it could be improved to more closely represent clinical specimens and extended to cover more edge cases.For example the inclusion of blood, mucin, a simulated concurrent bacterial infection and other interfering components and pharmaceuticals could be included in an extended reference panel.

Widespread test deployment
Potential process using synthetic specimens panel

Potential advantages of a synthetic patient specimen panel
Confronted with a new pathogen, or a mutant of an existing pandemic pathogen, it might be much faster to spike new synthetic or inactivated viral material into such a reference panel than wait for sufficient patient specimens to validate a diagnostic test.Such a panel would also offer the opportunity to include all the edge cases such a test should be able to handle.And finally with these consistent and characterised synthetic specimens a more consistent and reproducible validation of diagnostic tests may be conducted allowing the performance characteristics to be compared to each other.This would help lab staff make informed decisions about which diagnostic system to purchase or implement, this aids patient care, supports public health initiatives and provides a crucial piece of information to make the diagnostics market more competitive and efficient.

request for comment
This is a work in progress and any contributors who would like to join the draft with any size and scope of change are welcome.We look forward to receiving your thoughts and integrating them in this or a subsequent document.Please contact ajp@centerofci.org .

Figure 2 :
Figure 2: Potential model leveraging validation work being performed in CLIA laboratories to maximise speed, volume and accuracy of authorising emergency diagnostics.IFU: Instructions For Use.

Figure 3 :
Figure 3: How the use of a pre made clinically relevant synthetic patient specimen panel spiked with inactivated viral material may allow more rapid validation of diagnostics.Based on the assumptions that rapid access to diagnostics aids public health test, trace and isolate interventions to curtail community transmission and secondly that limited clinical samples delayed validation of new diagnostics.

4 .
The initial standard sets of validation data, including positive viral materials used, comparator test, etc. are uploaded to a centralised database 5. Any significant discrepancy between laboratory validation results are resolved 6.The company is authorised for a particular set of EUA performance characteristics 7.Only then can the submitting company know which labs are in possession of their test machine and may offer them to buy it This is one potential model for the distributed initial validation of diagnostics for emerging, high priority pathogens.It is coordinated by the FDA and uses the existing legislation but leverages the latent, capable and eager talent of our molecular pathology laboratories.