Public Meeting on Food Safety Risk Analysis Clearinghouse Data Quality Objectives
About the Meeting:
On December 5, 2000, the interagency Risk Assessment Consortium (RAC) together with the Joint Institute for Food Safety and Applied Nutrition (JIFSAN) held a public meeting to foster discussion and gain public and professional input on data quality issues as they relate to the Food Safety Risk Analysis Clearinghouse (Clearinghouse). The meeting served to fulfill the annual public meeting goals of the RAC. The meeting was held at the Crystal Gateway Marriott in Arlington, Virginia, which was also the site of the concurrent annual meeting for the Society for Risk Analysis. Several brief presentations were made, followed by a panel discussion and public comment period.
In accordance with the President's Food Safety Initiative, the
JIFSAN is charged with the development and operation of the Clearinghouse,
with guidance from the RAC. JIFSAN is a collaborative effort
between the U.S. Food and Drug Administration (FDA) and the University
of Maryland. The Clearinghouse is a specific program within
JIFSAN that is responsible for making information available for
supporting risk analysis. The RAC is comprised of many government
agencies, each with some role in food safety risk analysis. The
purpose of this meeting is to seek guidance and solicit input on
those standards that can be applied to data sets. It is a
means of addressing the ideal characteristics that should be made
available and should accompany a dataset to be used in food safety
risk analysis.
Agenda:
-Introduction, Dr. Will Hueston (Associate Dean, Virginia-Maryland Regional College of Veterinary Medicine) and Dr. Wes Long (Associate Scientific Director, Joint Institute for Food Safety and Applied Nutrition)
-Introduction to the JIFSAN Food Safety Risk Analysis Clearinghouse, Dr. Wendy Fineblum (Food Safety Risk Analysis Clearinghouse)
-Ideals in Risk Analysis, Dr. Dale Hattis (Clark University)
-Data Quality for Microbiological Risk Assessments, Dr. Marion Wooldridge (Veterinary Laboratories Agency, UK)
-Current Thinking, Dr. Will Hueston
-Panel Discussion, Dr. Marion Wooldridge, Dr. Dale Hattis, Dr. Bob Buchanan (Lead Scientist, FDA/Center for Food Safety and Applied Nutrition)
Presentation Summaries:
Introduction to the JIFSAN Food Safety Risk Analysis Clearinghouse, Dr.Wendy FineblumThe Food Safety Risk Analysis Clearinghouse is a collaborative effort between the Joint Institute for Food Safety and Applied Nutrition (a joint venture between the U.S. Food and Drug Administration and the University of Maryland), the Risk Assessment Consortium, industry, academics and international groups. The goals of the Clearinghouse are to promote risk analysis in food safety decision-making, encourage transparency and to provide access to data and information. The home page of the Clearinghouse contains links to such information as upcoming meeting and events, datasets, risk analysis terminology and completed risk assessments. The Clearinghouse faces many challenges, including access to resources, quality assessment of posted materials, and internet coordination and technical challenges. Access to resources and reluctance to divulge data can be addressed by communication, especially relating to confidentiality and management of data. Quality assessment can be addressed by requiring minimum standards and supporting documentation for all data to be posted on the website. This approach, as opposed to peer review processes, will allow users to determine if the data are appropriate for their needs. Visits to the Clearinghouse have been tracked and have dramatically increased in number and geographic diversity since its inception.
Ideals in Risk Analysis, Dr. Dale Hattis
There are four major relationships shared by risk assessors:
Relationship with our subject matter
Relationship with other disciplines
Relationship with our audience
Relationship with our colleagues
The relationship with our subject matter is the most controversial. The risk assessor must accept the reality of the world and realize that that reality matters. Risk assessment is the product of social processes that are inherently fallible. It is often necessary to rearrange the pieces of our observations into new theoretical structures. This has a direct relationship to the Clearinghouse because models and concepts of data analysis may be different in the future. It is important to keep our observations in the database as free from model-laden issues as possible and close to the elemental observations such that data can be used in ways that the original investigator may never have intended. Each step in the pathway to analyze data provides an opportunity to make measurements and interventions.
Relationships with other disciplines is important since risk analysis crosses the boundaries of subject matter. Risk analysis needs to be an interdisciplinary endeavor rather than just a multidisciplinary approach. We need to facilitate interactions, get to know each other's language (e.g., microbiology and economics), analytical tools and methods in order to understand each other. We need to devote our time to review the work of our peers and to share data in sufficient detail to allow our conclusions to be independently reassessed. This need to collaborate is the motivating factor for contributing to the Clearinghouse, shows confidence in our own analyses and contributes to the resources of the community. There are several key implications for building community databases (e.g., for microbial risk assessment). Participating in data sharing enterprises will enhance advancing the community goals of seeking the truth and checking hypotheses. For a more complete discussion of this topic, please see the December 2000 edition of the Human and Ecological Risk Assessment Journal, or email Dr. Hattis at dhattis@clarku.edu.
Data Quality for Microbiological Risk Assessments, Dr. Marion Wooldridge
The purpose of a risk assessment is to assess risk, provide insight into the process, identify crucial data deficiencies and to target further studies. Risk assessments are iterative. Since the data are never perfect and are rarely collected specifically for risk assessment, many types of data must be considered for inclusion into risk assessment. Good data may include complete data, relevant data and valid data. Complete data would include such things as the source of the data and the related study information, such as sample size, species studied, seasonality, sensitivity, specificity and precision of microbiological methods, and data collection method. Relevant data may depend on the risk question that is being addressed. Characteristics of relevant data include age of data, region or country of origin, purpose of study and species involved. Valid data is that which agrees with others in terms of comparison of methods and development of tests. Valid data may also be peer-reviewed data. Data that can be eliminated from the risk assessment depends on the stage of the assessment and the purpose of the assessment. In the early stages of risk assessment, small data sets or those with qualitative values may be useful whereas the later stages of risk assessment may include only those data that have been determined to have high quality standards. Data to be included in a clearinghouse or warehouse should be complete and valid. Relevant data is difficult to judge and may be case-specific. Unvalidated data can be included in certain circumstances, such as unique studies that are being reported for the first time. In summary, the purpose and stage of the risk assessment process determines the data that will be used. Good data are complete, relevant and valid; complete data are objective, relevant data are case-specific, and validation is time-dependent. Complete data should minimally be required for inclusion into a data warehouse or clearinghouse.
Current Thinking, Dr. Will Hueston
In posting data, the Clearinghouse must decide whether to make a subjective judgement on the quality of the data posted or, rather, set a series of standards to characterize the data. The Clearinghouse has chosen the latter. The Clearinghouse will make available data sets that are sufficiently described in terms of characterizing the conditions in which data were collected in order to allow the risk assessor to determine the usefulness for inclusion into risk assessments. The following list describes several important characteristics that the Clearinghouse feels are helpful to evaluate the usefulness of data sets for risk assessment, such as submitter information, data source, methods, and confidentiality.
General information, including:
Complete name and correspondence address of principal investigator
Purpose of study
Submission of raw data
Source of data, including:
Funding source/affiliation of principal investigator or data collectors
Who collected/produced the data
For numerical data, provide numerator and denominator
Study design, including:
Type of study
Sample size
Sampling frame/sample selection
How does the sample relate to the population (is the sample from a particular
country, region or producer?)
Data collection, including:
Method of data collection/compilation
Age of data
Country/region of origin
Time frame for collection (seasonality)
Conditions of collection (field versus laboratory data)
Microbiological methods, including:
Testing methods (which tests were run)
Sensitivity and specificity of test(s)
Techniques used
Precision of measurement
Definition of units being used
Species of animals used, if any
Specific organism tested or studied
Evaluation of information
Validity with regard to findings of other researchers
Publications using these data
Peer review
Investigator's evaluation of data
Investigator's recommended limitations of data
Collegial atmosphere for data sharing
Blinding
Confidentiality
Each of the panelists was asked to comment on the above guidelines and address any issues with regard to omissions from the list or suggestions for changes. Responses included:
-Inclusion of raw data is very useful in assessing the usefulness of data
-Definitions or prompts will alleviate confusion as to the meaning of various terms. This is especially important for international audiences and widely varying disciplines.
-Complete data is not always a reality
-Expert opinion must be used cautiously
-Confounding factors must be adequately addressed
-Peer review does not necessarily mean that the data are 'good'
-Language bias should be avoided when searching for useful data for inclusion into risk assessment
-Data should be scientifically plausible
-It may be helpful to refer to professional societies that have pre-existing ideals defined for inclusion of data into journal articles
Summary of Public Comments Directed at Data Quality Characteristics
Public comment was solicited after the panel discussion to gain input on the data quality objectives. A diversity of views was presented, for example:
-It was recommended that the Clearinghouse continue to evolve as the data quality objectives progresses.
-In response to Dr. Fineblum's presentation, it is helpful to allow space for comments at each prompt rather than drop down menus when preparing a questionnaire for web-based data submission.
-The description and characteristics of data must be correct and complete rather than only focusing on the data alone.
-Current contact information from the researcher or other contact person is essential.
-The Clearinghouse should serve as the go-between when correspondence needs to be established between the submitter and the user.
-It is important not to delete or reject data that is novel or unique. Although it may not meet the criteria established for data submission, it may be the only data available at a given time.
-Exercise caution on using preliminary data since they may not be able to replicated and may be controversial in nature.
-Data sets must be readily identified on the website, especially when changes have been made to the data. For instance, code numbers can be used to identify data, including amendments to data sets.
-Questions were raised regarding auditing of websites and referencing data posted on websites. For instance, will the website serve as a reference when data are used in publication.
-Longevity of the site is an issue that must be addressed (e.g., will data be available if the website is accessed several years from now).
-Ideal or minimum data characteristics, as opposed to gold standard characteristics, need to be defined to make it worthwhile for researchers to submit data to the Clearinghouse. Objectives may need to be broken down into pragmatic versus ideal standards of acceptance.
-The Clearinghouse must make the submission process feasible and
reasonable for the submitter.
Wrap-up
Participants in this meeting expressed concerns of confidentiality, data management, user friendliness and longevity of Clearinghouse data bases. The Clearinghouse will take all comments into account while developing guidelines for submission of data for use in risk assessment.
This meeting serves to address the data quality issues that are important to all RAC members. Based on the information gained from this meeting, the RAC will provide advice on data quality guidelines and data submission into the Clearinghouse.
