As a Principal Federal Statistical Agency, ERS must be knowledgeable about the issues and requirements of public policy and Federal programs pertinent to the USDA mission and be able to provide objective information that is relevant to policy and program needs. The unique alignment of resources and expertise create specific capabilities to produce important and influential data products that would otherwise not exist. This is demonstrated by ERS being the sole source provider of these data, including instances where ERS adds value in the form of recompilation and/or subject-matter expertise or how the data product is pertinent to the USDA mission and its relevance to Federal policy and program needs.
Data products must be branded as coming from ERS (when they are released by ERS), as standard best practice for documentation, to more accurately measure impact and to improve ERS’s profile as a Principal Federal Statistical Agency. This is demonstrated by links or other evidence of ERS branding on data product web pages and in tables, charts, etc. As applicable, data products should also cite the source of the input data (such as from multiple Federal agencies).
To ensure equivalent and timely access to all users, a schedule and mode of release must be developed and publicly conveyed in the calendar year prior to the planned release of a data product.
Persons or organizations that have a vested interest in the data and information provided in a data product are considered stakeholders. Interacting with and knowing the interests, positions, alliances, and importance to ERS of key stakeholders enables data product authors to interact more effectively with these individuals and adapt to their changing needs.
Measures of content relevance and quality of communication can include degree of accessibility, customer satisfaction with ease of use, or other stakeholder interactions from venues such as user conferences, exhibits and other promotional materials, demos, and media citations.
Recognizing the diversity of data users and their importance, all data products should employ a feedback/input mechanism—based on a strategy of engagement with users to help facilitate and prioritize data release.
Website Contact Forms, for example, allow for the elicitation of feedback from users in a secure and organized manner. Other forms of communication with data users include public meetings that seek comments from data users on recent and pending changes or providing technical support. Practices to improve communication with users employed by various statistical agencies were highlighted in the 2009 CNSTAT review.
2.7 Web usage statistics for data products should be regularly reported to appropriate ERS staff (quarterly, annually, or as appropriate to the release schedule) and evaluated.
Statistics about use may provide data on visits, page views, average time on site, referrers, and number of new/unique visits. This type of information can assist with priority setting and product refinement.
Objectivity is a measure of whether disseminated information is accurate and reliable and whether that information is presented in an accurate, clear, complete, and unbiased manner. Agencies should inform the public as to the strengths and limitations inherent in the information disseminated (e.g., sources of error, degree of reliability, and validity) so that users are fully aware of the quality of the information.
3.1 All data products are reviewed for data quality prior to dissemination.
Data products produced by ERS are thoroughly reviewed by knowledgeable staff prior to dissemination to verify the accuracy and validity of the data. The procedure used to conduct this review must be documented and available. Data are checked for internal consistency, consistency with other similar data sets or prior year versions of the same data set, and sources of error. Knowledgeable ERS subject-matter experts conduct “reasonableness” checks of the data. Where necessary, the data are edited and missing values are imputed using established statistical techniques to improve the utility of the data.
3.2 Subject to DPRC recommendation, data products must undergo an independent external review of methods at least every 10 years.
External reviewers bring to the review process diversity of perspectives and expertise to ensure the data product is objective, meaningful, and credible. The breadth and extent of review will be determined by the type of data product. For example, for data products that employ surveys or models, an external review could evaluate procedural methods and statistical validity. For compilations of data, the review could focus on the appropriateness of the data used and the clarity and adequacy of the documentation.
3.3 Where statistically appropriate, all data products must report measures of accuracy that accompany data elements.
Different types of data products might use different accuracy measures. For example, forecast error would be reported for estimates or projections, and estimates of sampling error and nonsampling error components (coverage error, measurement error, nonresponse error, and processing error), to the extent practicable, should be reported for sample survey programs. On the other hand, a data compilation can refer users to source agencies for information on data quality.
3.4 Data products should have an ongoing research program that examines methods and operations.
Research on methods and operational procedures must be ongoing for statistical and economic agencies to be innovative and cost-efficient in methods or practices for data collection, analysis, and dissemination. For example, improvements could include methodological research (modeling improvements, such as refining forecast methods and simulations) and/or operational procedures (improving data fielding and increasing the efficiency of data processing).
3.5 The production/dissemination process for premier data products should receive priority for IT investment and must undergo an evaluation of IT approaches every 5 years.
ERS will continue to invest in activities that ensure agency data products are of high quality, meet OMB guidelines and practices of statistical organizations, and meet the highest priority needs of our customers and stakeholders. This includes modern technologies for data collection, processing, management, and dissemination.
OMB requires that Federal agencies offer a high degree of transparency about data and methods used to derive statistics. These requirements enable the American public maximum access to government data and ensure reproducibility of government statistics, meaning “the capacity to use the documented methods on the same data set to achieve a consistent result.” 
4.1 Decisions made by ERS to initiate, terminate, or substantially modify the content, form, frequency, or availability of premier data products should trigger appropriate advance public notice.
Stakeholders and the public should be made aware of upcoming changes to premier data products.
Major updates/upgrades should be announced on the ERS website calendar of releases, within the data product itself, and where appropriate, via email notification to stakeholders/subscribers, or other types of communication. (Where appropriate, the Office of Communications should be notified directly).
4.2 All data products must be accompanied by accurate, transparent documentation that describes the source of the data, the method used to produce the data, definitions of data items, variables contained in the data set, sources of error, and, if applicable, limitations of the data.
Many analytical problems and misinterpretation of data can be avoided by providing comprehensive documentation. OMB Statistical Policy Directive Number 4 states that “With the exception of compilations of statistical information collected and assembled from other statistical products, these [federal statistical] products shall contain or reference appropriate information on the strengths and limitations of the methodologies, data sources, and data used to produce them as well as other information such as explanations of other related measures to assist users in the appropriate treatment and interpretation of the data.”
OMB provides detailed guidelines for, and a comprehensive list of, necessary components to be included in survey documentation (and other types of government data to the extent they are applicable) in section 7.3 of the Standards and Guidelines for Statistical Surveys. Some sample documentation elements include a description of variables used to uniquely identify records in the data file; a description of the sample design, including strata and sampling unit identifiers to be used for analysis; and a description of sample weights, including adjustments for nonresponse and benchmarking and how to apply them.
4.3 If data in an ERS product are similar to data reported in other ERS or Federal sources, the data product must explain the differences in a guide to users.
ERS data products should endeavor to clear up potential confusion with other sources of data. In certain situations, they should contain a guide to users to explain how best to use and interpret the data. For ERS data products that contain values derived using similar concepts or contain information that cannot be easily differentiated from other ERS or Federal data sources, a guide to users provides a way for users to fully understand the intended purpose of the data product and assist in distinguishing the best statistical series for the user’s intended purpose. In addition, if the product contains similar concepts to those in other ERS data products, then the data product must explicitly discuss differences in a prominent place within the product.
4.4 Premier data products must provide information on the update and revision history.
Data revisions can occur for a variety of reasons, including inclusion of new data or a change in the data source; seasonal adjustment and/or elimination of calendar effects; transition to a new base period; improvement of methods or a change in classifications, concepts, and definitions; or elimination of errors. To help the public understand and use the data, information should be provided that describes the revisions made, reasons why, and any implications. To ensure transparency of the revision procedure, where applicable, information should be provided that describes the revision procedure and assesses quality changes (for example, differences in data sources and calculation methods).
4.5 All premier data products must have an archival capability going back 5 years.
For purposes of reproducibility, ERS should be able to provide users with previous releases/versions of data offered in data products, either as part of a data product on the ERS website or upon request.
“Integrity” refers to the security of information—protection of the information from unauthorized access or revision, to prevent the information from being compromised through corruption or falsification.
5.1 All data products that use underlying primary, proprietary, and sensitive data must have a defined procedure for pre-dissemination review to ensure that privacy and confidentiality of individual responses are fully protected and that data are properly secured.
Data products produced by ERS are thoroughly reviewed by knowledgeable staff prior to dissemination to ensure that information is protected commensurate with the risk and magnitude of harm that would result from the loss, misuse, or unauthorized access to or modification of such information. Procedures should be documented and available upon request.
5.2 Data storage and processing, prerelease security procedures, and release procedures will be reviewed every 3 years for all data products.
Procedures for data storage, security, and processing must comply with OMB guidelines and the ERS Data Security Policy, particularly for primary, proprietary, and sensitive data. Methods used for pre-release review must conform to applicable security requirements be documented and available upon request.
5.3 Staff assigned to production of premier data products will undergo training for all related policies and standards.
An effective Federal statistical agency has personnel policies that provide training to encourage the development and retention of a strong professional staff who are committed to the highest standards of quality work. Training can also include efforts to enhance understanding of these guidelines.
Beyond required USDA AgLearn training for all employees, there are several other types of training that are available for data integrity, such as short courses offered by the Joint Program in Survey Methodology (JPSM) for data confidentiality; training provided for Federal Statistical Agencies on the Confidential Information Protection and Statistical Efficiency Act (CIPSEA), such as courses available in AgLearn or offered by USDA’s National Agricultural Statistics Service; and courses provided for specific data access, such as Census Title 13.
Data products have their most value when they are made available to the widest range of users for the widest range of purposes and impose no barriers to any person or group of persons. Accessibility refers to the ability of any user to obtain, manipulate, and save data.
6.1 Data products must be released in common machine readable formats that facilitate ease of use by a range of audiences.
ERS data products must be released in common machine readable formats that facilitate ease of use by a range of audiences and minimize the obstacles to using information contained in data files. New data products must conform to this standard and existing data products will be migrated over time. All data products must also meet Section 508 Accessibility Standards to ensure full access by the visually or hearing impaired.
Adherence is demonstrated by: 1) evidence of ‘open’ formats (CSV, JSON, bulk downloads, etc.), 2) evidence of a data model that supports ad hoc analysis, and 3) evidence of 508 accessibility (e.g., well-structured for screen reader technology; chart legends, alt tags, and D-links available; and interpretation not reliant on color).
6.2 Premier data must undergo usability testing in the design/development to ensure they are intuitive, navigable, and produce expected results.
Usability testing can help ensure data products are designed to meet users’ needs.
6.3 Data products must take steps to conform to OMB Open Data Guidelines.
The above recommendations for data quality address many of the OMB principles for Open Data: Public, Accessible, Described, Reusable, Complete, Timely, and Managed Post-Release. Data management procedures will be adopted going forward to support the quality and openness principles.
To implement open data guidelines, ERS data products will be captured in agency and Federal Government metadata inventories. In general, metadata is data that describes data. Structural metadata is information about how the data are stored and presented. Descriptive metadata is about the data content. The role of the data product manager is to coordinate with the Web Steering Committee and/or Information Services Division to initiate the development of appropriate metadata for their data product, particularly in the case of premier products.
 44 U.S.C. 3504(e).
 These standards apply to “Federal censuses and surveys” and, to the extent they are applicable, they “also cover the compilation of statistics based on information collected from individuals or firms…, applications/registrations, or other administrative records.”
 National Research Council. Principles and Practices for a Federal Statistical Agency: Fifth Edition. Washington, DC: The National Academies Press, 2013.
 Guidelines for Ensuring and Maximizing the Quality, Objectivity, Utility, and Integrity of Information Disseminated by Federal Agencies.
 Statistical Policy Directive Number 4 states that “Prior to the beginning of the calendar year, the releasing statistical agency shall annually provide the public with a schedule of when each regular or recurring statistical product is expected to be released during the upcoming calendar year by publishing it on its Web site.”
 Section 508 of the Rehabilitation Act requires Federal agencies to make their electronic and information technology accessible to people with disabilities.
 National Research Council. "Part II: Commentary," Principles and Practices for a Federal Statistical Agency: Fourth Edition. Washington, DC:The National Academies Press, 2009, pp. 14-54.
 Guidelines for Ensuring and Maximizing the Quality, Objectivity, Utility, and Integrity of Information Disseminated by Federal Agencies.
 OMB Statistical Policy Directive Number 4.
 For more detail, see Guidelines for Ensuring and Maximizing the Quality, Objectivity, Utility, and Integrity of Information Disseminated by Federal Agencies.
 “Federal Statistical Organizations’ Guidelines for Ensuring and Maximizing the Quality, Objectivity, Utility, and Integrity of Disseminated Information,” Federal Register, June 4, 2002, pp. 38467-70.
 OMB Circular A-130 states that agencies should “provide adequate notice when initiating, substantially modifying, or terminating significant information dissemination products.” OMB Statistical Policy Directive Number 4 states that “Statistical agencies shall announce, in an appropriate and accessible manner as far in advance of the change as possible, significant planned changes in data collection, analysis, or estimation methods that may affect the interpretation of their data series. In the first report affected by the change, the agency must include a complete description of the change and its effects and place the description on its Internet site, if the report is not otherwise available there.”
 OMB Memo M-13-13.
 In general, data models that support one type of analysis may not be ideal for other types of analysis. Flexibility to the end user can include, for example XML, JSON or CSV, but must be devoid of format imposed by the data product manager or the software used.
 OMB Memo M-13-13, see especially Attachment, I. Definitions, Open Data.