Using Proprietary Data

USDA uses data from several sources to conduct research and provide economic intelligence to policy makers and the public about critical policy issues the Nation faces. To enable research on a variety of food-related issues (e.g., food access, health, and security), USDA has developed a data system called the Consumer Food Data System (CFDS). The system is made up of four disparate data clusters: USDA surveys, supplements to other existing surveys, administrative data, and proprietary data sources. The proprietary data sources described on this page are used by the USDA, Economic Research Service (ERS) for research related to USDA issues; however, access to these data sets is restricted.

 

Information Resources, Inc. (IRI) Scanner Data

About the data

To address policy and programmatic issues of interest to USDA, ERS acquires proprietary household and retail scanner data from Information Resources, Inc. (IRI), a global market research firm.

Retail-based scanner data

IRI OmniMarket Core Outlets (formerly IRI InfoScan) provide in-store consumer purchase data in the form of weekly revenues and quantities of each Universal Product Code (UPC) sold by store. The retailer point-of-sale data includes consumer food purchases and detailed information for both packaged food and random weight fresh food. The food retail sales data are available for individual store locations or market areas, covering a variety of outlet types (including grocery, club, convenience, dollar, drug, and mass merchandiser stores). The purchase data are then linked to detailed food product characteristics and nutrition data for food products. The dataset includes billions of transaction records, covering a large portion of retail food-at-home sales in the United States. The data are available for the years 2008–20.

For more information:

Understanding IRI Household-Based and Store-Based Scanner Data

Household-based scanner data

The IRI Consumer Network™ incorporates household panel data from the National Consumer Panel (NCP) and is a nationally representative panel of households continuously recording purchase information. More than 120,000 households use a handheld scanner or mobile application to record what food products they purchase and where they shop on an ongoing basis. Households record all food-at-home (grocery store) purchases, including random weight fresh foods. The purchase data link to detailed product characteristics and nutrition information. The data include household demographic information, and a subset of households also report health information (MedProfiler) and prescription drug purchases (RxPulse). The data are available for the years 2008–20.

IRI Weekly Retail and Monthly Household COVID-19 Response Data

The IRI Weekly Retail and Monthly Household COVID-19 Response Data contain information on nationally representative weekly food retail sales and monthly household food purchases at the product-level (e.g., subcategory) spanning the duration of the pandemic. The available levels of geography include national, regional, State, and market (e.g., Baltimore, Maryland/Washington, D.C.). The retail data include total sales value and volume-equivalent units sold by product and geographical unit. The household data include a projected number of buyers, expenditures, number of trips, expenditure per trip; variables are at the level of all households, as well as stratified by household income status. Year-ago values are included for all variables. The data are available for October 2019 through present (data through September 2022 are expected to be available).

For additional information, please contact Patrick W. McLaughlin, Xiao Dong, or Alexander Stevens.

Data access

Access to proprietary IRI data is limited to researchers collaborating on USDA-sponsored projects. USDA-sponsored projects include USDA grants, USDA cooperative agreements, and/or direct collaboration with USDA researchers on an issue of interest to the Department of Agriculture. For a sponsored research project, the institution affiliated with the research collaborator must enter into a Third-Party Agreement (TPA) with the data provider. The language of the TPA is specified by the Vendor and must be signed as is (TPA Template).

Steps to obtain a TPA:

  1. The ERS point of contact works with collaborators to complete IRI’s TPA form. The collaborators will fill out the appropriate section on the TPA and sign it. The form is returned to the ERS Data Steward to review and sign, then forwarded to IRI for review and signature.
  2. A senior official from the collaborating institution needs to sign the TPA for that institution. The confidentiality clause holds the collaborator/institution responsible for the use of the licensed materials by any third party with which the collaborator/institution shares the data. ERS and collaborators shall maintain a record of each recipient requiring access to the data.

ERS has established an efficient and secure data enclave that meets the Federal Information Security Management Act (FISMA) requirements, where access to external users is provided only through secure channels.

To obtain the price structure of accessing the data enclave, please reach out to the ERS Data Steward at david.dudgeon@usda.gov.

Documentation

Several reports (listed directly below) provide detailed information about proprietary retail scanner data—including methodology, characteristics, and statistical properties of the data:

Understanding IRI Household-Based and Store-Based Scanner Data

This report examines commercial scanner data, from the market research firm IRI, for use in food economics research (report authored by Mary K. Muth, Megan Sweitzer, Derick Brown, Kristen Capogrossi, Shawn A. Karns, David Levin, Abigail Okrent, Peter Siegel, and Chen Zhen). The report examines the methodology, characteristics, and statistical properties of the data sets. The report also provides an introduction to the data for new users and important considerations for advanced users (April 2016).

Linking USDA Nutrition Databases to IRI Household-Based and Store-Based Scanner Data

In this report, the researchers created a purchase-to-plate “crosswalk,” linking data between USDA data and household and retail scanner data to measure the overall healthfulness of Americans' food-at-home (FAH) purchases (report authored by Andrea Carlson, Elina T. Page, Thea Palmer Zimmerman, Carina E. Tornow, and Sigurd Hermansen). Substantial improvements in the healthfulness of Americans' FAH purchases would be needed to comply with Federal dietary guidance (March 2019).

Statistical properties

Other technical reports (listed directly below) provide independent assessments of data quality.

Examining Food Store Scanner Data: A Comparison of the IRI InfoScan Data with Other Data Sets, 2008–2012

This report looks at proprietary retail scanner data (InfoScan) that are used to examine food policy questions (report authored by David Levin, Danton Noriega, Chris Dicken, Abigail Okrent, Matt Harding, and Michael Lovenheim). To determine how representative the data are, this report compares the number of stores and sales where revenue is reported in the InfoScan data with the same information from other datasets (October 2018).

Food-at-Home Expenditures: Comparing Commercial Household Scanner Data From IRI and Government Survey Data

This report compares proprietary household scanner data to nationally representative Government survey data and finds that reported household food-at-home expenditures in commercial scanner data were lower than in two Government surveys (report authored by Megan Sweitzer, Derick Brown, Shawn A. Karns, Mary K. Muth, Peter Siegel, and Chen Zhen). The report details the comparison methodology and describes implications for using the commercial data in food economics research (September 2017).

Value-added products

ERS offers several products (listed directly below) to authorized users that enhance the use of the IRI scanner data. These products derive from both IRI scanner data and other proprietary and public data sources. A description of each product follows.

Scanner Data Store Weights

USDA (in collaboration with RTI International) developed weights that adjust store-level sales data to be representative of the retailer channel at the geographic levels of national, Census Region, and the top 10 most populated U.S. Metropolitan Statistical Areas (MSAs). Weights are currently available for 2012–2018 and will be available for 2019–2020, in mid-2022. User documentation and more information are available from RTI International (including user’s guide in appendix F). For additional information, please contact Xiao Dong.

Monthly Food at Home Price Database

USDA (in collaboration with RTI International) developed unit prices and a series of price indices for 91 ERS Food Purchase Groups (EFPGs) that are representative at the geographic levels of national, Census Region, and the top 10 U.S. metropolitan areas. The Monthly Food at Home Price Database is an updated version of the Quarterly Food at Home Price Database that was created for 1999–2010. The price indices are computed in six different ways, including common methodologies, as well as several complex formulations accounting for changing representative market baskets. The time coverage is 2016–2018, with updates to come for 2019–2020. The Overview of food code mapping for ERS Food Purchase Groups and the monthly Food-at-Home Price Database provides more information on the price indices and EFPGs. For additional information, please contact Anne Byrne or Megan Sweitzer.

Purchase to Plate Crosswalk

The Purchase to Plate Crosswalk (PPC) allows researchers to import the in-depth nutrition data from USDA into the IRI data. USDA nutrition data are more extensive than the nutrition facts panel information included in the IRI data. Among other things these data allow researchers to measure how well consumer purchases or store sales adhere to Federal nutrition guidance by calculating Healthy Eating Index (HEI) scores. The PPC is available for IRI years: 2013, 2015–2018. For additional information, please contact Andrea Carlson or Christopher Lowe.

More information can be found in the following reports:

Linking USDA Nutrition Databases to IRI Household-Based and Store-Based Scanner Data

Development of the Purchase to Plate Crosswalk and Price Tool: Estimating prices for the National Health and Nutrition Examination Survey (NHANES) foods and measuring the healthfulness of retail food purchases

Purchase to Plate Price and Ingredient Tool

The Purchase to Plate Price Tool (PPPT) estimates food prices for foods reported to be consumed by participants in the What We Eat in America survey, the dietary component of the National Health and Nutrition Examination Survey (WWEIA/NHANES). The Purchase to Plate Ingredient Tool (PPIT) breaks the food prices into ingredient amounts and quantities, based on market purchases. Researchers can use the PPPT and PPIT on all the retail IRI data or a subset of the data, such as geographic region or food store outlet types. The PPPT is available for NHANES 2011–12 through NHANES 2017–18, while the PPIT is available for NHANES 2013–14 through 2017–18. The Purchase to Plate National Average Prices (PP-NAP) for NHANES are publicly available data. For additional information, please contact Andrea Carlson or Christopher Lowe.

More information can be found in the following reports:

Estimating Prices for Foods in the National Health and Nutrition Examination Survey: The Purchase to Plate Price Tool

Development of the Purchase to Plate Crosswalk and Price Tool: Estimating prices for the National Health and Nutrition Examination Survey (NHANES) foods and measuring the healthfulness of retail food purchases

 

The NPD Group Data

About the data

In response to the COVID-19 pandemic, the U.S. Department of Agriculture (USDA) Economic Research Service (ERS) acquires new proprietary data related to U.S. households’ food-away-from-home (FAFH) behavior from The NPD Group. These data provide information related to consumer purchases and acquisition behavior and trends prior to and throughout the COVID-19 pandemic. For information about trends in FAFH consumption behavior, USDA receives NPD Consumer Reported Eating Share Trends (CREST) data. These data provide national level spending estimates, derived from consumer level surveys related to FAFH purchases. In addition, USDA receives CREST Performance Alerts, a GPS based data product that tracks trends in FAFH transactions across the United States. 

The NPD Group provide multiple data products to understand these trends. Below is a summary of each data product and some information therein.

  1. NPD CREST (Consumer Reported Eating Share Trends) – CREST contains NPD provided nationally representative estimates, across various demographic categories, that provide insights into consumer food behavior and consumption within commercial foodservice. CREST is a syndicated database that is designed to measure consumer restaurant behavior, captured through email surveys from a representative sample of U.S. consumers. These data include information on the total consumer reported individual visits to foodservice retailers, (nominal) dollars, and servings (for food items) consumed across different demographics and census regions/divisions. The data also contain estimates for information on how individuals obtain their food, and what share of money is being spent in different sectors of the food industry on a monthly 3-month rolling average basis. Current data began in January 2019 and are updated monthly, with a 2 to 3 month lag time.
  2. NPD CREST (Consumer Reported Eating Share Trends) Performance Alerts – CREST Performance Alerts provide information on restaurant transactions and share trends reported weekly (relative to the same week a year prior). This information is an aggregate of 75 major restaurant chains in the NPD database and is a look at how transactions change at a national level across different segments (quick-service restaurants, full-service restaurants) and channels (casual dining, mid-scale), as well as by census region and division. This service is fueled by modeled GPS data, along with NPD’s broad portfolio of assets for the foodservice industry—including CREST, ReCount, Checkout, and financial transaction data. ERS receives historical data (beginning the first week of January 2019 to the present) and the data are updated on a weekly basis, with a 1-week lag.

For additional information, please contact Keenan Marchesi.

Data access

Access to proprietary NPD data is limited to researchers collaborating on USDA-sponsored projects. USDA-sponsored projects include USDA grants, USDA cooperative agreements, and/or direct collaboration with USDA researchers on an issue of interest to the Department of Agriculture. For a sponsored research project, the institution affiliated with the research collaborator must sign a third-party agreement (TPA).

USDA relies only on secure data enclaves that meet the Federal Information Security Management Act (FISMA) requirements to provide access to USDA external collaborators.

Documentation

Given the novelty of these data, research is still ongoing. However, as part of the ERS COVID-19 Working Paper Series, several reports are in progress.

COVID-19 Working Paper: The Impact of COVID-19 Pandemic on Food-Away-From-Home Spending

A report which utilizes the NPD CREST and CREST Performance Alerts data product to discuss trends in FAFH spending throughout the COVID-19 pandemic (report authored by Keenan Marchesi and Patrick W. McLaughlin). This report covers spending patterns and changes in restaurant transactions from January 2020 through April 2021, by looking at average total spending (on a 3-month rolling average basis) and compares the patterns to previous years’ trends (March 2022).

 

Nielsen TDLinx Data

About the data

To study the extent and characteristics of people and places that lack access to healthy and affordable foods—and the relationships between food access, food shopping and spending patterns, and diet and health—Economic Research Service (ERS) licenses proprietary retail location data from TDLinx.

The NielsenIQ TDLinx database provides national Food At Home (FAH) retail location information, using a number of both internal and external resources. The database is updated on a continuous basis. The locations are maintained to account for changes within the food retail environment (i.e., openings, closings, changes in owner, banner, supplier, or other store characteristics over time). Available food retail channels include Grocery, Drug, Mass Merchandisers/Dollar, Wholesale Club & Convenience.

In addition to providing individual food store names and owner relationships (i.e., names of parent companies), TDLinx continuously verifies store location information and provides U.S. Postal Service standardized addresses, geographic codes (geocodes), and Federal Information Processing System (FIPS) codes. TDLinx also provides information on food store characteristics, including whether certain items are sold at a particular location (i.e., gas, liquor, wine, beer, and pharmaceuticals). TDLinx has a two channel classification system for each type of FAH retail location. Each location is assigned a channel and a subchannel, based on industry accepted retail channel classifications.

For additional information, please contact Alana Rhone.

Data access

Access to proprietary TDLinx data is limited to researchers collaborating on USDA-sponsored projects. USDA-sponsored projects include USDA grants, USDA cooperative agreements, and/or direct collaboration with USDA researchers on an issue of interest to the Department of Agriculture. For a sponsored research project, the research collaborator(s) must enter into a Third Party Agreement (TPA) with the data provider. The language of the TPA is specified by the Vendor and must be signed as is.

Steps to obtain a TPA:

  1. The ERS point of contact works with collaborators to complete the Nielsen TPA.
  2. All collaborators need to sign the TPA. The confidentiality clause holds the collaborator(s) responsible for the use of the licensed materials.
  3. The form is returned to the ERS Data Steward to review and forward the form to Nielsen.
  4. Nielsen will review the TPA based on the information provided and agree and sign the document.
  5. Nielsen will then send the signed TPA back to ERS.

USDA relies only on secure data enclaves that meet the Federal Information Security Management Act (FISMA) requirements to provide access to USDA external collaborators. For information on the cost of accessing the data on ERS data enclave, please contact David Dudgeon at david.dudgeon@usda.gov.

Documentation

Several ERS reports and data products provide detailed information about TDLinx—including methodology, characteristics, statistical properties of the data—and other technical reports (listed directly below) provide independent assessments of data quality.

The Food Access Research Atlas (FARA)

The Food Access Research Atlas (formerly the Food Desert Locator) is a mapping tool that allows users to investigate multiple indicators of food store access. This tool expands upon previous low-income and low access estimates to provide a spatial overview of food access indicators by census tract. The tool incorporates alternative estimates of low-income and low-access census tracts, and offers contextual information for all census tracts and many demographics in the United States (April 2021).

The Food Environment Atlas

The Food Environment Atlas is a web-based mapping tool (developed by ERS) that allows users to compare U.S. counties in terms of their “food environment”—indicators that help determine and reflect a community’s access to affordable, healthy food. Food environment factors—such as store/restaurant proximity, food prices, food and nutrition assistance programs, and community characteristics—interact to influence food choices and diet quality. The Atlas currently includes more than 275 indicators of the food environment. The year and geographic level of the indicators vary to better accommodate data from a variety of sources. The most recent county-level, State, or regional data are used whenever possible (September 2020).

Understanding Low-Income and Low-Access Census Tracts Across the Nation: Subnational and Subpopulation Estimates of Access to Healthy Food

This report estimates access to food stores for subsets of the population (report authored by Alana Rhone, Michele Ver Ploeg, Ryan Williams, and Vince Breneman). Data are also aggregated to the census-tract-level to show State and local estimates of low-income and low-access (LILA) census tracts (May 2019).

Capturing the Complete Food Environment With Commercial Data: A Comparison of TDLinx, ReCount, and NETS Databases

This report compares the TDLinx, ReCount, and NETS databases to each other and to the Economic Census (report authored by Clare Cho, Patrick W. McLaughlin, Eliana Zeballos, Jessica Kent, and Chris Dicken). The report aims to evaluate each dataset's relative coverage of the food environment, or the number and types of food outlets available in local communities across the United States (March 2019).

Low-Income and Low-Supermarket-Access Census Tracts, 2010-2015

This report updates estimates of low-income and low-supermarket-access census tracts (as found in ERS’ Food Access Research Atlas) (report authored by Alana Rhone, Michele Ver Ploeg, Chris Dicken, Ryan Williams, and Vince Breneman). The report uses a 2015 directory of supermarkets, 2010 Decennial Census data on population and subpopulation characteristics, and 2010–2014 American Community Survey data on household vehicle access and family income (January 2017).

Food Choices and Store Proximity

This report investigates the correlation between living in low-income, low-access (LILA) food store areas and the purchase of 14 major food groups, in order to estimate the effect on diet quality of living in LILA areas (report authored by Ilya Rahkovsky and Samantha Snyder) (September 2015).

Access to Affordable and Nutritious Food: Updated Estimates of Distance to Supermarkets Using 2010 Data

A report updates population estimates of indicators of spatial access to healthy and affordable foods in the United States (report authored by Michele Ver Ploeg, Vince Breneman, Paula Dutko, Ryan Williams, Samantha Snyder, Chris Dicken, and Phillip Kaufman). The report uses population data from the 2010 Census, income and vehicle availability data from the 2006–2010 American Community Survey, and a 2010 directory of supermarkets (November 2012).

Characteristics and Influential Factors of Food Deserts

In this report, ERS examines the socioeconomic and demographic characteristics of food desert tracts to see how they differ from other census tracts and the extent to which these differences influence food desert status (report authored by Paula Dutko, Michele Ver Ploeg, and Tracey Farrigan) (August 2012).

Access to Affordable and Nutritious Food-Measuring and Understanding Food Deserts and Their Consequences: Report to Congress

This report fills a request for a study of food deserts—areas with limited access to affordable and nutritious food-from the Food, Conservation, and Energy Act of 2008 (report authored by Michele Ver Ploeg, Vince Breneman, Tracey Farrigan, Karen Hamrick, David Hopkins, Phillip Kaufman, Biing-Hwan Lin, Mark Nord, Travis A. Smith, Ryan Williams, Kelly Kinnison, Carol Olander, Anita Singh, and Elizabeth Tuckermanty). The report summarizes findings of a national-level assessment of the extent and characteristics of food deserts, analysis of the consequences of food deserts, lessons learned from related Federal programs, and a discussion of policy options for alleviating the effects of food deserts (June 2009).

 

National Establishment Time Series (NETS) Data

About the data

To study the localized landscapes and dynamics of food-at-home (FAH) and food-away-from-home (FAFH) retailers in the United States (such as grocery stores and restaurants), commonly referred to as the local food environment, USDA’s Economic Research Service (ERS) acquires the National Establishment Time Series (NETS) Database. NETS is an annual census of all food establishments in the United States (across all sectors).

Walls & Associates partnered with Dun and Bradstreet (D&B) to convert their archival establishment data into a time-series database of establishment information. NETS includes the following detailed information:

  • Business name, address and contact information (including officer, title, phone number, CBSA codes and longitude and latitude), as well as whether or not the establishment is part of a publicly-listed enterprise.
  • Headquarters linkages (including a unique ID of the topmost domestic firm in a "Family Tree" of companies, as well as the parent company and headquarters; number of establishments reporting to a headquarters; and whether the ownership has changed 1990–2019).
  • Number of establishments related to its final year of business.
  • Years when business was active (“1989” is earliest year in Database and, currently, “2019” is the latest year of data that ERS has purchased) and year business started.
  • Industry classification (primary standard industrial classification (SIC), up to five secondary SICs, and whether the primary market changed 1990–2019).
  • A crosswalk between SIC codes and the North American Industry Classification System (NAICS) codes.
  • Type of establishment (single location, headquarters, or branch; public or private; legal status: proprietorship, partnership, corporation or non-profit and "cottage" businesses).
  • Employment at location and job growth relative to peers (3-digit SIC).
  • Estimated annual sales at the establishment and its sales growth relative to peers.
  • Special indicators: foreign-owned, import/export, government contracts, minority- owned, women-owned, and gender of officer.
  • Relocation information (origin and destination of significant moves, employment and sales in move year, origin and destination latitude-longitudes, distance of move and whether the establishment moved more than once 1990–2019).

For additional information, please contact Eliana Zeballos.

Data access

Access to proprietary NETS data is limited to USDA researchers and those collaborating on USDA-sponsored projects. Sponsored research projects must be conducted and published jointly with ERS staff. First drafts of manuscripts must be reviewed for disclosure risks prior to submitting to journals or USDA outlets to be considered for publication.

USDA relies only on secure data enclaves that meet the Federal Information Security Management Act (FISMA) requirements to provide access to USDA external collaborators. For information on the cost of accessing the data on ERS data enclave, please contact David Dudgeon at david.dudgeon@usda.gov.

Documentation

Several reports provide detailed information about NETS—including methodology, characteristics, statistical properties of the data. Other technical reports (listed directly below) provide independent assessments of data quality.

The Food Retail Landscape Across Rural America

This report examines the landscape of food retailers across the contiguous United States, focusing on the rural United States and its food stores (report authored by Alexander Stevens, Clare Cho, Metin Cakir, Xiangwen Kong, and Michael A. Boland) (June 2021).

Capturing the Complete Food Environment With Commercial Data: A Comparison of TDLinx, ReCount, and NETS Databases

This report compares the TDLinx, ReCount, and NETS databases to each other and to the U.S. Economic Census (report authored by Clare Cho, Patrick W. McLaughlin, Eliana Zeballos, Jessica Kent, and Chris Dicken). The report evaluates each dataset's relative coverage of the U.S. food environment, or the number and types of food outlets available in local communities across the United States (March 2019).

USDA’s Value-Added Producer Grant Program and Its Effect on Business Survival and Growth

This report combines data on the USDA Value-Added Producer Grants Program (VAPG) with NETS data (report authored by Anil Rupasingha, John Pender, and Seth Wiggins). The report studies the impacts of VAPG on business survival and employment growth of program recipient businesses (May 2018).

Local foods go downstream: Exploring the spatial factors driving U.S. food manufacturing

This journal article uses data drawn from the National Establishment Time-Series for 2013–15 (report authored by Sarah A. Low, Martha Bass, Dawn Thilmany, and Marcelo Castillo). The article explores how entrepreneurship, farm marketing channel innovations, and more traditional spatial factors influence the location decisions of food and beverage manufacturing establishment start-ups in the United States.