Skip to Main Content

Do Regional Fed Surveys Reflect National Manufacturing Conditions?

Regional Matters
April 4, 2024

Surveys play an important role in how research staff across the Federal Reserve System understand changing economic conditions. Several Reserve Banks conduct manufacturing surveys to gauge the health of the manufacturing sector in their districts. Since these surveys are published before national data, policymakers and market watchers often use them for a signal of where U.S. manufacturing activity is heading. In October 2014, the Dallas Fed published an Economic Letter in which they found that regional Fed manufacturing surveys improved forecasts for several key national indicators. In this post, we reexamine the Dallas Fed's analysis to understand if regional manufacturing data published by the New York, Philadelphia, Richmond, Dallas, and Kansas City Reserve Banks — 10 years later — still track leading national economic indicators.

How Well Do Regional Fed Surveys Track National Data?

To assess how each Bank's survey tracks national data, we look at the monthly "topline" number published from each Bank between January 2004 through December 2023. There are differences in several aspects of each Bank's survey that could affect overall performance against national measures, such as how the published topline number is derived. For example, New York, Philadelphia, and Dallas Reserve Banks publish a general business activity index (GBAI) based off a single question, whereas Richmond and Kansas City Reserve Banks publish a composite index (CI) derived from several questions. Richmond's CI represents a weighted average of the shipments (33 percent), new orders (40 percent) and employment (27 percent) indexes. Kansas City's CI is an average of its survey's production, new orders, employment, supplier delivery time, and raw materials inventory indexes.

The Institute of Supply Management Manufacturing Purchasing Managers Index (ISM PMI) is a monthly index that gauges economic activity in the manufacturing sector. This survey is seen as a reliable leading indicator of the U.S. manufacturing industry. Our analysis found that all Reserve Bank surveys correlate strongly with the ISM PMI: New York had that strongest overall correlation (.834), followed by Richmond (.808). Kansas City had the lowest correlation, although still strong (.714).

Table 1. Regional Surveys Correlation With ISM PMI
Bank Topline Manufacturing MetricsCorrelation With ISM PMI
Philadelphia GBAI.761
Richmond CI.808
Dallas GBAI.728
Kansas City CI.714
New York GBAI.834
NOTES: Sample period is January 2004–December 2023. GBAI = general business activity index. CI = composite index. A correlation is a statistical measure of the co-movement between two variables. A correlation of zero means there is no co-movement at all. A correlation of 1 means there is perfect co-movement between the two variables.

Interestingly, the strength of each Bank's correlation has not been consistent over time. During the COVID-19 period (2020-2023), correlation between the ISM PMI and surveys from Philadelphia and Kansas City strengthened, whereas the correlation from New York weakened.

Another measure of assessing national manufacturing activity is the monthly Industrial Production (IP) index produced by the Federal Reserve Board of Governors. This index measures real output in the manufacturing, mining, electric, and gas industries, relative to a base year. Overall, there is a moderate level of correlation with topline metrics and the FED's IP index, ranging between .353 (Kansas City) and .406 (Richmond).

Table 2. Regional Surveys Correlation With Federal Reserve's Industrial Production Index
Bank Topline Manufacturing MetricsCorrelation With Industrial Production
Philadelphia GBAI.389
Richmond CI.406
Dallas GBAI.385
Kansas City CI.353
New York GBAI.399
NOTES: Sample period is January 2004–December 2023. GBAI = general business activity index. CI = composite index. A correlation is a statistical measure of the co-movement between two variables. A correlation of zero means there is no co-movement at all. A correlation of 1 means there is perfect co-movement between the two variables.

When we looked at correlations across several economic periods, we found that the strength of the correlation declined post-Great Recession for all Banks. For example, the Richmond Fed index had a correlation of .6 during the Great Recession but dropped to less than .3 during the 2015-2019 recovery. However, starting in 2020 (COVID-19), the strength of the correlations for most Banks, except the Kansas City Fed, improved.

Which Regional Fed Survey Indexes Reflect National Manufacturing Conditions?

While correlations provide a useful gauge of the comovement of regional Fed surveys with national manufacturing indicators, it is also helpful to test their statistical power. In this section, we run multiple regressions of key national manufacturing indexes on their lags and each of the contemporaneous values of five selected regional Fed survey indexes.

When coefficients in the models are statistically significant, we have more confidence that a variable can provide a signal about national manufacturing indexes. The table below captures results from two multiple regressions used to assess the predictive power of each of the regional Fed surveys on the ISM PMI and industrial production growth. The "Yes" fields show which regional Fed indexes were statistically reliable in explaining month to month changes in the national manufacturing variables.

The first column shows that the Kansas City Fed's CI, Philadelphia Fed's GBAI, and the Richmond Fed's CI provided statistically significant predictive power for the ISM PMI. However, the second column shows that only the Richmond Fed's CI provides explanatory power for industrial production growth.

Regional Survey MeasureNational Indicator
ISM Mfg IndexIndustrial Production Growth
Dallas Fed GBAINoNo
Kansas City Fed CIYes***No
New York Fed GBAINoNo
Philadelphia Fed GBAIYes***No
Richmond Fed CIYes***Yes**
R-Squared0.920.33

Source: Federal Reserve Banks of Philadelphia, Richmond, Dallas, Kansas City, New York, Federal Reserve Board of Governors, Institute for Supply Management, author's calculations

NOTES: Sample period is January 2004–December 2023. Each column represents a regression of the dependent variable on the listed survey indexes and three lags of the dependent variable. GBAI = general business activity index. CI = composite index. "Yes" denotes statistical significance at the 1 (***), 5 (**) and 10 (*) percent levels, respectively. The R-Squared statistic shows the percentage of the variation in the dependent variable which is explained by the five regional surveys and three lags of the dependent variable.

How Useful Have Regional Fed Surveys Been in Forecasting National Manufacturing Activity Over the Past Three Years?

We assess the ability of each of the regional Fed indexes to aid out-of-sample forecasting of national manufacturing indicators from January 2021 through December 2023. In the table below, we compare the forecasting performance of a baseline model (that includes only the three most recent months of each of the national manufacturing indicator) to a model that also includes the current value of the regional Fed survey. The relative root mean squared forecast error (RMSFE) assesses how well each regional Fed survey index aids the forecasting accuracy of national manufacturing activity compared to the baseline model. When the relative RMSFE is lower than 1, the regional Fed survey improved the forecasting accuracy compared to the baseline model. We also report results on the statistical significance between the forecast errors of the baseline versus each regional Fed model.

ISM Manufacturing IndexIndustrial Production Growth
Regional Survey Man IndexRelative RMSFERegional Survey Man IndexRelative RMSFE
Dallas1.01Dallas0.96
Kansas City1.07Kansas City1.12
New York1.16*New York1.16
Philadelphia0.81*Philadelphia1.04
Richmond1.05Richmond0.95

Source: Federal Reserve Banks of Philadelphia, Richmond, Dallas, Kansas City, New York, Federal Reserve Board of Governors, Institute for Supply Management, author's calculations

NOTES: The sample period is January 2004 to December 2020; out-of-sample forecasts run from January 2021 to December 2023. Each entry represents a separate regression, and all include three lags of the dependent variable (national indicator). A lower relative root mean squared forecast error (RMSFE) indicates better forecasting performance than baseline model. The baseline model contains three lags of the dependent variable (national indicator) and no regional Fed survey measure. * Indicates difference in forecasts' mean squared errors is statistically significant from baseline model at 1% level according to Diebold & Mariano (1995) test.

The results from our forecasting exercise suggest that only the Philadelphia Fed's GBAI significantly improved the forecasting accuracy of the ISM PMI compared to a baseline model, and the New York Fed's GBAI significantly weakened forecasting performance. None of the regional Fed manufacturing indexes significantly improve or weaken the forecasting performance of industrial production growth compared to a baseline model.

Since we focus only on contemporaneous relationships between the national and regional surveys (comparisons within the same month), our analysis may be obscuring more dynamic relationships between the series. For example, an article by Santiago Pinto and Nika Lazaryan contends that dynamic forecasting models of the ISM index using the Richmond Fed survey series tend to be more accurate when they allow for different numbers of lags for each variables, since many of the series exhibit a high degree of persistence with differing correlations depending on the time horizon. Additionally, the ability of a regional survey to forecast national variables may ultimately depend on how representative the region (or the survey's sample) is of the nation's manufacturing firms.

Takeaways

While regional Fed manufacturing surveys are designed to capture conditions in their respective districts, they have also been useful in assessing national manufacturing activity. (See this Economic Letter published in 2014 by the Dallas Fed.) Using data from the past 20 years, we find that regional Fed surveys have been highly correlated with the ISM manufacturing index and to a lesser extent industrial production growth. The headline manufacturing indexes from the Kansas City, Philadelphia, and Richmond Reserve Banks statistically reflect national conditions, but only Richmond predicts IP growth. However, only Philadelphia's survey index improves the forecasting performance of ISM PMI compared to a baseline model over the past three years. Often released before national indicators, regional Fed surveys are real-time measurements of changing economic conditions, some of which provide information in assessing the direction of manufacturing activity for the nation.


Views expressed are those of the author(s) and do not necessarily reflect those of the Federal Reserve Bank of Richmond or the Federal Reserve System.

Phone Icon Contact Us

Joseph Mengedoth (804) 762-2285