Related Links

• Why Use a Diffusion Index?
• How to Get the (High-Skilled) Workers?
• Richmond Fed surveys:
  • Community Development Financial Institution Survey
  • Small Business Credit Survey
  • Fifth District Survey of Community College Outcomes
  • Fifth District Survey of Manufacturing Activity
  • Fifth District Survey of Service Sector Activity
• Regional Data and Analysis

Transcript

Tim Sablik: Hello, I'm Tim Sablik, a senior economics writer at the Richmond Fed. My guests today are Jason Kosakow and Santiago Pinto. Jason is the Richmond Fed survey director and Santiago is a senior economist and policy advisor in the Richmond Fed's Research department. Gentlemen, thanks for joining me today.

Jason Kosakow: Always great to be here, Tim.

Santiago Pinto: Thank you for having us.

Sablik: Today, we're going to be talking about how surveys are used to inform economists and policymakers about local economic conditions. Many of the economic metrics that we hear about on the news, like GDP or the unemployment rate, are aggregate measures that try to summarize conditions for the whole United States. But, of course, conditions can vary greatly from state to state or county to county.

How are surveys used by economists and policymakers to get a better picture of economic conditions? Jason, maybe you can talk about that first.

Kosakow: Sure, that's a great question. Many of the statistics we rely on as a nation to understand our economy are based on surveys. There are people who dedicate their entire careers [to] understanding how to capture this information and how to make sure it's accurate.

For example, the unemployment rate, which right now is around 3.5 percent, is derived from a survey of about 60,000 households. The same survey [provides] us with the labor force participation rate and a host of other vital economic stats. The same can be said with inflation data, as the CPI is a collection of about 94,000 prices per month, which about a third are done via surveys.

What's nice about that is it's a large sample size — 60,000 households — so you can break that out by different geographies. You can break that up by race, ethnicity, and also ages. Additionally, a lot of these statistics have been collected for decades, so policymakers are able to see shifts and changes throughout different business cycles. As I said before, these surveys have really large sample sizes, so it really makes the data more actionable for policymakers.

Sablik: Santiago, did you have anything you wanted to add there?

Pinto: We conduct several different quantitative and qualitative surveys that we use a lot. This information allows us to assess current economic conditions. It also provides some context about other things that are going on in real life. They are quite useful, and they are available at a more granular level. They validate some of the trends that we see more at the aggregate level, the national level. For instance, the Richmond Fed conducts surveys and we can track regional and national economic conditions, as well as many other regional banks that do the same.

Quite often we try to see whether they are good indicators of economic activities at the regional and national level. And, actually, they seem to be tracking the economy quite well.

Sablik: Yeah, as you both mentioned, surveys feed into these quantitative measures, like GDP, unemployment or inflation. They also provide, I think, what researchers would call qualitative information. Respondents might give written or verbal responses to questions that might be a little bit more open ended. That seems harder to summarize into a single metric.

Santiago, how do researchers use that kind of information from surveys to conduct analysis?

Pinto: Over the years, survey data have been used in a variety of different ways. For instance, they have been used to construct indices of consumer or business confidence to describe what is the state of the economy and the state of the business cycle at a certain point, and to examine the extent to which certain measures are leading indicators in predicting aggregate consumption, let's say aggregate production or employment. They also have been used as input variables in short-term nowcasting and forecasting models.

In macroeconomic analysis, they have been used a lot. They have been used also as a source of information for several microeconometric analysis of agents' behavior.

Here at the Fed, we have done some research with some of my colleagues where we tried to investigate specifically diffusion indices. [These] indices are basically statistics constructed based on survey responses. In our case, it's as simple as a measure that is calculated based on the number of people that respond whether a certain variable has been going up minus those that respond that the economic variable has been going down. We find … this statistic is a good indicator of what we call the extensive margin of a change [the number of firms that reported a change] in a variable.

We'll also use that diffusion index indicator to construct a measure of uncertainty of the economy. It just focuses on one single aspect — possible uncertainty, the one given by polarization of disagreement among people that participate or respond to the survey.

There's another interesting development — the increasing use of text analytics techniques to examine and quantify data that is provided to us in unstructured textual form. These kinds of techniques [are] becoming widespread in academia, in the government, and in even in the private sector. The idea is to try to uncover certain kinds of information that is actually hidden in the text or in the verbal statements.

Sablik: Yeah, that's very interesting. It sounds like some of those new techniques have been driven by the advances in computing technology and the Internet.

Pinto: That's correct.

Sablik: My impression is that economists have historically been a bit more reluctant to use survey data in their research because there's been some studies that suggest that it could be unreliable or misleading, the idea that respondents to surveys might be uninformed or misrepresent their true beliefs.

Has there been any sort of reevaluation of the usefulness of surveys and economic analysis in recent years?

Pinto: Right, you are correct. Economists have been very skeptical about the use of survey data in the past, especially about anything that involves any kind of subjective statement. We were taught when we were young economists that you should believe what people do rather than what people say.

This may have been the reason why, in the past, we have not been consistently collecting survey data and tracking, for instance, expectations in a more systematic way, due to this reluctance. Without having this data, economists have resorted to different ways of modeling expectations, just looking at what people have done, their preferences.

I would say that since the 1990s, we started to observe [that] this reluctance against survey data has been weakening. We see a lot more of that data collected in a more systematic way.

Let me point this out. Economists, we think that there are a lot of important factors that drive economic decisions and economic behavior. Many of those factors are really hard to measure and really hard to quantify. [Those factors include] uncertainty or how do people form expectations, especially about expectations on inflation or expectations about any other economic variable.

You have survey measures of expected inflation. In this way, you ask questions to different groups, different customers, different consumers, or professional forecasters or firm managers. We've seen that getting this information in this particular way has shown [to be] quite useful, especially when you use it along with other variables. In macroeconomic forecasting modeling, they tend to perform very well.

Sablik: That's a great point. As you brought up, it seems like surveys can be a particularly useful tool for policymakers to get timely information, particularly in uncertain and rapidly changing times.

Jason, how have you used this kind of flexibility in Richmond's regional business surveys?

Kosakow: Just to add on to what Santiago was saying, surveys have always been used in the economics profession. What's been happening though in the recent decade, two decades, is there has been more surveys that have been happening and have been reaching the media.

A lot of these surveys are done with something called a convenient sample. These are samples that are not randomly selected in the population. They tend to not be representative of the population as a whole and they tend to be a little bit more wrong than traditional probability-based samples, which is [when] people have an equal chance of getting selected. When those surveys are wrong, they tend to get more scrutiny, which is good. But it also makes people feel less confident in the results they see from the surveys.

A lot of the organizations — like here at the Richmond Fed and also the BLS, Census and other organizations — spend a lot of time on methodology. We spend a lot of time thinking about how to conduct quality research. That type of surveys [is] very helpful in economic analysis.

But your question [is] on how we've been using surveys here at the Richmond Fed. We administer about a half dozen different surveys every year to understand our economy. A few that do come to mind are the Community Development Financial Institution Survey, the CDFI Survey, that is run annually [and] our Small Business Credit Survey, which is a partnership between all 12 Reserve banks. We also created the community colleges survey recently. We also do a bunch of ad-hoc surveys that might be about specific things that we're trying to better understand.

Generally, the two surveys that get the most attention are our monthly manufacturing survey and our monthly non-manufacturing survey, which we call our service sector survey. These surveys provide real-time insights into changing economic conditions. They tell us if the business metrics have either increased, decreased, or stayed the same from the previous month. For example, if we're trying to understand employment, a firm will tell us if employment has increased, decreased, or stayed the same compared to the other month. That gives us a nice indicator of how things are going.

In the economics profession, you hear a lot of talk about the business cycle. It's nearly impossible to predict when changes in the business cycle happen, often because data produced from the government have lag. But the surveys that we administer at the Richmond Fed give us real-time data into how our regional economy is changing. I'm not saying we can see inflection points in the business cycle as a result of these surveys. But it does provide us with insights into changing economic conditions sooner than when the government agencies generally produce their data. For example, we can see if supply chain pressures are easing by reported inventory levels or backlogs of new orders. We had this information in real time as opposed to waiting for the data to get released by various agencies.

Sablik: That brings up a good question. What are you seeing on the horizon right now? What are the big takeaways from the most recent surveys you've done?

Kosakow: Our monthly surveys have two components. We have a set of core questions which we've been asking for the past 20-ish, 30-ish years. The second components are special questions, which change every month based on topical economic questions that we just generally have.

For example, the labor market is really interesting right now — really high inflation, interest rates are rising, but employment has been very strong throughout the country. So, the question that we're thinking about is, is this strength in the labor market consistent among skill levels? We found at the beginning of the year trouble hiring and retaining people that we consider low- or mid-skill. That was a lot more difficult to do for a lot of firms throughout our region. However, in our most recent September survey, we found this shifted from in the beginning of the year from the low- to mid-skill to the mid- to high-skill. What we're finding in the survey is these "lower skill workers" — I'm using quotation marks — are not as difficult to hire and retain. That difficulty is more at the higher end of the skill. These might be people that might need a CDL to operate a truck. It might be people that are your marketing executives or accountants. Things along those lines are just harder to hire and retain.

Sablik: Well, thank you both, Jason and Santiago, for coming on to talk about how surveys are used most broadly and some of the latest findings from Richmond surveys.

Kosakow: Always a pleasure.

Pinto: Thank you, Tim. It was a pleasure.

Sablik: For our listeners interested in keeping up with the latest results and analysis from the Richmond Fed's business surveys, you can head on over to the Regional Data and Analysis section on Richmondfed.org. And if you enjoyed the show, please consider leaving us a rating and review.