Cross-checking ShadowStats

Last week I wrote about Balaji Srinivasan's idea of creating a decentralized version of the Billion Prices Project. The post got me thinking again about the topic of alternative inflation indexes.

One of the most well-known of the alt-inflation indexes is John Williams' ShadowStats, often cited by gold bugs and bitcoin maximalists. As of August 2011, ShadowStats puts U.S. inflation at 13% versus official inflation of 5%, as illustrated in the chart below.

Source: ShadowStats

That's a huge gap. One of the two data series has to be wrong.

I've always dreamt about writing a blog post on ShadowStats, but never had the gumption or statistical chops for it. So I was happy to see that economist Ed Dolan announced on Twitter that he was  republishing a 2015 blog post in which he carefully critiqued ShadowStats. It's such a good article that I'm not going to bother writing my own ShadowStats post anymore.

ShadowStats attracts a lot of sneers from the econ commentariat. What makes Dolan's post so effective is that he gracefully takes Williams' arguments on their merits and then proceeds to analyze them. Put differently, he doesn't try to damn ShadowStats with straw man arguments. He steel-mans it (or steelwomens it).   

Anyways, do read the post. 

Dolan saves his best criticism for the end. When Dolan was writing his post in 2015, the gap between official inflation and ShadowStats inflation was a whopping 7% (see chart above). What Dolan finds is that the majority of this 7% gap can be attributed to a simple double-counting error committed by Williams. By correcting this double-counting error, the ShadowStats inflation number shrinks. And so the gap between it and the official CPI is actually far less menacing than Williams' anti-government fans like to make out.

Dolan challenges Williams to correct his double-counting mistake. But you can see why Williams might find this difficult to do. He has been selling his data for many years on a subscription basis. Admitting that his product contains errors could anger his customer base.

The other part of Dolan's blog post that I want to draw attention to is a set of simple cross-checks he performs to see whether official inflation or ShadowStats is more accurate. For instance, taking grocery prices from a 1982 advertisement and projecting them forward with both inflation indexes, Dolan finds that the official CPI does a better job of predicting where modern grocery prices actually ended up.

It would be unfair to do just one set of crosschecks. Which is why Dolan does a bunch of them. It's worth reading through each one. ShadowStats does not make out well. (For instance, in order for ShadowStats to be right, you've got to believe that the U.S. economy has been in a recession for the last two decades.)

To finish my blog post off, I'm going to add to Dolan's list of cross-checks by adding one of my own. This cross-check is meant specifically for one of the main consumers of ShadowStats data: gold bugs.

If gold investors think ShadowStats data is right, and many of them do, then they also have to accept that gold has lost 91% of its value since January 1980 (see chart below of the gold price adjusted for ShadowStats inflation). Which means that the yellow metal is an awful hedge against inflation, and anyone who buys it for that reason is making a big mistake.

Source: Bullionstar

The far more reasonable position to take is that the ShadowStats data is wrong, and that gold has actually been a decent hedge against inflation since 1980. Using official inflation numbers rather than ShadowStats, the price of gold today is almost even with its 1980 level.

So gold bugs, you can relax. You haven't lost your sanity -- gold is not an awful inflation hedge. Rather, ShadowStats is an awful measure of inflation.

A decentralized version of MIT's Billion Prices Project

Balaji Srinivasan, an angel investor, wants to kick start an updated version of MIT's Billion Prices Project. He will invest $100,000 in the project that best envisions how to create a publicly-available decentralized inflation dashboard, one that relies on scraped data from retailer websites.

A truly global inflation dashboard would be the next coinmarketcap. It'd be bigger than that, in fact.

So we're offering a little prize to build one. https://t.co/OtZJS8FRWA

— Balaji Srinivasan (@balajis) August 5, 2021

Many years ago I was a big fan of the MIT's Billion Prices Project, so I perked up when I read about Srinivasan's contest. Created by economists Roberto Rigobon & Alberto Cavallo, the Billion Prices Project collected, or scraped, data from retailers' websites and used it to generate an alternative version of various government-tabled consumer price indexes. (I wrote about the Project here.) Members of the public could get access to Billion Prices U.S. data, albeit with a small delay.

This was incredibly useful! Because government consumer price indexes are published monthly, but websites can be scraped 24/7, the Billion Prices Project was far more responsive to price changes than government consumer price indexes are. It gave you insights into tomorrow's CPI announcement, today.

The Billion Prices Project also garnered attention because it revealed how Argentinean authorities had distorted official statistics to make inflation appear more muted than it really was. Conversely, the Billion Prices Project regularly confirmed the accuracy of U.S. Bureau of Labor Statistics' consumer price indexes, making it a useful tool for whacking gold bugs and inflation truthers over the head.  

While I like Srinivasan's general idea of bringing real-time scraped inflation data to the masses, I see three big problems.

The first problem is over-reliance on scraped data. Scraping is fast and cheap, but only a portion of the global economy's prices are scrape-able. Amazon and Walmart may sell almost every type of physical good under the sun here in Canada and the U.S., but they don't sell services. So while it's easy to find scraped prices of laptop computers, forget about prices for haircuts, rent, or healthcare.

That leaves a pretty big hole. Government statistical agencies such as the Bureau of Labor Statistics (BLS) or Statistics Canada are able to capture services prices because they send out human inspectors to check the prices of things like haircuts and back-rubs. Lacking price data on these items, Srinivasan's inflation dashboard will never be as accurate as the dashboards published by Statistics Canada or the BLS.

Consider too that goods in many developing and undeveloped countries are not available online. Amazon, for instance, isn't going to provide any clues into what is going on with vegetable prices in Afghanistan, or shoe prices in Yemen. Srinivasan says that he wants an "internationally useful" dashboard, but he's certainly not going to get one by relying on scraping alone. He's going to get a rich folks' dashboard.

Which leads into the second problem: the business model won't work. Compiling inflation indexes is costly, but Srinivasan wants his decentralized inflation dashboard to be made public, and presumably free. That's just not possible.

Rigobon & Cavallo's own Billion Prices Project is a good example of this dilemma.

Mere grants weren't enough to fund the Billion Prices Project. Yes, scraping may be cheaper than using physical data collectors, but it's still expensive to compile price indexes. Bills had to be paid. And so the whole Billion Price Project sold out. It was folded into a company called PriceStats and sold as a proprietary product to rich investors and central banks.

At first PriceStats continued to offer some free public dashboards. But this was never going to last. Rigobon & Cavallo's data had commercial value because it was quicker than government data, and could be used by traders to beat the market. Making even a portion of that data available to the public destroyed its commercial value. And so over time the public-facing parts were all discontinued. The Billion Prices Project, at least the public service side of it, is effectively dead. 

How data from PriceStats/The Billion Prices Project overlapped with US consumer price indexes [source]

Srinivasan's proposal faces the same tradeoffs as the Billion Prices Project. Price data is expensive to collect, compile, store, and process. Government agencies like the BLS are funded by taxes, not profits, and so they can give it away for free. We all benefit from this public service. But the calculus is different for private companies. To fund data collection, they must implement some sort of pay-wall. Srinivasan wants to make a public inflation dashboard, much like the BLS does. But he can't. He's not a government. 

(And no, an inflation dashboard won't be able to rely on advertising revenues, say like how Coinmarketcap does. Frenetic gamblers are addicted to checking coin prices. Inflation data doesn't attract eyeballs).

The last problem with Srinivasan's project is the basket problem. The introductory page that describes the project focuses on how to scrape for data. But this omits one of the biggest challenges to compiling any consumer price index: determining what the consumer price basket actually is. That is, what exactly is the "basket" of goods and services that the average consumer consumes each month?  

Government statistics agencies such as the Bureau of Labor Statistics solve this problem by conducting national surveys. For instance, the BLS's baskets are based on interviews with 24,000 Americans each quarter about their spending habits. The BLS gets even more precise data by having 12,000 of those participants keep a detailed diary that lists all expenses for a week.

But that's an incredibly resource-intensive process.

To avoid having to run costly surveys in order to build a representative consumption basket, the Billion Prices Project had a simple solution: it borrowed the BLS's baskets. But Srinivasan's project has declared this solution to be out of bounds. The project's website describes inflation as a "government-caused problem," and so the project can't rely on "government statistics."

Which means that Srinivasan's project will have to build its own representative price basket using its own surveys. Unless it can bring the same amount of financial resources to bear as the BLS, I don't see how it can pull this off.

Alternatively, the project will have to use the BLS's "untrustworthy" data. But that means contradicting its stated philosophy.

To sum up, Srinivasan envisions his decentralized inflation dashboard as being a superior alternative to untrustworthy government dashboards. But government consumer price indexes are far better than he is making them out to be, given the huge amount of money, time, and expertise committed to statistics agencies. (Yes, there are exceptions like Argentina). If any inflation dashboard is likely to be untrustworthy, I fear it will be Srinivasan's built-on-the-cheap dashboard.

(By the way, you'll notice I didn't discuss the decentralized aspect of the inflation dashboard. The project has enough challenges already, before even getting to the decentralized bit.)

All that being said, I'm in the same camp as Srinivasan. Scraped inflation data is neat and useful, and I think the public should be getting access to it. But my preferred solution is different than the one put forth by Srinivasan. Hey, BLS and Statistics Canada! When are you ever going to unveil some sort of free real-time consumer price index that relies on scraped data?

---

Srinivasan responds. Joe Weisenthal blogs.

Esperanto, money's interval of certainty, and how this applies to Facebook's Libra

Facebook recently announced a new cryptocurrency, Libra. I had earlier speculated about what a Facebook cryptocurrency might look like here for Breakermag.

I think this is great news. MasterCard, Visa, and the various national banking systems (many of which are oligopolies) need more competition. With a big player like Facebook entering the market, prices should fall and service improve, making consumers better off.

The most interesting thing to me about Facebook's move into payments is that rather than indexing Libras to an existing unit of account, the system will be based on an entirely new unit of account. When you owe your friend 5 Libras, or ≋5, that will be different from owing her $5 or ¥5 or £5.  Here is what the white paper has to say:

"As the value of Libra is effectively linked to a basket of fiat currencies, from the point of view of any specific currency, there will be fluctuations in the value of Libra."

So Libra will not just be a new way to pay, but also a new monetary measurement. Given how Facebook describes it in the brief quotation provided, the Libra unit will be similar to other unit of account baskets like the IMF's special drawing right (SDR), the Asian Monetary Unit (AMU), or the European Currency Unit (ECU), the predecessor to the euro. Each of these units is a "cocktail" of other currency units.

Facebook's decision to build its payments network on top of a new unit of account is very ambitious, perhaps overly so. When fintechs or banks introduce new media of exchange or payments systems, they invariably piggy back off of the existing national units of account. For instance, when PayPal debuted in 2001, it didn't set up a new unit called PayPalios. It used the dollar (and for the other nations in which is is active, it used the local unit of account). M-Pesa didn't set up a new unit of account called Pesas. It indexed M-Pesa to the Kenyan shilling.

I couldn't find a good explanation for why Facebook wants to take its own route. But I suspect it might have something to do with the goal of providing a universal monetary unit, one that allows Facebook users around the globe to avoid all the hassles of exchange fluctuations and conversions.

Global monetary harmony an old dream. In the mid 1800s, a bunch of economists, including William Stanley Jevons, tried to get the world to adopt the French 5-franc coin as a universal coinage standard. Jevons pointed out that the world already had international copyright, extradition, maritime codes of signals, postal conventions—so why not international money too? He wrote of the "immense good" that would arise when people could understand all "statements of accounts, prices, and statistics." It would no longer be necessary to employ a skilled class of foreign exchange specialists to take on the "perplexing" task of converting from one money to the other.

But the plan to introduce international money never worked out. (I wrote about this episode for Bullionstar).

Global money like Libra might seem like a great idea. But ultimately, I suspect that the decision to introduce a new unit of account will prevent Libra from ever reaching its full potential. Units of account are a bit like languages. If you are an English speakers, not only do you communicate to everyone around you in English, but you also think in English. Likewise with the dollar or yen or pound or euro. If you live in France, you're used to describing prices and values to friends and family in euros. You also plan and conceptualize in terms of them.

It's hard to get people to voluntarily switch to another language or unit of account once they are locked into it. For instance, in the 1800s L.L. Zamenhof attempted to get the world to adopt Esperanto as a language in order to promote communication across borders. To help facilitate adoption, Zamenhof designed it to be easy to learn. But while around 2 million speak Esperanto, it never succeeded in becoming a real linguistic standard. The core problem is this: Why bother learning a new language, even an easy one, if everyone is using the existing language? 

Facebook's Libra project reminds me of Zamenhof's Esperanto project. Nigerians already talk and compute in naira, Canadians in dollars, Indonesians in rupiahs, and Russians in rubles. Why would any of us want to invest time and effort in learning a second language of prices?

Let me put it more concretely. I do most of my families grocery shopping. Which means I keep track of an evolving array of maybe 30 or 40 food prices in my head. When something is cheap relative to my memory of it, I will buy it—sometimes multiple versions of it. And when it is expensive, I avoid it. But this array is entirely made up of Canadian dollar prices. I don't want to have to re-memorize that full array of prices in Libra terms, or keep two arrays of prices in my head, a dollar one and a Libra one. I'm already fluent in the Canadian dollar ones.

Nor will retailers like Amazon or the local corner store relish the prospect of having to advertise prices in both the local unit of account and Libra, plus whatever unit Google and Netflix choose to impose on us. 

So Facebook is inflicting an inconvenience on its users by forcing us to adopt a new unit of account. To make for a better user experience, it should probably index the Libra payments network to the units of account that we're all used to. 

If not, here is what is likely to happen. We'll all continue to think and communicate in terms of local currency. But at the last-minute we will have to make a foreign exchange calculation in order to determine out how much of our Libra to pay at the check-out counter. To do this calculation, we'll have to use that moment's Libra-to-local currency exchange rate. This is already how bitcoin transactions occur, for instance.

But this means that Libra users will lose one of the greatest services provided by money: money's interval of certainty. This is one of society's best free lunches around. It emerges from a combination of two fact. First, most of us don't live in a Libra world in which we must make some sort of last-minute foreign exchange calculation before paying. Rather, we live in a world in which the instruments we hold in our wallet are indexed to the same unit of account in which shops set prices.

Monetary economists call this a wedding of the medium-of-exchange and unit-of-account functions of money. This fusion is really quite convenient. It means that we don't have to make constant foreign exchange conversions every time we pay for something. A bill with a dollar on it is equal to the dollars emblazoned on sticker prices.

Secondly, shops generally choose to keep sticker prices fixed for long periods of time. Even with the growth of Amazon and other online retailers, Alberto Cavallo (who co-founded the Billion Prices Project) finds that the average price in the U.S. has a duration of around 3.65 months between 2014-2017. So for example, an IKEA chair that is priced at $15 will probably have this same price for around 3.65 months. This is down from 6.48 month between 2008-10. But 3.65 months is still a pretty long time.

Prices are getting less sticky thanks to online shopping: https://t.co/AVikQrJjJ6

A price change endured for 7 months back in 2008. It now lasts just 3.7 months. This effect is particularly strong in sectors where Amazon has high market share i.e. electronics & household goods. pic.twitter.com/ncAYkagA0b

— JP Koning (@jp_koning) June 25, 2019

Why do businesses provide sticky pricing? In the early 1990s Alan Blinder asked businesses this very question. He found that the most common reason was the desire to avoid "antagonizing" customers or "causing them difficulties." Blinder's findings were similar to Arthur Okun's earlier explanation for sticky prices whereby business owners maintain an implicit contract, or invisible handshake, with customers. If buyers view a price increase as being unfair, they might take revenge on the retailer by looking for alternatives. (I explore these ideas more here).

Anyways, the combination of these two factors—sticky prices and a wedding of the unit of account and medium of exchange—provides all of us with an interval of certainty (or what I once called money's 'home advantage'). We know exactly how many items we can buy for the next few weeks or months using the banknotes in our wallet or funds in our account. And so we can make very precise spending plans. In an uncertain world, this sort of clarity is quite special.

Given Libra's current design, the interval of certainty disappears. Store keepers will still keep prices sticky in terms of the local unit of account, but Libra users do not benefit from this stickiness because Libras aren't indexed to the same unit as sticker prices are. Anyone who has ≋100 in their account won't know whether they can afford to buy a given item two weeks from now. But if they hold $100, they'll still have that certainty, since dollar prices are still sticky.

If money's interval of certainty is important, it is particularly important to the poor. The rich have plenty of savings that they can rely on to ride out price fluctuations. The fewer resources that a family has, the more it must carefully map out the next few day's of spending.  The combination of sticky prices and a wedding of the unit-of-account and medium-of-exchange affords a vital planning window to those who are just barely getting by.

This clashes with one of Libra's founding principles: to help the world's 1.7 billion unbanked. Here is David Marcus, Libra's project lead:

3/ Because it's time for the internet to have a protocol for money, and it's time to try something new for the 1.7 billion people who are still unbanked 30 years after the invention of the web.

— David Marcus (@davidmarcus) June 18, 2019

Most of the world's unbanked people are poor. But Libra won't be doing the poor much of a favor by choosing to void the interval of certainty that they rely on. If Facebook and David Marcus truly wants to help the unbanked, it seems to me that it would better to index Libras to the various local units of account.

I suppose there is an argument to be made that Libras could provide poor people in nations with bad currencies a haven of sorts. Better Libras than Venezuelan bolivars, right? But the nations with the world's largest unbanked populations—places like India, Nigeria, Mexico, Ethiopia, Bangladesh, and Indonesia—all have single digit inflation, or close to it. Extremely high inflation is really just a problem in a few outliers, like Zimbabwe and Venezuela.

Besides, providing those who endure high inflation with a better unit of account isn't the only way to help them. Offering locally-denominated Libras that offer a compensating high rate of interest would probably be more useful. Not only would these types of Libra offer inflation protection, but they would preserve the interval of certainty.

Thankfully, I suspect that Libra is very much a work-in-progress. The current whitepaper seems to give only a hint of what the project might become. If so, one of the changes I suspect Facebook will have to make if it wants to get traction is to link the Libra network to already-existing units of account. A new unit of account is just too Utopian.

Are prices getting less sticky?

Sticky prices illustrated, from Eichenbaum, Jaimovich, and Rebelo (link)

What makes ride sharing firm Uber interesting is not just its use of new technology to mobilize unused car space, but the method it uses to price its services. Uber's surge pricing algorithm varies cab fares dynamically. To get from A to B, the car that you hired this morning for $10 could end up costing $100 this afternoon.

How unlike the traditional taxi fare it is displacing! In their 2004 paper on sticky prices, economists Bils and Klenow found that taxi fares tended to remain at the same level for 19.7 months before being adjusted. Getting from A to B pretty much costs you the same price day-in-day-out for almost two years.

In our internet age, are prices getting less sticky? 

At first glance no. Alberto Cavallo, who along with Roberto Rigobon created the Billion Prices Index (the bane of all inflationistas), has analyzed scraped data from the websites of retailers who continue to sell mostly through bricks & mortar stores, say like Walmart. Cavallo finds that U.S. online prices stay fixed for 42 days, about the same as offline prices.

On the other hand, Gorodnichenko, Sheremiro, and Talavera find that online prices exhibit more flexibility than offline prices. Unlike Cavallo, the authors analyze data from an online-only store, say like an Amazon (they aren't permitted to disclose which store). However, while Gorodnichenko et al find that online prices are less rigid than bricks & mortar prices, they still exhibit unusually long price spells, or periods of fixity. These spells tend to endure for about 7 to 20 weeks, two-thirds shorter than offline spells when the effect of discounts/sales has been removed. The result is counter-intuitive, say the authors, given that online stores have the technology to cheaply adjust prices as supply and demand change, yet for some reason choose not to.

Even though both papers were published in 2015, Cavallo and Gorodnichenko are using relatively stale data. The first dataset runs between October 2007 and August 2010 while the latter spans the period between May 2010 and February 2012. This delay is unfortunate as the online world is changing fast. Recent industry articles point to a large ramp-up in the use of dynamic pricing by retailers over the last few years. For instance, Profitero, a price intelligence provider, charts out a step-wise change in the pace of Amazon's price changes beginning in late 2012. According to competing price intelligence company 360pi, by 2014 some 18% of Amazon's prices were changing daily.

The same goes for an old dinosaur like Sears. While Sears' online prices rarely underwent changes in the earlier part of this decade, around 18% of its prices are now being adjusted each day, on par with Amazon. And now Sears is trying out digital signs in its bricks & mortar stores to ensure quicker offline price changes.

The moral economy

If we are indeed entering an Uber-style flex-price world, what underlying factors had to change for this to happen? It's not technology—we've always had the means to set rapidly changing prices, just look at financial markets. If anything had to bend in order for pricing patterns to change, it was the ethics of price setting.

To understand why, we need to explore one of the enduring questions in economics: why goods & services prices remain fixed in the face of continuously changing demand and supply conditions. When economist Alan Blinder polled businesses in the early 1990s to find out why they kept prices unchanged for long periods of time, the most common answer was the desire to avoid "antagonizing" customers or "causing them difficulties." Blinder's findings evoked Arthur Okun's earlier (1981) explanation for sticky prices whereby business owners maintain an implicit contract, or invisible handshake, with customers. If buyers view a price increase as being unfair, they might take revenge on the retailer by looking for alternatives. A retailer who promises to adjust prices rarely and only when costs justify it thereby avoids antagonizing customer sensibilities, and in return the customer provides a degree of loyalty.

The idea that prices are set within an overall moral framework predates Blinder and Okun. Nobel Prize winning economist John Hicks, for instance, once wrote that the notion that all prices are perfectly flexible was highly unrealistic and attributed rigidity to legislative control, monopolistic action "of the sleepy sort which does not strain after every gnat of profit, but prefers a quiet life," and "lingering notions of a ‘just price’."

Hicks' use of the word 'lingering' refers to the extended lineage of the concept of the just price. The belief that it is in some way sinful to sell a product for more than its fair price is a very old one, going back to early economic thinkers like Thomas Aquinas. In the age that Aquinas inhabited the economic roles that individual were permitted to play and the prices they could set were determined by tradition and custom. Historian E.P Thompson once referred to this as the "moral economy." For example, medieval English farmers could not sell their corn directly from their fields but had to bring it in bulk to the local "pitching market." Speculation, or the practice of "withholding" in the anticipation of better prices, was prohibited. Once at market, no sales of corn could be made before stated times. When the bell rang, the poor had the first chance to buy, and only after could larger dealers make purchases. These various market structures were designed to ensure a just price and fair profits.

Even as these structures were slowly unwound, writes Thompson, the English populace clung to the old morality, the physical incarnation of this being food riots which swept the countryside during the 18th century. These riots weren't random attempts to pilfer. Rather, they were relatively sophisticated affairs whereby rioters would organize to set the price of a good, in effect forcing the offending retailer to sell their wares at the level deemed just rather than at its much higher market-determined rate.

In the same way that 17th century rioters self-regulated markets by threatening to set the price for corn or bread, modern shoppers who encounter an unjust price threaten to cross the aisles towards the competition. Eager to avoid being punished by their customers' wrath, retailers implicitly promise to keep their prices fixed for long periods of time.

I find it interesting that even when we start from scratch, the notion of a just price quickly emerges. In his account of a temporary P.O.W. camp economy in which cigarettes circulated as money, R.A. Radford notes that:

There was a strong feeling that everything had its "just price" in cigarettes. While the assessment of the just price, which incidentally varied between camps, was impossible of explanation, this price was nevertheless pretty closely known. It can best be defined as the price usually fetched by an article in good times when cigarettes were plentiful. The "just price" changed slowly; it was unaffected by short-term variations in supply, and while opinion might be resigned to departures from the "just price," a strong feeling of resentment persisted. A more satisfactory definition of the "just price" is impossible. Everyone knew what it was, though no one could explain why it should be so.

Behavioral economists also find evidence of a just price mentality. Using telephone surveys, Kahneman, Knetsch, and Thaler were able to isolate community standards of price fairness. Generally, consumers feel they are entitled to their reference price, or past price. They also believe firms are entitled to their reference profit and deem it fair for a firm to raise prices to protect that profit, say because the firm's costs have increased. A firm that takes advantage of an increase in demand by raising its price and makes more than its reference profit is, however, breaking the rules of the game and acting unfairly.

----

So let's bring this back to Uber surge pricing and Amazon/Sears dynamic pricing.

I see two angles here. After centuries of a just price morality, perhaps we are inching towards an alternative framework. Maybe we've finally overcome our revulsion to an 'unfair price' that varies according to fluctuations in demand. Instead of the 19.7 month price spells of yore, we're now willing to endure 19.7 minute price spells. Morality changes, after all; slavery and death penalties used to be common, abortion was prohibited. If so, Uber and Amazon's pricing policy are emblematic of this underlying morality switch.

Or maybe we haven't switched at all and are still operating under the old rules of the moral economy. If so, the new pricing technologies adopted by Amazon and Uber are destined to be met by a titanic wave of consumer revulsion. This may be already happening; Uber's surge pricing policy has attracted plenty of negative press (here and here). Morality is a powerful force; unless they want to be dashed to pieces, the offenders will have to relent and make their prices more sticky.

Not your father's price index: the Billion Prices Project

The price of 52 Samsung TVs gathered by the BPP, April 2008 - November 2009 (Cavallo)

In a previous post, I mentioned that the Billion Prices Project (BPP) contradicts the claims of those who believe that the government understates inflation data. The BPP crawls major US retailers' websites and scrapes them for price data, compiling an overall US Daily Index that is available on its website. The deviation between this index and the official CPI is minimal, as the above link shows.

The BPP isn't your father's price index—it shouldn't be viewed as a perfect substitute for the CPI. So use it wisely. What follows are a few details that I've gleaned from several papers on the topic of online price indexes as well my correspondence with Roberto Rigobon, one of the project's founders.

The most obvious difference between it and the CPI is in the datasets:

1) Online vs offline: The price data to generate the CPI is harvested by Bureau of Labour Statistics (BLS) inspectors who trudge through brick & mortar retailers. Rigobon and his co-founder Alberto Cavallo get their data by sending out lightning fast algorithms to scrape the websites of online retailers.

2) Wide vs Narrow: BLS inspectors compile prices on a wide range of consumer goods and services. According to Cavallo, only 60% of the items that are in the CPI are available online. The ability to track service prices online is particularly limited given the fact that most large retailers' websites only sell goods.

Let's get into some more specifics about what is included in the BPP, because there seems to be some confusion about this in the online discussion. Some commentators have mentioned that the BPP doesn't include gasoline prices. Rigobon informs me that this is wrong, gas prices are included in the US Daily Index. As for the cost of housing, my understanding is the BPP does track real estate data. It incorporates these prices using the same methodology as the BLS. So any deviation between the BPP and CPI should not be attributed to the BPP's lack of either gas prices or housing.

Lastly, despite the fact that service prices are under-represented online, the BPP's US Daily Index does include a number of services. According to Rigobon, the easiest ones to track are things like health insurance, transportation, restaurants, hotel, and haircuts. Others are hard to track, like the cost of public education. My understanding is that Rigobon and Cavallo may use proprietary methods to calculate service prices by referring to various goods' prices as proxies (see here). For instance, in this BIS comment on the BPP, it is noted that the price of education can be computed from prices of text books, uniforms, energy and construction materials, all of which represent 75% of cost of education.

3) Often vs rare: The BPP's algorithms trawl retailer websites every day. BLS inspectors stroll through the malls just once each month.

Another big difference is in the publication of the data:

4) Now vs later: The BPP is reported three days after the data has been gathered, and ten days for non-subscribers. CPI is reported with a long delay, usually the second or third week following the month being covered.

The next few differences are a little more technical:

5) Fixed vs Responsive: Both indexes measure entirely different consumption baskets. The BLS surveys U.S. households every few years in order to gather information about their spending habits. It uses this information to construct a fixed representative basket of goods & services consumed by Americans, and then proceeds to fill in the data each month. This survey approach results in a CPI basket that takes time to adjust to new products. Should a revolutionary device, say a universal mind reader, suddenly becomes popular, it won't be reflected in the CPI till the next survey.

Think of the BPP as capturing a dynamic market-determined consumption basket. The BPP basket is comprised of whatever goods retailers happen to be selling online that day in order to meet customer demand. Because retailers are constantly updating their websites, August 7's basket could be different from August 8's. This means that new goods will be quickly incorporated into Rigobon and Alvarez's inflation calculation. In other words, when universal mind readers do catch on, the BPP will incorporate this data way before the BLS will.

One of the most interesting differences is the difference in methodology:

6) Small vs Large Sample size: The BLS delicately samples offline prices whereas the BPP bulldozes through a large percentage of the entire population of online retailers' prices.

There are millions of goods sold in the US, and it would be cruel to force BLS inspectors to collect prices for all of them. To simplify the calculation, the BLS brain trust chooses individual products to serve as ideal representatives for given product categories. Take dishwashers. To represent the category, they might select the Whirlpool WTD-10 or some such model. A BLS data collector in New York City will go every month to a specific store, say Macy's on West 34th St, and grab that specific model's price. The repetitive use of the same product and location ensures that the New York City dishwasher price index is not corrupted by changes that have little to do with purchasing power. (The alternating collection of prices from Macy's on West 34th and Nordstrom's on Union Square might introduce price changes having little to do with inflation.)

Because their algorithms are whip fast and don't require salaries, Alvarez and Rigobon can afford to send them out each day to Macy's website to gather the price of every single dishwasher. They do this for each of the major online retailers, say Walmart, Target, and Best Buy. The final assemblage of prices represents something close to the entire population of online US dishwasher prices on every single day!

This segues into the thorny problem of adjusting for quality changes. They both use different techniques:

7) Statistical vs market-based quality adjustments: As I pointed out, the BLS samples one good to represent a given category rather than canvassing the full range of products within that category. This causes some difficulties in accounting for quality changes when that one good is replaced by another product.

Let's return to the Macy's example. Say Macy's stops stocking the Whirlpool WTD-10. On arriving at Macy's a few weeks later, our flummoxed BLS data collector has to find a replacement in order to keep the dishwasher price category up to date. Let's say she grabs the price of a General Electric XK-400 from across the aisle. The GE is priced $50 higher than the missing Whirlpool was during the inspector's previous visit. The problem is this: how does the BLS determine how much of that $50 increase is due to changes in quality and how much is due to changes in inflation? If the GE is the same in every way to the Whirlpool except its boasts a turbo wash option, then some portion of the $50 increase is due to the higher quality of the GE. But how much?

Because Cavallo and Rigobon's tireless algorithms regularly retrieve multiple product prices for each category rather than single monthly representatives, they can use the overlapping nature of the data to seamlessly splice in new products. Let's say that the expensive GE dishwasher is introduced to Macy's website. It is sold on the same page as the existing and cheaper Whirlpool for a few days at which point the latter is removed. On the day the GE first appears, the BPP ascribes its higher price to its superior quality. While the GE drives up the average price of dishwashers on Macy's dishwasher page, the purchasing power of a Macy's shopper hasn't been altered, rather, a given dollar buys more 'dishwashing services' than before. Only on day 2, after the GE's price has been retrieved a second time by the algorithms, is it allowed to start affecting the index, since any price change thereafter is considered to be due to inflation, not quality.

The assumption that the GE's premium is due entirely to quality is based on the idea that market prices are accurate measures of all that is known by producers and consumers about a given set of products.

Because CPI collectors have limited resources and typically only collect the price of one representative dishwasher, they usually can't rely on the overlap between dishwasher model prices to measure quality changes. One method they have developed to compute quality changes is hedonic regression. In brief, a dishwasher is conceptually broken up into a package of characteristics, including its size, time per run, energy efficiency, etc. When the Whirlpool is suddenly dropped by Macy's and the GE added, CPI data collectors try to determine what sorts of new characteristics have been incorporated in the GE and then use regression methods to determine the dollar value of that characteristic.

So to sum up, to calculate quality changes, the BPP piggy backs on the power of the market to price differences in quality. The BLS uses econometric methods (among other tools) to control for quality changes.

Here is a big one, the difference in ownership of the indexes:

8) Private vs public: The CPI is compiled by the BLS and funded by taxpayers, whereas Rigobon and Cavallo have incorporated a private company called Pricestats to compile the BPP US Daily index and its many other indexes. PriceStats work in partnership financial-giant State Street to distribute the data to paying subscribers.

Which leads into the last major difference:

9) Transparent vs opaque: There is loads of documentation on the CPI. If you have any questions, call up the BLS and a researcher will walk you through it—it's your right as a taxpayer. PriceStats can only reveal so much information because their methods are proprietary (although Dr Rigobon was kind enough to answer a number of my questions). I suspect they are hesitant to reveal too much of information because the retailers on which they have gathered data might view this as a potentially threatening action. Not so with the CPI.

So those are some of the features of each index. In the case of the BPP, the difficulty of getting public information on their methodology is probably the biggest bug, although the founders are forthcoming on general questions. Maybe if national statistics agencies start adopting BPP data collection methods, the transparency problem will be solved, since public agencies have no competitive reasons (and less legal ones) to hold back information on methodology. There seem to be rumblings in this direction: Statistics New Zealand says that they are in the early stages of a collaboration with with PriceStats to develop online price indexes (link).

For now the public is lucky to get access to the US Daily Index, even on a 10-day delay. When CPI numbers are reported, the bond market quakes. For hedge funds, getting a hint of what the upcoming government inflation print will be before anyone else is probably worth a lot of money. No doubt that's why they are willing to pay to subscribe to get PriceStat's numbers. These funds would probably prefer if the public were not privy to the US Daily Index as it reduces the information's value. The amount they'd be willing to pay PriceStats to yank the US Daily Index from the public domain would be a good indicator of the value the public gains by getting free access to it. It could be a substantial number.

------------------------
Back to initial reason for writing about the BPP; the gold bugs (Gulp, you thought I'd forgotten about you, right?). Your typical gold bug will sagely mention some esoteric price that has risen at an incredible rate over the last few years, like the price of shitaake mushrooms or a 1982 GI-Joe Snake Eyes collectors action figure (here is Peter Schiff using the Big Mac). A gold bug is convinced that their preferred data series is sufficiently strong evidence to justify declaring inflation to be stratospheric and the entire CPI null and void.

What makes gold bugs think that their one or two pet prices are a superior measure of the dollar's purchasing power than the BPP US Daily Index? Crickets. That sums up the gold bug response to the BPP's existence. If not crickets, then desperate attempts to change the subject.

Gold bugs don't like to talk about the BPP because they don't want to be dissuaded from their views—they find too much comfort in them. With the BPP continuing to move in line with the CPI, the gold bug community's cognitive dissonance is growing. At some point, the squirm-level will get large enough that they'll have to do something about it. No doubt the easiest route will be to come up with a fiction that discredits the BPP US Daily index. Well, hey gold bugs, here's a conspiracy theory you can use to save yourselves some painful cognitive dissonance... the Billion Prices Index went offline for a period of time, just when it appeared to be showing a break with the CPI index. When it went back online, the two started to converge. Could it be that Rigobon and Alvarez were brought into some FBI dungeon and re-programmed, the BPP moving more in line with the party line after they emerged? Yeah, that's it.

Gilded cage

This blog wouldn't be around if it wasn't for gold bugs.

Many moons ago my former-employer (and friend), the truest gold bug you'd ever meet, would lecture everyone in the office for hours about imminent hyperinflation, the wonders of the gold standard, and why gold should be worth $10,000. Fascinated, but unsure what to make of his diatribes, I started to read about the history of monetary systems, all of which would eventually provide grist for this blog.

A gold bug will typically have the following characteristics. 1) An abnormally-sized portion of their investing portfolio will be allocated to the yellow metal; 2) they believe in an eventual 'day of reckoning' when gold's price rises into the stratosphere, the mirror image of which is hyperinflation; 3) their investing case for gold is twinned with strong moral view on the decrepitude of the current monetary system and/or society in general; and 4) they are 100% sure that the monetary system's collapse will lead to the flowering of a new and virtuous system, a gold standard.

One thing I discovered fairly early on from my interactions with the gold bug community is that there's no point in debating a gold bug. In any debate, you should be able to ask your opponent what evidence they'd accept as proving their idea to be wrong. Gold bugs are loathe to submit such a list. After all, to do so would open up the possibility that they might have to precommitt themselves to changing their mind, which is the last thing they want to do. A gold bug's ideas are comforting to them. They've structured their entire mental landscape around these ideas, not to mention their entire life's savings and often careers around them.

Gold bugs have a powerful set of defense mechanisms to protect their ideas from outside threat. These mechanism, I'll call them 'mental bodyguards', will kill on sight any idea or bit of evidence that runs contrary to the gold bug schema, thus saving the gold bug from the discomfort, and potential danger, of having to weigh each new bit of data on its own merit.

For instance, consider the fact that central bank money was unmoored from the gold peg in 1968 (almost 50 years ago!). The monkeys behind the wheel should have caused hyperinflation by now and all those financial Noahs who were smart enough to jump into the gold boat before the fiat flood should be fabulously wealthy. But gold trades at just $1200 or so, not far above $850 levels set in 1980. Except for a few exceptions like Zimbabwe, hyperinflation hasn't happened.

Gold bugs can rationalize this contradiction because they possess a 'mental bodyguard' that absolves them of any responsibility for the timing of their predictions. Like the Millerite movement—which predicted the second coming of Jesus Christ on March 21, 1884, only to have to push the date to April 18 when nothing happened, and when that day passed uneventfully, bumped the event to October 22—gold bugs can keep pushing the day-of-reckoning further into the future without suffering any mental dissonance. Using an even more impressive bit of mental-Aikido they turn disconfirmation into a positive. The longer gold's meteoric rise is forestalled, say gold bugs, the more time it provides true believers with an opportunity to accumulate a larger stash of the stuff.

Another powerful mental body guard is the invocation of "them". Gold bugs invariably blame vague external and impersonal forces for wreaking havoc on the noble intentions of gold bugs and the upwards trajectory of the metal's price. They  may be the Federal Reserve, the plunge protection team, or a cabal of Jewish bankers (politically-correct gold bugs just blame Goldman Sachs). When gold falls in price it's always because of the the machinations of these oppressors, without which the metal would be worth $12,000 or $13,000 by now. (Yes, gold bugs like to refer to gold as "the" metal, presumably to differentiate it from all the plebeian metals)

Thanks to the them mental body guard, the inability of gold bug predictions to be borne out in reality is never due to any inherent weakness in the ideas themselves, but to outside interference. Doubts are conveniently refocused on something external like Ben Bernanke and the Fed, upon which gold bugs regularly bestow two minute hates.

Other mental bodyguards that prove useful in protecting the core gold bug ideology include the knee jerk discredit that gold bugs level at both the economics profession and economic data. Gold bugs screen out economists by deriding them as mainstream and therefore (obviously!) puppets of the system. The shoot-first assumption of guilt spares gold bugs from having to engage with these economists' potentially contradictory ideas on a level playing field. The same goes for inflation data, which they dismiss out of hand as being 'cooked'. And if you try mentioning the MIT Billion Prices Index to them, they hum loudly and put their fingers in their ears. (Although when there's any sort of divergence between the BPI and CPI, they suddenly start to make noise).

The awful returns that gold and especially gold shares have provided over the decades have impoverished many gold bugs as well as those unlucky enough to listen to them. Yep, I've seen the year-end statements. Yet somehow the gold bug meme continues to limp on. That's because gold bugs are less concerned about making money than upholding "the cause", as they like to refer to it. The cause is a vague combination of the promotion of a gold standard and a +$10,000 gold price, where simply holding gold through all downturns is an expression of support for that cause. Mere financial losses cannot keep them down.

Now I've been tough on the gold bugs in this post, but the fact is that gold bugs would probably say that both myself and any of their many accusers harbour mental body guards of our own. And the gold bugs probably wouldn't be entirely wrong. With so much time and energy having been invested in the various things we know and believe, a bit of cognitive dissonance is only natural. I'd argue that the gold bugs having walked much further out along that plank than their critics.

This post won't change the minds of any gold bugs—as I already pointed out, they've made up their minds long ago. But if you're a busy individual with some money to invest, and you're considering a gold bug advisor, remember that the fate of your investment may take second seat to the gold bug's devotion to the cause. Be wary.