Adelman-Frankel Award Winner

The U.S. Energy Sector:  Progress and Challenges



Paul L. Joskow[1]


I am honored to be the 2009 recipient of the Adelman-Frankel prize given by the United States Association for Energy Economics (USAEE).   Unfortunately, I was unable to attend the annual meeting to receive this award in person.  Instead, I have been invited to write this essay containing some of my reflections on changes in the energy sectors and public policies toward these sectors during my career to date.  This is now a time period of over 35 years. Hard to imagine that it has been this long.

I did not start my academic career with a special interest in energy economics and policy.  Of course as an undergraduate student and advisee of Alfred E. Kahn at Cornell, I could not avoid learning something about electricity pricing, the field prices of natural gas, and competition issues associated with the petroleum industry.  However, as a graduate student my primary interests were in the areas of government regulation of industry, antitrust policy, political economy, the organization of firms, and industrial organization more generally.  My later interests in energy grew out of my work on government regulation and industrial organization and, in turn, some of my work on government regulation and deregulation, the organization of firms, contracts, industrial organization, and environmental policy were initially stimulated by my deepening understanding of the energy sectors and the public policy environment in which they operated.  Thus as my interests in energy economics and policy expanded, my research and teaching on energy economics topics have helped to reinforce and expand my broader interests in issues related to industrial organization, the organization of firms, regulation, antitrust policies, political economy and economic institutions.


When I began working on energy-related topics in the early 1970s, a large fraction of the energy sector was subject to price regulation and in some sectors the government also restricted entry of new suppliers.  Retail and wholesale electricity prices were regulated by state and federal regulatory agencies and electricity was supplied by regulated vertically integrated geographic monopolies.  The field price of natural gas sold in interstate commerce was regulated by the Federal Power Commission (later the Federal Energy Regulatory Commission or FERC), as were prices and entry for natural gas pipeline service and contractual arrangements between interstate pipelines, gas producers, and customers (local distribution companies and large retail customers).  Retail gas prices were regulated by state public utility commissions and commodity gas and transportation service at both the pipeline and local distribution levels were bundled together.  The federal government began regulating oil and petroleum product prices in 1971.  Coal prices were not regulated, but railroad transportation charges were heavily regulated and deficiencies in railroad price regulation and related policies affecting investment in railroad infrastructure and the quality of service adversely affected access to coal supplies.  These price and entry regulations led to energy shortages, effectively subsidized energy imports, adversely affected domestic energy supplies, decreased the ability of the U.S. energy sectors to respond efficiently to supply shocks, and distorted energy consumption decisions.

Perhaps the most significant change in the U.S. energy sector since the early 1970s has been the gradual demise of price and entry regulation in most of these sectors.  Petroleum prices were finally completely deregulated in the early 1980s (though lawsuits over “overcharging” during the regulated price era continued for many years).  The complete deregulation of natural gas field prices took longer, but the deregulation process for commodity natural gas was finally completed in the early 1990s.  The oil shock(s) of 1979-1982 led to serious domestic shortages (properly measured) of both petroleum and natural gas and complex and inefficient government administered allocation mechanisms were introduced to allocate scarce supplies. The inefficiencies of the natural gas regulatory framework were especially large and widely documented.  Concerns about “windfall profits” and interregional wealth redistribution led to relatively long deregulation “transition periods,” especially for natural gas.

The dramatic decline in natural gas prices (at least on the margin) that began in the mid-1980s also stimulated a completed restructuring of the rest of the natural gas industry.  Long term contracts between producers and pipelines and between pipelines and local distribution companies were completely restructured and the sale of commodity gas unbundled from the sale of pipeline transportation service.  In some states unbundling has been extended to the retail level as well.  The Federal Energy Regulatory Commission (FERC) also adopted more “light handed” regulation of pipeline transportation charges and the hoops investors had to jump through to build new pipeline capacity were relaxed.  As a result, prices for natural gas pipeline service and entry of new pipeline capacity in the U.S. have largely been deregulated “under the shadow of regulation as a backstop.”  I have previously referred to the current regulatory framework for pipeline charges and entry as a system that works in practice but not in theory, since it is difficult to define a coherent theory that describes how the current pipeline regulation actually works in practice.  In theory it looks quite rigid but in practice it is quite flexible.  There is, of course, a danger that the deviation between the theoretical principles of pipeline regulation and how they are applied in practice can lead to unintended consequences down the road as the political winds of regulation shifts and memory fades.  Overall, however, structural and regulatory reform in the natural gas sector has created a flexible efficient North American market for natural gas that is a far cry from the mess that was created by regulation during the 1970s.

In 1982, the Staggers Act led to virtually complete deregulation of railroad charges for transporting freight.  While some coal suppliers have complained repeatedly about being overcharged for service after deregulation, the overall consequences of this deregulatory initiative have been very good.  It has allowed the railroad industry to restructure, to operate profitably after decades of bankruptcies, to expand and modernize the railroad infrastructure and to support the transportation of growing volumes of low-sulfur coal over much longer distances than had been feasible in the past.

The hardest nut to crack has been the restructuring and deregulation of the electric power sector.  Low natural gas prices during the late 1980s and the 1990s were a driving force for restructuring here. Low natural gas prices and developments in combined cycle gas generating technology (CCGT) led to a situation where the total cost of producing electricity from these new facilities was lower than the implicit price of generation service reflected in regulated prices based on the embedded costs of existing facilities (including the costs of expensive nuclear power plants entering service during the 1980s).  This gap was especially large in the Northeast, portions of the Mid-west and California. Industrial customers in particular argued for “open access” and retail competition, as was emerging the natural gas sector, in order to bypass paying regulated prices in favor of buying power directly from lower cost sources.  (There was and is significant hypocrisy among some segments of the large industrial customer interest group as their policy seems to be to get the lower of the regulated or competitive price --- an unsustainable policy.)  At the same time, PURPA had created a growing group of independent power suppliers which had an interest in expanding their opportunities to supply electricity in competition with utility-owned generation in a wholesale a retail market.

So, the pressure was on to restructure and deregulate potentially competitive segments of the electric power industry.  The restructuring and competition program for the electricity sector adopted in the UK in 1989-90 provided a model for how it could be done and also provided some examples or where care had to be taken (e.g. generator market power).  This model had a big effect on the pioneering restructuring and competition model debated and implemented in California in the mid-1990s.  FERC introduced new transmission access and wholesale market rules in 1996 that supported the development of a competitive electricity sector. Variants of the California/UK model were then adopted in a number of other states including New York, Massachusetts, New Jersey, Pennsylvania, Texas, Michigan, Illinois and Ohio.  Indeed, by 2000 it appeared that restructuring for competition was sweeping the electric power sector in most parts of the country.  Then the California electricity crisis hit, the unfortunate consequence of a sudden spike in natural gas prices, a decade of underinvestment in new generating capacity preceding restructuring, a poorly designed stranded cost recovery mechanism, a poorly designed retail transition pricing mechanism, and a poorly designed wholesale market.  The resulting chaos effectively stopped and even reversed the spread of restructuring and competition in electricity to additional states.

Today we have states that have continued their commitment to the competitive model, states that are unenthusiastic and seeking ways to reverse it (usually leading to the worst of both competitive and regulated models), and states that have never departed from the traditional model of regulated vertically integrated monopoly.  This situation of a very diverse industrial organization and regulatory framework for the electric power sector that is physically integrated into three AC networks is in my view inefficient, unsustainable, and will make crafting a good greenhouse gas mitigation policy for the electric power sector especially difficult.

All things considered, one cannot but be impressed with the dramatic changes that have reduced the heavy hand of price and entry regulation and increased the role of competitive markets in the U.S. energy sectors.  And these developments have served the country well by getting the prices right, stimulating more efficient supplies, and providing a framework that allows the energy system to respond quickly and efficiently to major supply shocks, such as hurricane Katrina.  It also seems to me that it is a sustainable change as long as the lessons learned from the 1970s and early 1980s era about the costs of heavy handed regulation are not forgotten.  While we do and will hear lots of speeches from politicians complaining about rising oil, gas and electricity prices during periods of time when supplies are tight, one sees few serious efforts to reregulating prices for natural gas, petroleum, or railroad transportation.  The future of electricity sector reforms remains more in doubt, however and deserves more attention by researchers.


The first major national environmental laws were passed in the early 1970s just as I began my academic career.  The energy sectors were heavily affected by these new laws since these sectors are major emitters of conventional air pollutants, water pollutants, toxic wastes, and have noticeable and often unappealing affects on land use.  Their large facilities were also easy targets for regulators.  New environmental statutes and new environmental regulations tightened the constraints on the energy sectors.  As climate change has become a national and international issue, the energy sectors have been at the center of policy debates since the combustion of fossil fuels in the primary source of greenhouse gasses in the United States.

Accordingly, as price and entry regulation of the energy sectors has faded away, environmental regulation has gained increasing importance.  Indeed, environmental policy and energy policy are now so closely related that it is almost impossible to separate them.  This fusing of environmental policy and energy policy represents a gradual but overall very dramatic change for the energy sectors.  Tightening environmental constraints have affected supply costs and prices, affected fuel choices, and affected technology on both the supply and demand sides of the market.  These effects will only become more significant as constraints on CO2 emissions are tightened.  As time goes on, effective CO2 mitigation policies will lead to much less use of coal compared to business as usual, more use of natural gas, more reliance on low-carbon supply technologies, innovation to reduce the costs of low-carbon technologies, and innovations to improve the end-use efficiency with which energy is used.  These changes are necessitated by the simple arithmetic of the options that are likely to be available to meet the aggressive goals to reduce CO2 emissions by up to 80% by 2050 that we see in recent legislative proposals.

In the last 15 years we have seen a major change in thinking about the regulatory mechanisms that would be used to meet environmental goals.  For 25 years the major mechanism used by federal environmental regulators was what economists refer to as “command and control.”  Command and control regulation refers to regulations that require emissions sources to install particular abatement technologies or to meet specific emissions constraints without regard to relative costs of emissions reductions, let alone costs vs. benefits.  Two decades of research demonstrated that these policies were inefficient, slowed progress in achieving environmental goals, and often simply did not meet those goals at all.  Economists favored more flexible market-based mechanisms in the form of emissions taxes or cap and trade systems that placed a price on emissions and provided incentives for sources to respond to these emissions costs in the most economical ways.  These mechanisms also provided dynamic incentives for technological innovations to reduce the costs of reducing emissions.  For many years the use of economic mechanisms was not taken seriously by environmental regulators.

The Clean Air Act Amendments of 1990 which created a national cap and trade system for SO2 emissions to reduce emissions in response to concerns about the damages caused by acid rain changed this situation.  The SO2 cap and trade system was successful in all important dimensions.  Its success in turn stimulated the use of cap and trade mechanisms to control NOX emissions in some regions of the country and CO2 emissions in the European Union.  Cap and trade is a key feature of the greenhouse gas legislation being considered by the U.S. Congress as this essay is written.  This is indeed, a major and positive change in the approach to environmental policies affecting the energy sectors and is compatible with the earlier policies to deregulate prices and entry in most of the energy sectors.

However, I fear that the political reality is that the faith in market-based mechanisms for controlling emissions is broad but not very deep.  Economists have not helped matters by getting into heated debates about whether the right mechanism is an emissions tax or a cap and trade system.  This debate is like arguing about how many angels can stand on the end of a pin and provides fodder for those who would like to see no greenhouse gas mitigation policy at all.  To a first approximation an emissions tax and a cap and trade system are the equivalent if they are well designed.  The “simple” emissions tax with efficient recycling of revenues that some economists favor is not the kind of emissions tax we would get in reality and is little different from a cap and trade system that auctions all emissions permits (a proposal made by President Obama in January 2009 that survived political backlash for about three weeks).  A CO2 emissions tax system would have exemption, loopholes, and wealth redistribution provisions just like the rest of the tax system.  The revenues are unlikely to be recycled efficiently.  A cap and trade system that clearly separates the wealth redistribution issues associated with “allocation” of allowances from the efficiency effects of free emissions permit trading and abatement incentives created by the prices from trading emissions permits in a well designed trading system (no updating, no free allowances for new facilities, no confiscation of allowances for closing facilities, etc., as with SO2) can do just as well as an emissions tax system that must respond to the same political constraints.

I would argue as well that while it appears superficially that market-based mechanisms for controlling emissions have won out over command and control, the legislation that is now being considered in Congress is far from a pure cap and trade system.  Indeed, the cap and trade system included in the leading bills could turn out to be a side show to a massive introduction of more command and control regulation.  Renewable energy portfolio standards, automobile, appliance and building energy efficiency regulations, subsidies for all of them, and emissions permit allocation rules that violate the principle of separating wealth distribution and efficiency consequences may seriously undermine the relevance of the market-based mechanism in the form of a cap and trade system.  So, while policymakers have certainly become more receptive to market-based mechanisms that place prices on emissions, they are not sufficiently convinced that they will work to move forward without a lot of mandates and command and control regulations.


When I began my energy economics research career in the early 1970s, one took a standard approach to thinking about and measuring energy demand and the factors that affected it.  We estimated price elasticities, income elasticities, cross-elasticities, weather effects, effects of innovations in energy-using equipment, etc.  Energy was thought of more or less like any other good or service, except that research recognized that consumers did not get utility from energy itself, but rather from the useful services that it provided.  Consumers were free to make their consumption decisions given their preferences and budget constraints.

This “caveat emptor” situation began to change after the second oil shock as legislation and regulations began to be implemented to mandate minimum energy efficiency standards for automobiles, appliances, and buildings.  They were first implemented for automobile fleets, and gradually for household and commercial appliances, and for buildings.  Utilities in many states were induced to adopt energy efficiency programs to subsidize the installation of energy equipment appliances, lighting, insulation, etc. Utilities spent over $30 billion on these programs (unadjusted for inflation) between 1990 and 2007.  Appliance labeling regulations were implemented to better inform consumers about the cost of operating appliances like refrigerators.  Pending legislation will significantly tighten and expand these regulations in the future. And the energy efficiency of the U.S. economy has improved significantly over time, though the rate of improvement was higher during the 1970s and early 1980s when prices were rising than since the mid-1980s as these regulations began to bite.

There are classical arguments to rationalize regulations such as these.  It is hard to argue with regulations that provide consumers with more and/or better information to guide wise decisions about buying and using appliances and equipment, insulating homes, and so on since information is a public good and information markets are imperfect.  Another classical argument for energy efficiency regulations is that energy prices are lower than the true social cost of supplying energy due to regulations that keep prices below competitive market levels, and external costs, including energy security costs.  While it would be better to get the prices right, doing so often confronts political difficulties. Efficiency standards are more popular politically than are taxes (true from poll data comparing the public’s attitudes toward gasoline taxes vs. mileage standards).  However, if these were the primary rationales for energy efficiency regulations one would have thought that deregulation of energy prices, tighter environmental regulation, pricing emissions, and 20 years of experience with efficient appliances and buildings would have reduced the need for regulations by reducing significantly the market imperfections that may make such regulations desirable.

But just the opposite appears to be the case.  The government is placing more emphasis on mandatory efficiency standards today than in the past --- even outlawing incandescent bulbs just as we put the price of CO2 into electricity prices.  This is the case because the classical arguments outlined above are not the ones that are used by proponents to justify energy efficiency regulations.  Instead, the basic argument is that consumers face numerous “barriers” to making rational long term energy-related investment and utilization decisions.  When they make investment and utilization decisions they routinely leave $100 bills on the floor that are just lying there to be picked up if consumers just acted in their own self-interest.  Since they do not, the government will force them to do so with mandatory minimum energy efficiency standards or straight bans on certain types of energy-using equipment.

This widely accepted perspective leads to a number of questions.  First, if these decision-making imperfections are true for investments in energy-using equipment why are they not true about every other investment in the economy that involves a tradeoff between up-front costs today for benefits of some kind in the future?  What is special about energy?  If it is a more general problem it raises more fundamental questions about market economies.  Second, are there really $100 bills lying on the ground ready to be picked up?  My own research suggests that the answer is often “no.”  The studies upon which these numbers are based often fail to account for the real economic costs, including transactions costs (e.g., waiting at home instead of working for the installer who never shows up), of the equipment, underestimate installation and maintenance costs, and fail to account for wide variations in consumer utilization and equipment replacement behavior.

This being said, I believe that there is compelling evidence that while there may not be $100 bills lying around for everyone to pick up there are a lot of $50 bills lying around for a significant fraction of homeowners and businesses to pick up.  This in turn suggests that before imposing mandatory minimum energy efficiency standards we should better understand exactly why these $50 bills are not being picked up by those who can benefit from them.  This is the case because there may be better mechanisms to “nudge” (to use Sunstein and Thaler’s term) consumers to do what is in their best interests than mandatory standards or bans on appliances.  Yet there is surprisingly little research that has been devoted to understanding exactly what the relevant barriers are or alternative approaches to help consumers to make decisions that are supposedly in their own self-interests.

So, in forty years we have moved from leaving it to the consumer to decide how to consume energy based on standard private cost and benefit calculations to leaving these energy consumption decisions instead to Congress to decide.  It is hard to be convinced that Congress is likely to get it right more often than not.


There is one thing that has not changed since the early 1970s.  If you cannot think of a reasoned rationale for some policy based on standard economic reasoning then argue that the policy is necessary to promote “energy security.”  Many people then stand up and salute when “energy security” is at risk.  Unfortunately, it is not clear exactly what energy security means when the term is thrown around, or that it even has a unique definition.  It usually has something to do with importing oil from “unstable” areas of the world, associated concerns that the U.S. will somehow be cut off from global oil supplies, and that we will have to live through the gasoline lines, shortages, inflation, unemployment, slower productivity growth, and so on generally associated with the 1973-74 and the 1979-82 oil shocks.  The energy security case for energy policy reform reached an especially low level in the last presidential campaign.  President Obama argued that renewable sources like windmills would contribute to energy security.  Candidate McCain argued that nuclear power would contribute to energy security.  Neither argument made any sense.  Windmills and nuclear power plants produce electricity.  But the U.S. uses almost no oil to generate electricity.  Both windmills and nuclear power plants would largely displace fuels that are securely supplied by the U.S. and Canada.  Both windmills and nuclear power may be desirable for other reasons (e.g., for reducing CO2 emissions), but certainly not for energy security purposes.  Nearly 70% of the petroleum consumed in the U.S. is used in the transportation sector and if one thinks that reducing imports of oil is a good idea it is the transportation sector not the electric power sector where the action is.

Nor does the focus on “U.S. imports from unstable parts of the world” make much sense.  Only about 15% of U.S. oil imports and less than 10% of U.S. oil consumption comes from the Persian Gulf.  Most of the rest comes from North America, South America and West Africa.  This makes perfectly good sense since there is a well integrated world oil market and the distribution of that oil from producing to consuming countries will largely reflect transportation costs.  The fact that 10% of U.S. oil consumption comes from the Persian Gulf is a fact that is irrelevant for understanding the economic impact on the U.S. and other oil importing countries of a major disruption of supplies in the Persian Gulf.  This is because if there is such a disruption it will affect world oil prices not just the prices charged to the U.S. from oil produced in the Persian Gulf.  The U.S. could import nothing from the Persian Gulf and still face the economic consequences of a major global oil supply disruption.  Moreover, the oil shortages observed in the U.S. during the first and second oil shocks were due primarily to the price regulations and administrative rationing schemes that were in pact, not because these is a necessary feature of oil supply shocks.

It is clear to me that thinking and rhetoric about energy security has not advanced very far in 35+ years.  We need fresh thinking that clearly defines exactly what we mean by energy security, takes a global perspective that incorporates the attributes of world oil markets into account, and policies that are ultimately based on reducing the economic and wealth redistribution effects of oil supply shocks on the U.S. economy and those of other oil importing countries.


Over the last nearly four decades major progress has been made in removing costly price and entry regulations affecting almost every energy sector directly or indirectly.  I consider the reforms affecting the natural gas industry to be most impressive.  Those affecting the electric power sector the most disappointing.  Environmental policy and regulation has replaced price and entry regulation as the most important menu of regulatory policies affecting all energy sectors.  The role of environmental policy will only become greater as the U.S. becomes serious about climate change.  There is much to say for the argument that policymakers have gotten religion on the importance of relying on market mechanisms for allocating scarce resources to and within the energy sector and for relying more on market-based instruments to deal with environmental problems.  However, the support for markets and market-based instruments is not as strong as may first meet the eye.  We can look forward to a growing role for government regulation in choosing the technologies used to produce and especially to consume energy even as policy moves forward placing prices on CO2 emissions.  Thus, in the last 35+ years we have moved from heavy handed reliance on one type of regulation of energy markets motivated by one set of economic concerns and interests to heavy reliance on another type of regulation motivated by environmental concerns.  Energy security policy rationales remain a refuge for rogues who have difficulty making more respectable and coherent cases for the policies they favor.



[1] President, Alfred P. Sloan Foundation and Elizabeth and James Killian Professor of Economics at the Massachusetts Institute of Technology.  The opinions expressed here are my own and do not reflect the views of the Alfred P. Sloan Foundation or the Massachusetts Institute of Technology

Click to view a printable version of this article.


Volume 24, Number 3 - 2016