This five-part series dives deep into the economics of electricity markets and the role that markets can and must play in the decarbonisation journey. But in reality, do they? And what can be done about it?
In this last part, Towards future ready markets, let’s explore some ideas on adaptations that could be made to electricity markets to transform them into drivers of the clean energy transition. We discuss how such adaptions, supported by a strong regulatory mandate, could help to generate long-run market signals that facilitate fast, large-scale, and sustained growth in renewables.
Read previous parts: Part I, Part II, Part III, Part IV
In Part IV, we discussed how electricity is likely not a commodity in a strict economic sense. While that may lead to the conclusion that it’s provision and transition to renewables is best centrally managed by tightly-regulated, governmental agencies in contrast to the current wave of liberalisation and deregulation, I argue that a middle ground is possible.
We discussed that a free-market and a well-designed policy framework which incentivises private and public investment into clean energy must co-exist and cooperate to facilitate the transition. I argued how regulation must be the steward for future generations to ensure the necessary investments are done.
In this final part, let’s look at how policy and market design could work together more closely to generate the “correct long-run market signals” for those investments.
Let’s break that phrase down.
Market signals commonly refer to the events or trends resulting from market clearings that trigger (by incentivising or disincentivising) market participants to take any action in the market, e.g., to buy or to sell, by how much, where to offer or bid, etc.
Here’s a simple example in the electricity context. A short-run market signal is the occurrence of high prices of electricity, let’s say due to an unplanned outage of a nuclear power plant. This deficit in supply immediately triggers more participants to offer to sell in the next market iteration— which in turn, has the effect of bringing the price down and a stable market equilibrium is reached. Therefore, short-run market signals provide generators and loads incentives to change their decisions today or in the coming week (e.g., when responding to weather conditions, planned grid outages or maintenance, etc.).
In contrast, long-run market signals refer to those which guide investment decisions, e.g., when and where to build new power plants or loads such as data centers or storage units, how to dimension such investments, which technology to prefer, and so on. A simple example of a long-run signal is the congestion in the grid connecting, say, two countries, which leads to higher prices in one of them. This incentivises investors to, over time, build new power plants in the country with higher prices. This leads to: (a) increase in social welfare in the long term as consumers in the country pay lower prices, and (b) alleviation of congestion in the grid, thereby deferring the investment costs of upgrading it.
A market signal, both short-run and long-run, is correct when it aligns the social welfare goals with the rational, utility-maximising goals of individual market participants. In other words, a correct market signal , when generated by a market clearing mechanism, incentivises participants to act towards improving the efficiency of the market while taking actions towards improving their own profits.
Long-run market signals are extremely relevant when it comes to bringing changes in markets in the long term.
Pertinent to our discussion, long-run market signals in electricity markets must incentivise rapid and large-scale capacity growth in renewables.
Referring to long-run market signals in the current energy market paradigm, this article discuss the role of long-term forward (or futures) electricity markets in generating them. Purely financial products, i.e., those without any associated expected physical delivery of energy, are traded in such forward market which are settled by financial transactions instead of energy flows.
The primary goal is to enable market participants to hedge against adverse realizations and volatility in the short-term (day-ahead, intra-day and real-time) physical energy markets. A secondary goal is to facilitate long-term capacity planning and investment, by shielding investors from energy price volatility in short-term markets, discussed in Part III. Large volumes of energy (and energy-based derivatives) are traded in such forward markets, typically with delivery periods up to a few years ahead and through bilateral contracts between buyers and sellers.
However, they suffer from several drawbacks which implies that the above goals are attained to a limited extent. First, such forward markets have traditionally been energy-only and do not have an explicit focus on facilitating new capacity build-up. Furthermore, given the nature of products traded and the primary goal, i.e., hedging against short-term prices, they suffer from liquidity issues as the contract horizons increase. A recent policy paper from ACER highlights the challenges for new investments in energy, owing to the lack of liquidity in energy-only forward markets beyond 3 years-ahead.
One solution approach is to supplement the existing markets with an additional layer of auctions-based forward capacity markets with the specific intention to provide guaranteed capacity payments to new investments, e.g., such as those implemented by the system operator, ISO New England.
In the following, we do not focus on the long-term forward markets. This is reasonable because prices in energy-only forward markets are anchored to and move according to the underlying energy prices in short-term markets. Therefore, if prices are “incorrect” in those markets, long-term markets cannot be expected to fulfill their objectives fully.
In other words, the above approach implies that policy interventions and market adaptations proposed in the following operate under the paradigm that marginal prices arising from clearing of short-term electricity markets are responsible to generate the long-run signals, such as discussed in the congestion example above.
Given that and the market deficiencies we’ve discussed so far, several academic and non-academic voices have recently raised the topic of a possible redesign of energy markets necessary for a renewable future.
These works have been nicely covered in this article, which addresses the question of how future electricity markets may look like, from a well-rounded socio-techno-economic perspective and provides several interesting recommendations.
Of particular relevance to our topic here, is the recommendation that electricity markets should go from the current deterministic market clearing practices towards a more stochastic approach.
What does that mean?
It means that uncertainty and variability, inherent to the power produced by weather-dependent renewables, is accommodated in the market not via ad hoc patch-fixes as is the current norm, but should instead form a structural element of the price formation process itself. The risk arising from the uncertainty and variability is, therefore, priced into the market clearing process.
In the interest of full disclosure, I must clarify that this was also one of the topics I examined during my Ph.D. research.
For example, this article proposes a multi-commodity and multi-period day-ahead electricity market that not only clears quantities of energy (MWh) to be traded, but also flexibility products, e.g., “contracts” that flexible producers and consumers can enter into to provide response to forecast errors during real-time operation next day.
These additional commodities (flexibility products) are priced according to a competitive, spatial price equilibrium framework, i.e., at periods and locations where less flexibility is available, the price of these flexibility products is higher (thereby, sending the correct long run market signals for investment in flexible resources e.g., energy storage) and vice-versa.
As an interesting side note, the simulated, simple case study in this work (see Figure 1 below) already gave glimpses of price depression in day-ahead markets and simultaneous demand for and high prices expected for flexibility products, which we discussed at length in Part III.
The fact that flexibility products must play a crucial role in the grids with high shares of renewable energy is highlighted in this recent report released by the Danish Transmission System Operator (TSO), Energinet. Furthermore, the TSO provides some rather interesting data and insights into the historical and projected procurement quantities and prices of flexibility services purchased by the grid operator (i.e., ancillary services) here.
Additionally (yet, perhaps more fundamentally), one must pose the question whether pricing electricity based on marginal (fuel) costs is compatible with a future where the energy system is dominated by renewables which not only have no fuel costs but also no opportunity costs.
What is the marginal cost of power production from a wind farm or a solar PV park?
Without the fuel used up to produce that unit of energy, could it be a combination of the “cost of capital” and the “cost of supply firming”?
Here supply firming refers to the operation of flexibility provider, e.g., energy storage unit or demand response aggregator, that’s on standby to step in to produce/consume depending on whether the renewable power producer under-/over-shoots the predicted production.
While cost of capital is fairly easy to calculate, how does one quantify the cost of firming?
The answer to the above question could lie in a market redesign, as illustrated in the following.
Firmed supply bidding into the market is expected to play a key role in a future energy market and enable a market-based discovery for the price of firming. At the same time, this generates correct long-run investment signals for flexibility providers to ensure renewables can “truly replace” the dispatchable fossil-fuel based power producers.
This is relevant because renewable power producers are already (or will soon be) balance responsible, i.e., they will be required to inject into the grid exactly as much as they had previously bid and were cleared by the market, otherwise face penalties.
Such a change in paradigm is well-supported by firmed supply bidding, since it contributes to internalise the negative externality of uncertainty and variability from renewable energy.
Incorporating firmed supply bidding into electricity markets could take several forms; three of which are briefly outlined in the following.
Hybrid and virtual power plants: One possible adaptation to the energy markets is to enable new flexibility-centric market participants such as wind-solar-storage hybrid power plants, power-to-x hybrid power plants, virtual power plant aggregations, demand response aggregations etc. to bid while expressing their full range of operational feasibility and therefore, their ability to deliver a firm supply. This would require a significant change to the market-clearing algorithm and the bidding framework.
Such hybrid power plants act as a physical hedge against adverse renewable production forecast realisations, while additionally bringing new revenue streams to the renewable projects, as discussed in Part IV. More importantly, they help alleviate the high price volatility we discussed in Part III, thus bringing revenue stability for renewable energy projects. As an immediate effect, one can expect occurrences of negative prices and curtailment to become less frequent, which will be a great booster for profits of renewable energy projects.
Uncertainty-aware market-clearing algorithms: Another possible incorporation of firmed supply bidding could be the adoption of market clearing algorithms that endogenously consider uncertainty and perform a joint clearing of energy and flexibility, as discussed previously. In such a future market, renewables may be mandated to hold physical trade positions that stack up to a firmed supply delivery across the various jointly-cleared market products. Several academic works have delved deeply into such market redesign proposals in recent years; a survey can be found here.
Mandated flexibility procurement: Lastly, without necessitating a complete overhaul of the market clearing algorithm, firmed supply bidding could be implemented within the existing market framework via regulation-mandated and location-specific flexibility procurement linked to renewable production bids.
In such a future, renewables are responsible for and are explicitly charged for (preferably in a market-based price discovery framework) firming their production to match their energy market bids. They would be required to participate in flexibility trades (and incur risk payments associated with those), where they take positions based on their confidence on forecast accuracy, i.e., asymmetric positions of over-/under-production estimated by them are matched by the flexibility trades they undertake.
Flexibility trades, in the above context, behave similar to a combination of state-price securities or Arrow-Debreu securities of some form, which have been used in the financial markets for a long time as effective risk hedging instruments.
This is particularly relevant and has strong foundations in microeconomic theory, since for an energy system dominated by weather-dependent producers, internalising the forecast risk is an essential contributor to reach completeness of the market.
In the long-run, such a market sends signals that eventually penalise the risk payments, i.e., incentivise investments in improved production forecasting methods, which, in turn, improve the overall social welfare.
Needless to say, the adaptions discussed above are incomplete, non-exhaustive, and need further evaluations. Flexibility and firmed supply bidding aside, I can highly recommend this very recently published review article which provides an excellent account of the challenges, recent works, and the open research questions on market redesign and price formation in the new paradigm of energy systems, primarily composed of zero-marginal cost power producers.
Now that we have discussed a few possible pathways to enable energy markets to become future-ready (or rather, just “catch up”, because a renewable-dominant energy system is already a reality in several geographies), let’s reflect on the relevance of this topic.
Recalling the role of markets in steering rational and self-interested actors towards building social utility as discussed in Part I, I feel it necessary to elaborate the other facet of that story here.
It is also well-known that markets are typically not great at accounting for long-term risks, i.e., rational and self-interested market participants act while excluding such risks from their decision-making, since the impact of such risks on their current utility is negligible, when considered time discounted over the long term.
Adverse impacts of climate change, which poses a long-term socio-economic risk, are often neglected in the decisions made by actors in capitalist markets.
Therefore, if energy markets have to facilitate the transition to cleaner energy transition, they have to be designed not only with carrots (financial support) for renewable energy but also have proverbial stick (taxes) for fossil-fuel based energy sources. This could, for example, take the form of a carbon tax which is levied on all goods and services based on the emissions incurred in delivering them. Undoubtedly such a step is bold and has been talked about for several years now and requires a global consensus to be built for it to be effective.
However, above all, it requires tremendous courage and conviction from general population — the willingness to sacrifice short-term utility to preserve the long-term health of our planet.
We must agree and accept as a society that consuming goods and services, that are cheap in monetary terms, but whose bill of the associated carbon emissions is charged forward to the next generations, is not sustainable. Bitter pills and compromises must be swallowed by all of us to ensure a future for our planet.
How do we put that into play? First comes awareness, and with it comes action. We must make our local leaders and decision-makers accountable for climate change mitigation steps; ensure that our planet’s long-term future must always be a part of the agenda, irrespective of the short-term issues facing us.
Ultimately, given that societal de-growth is unpalatable to almost everyone, the only solution we have is faster decarbonisation, driven by a massive upscaling of investment in renewable capacity growth.
For that to happen, we have to urgently start treading along one of (or a mix of) the two paths: either we correctly internalise the price of climate change externality in all our economic decisions as individuals and economies, or we welcome regulation that overrides the shortsighted profit-seeking behaviour that is characteristic to actors in market frameworks.
Call me ruined-by-education, but I would much rather prefer only the first solution.
And I believe with an open and constructive dialogue among all stakeholders involved, we can get there.
We must. We cannot afford not to.
Thank you for making all the way through to the end of this rather long article series. Hope you found it useful, engaging, and informative.
I would love to hear your thoughts, insights, and reflections in the comments.
<hr><p>Are electricity markets fit for purpose? Part V: Towards future-ready markets was originally published in Clean Energy for Billions on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>