US economists Leonid Hurwicz, Eric Maskin and Roger Myerson have won the Nobel Memorial prize for economics.
The theory has been applied to the economics of climate change
The three researchers' work "laid the foundations" of mechanism design theory, the prize committee said.
The theory allows economists to distinguish situations where markets work well from those where they do not.
The Royal Swedish Academy of Sciences said the winners' work played a central role in many areas of economics and parts of political science.
The three winners will share a prize of 10m Swedish krona ($1.5m).
The committee said the theory they helped to develop "allows us to distinguish situations in which markets work well from those in which they do not. It has helped economists identify efficient trading mechanisms, regulation schemes and voting procedures."
It is an application of game theory when applied to situations where perfect markets do not exist - for example, within firms or in political bargaining between interest groups.
The economists sought to explain how to reach an optimum outcome, such as improved social welfare or greater profits, and to analyse what sort of government regulation should apply to these types of markets.
In doing so, they were further developing the use of game theory, pioneered by John Nash, to economics. Mr Nash won the Nobel prize for his work in 1994.
But unlike Mr Nash's original theory, they were also analysing situations were players could play the game again and again, learning each time.
In 2005, Thomas Schelling and Robert Aumann won the economics prize for similar work, applying game theory to politics.
Professor Roger Myerson, 56, an economics professor at the University of Chicago, has used the theory to compare how different electoral systems might function.
He also developed the theory of "revenue equivalence", the backbone of the approach. One branch of the theory looks at auctions - for example, the government's sale of the 3G mobile phone spectrum - as an efficient way of allocating resources.
Professor Myerson developed an approach to auctions which could help maximise the sellers' revenue even when, as often is the case, there are differences in the amount of information each side has - something which could distort the market.
Professor Myerson was born in Boston and educated at Harvard, where he received a PhD in mathematics in 1976. He has also taught at Northwestern.
Professor Eric Maskin, 56, who works at the Institute of Advanced Study at Princeton, has concentrated on how to make auctions more effective and how firms actually compete in the real world.
He has also looked at how effective the legal regulation of economic activity might be under conditions of imperfect information.
Among the issues that he applied the theory to, with his collaborator Partha Dasgupta, is how to regulate climate change.
His theoretical work points to the existence of equilibrium solutions in the case of discontinuous games that are played again and again.
Professor Maskin was born in New York City and was educated at Harvard at the same time as Mr Meyerson, receiving a PhD in mathematics.
He has taught at Cambridge, MIT and Harvard.
Professor Leonid Hurwicz, who was born in 1917 in Russia, is one of the early pioneers of the application of game theory.
His interests include the theory of the firm, mathematical economics and modelling.
He served on the Cowles Committee - which sought to link mathematics and economics by creating mathematical models of the economy - in the 1940s.
He worked with Nobel prize winner Kenneth Arrow in the 1950s on non-linear programming.
He has been emeritus professor at the University of Minnesota and received the National Medal of Science in 1990.
The prize has awarded by the Swedish central bank since 1969, and is not part of the original legacy of Alfred Nobel when he established the prizes in the early 1900s.
It is officially known as Sveriges Riksbank Prize in Economic Sciences in Memory of Alfred Nobel.