David Cameron has promised that if the Conservatives win the next election, they will renegotiate the terms of Britain’s EU membership and then hold an in-out referendum before the end of 2017. In his long-delayed speech on Britain and the EU, he pledged to campaign for a Yes vote "with all my heart and soul". The speech contained much that is sensible. But its implicit message to Britain’s partners was: "Give us what we want, by the deadline that we specify, or we may well leave the EU." Many other Europeans consider that not far short of blackmail. Given that Cameron cannot get what he wants without the co-operation of the other member-states, the strategy he has adopted is risky. The speech made many optimistic assumptions and was riddled with ambiguities.

Cameron had to promise a referendum in order to maintain control of his own party. Had he failed to do so, the Conservatives' most eurosceptic backbenchers – who want a referendum to propel Britain out of the EU – would have become even more truculent and rebellious than they are already. And some Conservatives who want to stay in the EU believe that only a referendum pledge can undermine the surge of support for the United Kingdom Independence Party, which threatens to deprive the Tories of many seats at the next general election.

But although the necessities of party management underlay what Cameron said, the speech was much more thoughtful than many pro-Europeans expected. Cameron was right to say that much in the EU needs to change; that the Union should accept the principle that powers can flow not only from member-states to institutions but also the other way, too; and that national parliaments should become more closely involved in EU decision-making. Cameron stated very clearly the economic and strategic benefits of EU membership for the UK. And he hit on the head the silly argument made by some eurosceptics that the Norwegian and Swiss models of association – whereby they have access to parts of the single market, without being able to vote on its rules – could be suitable for Britain.

Cameron did not, at least overtly, ask for UK ‘opt outs’ from specific areas of EU policy-making; many of his party's eurosceptics argue that such opt outs should be the price of Britain staying in the EU. The thrust of the speech was to ask for reforms that benefited all member-states, rather than only the British. This was a wise approach, since Britain would be unlikely to find many allies if it demanded changes merely for itself.

Cameron said very little about what his demands would be. One understands why: he did not want to annoy either Conservative europhobes, who want him to seek the 'repatriation' of EU powers, or the other member-states, which do not want to let Britain cherry-pick the policy areas it takes part in. The government’s review of EU competences, due for completion in autumn 2014, will analyse the beneficial and harmful effects of EU laws and actions on the UK. This review will feed into the specific demands that the Conservatives eventually make.

At some point Cameron will have to resolve the ambiguity over whether he merely wants to reform the EU, or engineer a significant repatriation of powers from it. If his ambitions are modest and he goes for the first option, he might succeed in obtaining a 'new deal for Britain'. With some deft British diplomacy, for example, the other governments could conceivably agree to reform the working time directive (singled out for criticism in the speech), deepen the single market in areas like services and the digital economy, give some protection to the special status of the City of London, enhance the role of national parliaments and put into the treaties a new procedure for allowing powers to return to member-states.

Such a modest settlement would not satisfy hard-line Conservative eurosceptics, many of whom would split their party by campaigning for withdrawal in the referendum campaign. But with luck, a Conservative-led government – backed by Labour and the Liberal Democrats – could persuade the British people to vote to stay in the EU. (In the past six months, most opinion polls have shown a majority of voters wanting to leave the EU, though some very recent polls suggest growing support for staying in.)

But Cameron may well – as many Conservatives assume – attempt a more ambitious renegotiation, in order to feed his back-benchers the 'red meat' they crave. He could, for example, ask for Britain to be exempted from EU labour market law; from rules on the free movement of labour; from the Common Fisheries Policy; and from all police and judicial co-operation (some of which Britain already has the right to opt out of). Leading Conservatives have already called for these and other opt outs. Boris Johnson, the mayor of London and a popular figure in the party, has said that Britain should stay in not much more than the single market.

However, any treaty change requires unanimity and Britain's partners have no intention of granting Britain-specific opt outs. They fear that Cameron would be opening Pandora's box: if Britain could spurn the bits of the EU that it disliked, others would demand the same privilege. The French hate EU rules that limit how much they can subsidise their car industry, while the Poles, with their coal-centred economy, find EU rules on carbon emissions irksome, and so on. Once you allow countries to treat the EU as an à la carte menu, the single market starts to unravel.

But might Britain's partners make an exception for labour market rules? After all, the then prime minister, John Major, managed to opt out of the Maastricht treaty's 'social chapter' in 1991 – which Tony Blair later opted back into. And some Central Europeans share the Conservatives' disdain for EU involvement in social policy. But the French and several other governments would never agree to reviving that British opt out: they regard labour market rules as intrinsic to the single market and think that allowing Britain an exemption would give it an unfair advantage in seeking to attract foreign investment. They want more social Europe, not less.

In his speech, Cameron said it was likely that eurozone governments would want a major new treaty – for the purposes of strengthening the euro – in the next few years. He argued that this would give the UK leverage: if the other member-states needed a signature on the treaty from Britain, it could demand concessions in return. Cameron is right that if a large treaty was on the agenda, covering many things that the Union does, the others could not easily sidestep a British veto by drafting a new treaty outside the framework of the EU.

A few months ago, Cameron's assumption seemed plausible. The Germans, alongside the European Commission and the European Parliament, were calling for negotiations on a 'political union' to start soon after the 2014 European elections. Most EU governments did not want a significant new treaty, partly for fear of the difficulties of ratification. But many assumed that what the Germans want, they get.

The mood has changed in recent months, notably in Berlin. Perhaps the Germans have listened to their partners’ opposition to the idea of a major treaty change – and especially to that of the French, who worry about having to ratify a new treaty by a referendum. Perhaps the Germans are recoiling from the prospect of having to commit to a 'political union' that could entail financial transfers to poorer members of the eurozone and the loss of sovereignty. And perhaps they are reluctant to give the British the possibility of blackmailing the rest of the EU into making concessions. Furthermore, in Berlin and in other eurozone capitals, governments now believe that the euro crisis has been at least partially sorted out. They therefore see little urgency in pursing the radical solutions that were on the agenda only a few months ago.

Whatever the reasons, most EU governments now see no need for a big treaty revision, of the sort that would entail a Convention on the Future of Europe and a grand inter-governmental conference, for many years to come. And if the health of the euro required an institutional fix – such as the establishment of an EU-wide deposit insurance scheme – they think a small and speedy treaty change, like those which set up the European Stability Mechanism and the fiscal compact, would suffice. Britain lacks the power to stop that sort of treaty: when Cameron tried to veto the fiscal compact in December 2011, 25 governments went ahead with a non-EU treaty to establish new rules on national budgets.

Cameron quoted the Commission president, José Manuel Barroso, to justify his assertion that a major new treaty was likely. But Barroso – though a believer in a more federal future – does not set the EU’s agenda. Evidently, moods can shift, and a major new crisis in the eurozone could revive talk of a new treaty. But in such circumstances the most likely response would be a mini-treaty that can be ratified quickly.

Cameron recognised the possibility that Britain's partners might not negotiate a new treaty in a timetable that suited Britain. In that case, he said, a Conservative government would seek a unilateral renegotiation of the EU treaties. But that would be difficult since its leverage would be limited and treaty change requires unanimity.

Many Conservatives make another assumption that may turn out to be false. They believe that the Germans do not want to be left 'alone' with the statist, protectionist French, that they want the British to be on the inside fighting for economically liberal policies – and that they will therefore do whatever it takes to keep the UK in the EU, including by offering exemptions from EU policies. The first two parts of the previous sentence may be true, but not the conclusion. Notwithstanding Angela Merkel's polite response to the speech, the officials around her – like most senior German politicians – say that although they hope the British will remain in the EU, they are not prepared to pay the price of opt outs. The Conservatives have form when it comes to misreading German intentions: in December 2011, at the summit which gave birth to the fiscal compact, Cameron wrongly thought that Merkel would support his demands for treaty changes to protect the City.

If Cameron did ask for opt outs, as many in his party hope, but failed to secure them, how would he handle a referendum? It is hard to imagine him campaigning for a No, given what he has just said about the virtues of EU membership. But he might also find it hard to campaign for a Yes, if he had failed to change fundamentally the terms of Britain’s membership.

All of the above assumes that Cameron will lead a Conservative government in the next Parliament. But recent opinion polls suggest that, although the next general election is still more than two years away, a Labour government is more likely. Labour leaders remain reluctant to promise a referendum. Like the Liberal Democrats – who in the past have supported an in-out referendum – Labour believes that with the EU in flux, this is not the right time to talk about referendums, and that to do so creates uncertainty and could deter foreign direct investment.

Labour may also worry about its ability to win a referendum on staying in the EU. Unlike the Conservative Party, Labour does not want to renegotiate the terms of Britain’s membership. So although a Labour government would seek to reform the EU, it could not claim during a referendum campaign that it had transformed Britain's relationship with Brussels. Moreover, the Conservatives in opposition would be likely to install a more eurosceptic leader and campaign for a No vote. Yet despite Labour's current opposition to an EU referendum, party leaders have not ruled one out. If the Conservatives appear to profit from their referendum promise, Labour may have to offer a similar pledge.

Even if Labour wins the next election and continues to oppose a referendum on EU membership, at some point in the future there will be another Tory government. That government would almost certainly hold an EU referendum. Therefore those who value Britain's membership should treat the Cameron speech as a wake-up call to come up with a convincing agenda for reforming the EU and explain to the British people why they are better off in.

Charles Grant is director of the CER.