UK Financial Investments (UKFI), which manages the state's bank holdings, said in a statement that it will seek to sell 4.28 billion shares to institutional investors.
The sale will see the Conservative-Liberal Democrat coalition government's LBG holding drop from 38.7 percent to about 32.7 percent.
The government is keen to recoup £20 billion ($32 billion, 24 billion euros) of taxpayers' cash that was ploughed into the group at the height of the global financial crisis.
"UKFI announces its intention to sell part of HM Treasury's shareholding in Lloyds Banking Group plc," it said in a statement issued after the closure of the London stock market.
"The disposal of these shares will be by way of a placing to institutional investors."
Bank of America-Merrill Lynch, J.P. Morgan Cazenove and UBS Investment Bank have been appointed to act as joint bookrunners.
Lloyds shares finished 0.98 percent higher at 77.36 pence on Monday. That was higher than the 63.1-pence which the government says it needs to break even.
Based on today's closing share price, the sale would recoup £3.31 billion for the taxpayer.
Antonio Horta-Osorio, the bank's chief executive, said he was "pleased" that the government had begun the process of selling its stake.
"I believe this reflects the hard work undertaken over the last two years to make Lloyds a safe and profitable bank that is focused on supporting the UK economy," he added.
Lloyds Banking Group was created by a merger of Lloyds TSB and rival British lender HBOS in the wake of the 2008 global financial crisis.
However, HBOS was saddled with toxic or high-risk property investments, and LBG subsequently received a vast state bailout under the then-Labour administration.
British finance minister George Osborne, a key Conservative member of the current coalition government, has authorised the sale.
"UK Financial Investments today advised the Chancellor it would be appropriate to begin the process to sell part of the government's shareholding in the Lloyds Banking group," said a Treasury spokesperson on Monday.
"The Chancellor agrees with that advice and has authorised the process to begin."
The spokesperson added: "We want to get the best value for the taxpayer, maximise support for the economy and restore them to private ownership.
"The government will only conclude a sale if these objectives are met."
Chancellor of the Exchequer Osborne had stated earlier this year that he was considering share sale options as the government seeks to return LBG to the private sector along with Royal Bank of Scotland (RBS).
UKFI was created in 2008 to manage the state investments in RBS, LBG, Northern Rock, and Bradford and Bingley, after the state stepped in to prop them up.
The government still holds 81 percent of Edinburgh-based RBS.
Analysts welcomed Monday's news and noted that the market was awash this month with speculation over a possible Lloyds shares sale.
"The sale sets in motion the first step of the re-privatisation of banks rescued during the financial crisis," said City Index analyst Joshua Raymond.
"This is yet another important milestone on the road to recovery for the financial sector and Britain's economic recovery."
Back in August, Lloyds revealed that it had returned to first-half net profit and was looking at resuming shareholder dividend payments.
Lloyds, under the stewardship of Horta-Osorio, has been boosted since the start of the year by rising income, deep cost-cutting, falling bad loans and recovering economic growth.
LBG posted profit after tax of £1.56 billion in the six months to the end of June, which compared with a net loss of £697 million for the first half of last year.
Earlier on Monday, meanwhile, UKFI had announced the appointment of James Leigh Pemberton as its new boss from October.
Leigh Pemberton, who is currently the UK head of Credit Suisse, had worked as an advisor on the state recapitalisation of Lloyds and RBS in 2008.
