Stolen Generations survivors say they are deeply disappointed the Prime Minister chose not to attend an event to mark the 10th anniversary of the apology.
Elderly survivors, their families and political leaders gathered at the event in Canberra to mark one decade since former prime minister Kevin Rudd's apology to the Stolen Generations.
Malcolm Turnbull was invited to be give an address to survivors, but showed up only briefly for a photo opportunity.
Indigenous Affairs Minister Nigel Scullion instead spoke at the gathering of about 300 people.
"Very disappointed the Prime Minister couldn't be here to speak and spend time with Stolen Generations survivors," Florence Onus, chairwoman of the Stolen Generations Reference Group, said.
"It's not just to hear our voices, but to be here in respect to all the Stolen Generations survivors, who've travelled from right across the country. We have elders who are here from their 60s to their 80s.
"The goodwill that was happening 10 years ago was fantastic, but the momentum of the action from government has been too slow for our people."
Minister for Indigenous Affairs Ken Wyatt said he would have a discussion with the Prime Minister.
"I have had a number [of Stolen Generations members] express to me their disappointment, and I'll convey that to the Prime Minister," Mr Wyatt said.
PM pays tribute during Question Time
Despite missing the morning event, the Prime Minister paid tribute to the suffering of the Stolen Generations during a speech in Question Time.
"To all those Aboriginal and Torres Strait Islander people, who despite immeasurable pain survived. We are grateful that you accepted our apology with such generosity of spirit," Mr Turnbull said
"A national apology recognised that skin colour had been used to control the lives of Indigenous people and diminish their value in society.
"We have the chance to write a new chapter of history, where indigeneity is celebrated, not derided and culture and language is embraced."
During Question Time Mr Turnbull was asked if he would reconsider his rejection of a constitutionally enshrined Indigenous advisory body to the Parliament.
It was the only recommendation that came from the Uluru Convention last year, a meeting of hundreds of Indigenous leaders to discuss constitutional reform.
"I did not believe that was a good idea because it was inconsistent with a fundamental principle of our democracy is that all of our national representative institutions are open to every Australian," Mr Turnbull said.
"I thought the prospects of such an amendment to the Constitution being successful were zero. Absolutely zero. And so that was the advice that I gave and that is the view that I and the Government hold today."
Last week Mr Turnbull was criticised for leaving a Closing the Gap Campaign event before it finished.
Tuesday's breakfast event reflected on the historic apology 10 years ago, but many acknowledged much more needed to be done to improve the lives of Stolen Generations survivors.
Mr Rudd said the apology was only a small step on the path to reconciliation.
"The uncomfortable truth is that this is only one part of the answer and perhaps, in the larger part, a whitefella's answer," Mr Rudd said.
"The time has come for us to listen to the considered voice of our Indigenous peoples themselves, rather than our interpretation."
Mr Scullion acknowledged in his speech that services were not meeting the needs of Stolen Generations survivors.
He said they were waiting to see the outcome of an analysis from the Healing Foundation before committing to any new policies.
"There needs to be a comprehensive needs analysis, because there is no current information on how many Stolen Generations members are still alive, or their demographic data that enables us to ensure we deliver the right services," he said.