The resident did not let Mr Holyoake into the apartment, but he stayed outside the building.
The resident later confronted Mr Holyoake, and asked what he wanted with the baby.
Mr Holyoake made a comment about the baby, and neighbours chased him down and held him until Burwood police could arrive.
He was spoken to by police but no charges were laid.
New South Wales police confirmed they responded to a welfare concern at the unit complex.
"A 34-year-old man was spoken to before NSW Ambulance attended and the man was taken to Royal Prince Alfred Hospital for assessment," a spokesperson said.
The news comes after local news outlet San Francisco Gate reported Mr Holyoake allegedly tried to grab a young boy from his mother in the middle of the day last week.
Witnesses said the mother began screaming for help but managed to free the boy from the 34-year-old's grasp.
A couple standing nearby with their own four-year-old son said they saw Mr Holyoake release the child, smile and run away.
During his court date Mr Holyoake's defence lawyer Stephen Olmo pointed out that his client had been to the US before to carry our charity work.
There has been no clear explanation about the charges laid.
"Overzealous police work, mistaken identity, wrong place, wrong time", Mr Olmo told TV news cameras in the US.
Mr Holyoake is expected to remain in the local county jail on a $US500,000 bond for at least two more days.
The 34-year-old is well known in Perth due to his former position on the Pride WA board of management and through his DJ appearances.
In response to his arrest, a statement from Pride WA uploaded to Out In Perth said his behaviour was "out of character".
"Roscoe was a general Pride WA committee member for two years. He did not stand for election at our latest [annual general meeting]," a spokesperson said.
"He has since been overseas pursuing his musical career.
"We appreciate that this is a very concerning time for the alleged victims and Roscoe's family. Right now our sympathy is with them during this difficult time.
"The current allegations and reported behaviour appear to be very out of character. However we simply do now know enough at this time to provide any further comment.
"This is a highly abnormal situation and Im sure you can respect that until further details emerge we are unable to provide any further comment until that time.
"While we recognise his affiliation with our organisation, this issue does not pertain to the function of PRIDE and the local LGBT+ community."
Mr Holyoake had also previously worked for RTRFM 92.1 as a presenter and was scheduled to be the closing act at the Joondalup Festival earlier this month.