You contact trace by finding those who are sick then testing everyone they came into contact with (as many as you can), thereby theoretically catching the asymptomatic carriers.
This is not that much different than a scenario that plays out every day across the country (thankfully in much less common numbers now): Someone tests positive for HIV. They provide a list of their sexual history/contacts. The local health department calls or sends a letter saying someone you were recently intimate with has tested positive, please proceed to the nearest testing place. These people aren't going to get tested because they have full blown AIDS, or any symptoms at all, its because they are told they could have been exposed. The COVID-19 version of that would involve thousands of people and mobile testing sites, rather than a few overworked people in a county health department.