Assume for the moment that we accept the legitimacy of the Constitution (although we shouldnt).
Assume, also, that the Constitutions census clause not only authorises the federal government to conduct a census but also authorises it to compel people to answer it (though, again we shouldnt accept that either).
Even so, as Rothbard reminds us, the freedom to speak includes the freedom not to speak. And since freedom of speech is guaranteed in an amendment to the Constitution, that provision automatically trumps anything in the body of the Constitution thats inconsistent with it. So, FWIW, enforcement of the census is doubly unconstitutional.