By Irwin Arieff UNITED NATIONS, Nov 21 (Reuters) - Lamenting that "stronger borders are not necessarily smarter ones," U.N. Secretary-General Kofi Annan urged governments on Friday to work together to harmonize their immigration policies. Some 175 million people, or 3 percent of the world's population, live outside the country where they were born, Annan said in a lecture at New York's Columbia University. The mobility this demonstrates, the $88 billion migrant workers send back to their home countries each year and the resulting cultural diversity should be cause for celebration. But the phenomenon has also raised concerns that migrants pose a terrorist threat, are trying to take away peoples' jobs or could deplete limited social services budgets, Annan said. "These are understandable concerns and they must be answered. The answers are not easy. But I have come here today to say that they do not lie in halting migration -- a policy that is bound to fail," Annan said in prepared remarks. "I say the answer must lie in managing migration -- rationally, creatively, compassionately and cooperatively." Annan's speech, billed as a major policy address, comes during a U.S. crackdown on immigration as part of its global war on terror and amid years of European efforts to exclude foreigners to protect domestic jobs. Other nations routinely tighten border controls to keep out refugees driven from their homes by war, hunger, poverty, natural disaster or human rights abuses. NO EASY LIFE Countries accepting immigrants often fail to accept them as equals or grant them legal status, even as they benefit from their labor, Annan said. Migrants "are usually not free riders looking for an easy life but courageous people who make great sacrifices in search of a better future for themselves or their families," he said. "Nor are their lives to be envied once they have left home. They often face as many risks and unknowns as they do hopes and opportunities. Many fall prey to smugglers and traffickers on their journey, and many more face a surly welcome of exploitation, discrimination and prejudice once they arrive." While "homeland security" was vital, it would be a tragedy for the United States "to deprive itself of the enrichment of many students and workers and family members from particular parts of the world, or if the human rights of those who would migrate here were compromised," he said. America was "a land where constant renewal and regeneration are essential elements of the national character. That character must never be lost," Annan said. While various U.N. and other international agencies deal with some aspects of immigration, no single global institution deals with all the issues raised by "this quintessentially global phenomenon," he said. For this reason, he endorsed a recent initiative by 12 countries to form a Global Commission on International Migration to study how to improve cooperation. The commission is co-chaired by Sweden and South Africa, and its other members are Australia, Brazil, Canada, Egypt, Italy, Mexico, the Philippines and Switzerland, U.N. officials said.