Team members may still be doing their "day jobs," with the same responsibilities, workload, and deadlines as before. This can lead to prioritization issues.
People might be reluctant participants, and may not be happy to take on the additional work and effort that being part of a cross-functional team often requires. (This may be true for you, too!)
It's more difficult to set priorities, make decisions, motivate people, and manage performance when you don't have direct authority over members of the team.
Team members may be required to use a different set of skills in a new environment. For example, a programmer who normally works alone may now be required to work with others.