The stress would be on the local computer and bandwidth, as you'd be syncing from BCM to Outlook, then Outlook would sync up to the Exchange server. After the initial sync, only changes would sync and this won't take "much" bandwidth unless you make a lot of changes to the contacts.
Syncing would increase each user's mailbox size (by an estimated 200MB for 20,000 average sized contacts), as each user would have a copy of all 20,000 BCM contacts. Unless the server quota is low, this would not be a problem; users waste a lot more space keeping junk mail. However, anyone who uses a smartphone and enables the contact folder for syncing, would have all 20,000 synced to the phone.
On option for sites that have SharePoint, is to link a SharePoint Contacts list to Outlook and set BCM to sync to the SharePoint list. Because only one user would need to sync the BCM Contacts, it would eliminate the need for each BCM user to sync all BCM contacts to their mailbox. The users can access the contacts through SharePoint.
Create a SharePoint Contact app and Connect it to Outlook.