We developed a two-pronged approach to establish safety thresholds for active monitoring. First, we adapted concepts from setting margins in non-inferiority (NI) trials ("NI approach"). Second, we summarized NI margins used in published randomized trials and reviewed publicly available data from the US FDA's website to identify the type and magnitude of evidence used in regulatory decisions involving withdrawals and black box warnings between 2009 and 2013 ("benchmark approach"). We applied the framework to a case study of dabigatran versus warfarin and major bleed.