In the access.log you will notice that it says SNC this means that the server is preventing the file from being cached.
In your access_log_format can you please add %P so we can see which policies are currently being matched so we can see if we can come up with a solution.