access control list - Apache virtual host allow NOT overriding server allow

I am running Apache 2.4 and according to the documentation I found on the apache website (http://httpd.apache.org/docs/current/sections.html), under the section "How the sections are merged" it says that the virtual host configurations should override the server configurations.

I have a filesmatch server config regex to prevent public access to important wordpress files using "order deny, allow" logic:

   Include conf/global_acl.conf

Then under my virtual host I have another filesmatch to allow specific access to certain files to enable a wordpress plugin:

   Order Deny,Allow

   Allow from wordpress.com
   Allow from 123.123.123.123

My goal is to have the server config allow process, be inherited by the virtual host, then have the additional allow directives parsed by the virtual host. It does not seem to be working. If I modify the server level by removing the xmlrpc file, the virtual host will work.

EDIT

To clear up confusion, my issue is that under the virtual host, I am being denied even though I am using IP 123.123.123.123. I am expecting the whitelist entries from the server config to be inherited by the virtual host. Then having additional whitelisted allows in the virtualhost config. I have a DENY FROM ALL at the top of my server config.

Any ideas what is going on?
Thanks in advance!