apache 2.4 - Adding HTTP Content-Type Header out of a file not working properly

I have a server running on Apache 2.4.
Clients can upload files, which are only stored compressed. For every file is also created a textfile containing the mimetype of the file.

When a client downloads a file I want the server to read the mimetype out of this textfile and set the HTTP Content-Type Header respectively.

What I have until now:
I made a python script, which gets the correct mimetype and writes it in stdout and added these directives to the relevant apache config file

RewriteEngine On     
RewriteMap mimetype prg:/home/user/scripts/mimetype.py
RewriteRule ^(.*)$ - [ENV=CONTENTTYPE:${mimetype:%{REQUEST_FILENAME}}]
Header append Content-Type "%{CONTENTTYPE}e" env=CONTENTTYPE

so the script gets called.

The problem: It only works after service apache2 reload.
But when a new file is uploaded the Content-Type header for the following downloads is empty (even for files, that worked before).

Update: Following the answer of Anson W Han the solution should be using a .htaccess file, because the config file gets only parsed once on service boot/reload.

Unfortunately RewriteMap is not allowed in .htaccess. I tried adding the

RewriteEngine On     

RewriteMap mimetype prg:/home/user/scripts/mimetype.py

directives to the sites config file and placed the .htaccess file containing

RewriteEngine On 
RewriteRule ^(.*)$ - [ENV=CONTENTTYPE:${mimetype:%{REQUEST_FILENAME}}]
Header append Content-Type "%{CONTENTTYPE}e" env=CONTENTTYPE

in the sites Document Root. But that doesn´t work at all. The Content-Type Header is now empty all the time.

htaccess overrides are allowed and http://www.htaccesscheck.com/ says the syntax is ok as does apachectl configtest.

Is there a way to get it working using a .htaccess file?