Server crashes when too much memory is allocated

my server crashes whenever one of my users is running a 'R' script (this script requires a large amount of memory). Below is the last top I saw:

top - 11:32:39 up 20 min,  4 users,  load average: 1.08, 0.85, 0.46
Tasks: 336 total,   2 running, 334 sleeping,   0 stopped,   0 zombie

Cpu(s):  6.1%us,  0.2%sy,  0.0%ni, 93.7%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:  65939968k total,  5131440k used, 60808528k free,    88256k buffers
Swap: 68124664k total,        0k used, 68124664k free,  1077612k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+ COMMAND
10392 cdina     25   0 3702m 3.5g 2428 R 100.0  5.6   7:51.82 R
10430 root      15   0 12872 1272  804 R  0.7  0.0   0:02.42 top
    1 root      15   0 10348  704  592 S  0.0  0.0   0:02.95 init
    2 root      RT  -5     0    0    0 S  0.0  0.0   0:00.00 migration/0

is there a way to prevent my server from crashing ("don't run that script" is not an option :-) ) ?
something like fixing a 'quota' for the memory allowed ?

(update)architecture:

uname -a
Linux myserver 2.6.18-194.3.1.el5 #1 SMP Sun May 2 04:17:42 EDT 2010 x86_64 x86_64 x86_64 GNU/Linux
#R was compiled on the server. just a simple configure/make/install
R --version

R version 2.11.0 (2010-04-22)
file /usr/local/lib64/R/bin/exec/R
R: ELF 64-bit LSB executable, AMD x86-64, version 1 (SYSV), for GNU/Linux 2.6.9, dynamically linked (uses shared libs), for GNU/Linux 2.6.9, not stripped

Answer

what about using /etc/security/limits.conf to limit the amount of cpu for the user.

Or with:

ulimit -m

in the shell, and then executing the process.