domain name system - How to create a high-availability application server?

I am working with some hardware units which uses mobile Internet to communicate to my application server.

Initially, the hardware had a hard-coded IP address (load balancer) in the hardware for communication which was used to route the traffic to my application server; now this creates a single point of failure if the load balancer goes down the whole redundant infrastructure behind it becomes useless since its objective is to share the load, not to maintain the availability.

Now I asked the hardware creator to switch it to using DNS, and by using multiple A records I got better results. In this case I can have applications servers directly working with the hardware; but if one server goes down some hardware units fail to communicate for a while, but it goes to the other server in 20-30 minutes; since DNS failover doesn't work always.

So how can I overcome this problem and keep my application server working somehow all the time?

Answer

As your question is quite generic and doesn't give any details about the involved software, my answer is short as well: Just use a set of two loadbalancers, which share a 'virtual IP' between them via the use of e.g. vrrp (the "Virtual Router Redundancy Protocol") or pacemaker (there are more options available as well). Use two machines for that, which are either bare metal or running from two different hosts. If one of the machine goes down, the other will take over the IP. Point your record to this IP (and remember: DNS isn't made to achieve high availability.)

Edit 1: As @Ondra Sniper Flidr mentioned in the comments: Another solution is keepalived.

Edit 2: You could also have a look at CARP, the "Common Address Redundancy Protocol". A software solution using this protocol would be UCARP for example.