ssl - After setting up Elastic Load balancer my https doesn't work anymore. Nginx error

I have a regular instance that works fine when not behind a load balancer.
I set up an ELB with 80 forwarding to 80 and 443 forwarding to 443 and sticky sessions.
Afterward I receive this error when going to any https page.

The plain HTTP request was sent to HTTPS port

I handle the process of forcing https on certain pages in my nginx configuration.
What do I need to do to get this working? I'm putting in a barebones version of my nginx config below.

http {

  include               mime.types;
  default_type          application/octet-stream;


  # Directories
  client_body_temp_path tmp/client_body/  2 2;
  fastcgi_temp_path     tmp/fastcgi/;
  proxy_temp_path       tmp/proxy/;
  uwsgi_temp_path       tmp/uwsgi/;

  server {
    listen              443;
    ssl                 on;

    ssl_certificate     ssl.crt;
    ssl_certificate_key ssl.key;

    server_name www.shirtsby.me;
    if ($host ~* ^www\.(.*)) {
        set $host_without_www $1;
        rewrite ^/(.*) $scheme://$host_without_www/$1 permanent;
    }

    location ~ ^/(images|img|thumbs|js|css)/  {

          root /app/public;
    }
    if ($uri ~ ^/(images|img|thumbs|js|css)/) {
        set $ssltoggle 1;
    }
    if ($uri ~ "/nonsecure") {
        set $ssltoggle 1;
    }
    if ($ssltoggle != 1) {
        rewrite ^(.*)$ http://$server_name$1 permanent;

    }

    location / {
      uwsgi_pass  unix:/site/sock/uwsgi.sock;
      include     uwsgi_params;
    }

  }

  server {

    listen      80;
    server_name www.shirtsby.me;
    if ($host ~* ^www\.(.*)) {
        set $host_without_www $1;
        rewrite ^/(.*) $scheme://$host_without_www/$1 permanent;
    }

    if ($uri ~ "/secure") {
        set $ssltoggle 1;
    }

    if ($ssltoggle = 1) {
        rewrite ^(.*)$ https://$server_name$1 permanent;
    }

    location ~ ^/(images|img|thumbs|js|css)/  {
          root /app/public;
    }

    location / {
      uwsgi_pass  unix:/home/ubuntu/site/sock/uwsgi.sock;

      include     uwsgi_params;
    }
  }
}

Answer

Ended up getting the answer from vandemar in the nginx IRC Channel.
Seems pretty simple, but I struggled with figuring it out. The ELB was handling the SSL, I had already given it all the cert information. The problem was trying to handle it again on the individual instances or in the configuration file.
The solution is to just eliminate all the SSL stuff from the config file.
So removing these three lines fixed everything:

ssl                 on;
ssl_certificate     ssl.crt;
ssl_certificate_key ssl.key;