This can occur if the certificate format you are importing is incorrect. The certificate supplied by the Certificate Authorities is normally in a PEM format. Glassfish requires DER format to import.
Instructions on how to generate a CSR code using Java Keytool.
Instructions on how to generate a CSR code with Java's Keytool in Tomcat server.
Instructions on how to install an SSL certificate in Java.