For anyone who is faceing this issue and the answer provided (local/<username>) thing didn't help: it's not your fault. It's the lousy EMC bootstrap script that fails to create a self-signed certificate (op mentioned the cert location, you can check it there but UnityVSA won't let you touch it without root permissions, which we don't have).
So, what did the trick in my case:
1. Create a new, VALID PEM formated certificate. I used a RSA 2048 key
Note: the "generated" self-signed cert of the UnityVSA was outdated (2015) and so vCenter wouldn't accept it EVEN if you tell vCenter to trust the cert. Which is just what should happen if a cert is outdated.
2. Enable SSH acces on UnityVSA: from the HTML UI go to service --> service tasks and --> enable ssh
3. Login to UnityVSA using the account service (you set the password for it at install time (default: same as the admin password)
4. Upload your PEM formated certificate named something like mycert.crt (crt extention is important) to UnityVSA. Make sure to contain the certificate chain in the certificate if it's not self-signed.
5. Upload your PEM formated private key for the certificate name something like mycert.pk (pk extention is important) to UnityVSA
6. On the UnityVSA run the command svc_custom_cert mycert
This will update the UnityVSA certificate and restart the UnityVSA webserver. The Webcertificate is also used for the VASA provider (since the initial VASA connection in UnityVSA is just a connection to http:// on port 8443, which is using the same cert as on port 443).
After I updated the certificate I was able to register the UnifyVSA VASA provider. Note that this was the latest version of UnityVSA as of today (4.2.0.9392909) on vCenter 6.5U1 and with ESXi 6.5U1.