When using vscode remote-ssh:
1- delete ssh config folder both in C:\Program Data\ssh and C:\\.ssh
2- In VS Code, press F1, choose Remote-SSH: Connect to Host...
3- Do NOT enter anything in the prompt, but instead choose + Add New SSH Host.. 
4- Enter the full ssh command, including the key (in case of Windows, you may want to enclose the path with double quote mark) ssh -i "C:\path\to\key" user@host. (you need to make sure the key has a limited permission. Remove all inherited permissions, and only give a full control to the owner.)
5- You will be asked to choose a folder in which a new config file will be created. Choose any of the two options.
6- There will a prompt notifying that the new config file has been created. Click connect