FTPeek, SFTP, Invalid Key Format

I ran into an interesting problem. A customer of a customer was using FileMaker with the FTPeek plugin to access an SFTP service hosted by our customer. Let's call our customer Foo Inc, and their customer Bar Ltd.

The Setup

Foo Inc had setup a CrushFTP server to accept data from their customers. They want to receive the data securely so they use SFTP. People from Foo Inc can connect using the typical tools like Filezilla and WinSCP. The developer from Bar Ltd was unfortunately having a little more difficulty. He could connect with Filezilla and WinSCP, but not with FTPeek within FileMaker.

Being a diligent developer, he setup his own CrushFTP service on his own server, mimicking as much as he could the setup used by Foo Inc. He had CrushFTP involved, and as I understand it, they setup a whole new set of keys using PuTTYGen. I initially thought this meant just a set of private/public keys for his FileMaker machine, but I suspect he also setup new private/public keys for CrushFTP's SFTP service as well. This is important, we'll come back to it.

The short of it is that he could get FileMaker to connect via FTPeek to his CrushFTP server. So the issue must lay on Foo Inc's side. Makes sense, except for Filezilla and WinSCP which had no problems connecting to either CrushFTP environment.

I wanted to focus on FileMaker and FTPeek, but doing so would involve tying up the developer from Bar Ltd. So we did a remote session, I got to see how the code in FileMaker worked, and off I went.

The Troubleshooting

I'm going to skip over a lot of troubleshooting steps that didn't end up giving us a whole lot of information.

In the end, what sealed the deal was setting up my own FileMaker environment with FTPeek, and attempting to recreate the errors, which I could do. What I ended up doing is skipping over the FTPeek_GetPublicKey function. That function stated that it was unable to determine whether or not there was an SSH service running on the specified IP address or hostname at the specified port.

FTPeek_ConnectSFTP, on the other hand, would throw an error immediately stating that the format of the key was invalid. I've seen this kind of rejection with SFTP and SSH applications on Android devices. The function isn't going out, getting the public key and saying that it doesn't make sense. It's saying that the public key I provided in order to check against the host public key doesn't make sense. Does that make sense?

For grins, I generated new keys. Since I'm a huge Linux nerd, I just generated them on my laptop using ssh-keygen. This generates keys in the OpenSSH format rather than the SSH2 format. I put the newly generated public key in the ConnectSFTP function, and sure 'nuff, the error moved on to an immediate close of communication. The public key I just generated didn't match the public key on the server, so that makes perfect sense.

If the developer generated new keys for his CrushFTP server using PuttyGen, then it would make sense that the functions work properly against his server as well, because PuttyGen also uses the OpenSSH format.

The Answer

FTPeek's SFTP functions don't support the SSH2 format. Generate new keys for your server, as well as your FTPeek client, make sure that they're OpenSSH format, get them in place and call it a day.