Birdwatching

Initial Finding

When you try to login to a MySQL server with incorrect credentials, an error is returned together with an SQL state. As is defined in the developer documentation here, this section of the packet should start with a hashtag (#) and should then be followed by exactly five bytes describing the SQL state. However, an error in the implementation by OpenCanary put two hashtags there. Something you’d miss if you weren’t comparing error messages to each other.

Standard MySQL error:

OpenCanary error (the hashtag changes the syntax highlighting):

The Code Behind It

So the code powering this mock-mysql service is quite simple:

The error is essentially in the \x23#, as ASCII character 23 already is a hashtag. There was another error with the access denied state which was defined as 2800 while it should’ve been 28000. This is also the reason why the double hashtags didn’t mess with the format as code was one character too short making the effective length six again.

Turning It Into an Attack

A honeypot can be used for multiple purposes, it’s a device that acts as if it were a real server making it a potential target for hackers. A honeypot’s purpose however, is to be attacked. This has a couple of advantages:

1. You know there are hackers poking around your network leaving you able to act on them.
2. You can potentially figure out what exploits they’re using by examining the data sent to the honeypot.
3. They waste time on your honeypot instead of attacking real production servers in your network.

To put it simply: if your honeypot is attacked, it will be able to notify you and you can stop the attackers dead in their tracks. So it’s pretty essential that your honeypot isn’t found out. The error in the implementation shows us that we’re not dealing with a genuine MySQL server. We do need to interact by logging into the server however, which would make the honeypot notify the system administrators. So we’d know that we were dealing with a honeypot, but the owners of the network are alerted to you presence which is something attackers want to avoid.

So is there a way to look at this error message without triggering the alarm? As it turns out: yes! MySQL’s protocol requires packets to be numbered. When a connection is established the Mysql server will initiate the conversation by sending a packet with its banner and capabilities. This is packet zero:

The client will then send an answer back to the server with its own capabilities and the username it wants to login with. The packet number has to be one now:

Whenever the client sends a packet that doesn’t have the right packet number MySQL will return an error saying Got packets out of order with an SQL state of 08S01. But due to the double hashtag in OpenCanary, we can notice two things:

1. The double hashtags.
2. The error message starts with the last digit of the full error code

The code behind this one is the following:

This code is part of an if statement, it checks whether the received packet has the correct sequence id. If not, an out of order error will be returned. The problem is that the code for logging a login attempt is after that check. So when a packet is sent out of order, no logging will be done but an error message will be returned. So an attacker is able to scan a compromised network and detect canaries without raising suspicion.

Mitigation

I created a patch which consisted of 2 small changes to the MySQL module:

An extra zero was added to SQL_STATE_ACCESS_DENIED and the \x23 version of the hashtag was removed from the data byte string.

I reached out to the people at Thinkst and the patch was published within 2 days.

• The commit is here
• The security advisory is here

Conclusion

A way was found in OpenCanary running the MySQL module to detect that they were honeypots. This detection could be executed without raising an alarm and therefore bypassing the benefits of using a honeypot.

The team over at Thinkst responded quickly to my mail and we had the issue resolved in two days. It’s best to upgrade to OpenCanary 0.6.1 or if that’s not possible for you at this time, disable the MySQL module.

Reference: https://​nielsvangijzen​.com/​p​o​s​t​/​b​i​r​d​w​a​t​c​hing/