Receiving QSOs over the network from N1MM

We previously outlined a method for using rotor control messages issued from N1MM, a Windows program, to forward control messages to rotctld, a Linux program for controlling rotors. The communication protocol is meant for allowing a second computer running N1MM to control a rotor which is connected to a first computer running N1MM, but we were able to exploit the messages to do something else.

N1MM also has other properties that can be sent to arbitrary addresses, like contacts. Every time a QSO is logged, N1MM can be made to send a UDP packet to an arbitrary address, which contains all relevant information otherwise saved to the log.

On our IRC channel, we’ve previously made our IRC bot post the last QSO in the log using the command !lastqso, and post some statistics when issued !qsostats. Example:

19:37:36 < asgeirbj> !lastqso
19:37:36 -la1kbot:#ark- 2018-09-29 17:36:04: CR3W @ 
14079.69 RTTY SNT:599 RCVD:599 (LA3WUA: None) 
(CQWWRTTY)

We first achieved this by exposing the folder containing all N1MM log files to the network, and mounting it as a samba share on the computer running the bot. We were then accessing all the files directly, and made them susceptible to file corruption and database mishandling, and there also were problems whenever the network was down. In the next iteration, we mounted the folder and also copied down the files to a local folder before accessing them. In order to access not only our daily log but also any contest logs, we had to add all logs to the same database in order to access them as a single file. Whether this was a good solution or not depended on the current traffic manager. Whenever we changed the name of the log database (typically once a year), we had to correspondingly change the name in the other end. And whenever we had to set up the computer running N1MM again, and it took some time before we had the network share ready again. And as mentioned: we were exposing all logs to the network.

All in all, this was not a very good solution, and we almost immediately ported it over to a solution based on receiving N1MM UDP packets once we were made aware of this possibility. This was first done by just receiving the XML packet and outputting it to file, which the IRC command read directly and posted to IRC, until we recently made a small prototype for pushing the log further on to a SQLite database to calculate more advanced statistics.

This might sound like a bit of a detour, since N1MM uses an SQLite database as its main log database file format. N1MM therefore writes a contact to an SQLite database, before it sends it to another computer over UDP, which also writes the contact to an SQLite database: but we avoid the need for the Windows computer to expose all log files to the network, and we don’t have to make the receiver end be aware of file paths on the Windows end. This is therefore a more elegant solution, as there are less complexities and dependencies and things to tune – we just need to set up a socket on one end, and ensure that the N1MM end pushes contacts to the correct IP address. The solution also makes us able to independently develop other kinds of triggers for logged QSOs. It has been surprisingly stable and apparently robust so far, as we’ve also experienced the rotor control setup to be.

We can imagine that instead of a local SQLite database, we could push it to a centralized postgresql or mysql database. A couple of nice applications can be made possible using this approach:

• Independent backup of QSOs (kind of, we could miss QSOs. Trustworthy backup should still be done by copying the log files to a remote location).
• The possibility for having an OS-agnostic “master log” and combine contacts from multiple applications, or even slowly create a web-based QSO logger that can be interfaced with/from N1MM.
• Live view of statistics, independent from N1MM.
• Live view over QSOs as they are run, along with placement of markers on a world map for each contact, to be displayed on a screen placed at ARK to show QSO activity in real time (can also be achieved by direct reception of UDP packets from N1MM without a log middle-man)
• Automatically uploading QSOs to logbook of the world, clublog, etc, for live update over finished DXCC’s.
• A nice database available for playing around with old logs, and for quick development of statistical methods or plotting tools for such.

A lot of nice stuff is possible once the contacts are made available on a server or in a database independent from N1MM. We’ll come back to some of these later (when we have implemented them). We’ve only pushed the contacts to an SQLite database so far, a local file on the computer running our IRC bot, but once we make this available on a centralized database other possibilities open up.

Creating a socket for receiving UDP packets from N1MM is easy enough:

n1mm = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
n1mm.bind(('', 12060))

On the N1MM end, N1MM should be configured at menus Config -> Config Ports (…) -> Broadcast Data to push “Contact” to [IP of the host that should receive packets]:12060. Receiving packets is then a matter of

while (True):
    msg = n1mm.recv(buffer_size)

The received message is an XML message, and is easily parsed using

    contact = xmltodict.parse(msg.decode('utf-8'))

, and then we can do the stuff we want to do with it. It is specified in the N1MM manual what the XML message looks like. The structure is

<message type>
(... message contents)
</message type>

, where message type is either contactinfo, contactreplace or contactdelete, depending on whether it is a new contact, a contact in the log was edited or a contact was deleted. Assuming a contactinfo message, we could print the timestamp, operator, call and frequency of the contact using e.g.

contact = contact['contactinfo']
print(contact['timestamp'], contact['operator'], contact['call'], contact['frequency'])

Picking our fields of interest, we can create an SQLite database based on the N1MM fields by

connection = sqlite3.connect('qso_database.sqlite')
c = connection.cursor()
c.execute('''CREATE TABLE qsos (contestname VARCHAR(10), timestamp DATETIME, band FLOAT, rxfreq DOUBLE, txfreq DOUBLE, countryprefix VARCHAR(8),
operator VARCHAR(20), mode VARCHAR(6), call VARCHAR(20), snt VARCHAR(20), rcv VARCHAR(20), comment VARCHAR(60), continent VARCHAR(2))''')
connection.commit()

(The data types of the fields have been copied from the schema in the N1MM sqlite database, and haven’t been willfully chosen in any meaningful way. We’ll design a better setup in a future post. 8) )

Inserting QSOs into this database is done by

c = self.connection.cursor()
c.execute('''INSERT INTO qsos VALUES (:contestname, :timestamp, :band, :rxfreq, :txfreq, :countryprefix, :operator, :mode, :call, :snt, :rcv, :comment, :continent)''', contact)

We can for example fetch the latest QSO by

c.execute('SELECT timestamp, call, rxfreq, mode, snt, rcv, operator, comment, contestname FROM qsos ORDER BY timestamp DESC LIMIT 1')
last_qso = c.fetchone()

This is mostly it, really. We’ve wrapped this in a class and added some specific functions for calculating various statistics and (all-time best over a two-month period) QSO rates:

12:46:27 < oyvkatt> !qsorate
12:46:27 -la1kbot:#ark- 33 QSOer siste timen. (Best rate siste to måneder: 44 QSOer per time, 2018-09-29 13:49:43, CQWWRTTY. Operatører: LA3WUA, LB2AI)

(The best rate here was at September 29 because we only started recording the QSOs over the network at September 29 and this post was written a couple of days later.)

We’ll come back to this database and related interface once it has become more mature and we have separated it from our IRC-bot to stand on its own feet.