Decoding darkstat’s export format
While pondering why darkstat all of a sudden shows corrupted timestamps when running on FreeBSD/amd64 stable/9, I wrote a small program to decode the export format. The program is available using
svn co svn://svn.ximalas.info/darkstattype
or http://svnweb.ximalas.info/darkstattype/. The license is the 2-clause BSD license.
Everything is shown correct in the web interface when darkstat starts without reading any export file. As soon as darkstat has written its export file, the computer has been rebooted, and darkstat later reads that file back into memory, the corruption is very visible in the timestamps shown in the web interface:
(clock error: now = 1181, last = 18446744073707737811)
It’s interesting to take a closer look at the “last” value. 18446744073707737811 in decimal, is 0xFFFFFFFFFFE452D3 in hexadecimal. Could this strange value be the result of a negative value, expressed as a 64-bit signed integer, and finally interpreted as a 64-bit unsigned integer? Interpreted as a 64-bit signed integer yields -1813805. Further interpretation as seconds yields a time difference of close to -21 days.
Inspection of the darkstat source code reveals that darkstat has deviated from the format specified in the export-format.txt file. darkstat writes the lastseen timestamp ahead of the MAC address. Luckily, darkstat reads the file the same way, so its handling of the export file is consistent at least. I have made the necessary changes to my own program.
Here’s a censored sample of my program’s output:
File header 0xda314159
Section header host_db v1 0xda485301
Host count 2
Host #1 of 2:
Host header v3 0x48535403
IPv4 address family (0x04)
IPv4 address 157.56.93.40
Last seen 0x532875d1 = 1395160529 = 2014-03-18T16:35:29+0000
MAC address 00:17:e0:77:14:45
Hostname length 34
Hostname msnbot-157-56-93-40.search.msn.com
Bytes in 552
Bytes out 476
IP Proto count 1
Protocol #1 of 1:
Protocol 0x06
In 552
Out 476
TCP proto count 1
Port 29300:
SYN 0
In 552
Out 476
UDP proto count 0
Host #2 of 2:
Host header v3 0x48535403
IPv6 address family (0x06)
IPv6 address fe80:0000:0000:0000:0217:e0ff:fe77:1445
Last seen 0x532aaba4 = 1395305380 = 2014-03-20T08:49:40+0000
MAC address 00:17:e0:77:14:45
Hostname length 12
Hostname (link-local)
Bytes in 501120
Bytes out 650984
IP Proto count 0
TCP proto count 0
UDP proto count 0
Section header graph_db v1 0xda475201
Last time 0x532875e9 = 1395160553 = 2014-03-18T16:35:53+0000
Graph #1 of 4:
Number of bars 60
Index of last bar 53
Bar #1 of 53:
In 0
Out 0
Bar #2 of 53:
In 0
Out 0
Bar #3 of 53:
In 0
Out 412316860416
Bar #4 of 53:
In 687194767360
Out 309237645312
Bar #5 of 53:
In 0
Out 0
Bar #6 of 53:
In 618475290624
Out 592705486848
Bar #7 of 53:
In 0
Out 0
Bar #8 of 53:
In 326417514496
Out 171798691840
Bar #9 of 53:
In 1546188226560
Out 927712935936
Bar #10 of 53:
In 0
Out 0
Bar #11 of 53:
In 0
Out 0
Bar #12 of 53:
In 0
Out 0
Bar #13 of 53:
In 0
Out 0
Bar #14 of 53:
In 0
Out 0
Bar #15 of 53:
In 0
Out 0
Bar #16 of 53:
In 0
Out 0
Bar #17 of 53:
In 2284922601472
Out 223338299392
Bar #18 of 53:
In 7314329305088
Out 13602161426432
Bar #19 of 53:
In 0
Out 0
Bar #20 of 53:
In 554050781184
Out 485331304448
Bar #21 of 53:
In 0
Out 0
Bar #22 of 53:
In 0
Out 0
Bar #23 of 53:
In 412316860416
Out 412316860416
Bar #24 of 53:
In 0
Out 0
Bar #25 of 53:
In 0
Out 0
Bar #26 of 53:
In 0
Out 0
Bar #27 of 53:
In 309237645312
Out 274877906944
Bar #28 of 53:
In 807453851648
Out 687194767360
Bar #29 of 53:
In 3848290697216
Out 2602750181376
Bar #30 of 53:
In 2456721293312
Out 223338299392
Bar #31 of 53:
In 6764573491200
Out 7490422964224
Bar #32 of 53:
In 687194767360
Out 6158983102464
Bar #33 of 53:
In 2272037699584
Out 223338299392
Bar #34 of 53:
In 5570572582912
Out 7425998454784
Bar #35 of 53:
In 343597383680
Out 5987184410624
Bar #36 of 53:
In 0
Out 0
Bar #37 of 53:
In 0
Out 0
Bar #38 of 53:
In 326417514496
Out 171798691840
Bar #39 of 53:
In 2284922601472
Out 2284922601472
Bar #40 of 53:
In 1649267441664
Out 1305670057984
Bar #41 of 53:
In 498216206336
Out 326417514496
Bar #42 of 53:
In 996432412672
Out 790273982464
Bar #43 of 53:
In 824633720832
Out 790273982464
Bar #44 of 53:
In 1580547964928
Out 6786048327680
Bar #45 of 53:
In 171798691840
Out 0
Bar #46 of 53:
In 0
Out 0
Bar #47 of 53:
In 0
Out 0
Bar #48 of 53:
In 0
Out 0
Bar #49 of 53:
In 910533066752
Out 755914244096
Bar #50 of 53:
In 652835028992
Out 343597383680
Bar #51 of 53:
In 0
Out 0
Bar #52 of 53:
In 0
Out 0
Bar #53 of 53:
In 1047972020224
Out 2095944040448
Graph #2 of 4:
Number of bars 0
Index of last bar 0
Graph #3 of 4:
Number of bars 0
Index of last bar 0
Graph #4 of 4:
Number of bars 0
Index of last bar 0
The file format and the contents turns out to be sound after all. How come darkstat misinterprets the stored timestamps? A system reboot is the only thing that provokes this bad attitude, simply restarting the darkstat process without a reboot produces no ill side effects. The monotonic time corresponds in reality to the system’s uptime, at least in FreeBSD and quite possibly in every POSIX compliant system.
In conjunction with darkstats’s author, Emil Mikulic, we gather that darkstat’s fails to take negative values into account, both when doing the necessary arithmetic on monotonic time, and when formatting such values for use in the web interface. Improvements might appear within the next few weeks. Stay tuned.