Missing messages?

This is an archive of a topic from NESdev BBS, taken in mid-October 2019 before a server upgrade.
View original topic
Missing messages?
by on (#47468)
It seems the BBS had a few issues during the night and some message disappeared. Is it because of some maintenance or the BBS software crashed and a few mail were lost? A few threads seems affected.

by on (#47475)
I tried posting last night but failed. Some debugging information regarding keywords showed up and the post wasn't made. I haven't noticed anything missing yet.

by on (#47477)
This was the result of somewhat unplanned maintenance on all the servers in our co-location facility. (We had a lot of work to do, and were there for about 4 hours doing numerous upgrades, re-wired the entire cage, and many other things).

There was only one piece of fallout: the MySQL server began acting oddly prior to the maintenance beginning, specifically with regards to NFS. Anyone familiar with UNIX knows how NFS timeouts can more or less indefinitely stall a userland program, and we had many of those. I'm pretty sure I found the root cause and fixed it, but by that time the damage had been done.

We had to reboot the MySQL server without cleanly shutting things down. Specifically, shutdown, reboot, etc. would all cause disk buffers to get flushed -- and that includes NFS -- so we had to tell the kernel to shut down without flushing any I/O buffers (e.g. any cached I/O transactions would be lost) using "reboot -q -n". This is a big no-no in the BSD world, but the circumstances justified it.

Sadly, this had a major effect on MySQL. There were 8 or 9 tables which mysqlcheck reported as corrupted, and using --repair fixed them, but some rows were lost. Thus, there could be some table integrity problems (e.g. two INSERT statements were made in 2 different tables, one went through but the other was lost, so the board software becomes confused).

Two other users saw similar impact -- one lost most of his entire website (but had backups of the Wordpress content so he should be fine), and the other also lost forum posts (also phpBB).

This is the first time we've actually seen data loss of this severity before. There was not a hardware malfunction or anything of that sort -- no disk errors/bad sectors/etc.. This was purely due to the machine becoming more or less deadlocked due NFS problems on the MySQL server.

This shouldn't happen again in the future given the modifications I've made to the NFS configuration on the server. But if there's any brokenness that people find here (besides missing posts), please let me know and I'll see if I can reverse engineer phpBB and delete or fix whatever rows are broken.

by on (#47498)
tokumaru wrote:
I tried posting last night but failed. Some debugging information regarding keywords showed up and the post wasn't made. I haven't noticed anything missing yet.


One of your mail on my thread regarding interleaving was deleted, my answer to it too. Some of the message in the gemventure thread, a few in Nescicide new screens. Except for that, I don't remember anything else was deleted.

It's not like it was that much important information so we will survive.

by on (#47499)
Yeah, I ended up noticing. Kinda scary to think that we could loose all the content overnight like that! O_O

by on (#47500)
tokumaru wrote:
Yeah, I ended up noticing. Kinda scary to think that we could loose all the content overnight like that! O_O


Since they're doing backup of the database (I think), the worst that could happen is losing one day of message.

by on (#47512)
Tom reported some threads were showing as multi-page, even though their posts were missing (as a result of the mishap described above).

A simple MySQL query can get me the list of post IDs which exist in the phpbb_posts table but not in the phpbb_posts_text table:

Code:
mysql> SELECT a.post_id,a.topic_id FROM phpbb_posts AS a LEFT JOIN phpbb_posts_text AS b ON a.post_id = b.post_id WHERE b.post_id IS NULL;
+---------+----------+
| post_id | topic_id |
+---------+----------+
|   47427 |     5230 |
|   47428 |     5229 |
|   47429 |     5217 |
|   47430 |     5232 |
|   47431 |     5217 |
|   47432 |     5217 |
|   47433 |     5230 |
+---------+----------+
7 rows in set (0.00 sec)

And then to determine what the threads were which were affected:

Code:
mysql> SELECT topic_id,topic_title FROM phpbb_topics WHERE topic_id=5230 OR topic_id=5229 OR topic_id=5217 OR topic_id=5232;
+----------+----------------------------------+
| topic_id | topic_title                      |
+----------+----------------------------------+
|     5217 | GemVenture                       |
|     5229 | NESICIDE v? screenshots          |
|     5230 | Map data interleaving, worth it? |
|     5232 | converting problem               |
+----------+----------------------------------+
4 rows in set (0.01 sec)

So there you have it -- those were the 4 threads which lost some posts.

I've since deleted all of the post_ids shown in the above list.

by on (#47524)
Thanks for doing the maintenance and repairs, man. Parodius is like my home, it's the first and last place I go (online) everyday. I know many people appreciate it, but probably fewer say it (or even know where this stuff is hosted). So, I thought I'd say so.

BTW I want to remind people that a possible way to show appreciation is to donate to our hosts - there's a paypal link on http://www.parodius.com/ on the upper right. This site has been on this server for over a decade, without ever - not even once - having paid advertisements. I don't know if that can be said for many other sites.

And surprisingly, as far as counting hits on the mainpage of NESdev goes, I recently noticed it has actually turned around and is trending upwards after years of slow and steady decline (and no HTML updates).

by on (#47543)
This seems to cover the post that I remember going strange and some extra that I didn't remember.

Thanks for fixing the DB.

It's a nice little BBS here. This is the only one that I go everyday.

by on (#47547)
Banshaku wrote:
This seems to cover the post that I remember going strange and some extra that I didn't remember.

Thanks for fixing the DB.

It's a nice little BBS here. This is the only one that I go everyday.

me too!