Well, here we are, back up again. After over a week of downtime, we feel that we are ready to give this another shot as well as explain what had happened.
Last month, we were informed that our server was hosting a PayPal scam site. Our host was working with us to close up the exploit. It seems that whoever got in may have used more than an exploit. I found a couple of new domains on the server that were not set up by us. After further investigation, it was thought that they had the root password to the server. This fear was confirmed when the site went down.
Our initial check led us to the fact that our database tables had been dropped. That's bad.
This effectively removed our forum and game databases. We have been working to try to restore the forum from the information that was left on the server, but after all of the effort, we were stuck. Eddie was able to get the game database back from early December. This also included the blogs. The forum, however, was not so lucky.
The initial forum restoration that we found was from 2009. This included the member table. I happened to attempt to make a backup back in January, but it only partially backed up. Fortunately, this was successful in including the member list and even PMs. So, while we have now lost a ton of the forum, that can be recreated. The important part is that we were able to recover a fairly recent backup of the game DB and almost all of our members (not sure how many registered in the few weeks since my backup).
Now what happens? This experience has certainly left an impression on me as well as Dave, Eddie, and Mike. We will be implementing means to do (starting off) biweekly backups and as we get things honed in, we hope to get this down to possibly daily. We will also be working to have a system in place to test these backups and make sure they are good to prevent this sort of downtime in the future as well as making it easier for us to move hosts in the future should we need to do so.
We are very sorry that this has happened and that it was down for so long. We hope that it hasn't tarnished our image for you and that you continue to come back. Personally, I appreciate everyone that came in to the IRC room during the down time to keep the conversation going and your words of encouragement really meant a lot. The community has always been fantastic here and now that we're back up, I hope we continue to get stronger as a result. Thank you for your patience with me and my stubbornness to try everything I could to try to get it back to where it was before the disaster struck.
Welcome back to Channel 3!
|
Ouch!
Glad to see RF Gen back up and running though.
|
|
I just wanted to say a great big "THANK YOU" to Adam, Dave, and Eddie for getting (almost) everything back up and running in a timely manner. Without the dedication of these fine gentlemen, we'd still be down and in much worse shape when/if we did come back. You guys all have your own lives to take care of outside of this site so its awesome to see you all work together in your precious free time to fix this mess.
And sure, we lost some forum posts along the way, but those are easy enough to replace and in the mean time, we all get to pretend its 2009 again. Hey, did you hear Circuit City is closing?
|
|
Thanks for all of the effort, guys!
I'll be fairly active trying to get the game DB issues sorted out as they come up, but there will be a few things to note:
1) The restore is from Dec. 2nd. Any additions, edits, or deletes after this time will need to be re-done.
2) Images submitted or moved after Dec. 2nd will still be tied to the ID they were submitted against. This means that images may suddenly appear for an addition if it is using an ID that was previously used before the downtime. If this occurs, please report it in the Announcements and Feedback forum.
3) Extra Media images submitted after Dec. 2nd will not show up on a game page. If you submitted an Extra Media image for a title after Dec. 2nd, please let us know and we'll try to get it routed to the correct page.
|
|
Thanks for all the hard work guys. I don't know what I would do without my daily dose of Channel 3. 
|
|
Great to see you again!
|
|
Top work fellas, nice to see the site back up and running.
|
|
Thanks so much for the hard work Adam, Eddie, Dave and whoever else was working hard behind the scenes. It sucked not having rfgen, but at least I could sleep well knowing there were some awesome people working on it.
I was really hoping the silver lining with the forum issues would have wiped Izrets post count. Oh well:/
|
|
Congrats on getting the site up and running again. Thanks for all your hard work!
|
|
I've never missed a website before now. Glad its back, thanks for the work.
|
|
Thanks for all of your hard work guys. Really missed being a part of this community for the past several days. Know that everything you have done is greatly appreciated and any thing lost, will be back better than ever.
|
|
So glad the site is back up. I will finally be able to enter my backlog of games!
|
|
I knew I would miss the site should it ever go down, but this sure was an example of not realizing how much something is a part of you until you don't have it.
Thank you, thank you, thank you Adam, Eddie, Dave, and everyone else who worked hard so we could enjoy the fruits of this fine internet community and resource site.
Everybody, we should be proud to pitch in on the database and everywhere else we can to help rebuild anything lost and then some, as not only good showmanship to those who put their own time and work into getting us back running again, but as a general respect to all this site offers free of charge.
Except the coffee. Please put 50 cents per cup in the change jar on your way out. 
Glad to be back!
...guess I should clean up my desk now...
|
|
So happy to see things back up and running. I've been missing this community, and I'm ready to dig back into some submissions. Thanks to Eddie, Adam, and Dave and everyone else!
|
|
Yay! RFG is back! Now to redo all my N64 edits. 
|
|
It's good to hear things are back!!
|
|
Glad my home on the web is back up. I missed the site and the members.
|
|
Wahoo! Thanks for saving so much!
Your hard work is definitely appreciated! I kept stalking the site waiting for it to be brought back up.
On the plus side of the forum loss I wasn't here in 2009 so now I get to read all that old stuff
|
|
BIG thanks to Mike and Adam, they really led the charge on this. Dave and I played more of a support role. Those two guys are the main ones responsible for getting this whole thing back up and running. Huge kudos to both of them.
It sucks that we lost as much data as we did, and I myself am partially to blame for not keeping as recent backups as I should have. On the plus side, now multiple of us will be making backups and should something this catastrophic ever happen again we will be ready for it.
|
|
Awesome that you're back!! Really sorry to hear all that happened. Hackers can be real pricks sometime.
|
|
Phew. Good job getting things back and running.
I will say I'm surprised that with the amount of work that has gone into creating the database over the past couple of years that you guys didn't had a regular backup schedule in place already. I guess live and learn.
I'm guessing that once the hackers realized you were onto them they trashed everything and got the heck out. Blah.
|
|
Thanks a lot for getting back online. I really missed rfgeneration the last days!
|
|
just to make sure, was anyone's passwords compromised, should we changed them ?
|
|
Thanks for all the hard work getting RFGeneration up and running!
|
|
@douglie007: Passwords are all hashed in the database and should not have been compromised. I have no reason to believe that they were stolen, but if you prefer to err on the side of caution, feel free to change it.
|
|
@douglie007: I changed mine, not because I thought it was compromised, but because it was the password I started with here and wasn't a good one at all. And this security hole made me realize how unsafe it could be to have a bad password.
|
|
@Lendorien: We actually had a January 15th backup for the forum. It had corrupted several corrupted tables. Fortunately, we were able to save the members table from that backup. But yeah, we will be developing a more redundant plan for backups.
Big thanks to Adam and Eddie. They kick ass. They did a lot of the manual labor editing the SQL files to be restored properly. Adam deserves much kudos. He was the lead restorer and he did a great job. We are grateful.
Also, I am aware the forum is a bit wonky (things beyond just the loss of posts in 2010 and 2011). I may be updating it to SMF 2.0 earlier than I thought to deal with these bugs.
|
|
Great to be back
|
|
Great to be back, only one thing I don't get. I lost some credits for scans. I just found out that scans for games which were added in january 2012, but added to an existing release are still in the database.
|
|
@sharp:
Images are stored on a separate server and are tied to a particular ID. If the title was added after Dec. 2nd with an image, you'll notice that sometimes an addition will occur that occupies the ID for the old submission and the images will get tied to it. Previously existing submissions will not be affected by this, and images will show up as normal.
The only exception to this is Extra Media images. They still exist like normal images, but had a Page Edit done to link them to the Extra Media section. This link needs to be added again (it's just an html link) for any Extra Media added after Dec. 2nd.
|
|
Wooh, you guys are back up!
The database isssue sucks, but it seems to happen to every "update-able" site I've seen.
|
|
@Shadow, I just found another issue with adding games
JUst re-addded the UK-version of Resident Evil: the Mercenaries and it got the scans of Ridge Racer
http://www.rfgeneration.com/cgi-bin/getinfo.pl?ID=E-182-S-00160-A
I think it will add the pics as they were previously added, but it's im[
ortant to find a solution for it.
|
|
Glad to see you back!
|
|
I never knew just how many times a day I accessed this website until it went down. I have the original count on my work terminal, but it stands somewhere around twenty-six visits during a ten-hour period (and that was while the website wasn't up!).
|
|
@sharp: That's going to be happening for a while. We're trying our best to catch them as they come up, but as you can see we will miss a few. Best thing to do is to make sure you let us know about it.
I got that title fixed, along with 4 more that kept doing the same thing everytime I added a new title for the existing scans.
|
