Regarding the SQL Rollback..

[block1]

 

I'm not 100% sure if this would be considered an admin complaint, I'm pretty sure it is but whatever.

 

In the future could you guys make an announcement before you proceed? A lot of work was lost on the wiki, character pages, posts, etc.. and it would've been extremely helpful if we knew that a few weeks worth of work was about to disappear, it could've gave us time to save all the work and be able to reapply it after the rollback was done. Instead you guys just went ahead with the rollback, not considering the two weeks of work the community put into the wiki and forum. Personally I will probably be taking a leave of absence from the wiki/forum due to a lot of work I did getting instantly obliterated and justified by "Oh well, won't happen again." and had no warning or at least a few hours to prepare and save the work I did. I'd appreciate if the admins could be a little more vocal about giant decisions like this that could affect the community instead of just doing it and saying "Shit happens, sorry." afterwards.

 

[Habalabam]

 

I assume the data was lost prior to the rollback.

That the database corrupted and this had to be done to get stuff (like the wiki and forums) back up and running.

I don't think they had a working web page with the updated and then pushed "the button".

 

[necaladun]

 

A lot of this is due to me pushing for it to happen ASAP, and I take full fault for the problems here. I had no idea how much had been corrupted, and pushed for the mySQL fix to be implemented without knowledge of exactly what would happen. This is partially due to the fact that security flaws were involved.

 

Had I known what this would cause, I would have given the word out to back stuff up, and even done some myself, as a lot of forum work from our end, etc has been lost.

 

Due to the difference in time zones, and being busy RL, I had to make a call on this issue quickly. It is regrettable how it went, and I apologise that all the hard work was lost. An announcement before we proceeded wasn't really possible, as the issues with the database had already happened. In the end I went with making it quickly due mainly to the security issue. I also knew it needed to be done sooner rather than later, before more data could accrue that would also be corrupted by the problem.

 

Had I the knowledge I did now, I would have done so differently - I would have copy pasted as much as I possibly could myself, for instance.

But due to the circumstances, I wasn't able to. Again my deepest apologies for how it turned out.

 

[mkenner]

 

A lot of really great work by a lot of people was lost in this incident, along with just the regular daily activity of the server. I for one had been extremely impressed with all the wiki work you've done recently and I know many others were as well. There was also a lot of great development on the forums themselves by Regens that's just disappeared. Not to mention all the individual threads people contributed on the forums.

 

We're talking hundreds of hours of people's activity, contributing to the wiki, forum, their characters, receiving karma, bans, appeals, etc all just vanishing in instant. This was in no way deliberate and as you can tell by how all our work has been lost as well, not something we had any warning about before it happened. I'm not saying it wasn't our fault, because it was. However offering warning wasn't something we could do, because we had none ourselves.

 

The 'rollback' wasn't really a rollback, but rather a "F*ing hell, we've just lost all data. Reload the most recent unaffected backup".

 

After it happened we didn't really communicate with the users about this on an emotional level. This was because we were collectively pretty shocked ourselves. The post we put up on the forum explaining what had happened was something we did immediately after discovering our own work into the forum had just disappeared into the ether. I think the reason that post didn't go into the emotional nature of the situation for the people affected, was that if it hadn't been detached from our own emotions it would have consisted of pages of screaming.

 

If we gave the impression that we didn't care about this, or that our reaction was "oh well", it was because we were desperately trying to come to terms with this and remain professional despite being pretty overwhelmed ourselves. Our private reactions were much more emotional.

 

Absolutely the only good thing to come out of this, was fixing it so that it can't happen again. Small consolation I know.

 

[PokeMoney]

Small sacrifices must be made to inch towards success. In this case the sacrifice was hours of work to restore everything so it wouldn't happen again.

[mkenner]

 

Small sacrifices must be made to inch towards success. In this case the sacrifice was hours of work to restore everything so it wouldn't happen again.

 

While that's probably true, it's hard for me to see it that way. Up until now, not everything was perfect but every step we've taken has been a step forwards. Taking steps backwards isn't something we're used to. All the progress we've made, has been because of people contributing so much great work and it feels particularly wrong to lose or waste that work when we owe it so much.

 

I know when it first happened, that going through the appeals forum and trying to reconstruct everything from memory was the last thing I wanted to do. Fortunately that was only an hour or twos work for me. I know a lot of the people who put time and effort into the wikis and forums have much more work to do to get back what they managed and I know they're going to be demoralized by this too.

 

I just hope people like block1, the original poster here, don't give up on contributing to the community all together. I can't blame anyone who does, but I would really miss their contributions. Seeing the wiki updated with brilliant information just because people love the game, always brought a smile to my face.

 

[andrewmeythaler]

 

Honestly I think the biggest issue is that only one person on the team understand the core structure that controls all the servers data. Who was allowed to implement it by himself, explain it to no one, then leave on vacation.

 

When shit the fan everyone just kinda looked around and hoped Ponies would fix it.

 

Don't rely on one person.

 

[block1]

 

Honestly I think the biggest issue is that only one person on the team understand the core structure that controls all the servers data. Who was allowed to implement it by himself, explain it to no one, then leave on vacation.

 

When shit the fan everyone just kinda looked around and hoped Ponies would fix it.

 

Don't rely on one person.

 

This.

 

[Habalabam]

 

I think that is a slippery slope.

 

Although it is a valid observation and something many people should take to heart when they are entering work and starting businesses, let's not forget that we are all here enjoying this game and server exactly because of these solo efforts.

 

So make sure the room is warm, friendly and appreciative when making that observation.

 

There is a saying in IT that it is only noticed when there is a problem. Since a well run IT department is not noticed, it is frequently a target for cost reduction.

Looking at the forums, the staff is pretty clear that the manpower is stretched thin.

I myself have dozens of posts pushing for new functionality, bugfixes, fluffs, in-game clarification, lore background etc. I don't have a single post pushing for better backup and recovery procedures. Or anything indicating that I would be appreciative of a functionality freeze while the staff worked on dealing with the contingency that key person would be away at the exact time of a server failure.

 

So, I have no problems seeing how this will happen.

 

[andrewmeythaler]

 

There is a saying in IT that it is only noticed when there is a problem. Since a well run IT department is not noticed, it is frequently a target for cost reduction.

Looking at the forums, the staff is pretty clear that the manpower is stretched thin.

I myself have dozens of posts pushing for new functionality, bugfixes, fluffs, in-game clarification, lore background etc. I don't have a single post pushing for better backup and recovery procedures. Or anything indicating that I would be appreciative of a functionality freeze while the staff worked on dealing with the contingency that key person would be away at the exact time of a server failure.

 

So, I have no problems seeing how this will happen.

 

I don't doubt that we have an excellent coding team hard at work.

 

But there shouldn't be a "key person"; you would be hard pressed to find any business that required one person who could not be replaced. Why? Because that is fragile, as we have seen.

 

I have more examples.

 

Working in year three of a computer game design class. Our teacher tells us over and over to make sure we have people ready to step in, incase on of us gets sick. Sure enough, the head designer, and the guy who came up with the idea for the game, dropped off the face of the earth. No one ever heard from him again. But the project went on because we commented on all our code, made sure everyone knew what we were working on and how to do it, and didn't rely solely on him.

 

Working in theatre we always, ALWAYS have more than one person who is trained to do more than one job. Because people come and go, and if suddenly the one person who can do lights gets busy you are shit outa luck, and the director isn't going to be very understanding. I'm working at a local community college as a lighting designer for a musical. We were supposed to also hire an in house tech, but they only had one, and low and behold he never showed up. This made my job infinitely more difficult. As there system is out of focus, has very few lights, and is kind of broken. Not to mention they keep their ladder under lock. This meant I had to go and break into my school, get our ladder, take some of our lights, and bring a spare dimmer box to work on their broken system.

 

The flip side of that story is working at a local university theatre as an intern. It was a union house, which meant you had to use their union team. The rules also stated we needed TWO in house lighting techs. The master electrician showed up but the other didn't. However it didn't matter, because we didn't rely on that one person. Everyone on the entire team was trained to do almost every job. Any one of them could have died and nothing would have changed.

 

Don't rely on one person.

 

[mkenner]

 

Honestly I think the biggest issue is that only one person on the team understand the core structure that controls all the servers data.

 

Our team only includes the people that we have. While it's an interesting subject that I would love to learn some day, I don't have time at present to learn MySQL programming. Necaladun has learned a lot of SQL, and has been able to handle many minor problems and even some major patching. This particular issue was far from simple though and no one else on our team was able to volunteer a solution. Access was never an issue, if a fix had been invented by any of our coders, Mel or Necaladun could have accessed the server to implement it.

 

Ponies has the position he does, because he's a qualified computer professional with coding experience. He can do more than we can in this field and it's been largely due to his singular efforts that the server exists in its current state. We'd be a lot more limited if he hadn't volunteered his time and efforts for free over the almost a year that this server has been running.

 

As this particular issue occurred within our security systems as part of safeguarding our data from the latest round of newly developed or discovered exploits, this particular system cannot be made open source or public access for obvious reasons.

 

Regards,

M Kenner.