Thoughts on the delisting system and possible alternatives and accompanying features.

[mrturkeytoe]

So, there's this new delisting system I noticed got that put in, delists the server at like 110 or 120 population. That's pretty neat and dandy but what about the alternative approach of if the population goes over 60-80 the server will run Polaris' map or metastation instead of Box and just do away with the unlisting system entirely?

Edited March 20, 2017 by mrturkeytoe

[Birdtalon]

That would be good except we don't have a suitable "larger" map yet.

[necaladun]

While a nice idea, no other maps have yet surfaced that are up to scratch and complete with our code. Metastation is the closest currently, but it still has a lot of work to be done, and even there's still huge amounts of design elements to look at.

[froz-t]

I feel like the cap should be way lower, probably delisting at like, at the very least, 90. By the time that 120 people are on, it's already far too many, and there's hardly any more that I expect to see join via the Hub.

[Urlance Woolsbane]

25 minutes ago, froz-t said:

I feel like the cap should be way lower, probably delisting at like, at the very least, 90. By the time that 120 people are on, it's already far too many, and there's hardly any more that I expect to see join via the Hub.

Seconded. The difference between 120 and 150 is like the difference between the deep sea and the Devil.

[ZN23X]

I also think there should be some sort of guidelines about the HOP opening more security and doctor slots when the population grows. I went to join a round yesterday that had a total population of over 150,  60 civilians, 4 officers total, and I wasnt able to join as an officer because there were no more slots. Sec can't trust hiring from crew so more slots NEED to be opened so we can get "pre-screened candidates sent by central" who are mindshielded. 

Those are the types of rounds I join as a civilian and apply to be Medbay Guard but that only works if the HOS is familiar with me.

Edited March 20, 2017 by ZN23X

[Shadeykins]

1 hour ago, ZN23X said:

I also think there should be some sort of guidelines about the HOP opening more security and doctor slots when the population grows. I went to join a round yesterday that had a total population of over 150,  60 civilians, 4 officers total, and I wasnt able to join as an officer because there were no more slots. Sec can't trust hiring from crew so more slots NEED to be opened so we can get "pre-screened candidates sent by central" who are mindshielded. 

Those are the types of rounds I join as a civilian and apply to be Medbay Guard but that only works if the HOS is familiar with me.

Captain is responsible for doing this as well. Removing the ID console from the bridge made this a little more tedious though.

[Ziiro]

I advocate that the server at those numbers needs something on the scale of DeltaStation, but as far as I know that map is still a Work-In-Progress on /tg/ even.

[Doukan]

All command have to do is not be anal about paperwork/vetting for basic jobs, always a long line at the office, and BOOM no more job shortages.

It's an entirely IC issue with entirely IC solutions, if only people would apply themselves.

Edited March 21, 2017 by Doukan

[Birdtalon]

17 hours ago, Shadeykins said:

Captain is responsible for doing this as well. Removing the ID console from the bridge made this a little more tedious though.

The ID console isn't actually removed. The thing they replaced it with has an ID console built into it which you can select.

[Love-To-Hug]

I think the threshold needs to be lowered to 100, and I also think it needs to be less buggy.

[Kyet]

Delisting is just sticking our head in the sand, and refusing to make a serious attempt at tackling our problem.

The problem is that we don't have enough jobs, job slots, or, sometimes, space, to handle our popularity. That's the real problem, and we need to address that.

 

Here's how we could improve things:

- Merge all the job slot PRs on Github (+11 job slots for existing jobs, including several currently understaffed jobs, like Security). We could probably add more than this.

- Merge the new job PRs on github (up to +6 slots)

- Get cracking on implementing Lavaland, complete with lavaland roles (up to +10 new job slots, e.g: tribesmen, syndi researchers, etc)

- Look beyond that for more new jobs that we could add (+5 or so, e.g: Coroner, Goon Mechanic, various gimmick/RP jobs like regional director, wrestler, movie star, etc). They don't need to be super fleshed out. RP roles are fine too.

- Decentralize the handling of Civilians seeking jobs. Merging https://github.com/ParadiseSS13/Paradise/pull/6766 for example would enable heads of staff to handle some of the jobseekers, eliminating the massive queues at the HoP line, and reducing riots/greytide dramatically.

- Expand the current map with more things for civilians to do, like Goon VR, and more space in general, so at ~150 pop the station doesn't feel so crowded.

 

If we do these, we'll massively reduce the greytide, with over half of civilians getting jobs, and even the remainder having other options besides causing trouble. That would make an enormous difference to the number of concurrent players we could support. It would also open lots of new RP options, from small-scale things like interfaith rivalries between multiple chaplains, to big-scale event options like a massive engineering team being charged with a huge construction project.

 

Paradise, like SS13 in general, is a multiplayer game. As the number of players increases, so does the potential for interaction, and fun, unexpected things. In short: the more players, the more awesome it can be. Looking for ways to turn away players is the *wrong* approach. Instead, we should be looking for ways to increase our capacity so that we can have one hell of a party, as the biggest, most popular SS13 server out there. Full of curious humans, shrieking Vox, beeping IPCs and mysterious Plasmamen. We should be the rockstars on stage, the feature event that packs the hall. Let's not be the shy kid at the back, hiding in their little clique, too scared to handle the crowd, or meet new people. That's not Paradise. Paradise is meant to be awesome. We can be awesome... if we embrace our popularity and run with it, instead of hiding from it. We have player numbers other servers can only dream of. Let's not flee from success. We've earned it.

 

[Love-To-Hug]

I don't really agree with the philosophy that the delister is 'sticking our heads in the sand'.

Most multiplayer game servers (and I'm talking in all sorts of games) have a player cap for good reason, there comes a point in which gameplay starts to degrade if you have too many people.

Instead of having a hard cap, this is more like a soft cap.

That said, I completely agree with the codebase changes that need to take place so we can give support to these higher populations. I would also look into some of the jobs being 'high peak exclusive' so that we don't get a bunch of filler jobs at low peak when essential jobs are vacant.

The delister can continually get re-evaluated as our circumstances change - I'm a believer in adapting to the current situation rather than endlessly planning for the future.

Edited March 22, 2017 by Love-To-Hug

[Fruerlund]

While I do not believe we should opt out lf the delisting mechanism as of right now, I agree that it is not a sustainable solution for the long term. 

Our playerbase is growing and growing which and as we are one lf the largest server our base is not going to shrink anytime soon. (and summer has not started yet) 

In my opinion there are two major focus areas where we should start dedicating our resources so we can get back to working on nice new features ASAP.

1. Map rotation. I am not sure what the state of this mechanism is at currently, however it is ignorant of us to ignore the fact that our map is not suitable for so many players. We should focus on finishing meta station,  add a rotation system based on cur. players and then we can add more maps to the rotation later(either some that is created by us or future ports of other station codebases. That should be our major goal, even if it should be the result of a minor feature freeze(yes I said that) 

 

2. while number one is a bigger goal for now(and our most important one) lets get the job PRs merged and come up with even more alternative job titles for RP purposes. This goal is simple and doesnt require a lot of explanation. 

[Spacemanspark]

I'd like it if we lowered the cap before delisting to around 90 or so, similar to what others are saying.

Additionally, I also agree with raising the job limits.

[Allfd]

A number of potential solutions have been brought up, but I want to address the delister itself as I am its designer.

Currently three things are evaluated by the listing system,

1. Total player count
2. Players to Admins as a ratio
3. CPU 

The elephant in the room is that the hardware the server runs on cannot sustain more then around 120 players before things start to noticeably degrade.



This graph is the last 48 hours

The blue shaded area is the server playercount, the redline is the moving average the system uses to determine if it should take action based on CPU, the brown line is the CPU for that minute (the sample rate of this data)

The blue line that bounces from bottom to top is our listed status, if its on the top, we are listed, at the bottom, delisted.

CPU  budget is one of the many things that needs to be taken into account when we talk about the servers listed status at any given time.  Our player count is the key factor limiting certain CPU intensive game changes everyone would like to see.


In the next few days we will be upgrading from an i7-6700k to a i7-7700k.  This will get us about 5% more single threaded performance.  We will continue to upgrade processors when faster ones come on market, but at the moment short of moving from 14 nano-meters down to 7, things are not looking promising.

With all the problems of making a larger map, that is easy compared to rewriting byond to take advantage of more then one CPU core.

Edited March 22, 2017 by Allfd

[Love-To-Hug]

Allfd, this is all assuming current code, right? Would optimizations allow us to support a higher player count? I mean, I'm sure the answer is 'yes', but, to what extent of an improvement could we expect?

Edited March 22, 2017 by Love-To-Hug

[Allfd]

I need to defer the optimization question to the maintainers,  Fox was working on some optimizations involving reusing objects instead of deleting, but those did not pan out performance wise.  I bring this up as its not as simple as fixing bad algorithms or bugs that just need to be worked out, instead new approach to existing functions and code may be required and that will require experimentation.

The way I view these trade offs is that on the most basic level, certain features come with certain costs.  This is an over simplification but essentially CPU heavy features can be measured against specific player counts in a sense.  This was the purpose behind a couple of rounds where backend functions were tweaked to measure the corresponding impact over the course of the round.

Right now we are fairly stable at 120 players,  if we reverted goon lights we could take that up to 160.  4x'ing LINDA would take us down to 90, however reverting goon lights and 4x'ing Linda would take us up to 130.

The LINDA numbers come from a meteor round, and all this is of course subject to very different pressures as to what drives the exact cost, but back of the napkin numbers put the cost of 

• Goonlights = 40 Players
• 4x LINDA = 30 Players

I am not advocating changes to lighting or atmospherics, this is to illustrate the problem paradise faces when looking at any PRs that significantly alter how CPU is used, we are operating in an environment which is far more constrained performance wise due to our popularity. 
 

What I hope to instill however, is that we are at the point where,

• Neat new feature = ## Players.

There is some hope, for instance I think that some of the strange behavior within each round CPU wise may be fixed in an upcoming Byond patch.  However we lack detailed metrics from before the issue was introduced, so we don't have a great understanding of what that may mean for us. 


I have not addressed the issues of the how many job slots are available or any of the other factors player count brings, maps/ect.  Those are best left to the maintainers and admin staff to address.

[Shadeykins]

4 hours ago, Love-To-Hug said:

Allfd, this is all assuming current code, right? Would optimizations allow us to support a higher player count? I mean, I'm sure the answer is 'yes', but, to what extent of an improvement could we expect?

The issue with optimizations is...

1) We have few people interested in doing these.

2) We have few coders in the first place, which makes the # for 1 even smaller.

3) BYOND code is spooky.

Also, IIRC /tg/ has fast LINDA that's relatively optimized, no?

Edited March 22, 2017 by Shadeykins

[Dinarzad]

/tg/ does yes, but we're missing some backend pieces that would be sorta required for it.
I think it's a Scheduler of a sort (if someone more knowledge could elaborate/correct me here would be tops.)

[Shadeykins]

We use SHECK- which is apparently superior to /tg/'s scheduling system, as /tg/'s is more prone to stalling out when you max the CPU.

 

IIRC.

[Love-To-Hug]

While TG has faster atmos I do notice that their server stalls a lot compared to ours. Our code seems to handle 2-3 dozen more people before it starts to stall.

We have the best performing server out there as far as I can tell.

Edited March 23, 2017 by Love-To-Hug

[apneaman]

On the issue of massive civilian numbers there is a way to split some departments roles to make room for new jobs. Mainly taking the tasks many crew have no time for and using it to fill a position.

E.x.

Sec: evidence room clerk/records, labor camp warden

Atmos: fire marshall

Cargo tech : delivery boy,postmaster,garbageman

Medical: orderly, coroner,surgeon

Robotics: bot maker

Chapel: altar boy

Misc: comedian, artist, naturopath, waiter, paranormal investigator

As for the delisting. Its really a balance of large server numbers vs quality of play. I'll take quality over quantity any day

 

 

[Shadeykins]

Quote

Another reason is what Tigercat touched on; TG's subsystems are, from what I've gathered, superior to the process scehduler.

 

How the process scheduler works: it kinda breaks apart individual controllers so that if one is hung up, it doesn't impact the others; you know the times you get spammed with "obj processing hung, and was killed at XX:XX"--note how youd could still move, walk talk, and do things---with our previous controller, if this happened, the entire game would lock up.

 

Another feature of it is a thing called "SCHECK". Scheck is a command that references and external .dll called 'btime'. BYOND has no way of determining how far into a tick the server is nor does it have an accurate way of measuring CPU (the CPU displayed in game is an average over the past few ticks...). What scheck does is after every so many things in a loop (ie: every 50 mobs) it'll check this external .dll and see if it *thinks* the current tick *may* be driven into overtime. If it looks like it may, it'll sleep for 1 tick and defer the rest of the calculations for that loop to the next server tick (this is also why it's kinda pointless to break a loop up into quarters). Oh, that reminds me, referencing and checking btime.dll is hyper expensive.

 

That said, we don't live in a perfect world; it's still not 100% possible to determine if a tick really is going to go into overtime andddd sometimes scheck is a bit too aggressive and will defer when it doesn't really need to (thus wasting CPU). If everything in SS13 were perfectly coded, the scheduler would actually be a net drain on performance and incur a loss from scheck.

 

TG's subsystems don't have btime and they handle adjusting their systems in a (I think--though I could be wrong) more optimized way in calculating costs and adjusting things (on the fly) internally; LINDA, for example, will actually end up running slower than our implementation if their CPU load is very high (it just runs at every 5 decisecodns by default).

 

When we were initially going with the scheduler, I had a discussion with some of the TG station devs and why they weren't going with it--ultimately, in a perfect world, the scheduler is dead weight--and thus why they viewed their subsystems as superior (and this was before they made major strides into internal system adjustments for controllers). I do believe there's aspects of the scheduler that are superior, but on the whole, the subsystems appear to handle lag better.

 

Performance is a constant battle on Paradise as we don't have huge margins to work with, unfortunately; for example, when I was porting the Tesla engine, I was constantly keeping an eye on how costly the orbiting energy balls were---I even ran the server for 2 hour durations (multiple times) to see what performance impact was likew ith tons of balls orbiting it---this all went in mind before I decided if it was actually an "ok" thing or not.

Here's the explanation on the scheduler for those curious.