A New Proposal on Judging

[Storm Veritas]

Well, I read the entire judging thread in "Your Word", and lots of good points were brough up. I think the issue with judging in Althanas is due to a series of connected issues: Althanas administration grossly underestimates what they ask from in their judging volunteers. Judging is perceived as a thankless job. Judging is no longer perceived as an honor, but rather a duty.

1. Althanas administration grossly underestimates what they ask from in judging volunteers.


WANTED

Judges!
Must be skilled writers, assessed as creme de la creme from a crop of very good writers. Must be willing to commit to an enormous, endless stream of work. Must be willing to be consistent, fair, patient, and dilligent.
If you get tired, take a fucking hike, we don't need you. There's plenty more great writers where you come from that would rather do this.

Now certainly I exaggerate for effect, but I'm not long off. Hence massive turnover being viewed as healthy.

2./3. Judging is perceived as thankless / it is not an honor but a duty.

It appears that at any given time, there are roughly 3 or 4 judges who are good and are working dilligently. They strive to maintain a high standard and work to keep the volume down.

Here's the problem: there's also about 20 other judges who aren't doing dick at any given time.

I think we need to figure out who -REALLY- wants the job. Not more than 3 or 4 people. We need to make sure that they are the people judging, and that they are capable. We need to stop just hiring more when more are needed, and remove judges who are not working.

If the entire judging staff was reduced to only a select few, that select few quickly gains a huge bump in prestige. They are empowered; they are the decision makers.

Dilution needs to be dealt with. If there are only a few judges, their respective input becomes MUCH more valuable, and their standing in Althanas is more highly sought after. Tournaments could be run to determine the winners ALLOWED to be judges, because people should WANT the job. When almost anyone who wants the job is granted access, there is no longer prestige associated with the power entrusted.

I know this isn't foolproof, but I firmly believe that to re-establish productive judging, we need to redefine the judging role and re-establish it as a true privilege, acknowledging the judges as being more clearly delineated as the "Chosen Few".

Oh, and hi. Hope all is well. Good to see you, and what-what.

[None So Blind]

Ha, is it just me or does this seem pretty random? I'm not trying to undermine your call for productive judging, but there's not a backlog right now, with the exception of a few threads who are all requesting the same judge.

EDIT: Whee, I guess I didn't really address any of your points.


When almost anyone who wants the job is granted access, there is no longer prestige associated with the power entrusted.

Actually, when people submit an application, it has to pass through a debate of all staff. People get turned down all the time, and applications are really put through the ringer. You really can't get into a moderator position for just having plain enthusiasm in your application. Granted, being nonchalant also counts against, but not just "anyone" is let in. Even well known players.

As for "20 judges who do dick", I haven't really seen that. Everyone on the judging staff tries their best to work their duties around their own real lives and characters that they want to nurture on here. I think we've got a great staff right now, and I can't really think of anyone on it who doesn't do work. Except maybe me. I work in spurts, like an orgasm.

[Serilliant]

It seems to me as if your two points (or rather three points organized into two subheadings) conflict. Your first issue is that we expect too much, and then the second issue is that we should expect more?

Perhaps I'm missing what your distilled point is. If it is merely that we need to reevaluate the current staff, I agree. However we are currently short 1 staffing administrator which puts the breaks on such projects. If there is more that you are suggesting, however, could you possibly rephrase your ideas?

[Karuka]

What I've noticed is that it tends to cycle. Judges who work hard are at high risk for burn out, simply because they perceive that there's no one else to do the job, so they work hard to keep the backlog down because backlog is stressful.

I remember that at one time, it was Letho and Advent doing most of the work. Then I came on staff, and it was Letho and Witchy. For a while after I got into judging, there were three or four of us who took most of the threads. For a while it was me, Manda and Witchy, and then it was me and Shyam. Now it looks to be mostly Numbers and Raelyse, and eventually they'll cycle off and be replaced by other, competent judges. I know that I'd rather see people cycle on and off staff and judging than get too fed up and say "screw this" after a while. I remember that I really enjoyed doing it for a while, because despite all the threads you come across that make you want to brain yourself on your own keyboard, there is the occasional truly great one, and the rare thank you PM that makes it worthwhile. But then it just got to be too much work with too many anti-thank you notes, and pretty soon you don't want to write, much less even look at people.

While I agree that it would be good to have long-lasting, hard-working judges on the site, I don't know if there's really a feasible way to accomplish it. And our staffing director is MIA, which makes getting new judges on staff somewhat difficult. -_-*

[Breaker]

And our staffing director is MIA, which makes getting new judges on staff somewhat difficult.

Maybe we need a new staffing director... just a thought.

Good points, Storm. I don't exactly cherish the idea of "cycling off" the staff or getting burned out, but hell, I already feel my desire to write waning. I really don't think that our system should promote that.

[Max Dirks]

I think a good way to solve most of those problems is to offer moderators substantive rewards, Storm.

Check here (http://www.althanas.com/world/showthread.php?p=104691) for our previous discussion on the matter.

[Lavinian Pride]

Or perhaps make their opinion count, or once again segregate the mods.

Saying that any mod can do any job creates the problem I see on the boards now. The jobs that no one wants to do, certain people force themselves to do, not because they want to, but because they know it has to be done and do it. While other people do what they want, or even nothing.

Before you even open that can Dirks, lets get the work done without forcing mods to cycle out every few months from burn out. Whats the point of having the system, when the people its designed for leave before they can really benefit from it?

[Storm Veritas]

Sorry, had a few ideas in my head yesterday and they came streaming, so I can rephrase if needed.

None So Blind: Perhaps I didn't introduce myself properly, but then I didn't think I had to. I'm no rookie anymore. I was a judge for over 2 years, and worked on that "ringer" that you speak of (which is far from as scientific as you imply). Your claim that there is no backlog would hold more water were there not LITERALLY hundreds of unclaimed threads in queue. Perhaps I'm looking at that list incorrectly, or they are not removed after judgement. Otherwise, wtf.

The "20 judges not doing dick" is something I've seen firsthand. For a long time, I was one of the few that was really busting ass to keep the wait time down. After that, various conflicts (my job, my marriage, my house) took priority and I became one of those judges not doing dick. It's an easy trap to fall into.

Serilliant: There are two cultures here, and it's evident in both the thread Max linked to and the offspring of that, the "Let's pay judges in EXP" thread that was eventually locked due to fire and brimstone. One is the voice of the people, the other is yours. It's not so much a difference of values as it is a perspective issue. When the administration is so quick to remind everyone that judges are a highly replaceable role, it undermines the value of these judges tremendously.

It's also simply not true. This site is highly dependent on a small core of ultra-skilled judges, in particular Shyam (who I am convinced is an alien with magical speed readerish powers). Over time, various conflicts come up that either burn out the judge or restrict the judge's power.

Here is the big problem: historically, instead of dethroning a judge no longer capable or willing to judge prolifically, we allow them to stick around and linger while we also hire someone new. This creates a mass of judges, and that isn't useful when one judge only scores a thread once a year. The active judges are left to look at the long queue and the list of "active" judges and wonder why they are forced to carry a huge burden while other layabouts carry the same title.

Max, I agree, but have a different proposal. The "Judge" title carries no clout anymore, and thus the demand for the Judge position from players has plummeted. It really IS a wonderful job if you have time for it, and it obviously is something that requires significant skill and time to execute.

I think there should NEVER be more than 3 - 5 judges, and these should only be the very best of all applicants. If there are only a few, and there is a new prestige afforded the role, I think Althanas can reasonably expect more out of the position, and reap benefits from the realignment. I think a lot of competition over who is ALLOWED to be the judge is healthier than the huge turnover of people who happen to fill the role of competent and willing.

[Zook Murnig]

I'm sorry Storm, but since when are you capable of viewing the modcp? What you're likely referring to is the Judging Requests forum in the Mod section, and I'm not entirely sure those threads are affected by what we do in the CP. Especially now that we've gone through an update of the system.

There, really, is no backlog. A couple of threads by Godhand are all that remains, and that's only because he stubbornly requested Shyam (that alien you spoke of who apparently can read things at light speed, but has been gone, except for a brief visit, for MONTHS). We're doing fine.

[Skie and Avery]

None So Blind: Perhaps I didn't introduce myself properly, but then I didn't think I had to. I'm no rookie anymore. I was a judge for over 2 years, and worked on that "ringer" that you speak of (which is far from as scientific as you imply). Your claim that there is no backlog would hold more water were there not LITERALLY hundreds of unclaimed threads in queue. Perhaps I'm looking at that list incorrectly, or they are not removed after judgement. Otherwise, wtf.

The "20 judges not doing dick" is something I've seen firsthand. For a long time, I was one of the few that was really busting ass to keep the wait time down. After that, various conflicts (my job, my marriage, my house) took priority and I became one of those judges not doing dick. It's an easy trap to fall into.



I know who you are, dumbass. I've been around six years myself. It might help your point that moderators should get more respect if you didn't go attacking one, trying to throw around clout that your points are infallable and nothing has changed for the better since the days when you stopped doing stuff behind the scenes.

The ModCP is telling me that the only things in the judging queue are two threads claimed/requesting Call Me J, one claimed by Advent Wings and one claimed by me. Nothing unclaimed. Where you're getting these hundreds of threads is beyond me.

[Karuka]

There's a new system for claiming/filing submitted threads, Storm. The "judging requests" subforum hasn't been cleared out since it was implemented. No one keeps house anymore, it's a little sad.

[Storm Veritas]

http://www.althanas.com/world/forumdisplay.php?f=73

16 pages of Judging Requests, some as recent as yesterday, some over a year old.

Call me crazy for thinking that "Judging Requests" that I could see resembles the old queue, seeing as the format is identical and the names of judges taking threads all looked familiar.

Skie. Pot. Kettle. Black.

If you're going to criticize, at least give me the effort of READING first. Without seeing the "new" list, I thought that the system was not now broken, but rather still broken, and that the flaws that made things difficult before still remained.

[Karuka]

Yeah, that's just house keeping that hasn't been done. We Emeriti can't see the modCP and know what's really going on.

[Storm Veritas]

Oh, well that's good news then!

Having come back after another hiatus, it certainly appears through that link that the request / judgment volume balance has gone completely out of control.

Given that it isn't, then there certainly isn't need for sweeping reform. Always room for improvement, I suppose, but not a total overhaul.

My mistake, carry on.

[Serilliant]

Note: I know you were mistaken about our backlog and that you are stepping back from your position of needing sweeping reform. I did want to take a moment, though, to talk about the substance of your position to hopefully arrive at some ideas for improvement, as you mention.


...One is the voice of the people, the other is yours. It's not so much a difference of values as it is a perspective issue. When the administration is so quick to remind everyone that judges are a highly replaceable role, it undermines the value of these judges tremendously.

I'm not sure I fully understand your accusation that my voice is distinct from that of "the people". I took a cursory look at the EXP rewards thread that you mentioned and counted 6 people in favor and 15 people against. How am I subverting democracy when the decision was made based on more than 70% of the vote?

I also feel I must correct any sort of misperception that "the administration" believes that judges are highly replaceable; I know that judges are highly irreplaceable. While there is no shortage of writing talent on the site, finding someone with the unique trifecta of time, writing ability, and teaching ability is no easy task. Judges are important to the site indeed. However, your proposal seems to leave us worse off.

The inconsistencies that I noted before still seem to be there. For example, you assert that we should make judging a more prestigious position by having only a handful at a time, but then go on to suggest that our backlog is too large. Do you believe that by shrinking the judging staff, threads will be judged faster?

You further assert that the judging position is undervalued by the administration, but also recommend that judges not pulling their weight be immediately replaced. Wouldn't it be contrary to the judges-are-irreplaceable philosophy if we canned every judge the moment he or she got busy or burned out?

Right now we run on a rotating system that you pointed out: judges swoop in, judge a bunch, get tired, and vanish for a while before later coming back, judging a bunch, and vanishing again. It's certainly not the ideal system, and it perhaps may not even be a good one, but so far it's working better than any other method we've experimented with. Further, it inherently grants respect for the judging position by understanding that it's tough work and that breaks are a necessity, thus the job will be waiting for you when you're ready to come back. I really don't see how mass-firings would add "prestige" to the role, nor do I see how this amorphous "prestige" will make judges more committed to it.

It could be your point that judges burn out because they're not appreciated. It's a reasonable thesis. However your solution is predicated on the assumption that all judges burn out because they're not appreciated. Certainly some do, but others stop judging for reasons unconnected to their level of appreciation. Your solution may raise prestige, but also exacerbates other reasons judges may have to quit. In the end, I don't see how we would be better off.

Perhaps I'm misunderstanding your position again, and I hope that I am. Behind your philosophies, there seem to be some solid substantive solutions. I fear, though, that if you are basing your criticism on what you perceived to be a huge backlog that we (thankfully) do not actually have, that your ideas may end up being shortsighted. As you mentioned, your call for "overhaul" is now unnecessary, but I would still like to hear your ideas for improvement if you have them.

Though really, if we were actually a year and 100 threads behind, don't you think there would be a lot more threads like this one?

[Godhand]

There, really, is no backlog. A couple of threads by Godhand are all that remains, and that's only because he stubbornly requested Shyam (that alien you spoke of who apparently can read things at light speed, but has been gone, except for a brief visit, for MONTHS).

During his peak Shyam did more in a day than you've done in your entire time on staff.

[Storm Veritas]

I'm not sure I fully understand your accusation that my voice is distinct from that of "the people". I took a cursory look at the EXP rewards thread that you mentioned and counted 6 people in favor and 15 people against. How am I subverting democracy when the decision was made based on more than 70% of the vote?

Well, in the case that you mention, tone was the decisive factor. Those in favor of rewards gave substantial evidence backing their idea. Those opposed discussed the tradeoffs or other potential middle ground ideas. Your tone was alarmingly totalitarian: (paraphrasing) "No, this will not happen, case closed."


I also feel I must correct any sort of misperception that "the administration" believes that judges are highly replaceable; I know that judges are highly irreplaceable. While there is no shortage of writing talent on the site, finding someone with the unique trifecta of time, writing ability, and teaching ability is no easy task. Judges are important to the site indeed. However, your proposal seems to leave us worse off.

The inconsistencies that I noted before still seem to be there. For example, you assert that we should make judging a more prestigious position by having only a handful at a time, but then go on to suggest that our backlog is too large. Do you believe that by shrinking the judging staff, threads will be judged faster?

In short, yes.

Currently there is very little impetus to be a judge; there are a few who do it because they -love- to do it, and they move at a fair clip. This is also a stance which is compromised by the fact that the enormous apparent backlog is not accurate; that would not lend one to believe that the current system worked at all.

I think that the Judge spots should be viewed as prestigious rewards again, and not just something a few people do out of the goodness of their hearts. Perhaps there is a better practice to reestablish this worth, but I can't think of one.


You further assert that the judging position is undervalued by the administration, but also recommend that judges not pulling their weight be immediately replaced. Wouldn't it be contrary to the judges-are-irreplaceable philosophy if we canned every judge the moment he or she got busy or burned out?

Not necessarily. A reasonable quota could be installed to objectivize the position, something which could be easily attained. When that isn't reached, the judge could be replaced without arguement; you aren't replacing based on technical skill but rather output in the best interest of the site.

If we only had 4 or 5 judges, then (hopefully? theoretically?) there would be several current, competent judges hoping for a spot. These are people we know can do the job, and would have the key element that appeared to be (but apparenly is in fact not) lacking - desire to judge.


Right now we run on a rotating system that you pointed out: judges swoop in, judge a bunch, get tired, and vanish for a while before later coming back, judging a bunch, and vanishing again. It's certainly not the ideal system, and it perhaps may not even be a good one, but so far it's working better than any other method we've experimented with. Further, it inherently grants respect for the judging position by understanding that it's tough work and that breaks are a necessity, thus the job will be waiting for you when you're ready to come back. I really don't see how mass-firings would add "prestige" to the role, nor do I see how this amorphous "prestige" will make judges more committed to it.

I won't pretend to be a champion of human motivation, but my thought was that the "grass-is-greener" prediliction we share would make judging more attractive when the responsibility is more difficult to attain. The idea was that from the perspective of the position, decrease supply to drive demand.

Perhaps the "amorphous prestige" wouldn't motivate commitment, but I would expect it would increase attraction to the role. It would certainly aid in the cases wherein judges felt unappreciated.


It could be your point that judges burn out because they're not appreciated. It's a reasonable thesis. However your solution is predicated on the assumption that all judges burn out because they're not appreciated. Certainly some do, but others stop judging for reasons unconnected to their level of appreciation. Your solution may raise prestige, but also exacerbates other reasons judges may have to quit. In the end, I don't see how we would be better off.

The first part is true; this would do nothing to aid in the judges that burn out due to lack of time, life changes, etc. I counter that there is -nothing- that can be done to aid that problem. Perhaps the overall aging of Althanas leads to an inevitable decline in activity and the need for fresh input (as we have seen in the judging role with relative newcomers doing very well). Additionally, newer players are more likely to be motivated by "amorphous prestige" than those jaded by an intimate knowledge of both all the main players and the system itself.


Perhaps I'm misunderstanding your position again, and I hope that I am. Behind your philosophies, there seem to be some solid substantive solutions. I fear, though, that if you are basing your criticism on what you perceived to be a huge backlog that we (thankfully) do not actually have, that your ideas may end up being shortsighted. As you mentioned, your call for "overhaul" is now unnecessary, but I would still like to hear your ideas for improvement if you have them.

Though really, if we were actually a year and 100 threads behind, don't you think there would be a lot more threads like this one?

I think that since the system in place works enough to keep the backlog down, then it makes sense to leave it alone and not risk fouling up something that works, even if the mechanism is awkward.

And I thought that the two threads WAS in fact low, but then if there were dozens of threads redundant to the two threads discussing proposed change, they would likely have been groomed or redirected to productive threads anyway.

[Canen Darkflight]

If I may throw my twopenneth in here a moment...

The issues with having the judging role as a symbol of presitige, and something to honour, is that currently most evidence points to the role as something that is often thankless, more often than not time consuming and certaintly difficult to do. Most people would probably be turned off the role given these factors.

But from a practical point of view, the member base of Althanas need a small base of competant judges to motivate and drive members to improve and carry on, giving them a fair and accurate estimate of their work. I honestly believe that whilst I understand what Storm is saying, trying to turn the role of a judge into a sort of reward (if that is my correct perception of his comment) is a bad idea. We need tried and tested judges to keep this place running.

I know that there are a select number of judges on here that as a member of the community, I can put my trust in to do a good job. Perhaps the public should vote for the judges they believe are doing the best job via an election, or something? Like a select judging panel, a central focal point?

[Skie and Avery]

Perhaps the public should vote for the judges they believe are doing the best job via an election, or something? Like a select judging panel, a central focal point?

Doing that would most likely come down to a popularity contest, with people voting for judges who had given them the best scores in the past, or judges that they were friends with and wouldn't want to see sacked.

[Call me J]

I'm sorry Storm, but since when are you capable of viewing the modcp? What you're likely referring to is the Judging Requests forum in the Mod section, and I'm not entirely sure those threads are affected by what we do in the CP. Especially now that we've gone through an update of the system.

There, really, is no backlog. A couple of threads by Godhand are all that remains, and that's only because he stubbornly requested Shyam (that alien you spoke of who apparently can read things at light speed, but has been gone, except for a brief visit, for MONTHS). We're doing fine.

Shyam has left the building. Rumor has it he laughed at danger and broke all the rules one too many times and now Florida is after him.

Mary, there is no hope for us!

IMO the problem with judging is simple: demand exceeds supply, and there is nothing that can be done about this, outside of paying judges genuine cash.