Judging work in progress bars...

[Shinsou Vaan Osiris]

EDIT: The subject has moved on from progress bars to a PM system. Original post below.

I don't know how feasible this is but one function I would quite like to see implemented on the site would be some sort of "progress bar" on either how far through a thread has been judged or, if it hasn't been started, how long the staff predict they will take to get to it.

I don't know how anyone else feels about it and i'm sure there will be members of staff rolling their eyes at me and sighing deeply, especially those who are working very hard to keep everything up to date.

Of course this is not intended to be a subtle dig at timekeeping as the judges work very, very hard at getting everything done. It's more a suggestion to help give some sort of visual representation to the members of their thread's progress through the system.

Thoughts? Constructive ones only, though, please.

[Bard]

Well just to show how I'd do something like this, for a workshop, mine would look something like this. Initial lesson through, completed (date), second listen through, assessment for noticeable grammatical errors, review begun, first secon, second section, third section, completed. Thus, a workshop might take me two to three days to complete, with the various stamps being reported as that stage is completed. At present I use this system in my personal notes only. Since the topic has been brought up, thought I'd mention it.

[Karuka]

I think he means for mod-judged threads, Bard. With workshops, you just take what you get when you get it.

I'm opposed to the progress bars, personally. I feel like it would create more pressure on our volunteer staff and increase nagging. Not good. It would be nice, though, if the judge sent a pm to all parties when they started a thread and gave a rough estimate on when they thought they'd have it done.

Like "holy geeze, guys, this is a novel and I'm a bit busy these days. I've got this thread, but don't expect judgement for 2-3 weeks." Or "this is a quick little thing and I'm off today. Barring something weird like my cat exploding, expect judgment by lunch time, my time."

[Gnarl & Root]

We were just talking about this on Althanas chat.

While from the writers point of view it's not a bad idea, it's nice to see how things may be getting on for sure, or see your thread's score's gonna be unveiled soon.

However I also spoke with some of the mod's/judges and I worry about the pressure it would potentially put on them. I mean, seeing something at like maybe 50% or something for a while I could be nagging asking why has it stopped? Or why has it not been started yet? Nag Nag Nag.

Waiting times have always sort of been a factor, I'm kinda used to it, so it hasn't bothered me. But seeing a sort of progress bar could encourage me to want to question more. Or look at it the other way round, the Judge's will know there progress is being monitored constantly.

So i'm not so sure it's the best idea. But Maybe that comes down to how the Judges feel about it.

[Bard]

It was meant as an example, a sort of "if it were done, how it might be implemented," sort of thing, nothing more.

[Shinsou Vaan Osiris]

Again just want to point out that this wouldn't be a mechanism for trying to hurry judgements. Just literally a simple "rough idea" representation of how far through a judgement is.

The judges here work hard and are very patient, have lives, and the last thing they need is nagging.

As mentioned above maybe even a simple "It's en route, don't expect anything earlier than a week" would be effective, or a "thanks for posting up a million pages of text, it's going to be a month before I can sort that".

No one minds waiting as long as they know that the judgements are "on their way".

[K-Zu-Ziro]

i'd like to make a suggestion inspired by your original idea. rather than a progress bar for individual threads, a feature displaying the current TOTAL workload and overall average wait time for a response would be really cool. it would be really simple, just three stats.

1. x amount of threads awaiting judgement.
2. y average response time.
3. z oldest judgement request in the system.

it could go somewhere in the header area of the website. just so everybody has some perspective of where you are in the queue when you initially submit a judgement. think of times you've submitted a ticket on some kind of website to some kind of company. the best experience i've had is where they tell me, the average wait time for a response is "blah blah days" or "blah blah hours" i.e. actively managing expectations.

but about your original suggestion i do think there might be some real difficulties in coming up with a way of a progress bar for processing a submitted thread. as mentioned by bard, you'd have to assign like a standard order to doing it and then allocate %s to each part. when i'm guessing different mods do it different ways? and then just because somebody does 90% of the work on thursday doesn't really tell you much about when the last 10% will be done. it might be done on friday or it might be done two weeks later.

if you could solve some of those problems then it would be a solid addition to improving the transparency of the administration of this community etc.

[Logan]

I think the problem would be as Karu stated, even if unintended it would put unnecessary pressure on the judges. Which in turn would lead to more hastily completed judgements.

I will, however, say this could be done for No Judgements, though I am unsure the effectiveness of it from a member's point of view since those should take no more than 1-2 days at most to be no judged as it is mostly just paperwork, so to speak.

The actual judgement process, however, isn't really able to tracked and most judges have their own personal process for it. There wouldn't be a fair way to assess a judge's progress on a judgement accurately enough, so any sort of percentage system would be just simply for show -- which in my opinion is even more pointless.

It would be the equivalent of writers having a system that tracks how close to 3000 words they are so they know when they get close to the breaking point for what would traditionally constitute a thread. We could arbitrarily say 3000 words is a thread, and then writers would mostly find themselves writing to that bar instead of just plain writing to the story's ending point. Does that make sense?

[Shinsou Vaan Osiris]

I think the problem would be as Karu stated, even if unintended it would put unnecessary pressure on the judges. Which in turn would lead to more hastily completed judgements.

I will, however, say this could be done for No Judgements, though I am unsure the effectiveness of it from a member's point of view since those should take no more than 1-2 days at most to be no judged as it is mostly just paperwork, so to speak.

The actual judgement process, however, isn't really able to tracked and most judges have their own personal process for it. There wouldn't be a fair way to assess a judge's progress on a judgement accurately enough, so any sort of percentage system would be just simply for show -- which in my opinion is even more pointless.

It would be the equivalent of writers having a system that tracks how close to 3000 words they are so they know when they get close to the breaking point for what would traditionally constitute a thread. We could arbitrarily say 3000 words is a thread, and then writers would mostly find themselves writing to that bar instead of just plain writing to the story's ending point. Does that make sense?

Absolutely, Logan mate. No-one wants to put additional pressure on the judges, not by a long shot. We all have to remember that these people are volunteers as well.

However, as an alternative to my original suggestion, how do you feel about a judge perhaps PM'ing the participants of a thread when it has been claimed as a courtesy to say:

"Hi! Your thread is being judged, I expect as a rough guide for it to take roughly [insert time here] as I have a few things on, but just to let you know I've started it."

Or

"Hi! Your thread has been claimed but there is a little bit of a backlog with threads, so we estimate it might be a week before we can make a start".

This way, the Judge is in control of their schedule and there is no need for them to feel pressured. The member knows that the thread is being looked at and what timescale they can expect on a rough basis (experienced judges should be able to look at a thread's length and post lengths and have some sort of idea, even if it's only a guide, as to how long it will take).

Everyone is a winner.

I don't think that would be particularly unreasonable as it also keeps it out of the public domain. What do you think?

The objective of my suggestion is more to give the users a "rough estimate" of a timescale based on the judge's personal circumstances and, of course, site backlog.

That's the key bit. If a judge just sends a quick pm with a bit of time guesswork on it (even if its two to three weeks), then the participants know where they are up to and won't have to chase and chase.

So, yeah. I may be pissing in the wind with all this, I don't know. Judges will have their own opinions on whether this is a good idea or not, but I also think it is important for them to look at it from a member's point of view as well as theirs, and members to also understand that their judgements won't be done overnight.

[Cards of Fate]

I kinda like the idea of a courtesy PM. Its pretty low pressure and we could literally just have a copy paste message.

[Sulla]

My biggest issue with a progress bar would be how to quantify it into numbers. I don't know what half complete would be, because the work can vary so easily. We could come up with some arbitrary markers - finished reading it, judging action, scoring, etc - but those would be difficult to mark on the bar given every judge's different process.

Judges PMing writers, especially for full judgments, sounds like a much more sound idea. It wouldn't create the same amount of pressure as a progress bar and it would help put a face to the writing their reading, perhaps motivating them to further push it along without the same amount of stress. And, of course, it shows some availability to communicate between community and staff, and that's always a good thing, before and after judgement, in case any questions arise.

[Shinsou Vaan Osiris]

Judges PMing writers, especially for full judgments, sounds like a much more sound idea. It wouldn't create the same amount of pressure as a progress bar and it would help put a face to the writing their reading, perhaps motivating them to further push it along without the same amount of stress. And, of course, it shows some availability to communicate between community and staff, and that's always a good thing, before and after judgement, in case any questions arise.

Exactly this.

The progress bar is unworkable but if everyone is willing to support the PM idea, it will work just as well, if not better for all involved.

[Bard]

Very good idea Sulla.

[Visla Eraclaire]

I actually advocated something similar when I was a judge. While an actual progress bar or any kind of technical work is an unnecessary expenditure of resources, some level of status isn't too much to ask for.

I imagine the judge forums work much like they did years ago. When a thread is submitted, a post shows up in the forum, a judge claims it, and then they submit their judgment when they're done.

I don't think there's anything to be gained by requiring that judges give status updates or pressuring them, but at the same time, obfuscating the system entirely isn't super helpful to anyone. When I was a judge, I could at least see when my thread was claimed and how many other threads that judge had claimed.

I don't think it would hurt for a judge to post in the public thread that they have claimed a thread and what number it is in their judging queue. This would prevent one thing that I saw a lot and was myself guilty of at the time I judged, letting a thread slide behind. Threads really should be judged on a first come first served basis. A lengthy or complex thread could easily get passed over for a while in terms of people claiming it and then once claimed the judge was very likely to read a bit, take a break, judge a shorter thread, read a bit, take a break, etc. etc.

Volunteer judges don't need people policing their work habits or harassing them, but I think showing that you claimed it and giving people a "You are currently behind 2 other threads" notification is a courtesy that could go a long way in staff/poster relations.

Now, if things work differently now, this could be a bunch of nonsense, but I'll take the risk of looking stupid. It wouldn't be the first or second time.]

EDIT: The suggestion above of PMing sounds fine, but I think actually posting in the thread would give a degree of accountability and ensure that this was actually happening. Once again, not trying to be the judge police, but if you are supposed to give a PM and you don't, only one person knows. If you're supposed to post and you don't, it enforces itself.

[Logan]

It's less of a claim oriented system and more of assignment system now. There's still some layers of claiming, for sure, but it's less so than it was before.

As for the PM system, what I think might be more beneficial but would require a bit more back-end work (oh poor Dirks) would be if when a thread was assigned or claimed for a judgement (surely there's a vb hack for this type of thing), it would notify (automatically) the user who submitted the judgement with an auto-generated PM or notice stating their thread was being reviewed by "username". This would be the equivalent of the PM, but place a little less necessity on it falling to the judges to add another step to an already tedious and stressful process.

Thoughts?

[Shinsou Vaan Osiris]

It's less of a claim oriented system and more of assignment system now. There's still some layers of claiming, for sure, but it's less so than it was before.

As for the PM system, what I think might be more beneficial but would require a bit more back-end work (oh poor Dirks) would be if when a thread was assigned or claimed for a judgement (surely there's a vb hack for this type of thing), it would notify (automatically) the user who submitted the judgement with an auto-generated PM or notice stating their thread was being reviewed by "username". This would be the equivalent of the PM, but place a little less necessity on it falling to the judges to add another step to an already tedious and stressful process.

Thoughts?

The only problem with your idea is that it doesn't allow the judges any room for estimating a timescale based on their personal workload and outside factors, therefore affecting communication. However, as a notification system it's brilliant.

I also quite like the idea Visla posted about posting in the thread but as long as the objective is achieved it doesn't matter which method is used.

[Visla Eraclaire]

Save Dirks the backend work. If a judge is going to take a thread, they're going to spend a few hours with it. 2 minutes to say, "I'm judging this, it's behind 3 other threads, two are kinda long." is no biggie.

I don't think anyone wants to overburden judges, but we don't need to coddle them.

I'm curious how the assignment system is working. That was proposed before and I was against it then, but it had its pros and cons.

[Rayleigh]

I generally let people know via chat when I have claimed one of their threads. It takes only a second or two. I don't always provide a time-estimate (other than maybe "it is the first week of school, so I'll try to get to it next week"), but I would be happy to start doing that.

Honestly, it doesn't take a judge any time at all to send a PM or drop into chat. I would much, much rather we try to make that common practice than have some sort of bar. When I judge something, I do it on one sitting. My progress bar would say "zero progress," and an hour later, it would say "done." I don't think that it is worth the hassle. I do absolutely appreciate and support the idea of open communication between judge and writer when it comes to progress.

[Shinsou Vaan Osiris]

Thanks Sulla, Logan and Rayleigh. Good to get positive feedback on this from judges and moderators :)

[Logan]

What Shelby said is spot-on.

The PM process is fairly quick and painless, it is just a matter of making sure judge's are actively doing that extra step during their process.

[Max Dirks]

I actually secretly sampled a system like you guys are suggesting two years ago using a modified version of the Advanced Post Reporting System hack. The system would automatically send PMs when the status of an "issue" (in our case judgment) changed. Results were mixed. My sample judges often either forgot to update the system or too little time passed between updates (meaning they'd click "assigned" the day they completed the judgment), rendering it largely useless to players. The other downside is that it literally took over the post reporting system, meaning players couldn't report spam or inappropriate content. That doesn't mean a new system couldn't be created, it's just that I would prefer not to use the old one. I've attached a screen shot of what a player would see if looking at the system to this thread. It definitely looked cool.

Ultimately, I decided to give a variety of different judging methods with some fixed (but conservative) timeframes for judging. Giving you guys the freedom to determine the depth of review has definitely reduced pressure on our volunteer judges. Realistically though, I don't see a progress report system being particularly viable. Drum, every judge has a different process, so other judges might not update at the same things as you (I rarely re-read threads, for example). PMs are nice, but I wouldn't want to add anything that isn't automated. Argue they are easy all you want, but frankly I want judges focusing on judging rather than administrative matters. That's why I created the Head Judge position, after all. Matt, a general site counter is a cool idea, but I envision most people would submit a ton of full judgments when they saw the queue was low, ultimately resulting in a perpetual queue. Sorry guys!

[Bard]

Max if its easy, I'm probably not using it... that's a joke.

[Logan]

Pretty much anything put into place would add one additional step for judges no matter what. Either they take it on themselves to PM is one option, or they click a button once they grab or are assigned a judgement, is the other simple option. Either way, it is an additional step the judges have to take for each and every judgement.

I think so long as judgements are happening in a reasonable timeframe referenced by the varying levels of a judgement type (no judgement, basic, full, etc), then there really isn't a necessity for a judgement progress tracker. Like Dirks said, a big part of the tracking falls on the Head Judge to moderate, delegate and deal with any judgements, especially ones that lag beyond the aforementioned "reasonable timeframes" referenced in the judgement submission form.

[Shinsou Vaan Osiris]

Well at least there has been a proper, well rounded debate on the subject. Personally I do hope that the PM system is either implemented as a rule (unlikely I know) or adopted as habit, as I think it will do wonders for the member / judge relationship. On the other hand it won't ruin my life if it isn't.

I'm glad the staff got to have their say too.

[Philomel]

Here are my two cents as Head Judge.

At the current moment (especially currently with the giant back log as from Althanas Day) judgements are taking a long time to get through, as I am sure you have noticed. This is mainly down to the simple fact that a lot of our judges have other work on the site that they are doing (many are moderators in other areas). We try to get judgements done as soon as possible, on a two-week stretch once the judgement has been assigned, but this is not always possible, as I am sure you can understand.

Having a progress bar I think will put a lot of pressure on the judges themselves. Its something that will be staring them in the face and cause more stress when already there are enough things on their plate. We are working at the moment on speeding up judging in general, and getting more out, and I keep a constant eye on what is occurring and how people are doing.

The best thing, and possibly a very fine thing, would be the courtesy email. It would be fine to implement, and easy to do, with a email saying "your judgement has been assigned to/accepted by ..... who aims to have it to you by ....."

[Shinsou Vaan Osiris]

Here are my two cents as Head Judge.

At the current moment (especially currently with the giant back log as from Althanas Day) judgements are taking a long time to get through, as I am sure you have noticed. This is mainly down to the simple fact that a lot of our judges have other work on the site that they are doing (many are moderators in other areas). We try to get judgements done as soon as possible, on a two-week stretch once the judgement has been assigned, but this is not always possible, as I am sure you can understand.

Having a progress bar I think will put a lot of pressure on the judges themselves. Its something that will be staring them in the face and cause more stress when already there are enough things on their plate. We are working at the moment on speeding up judging in general, and getting more out, and I keep a constant eye on what is occurring and how people are doing.

The best thing, and possibly a very fine thing, would be the courtesy email. It would be fine to implement, and easy to do, with a email saying "your judgement has been assigned to/accepted by ..... who aims to have it to you by ....."

So is this something that is being discussed at any level at the moment?

[Philomel]

We're currently discussing other possible changes to the site. I can't reveal anything really at the moment, but certainly I can raise it as an idea in moderator forum.

I will work on a piece and possible ideas and ways to implement this over the next couple of days and get back to you, Shin. Then I can discuss it with the admins and super mods maybe, when they have time.