STAR Awards Judging – Rules and Guidelines August 2018

Creating the application

Advice we give to applicants when they are creating Star Awards Applications:

1. Create application(s) as if they were presenting to the CEO of ANOTHER company – it should have that level of polish and just the key insights needed.
2. Everything should be clear and concise – all graphs should be well-labeled, all acronyms should be clearly explained – the reader shouldn’t have to guess at what the content means or why the content is there.
3. Tell a story about innovation that is focused on a topic (category). Every good story has a beginning, middle and an end. They should open with the beginning and tell us why they are applying and what they are going to tell us, and then they should tell us the meat of the story by clearly explaining their innovation and outcomes. Optional is lessons learned and next steps which is a great way to end and summarize their story.
4. Key points: Look for details in the business impact and customer impact sections. Applicants should provide specific examples ($, % impact, etc.). The more specific the evidence is – the higher the score. General observations typically get a low rating.
5. In the application template for each category, applicants are provided suggested points to address; this is not a comprehensive list and not intended to limit their response or how they tell the story in any way. We want the applicants to have freedom of expression.
6. There is a thirty page limit for each application (2 required (title & admin) pages plus 28 pages of content for a total of 30). Applicants that exceed the page limit may be given deducted scores.
7. There are five (5) criteria that are given equal weight in scoring:

• The Pacesetter Factor of Your Application
• Business Impact: How Has This Helped Your Company?
• Customer Impact: How Has This Helped Your Customers?
• The Cohesiveness of Your Application
• The Effectiveness of Your Metrics 

Numeric Ratings

Numeric rating scores - score from 5 (best) to 1 (worst). The best analogy is that these are like letter grades, A, B, C, D, and F. We only use whole numbers (no decimals).

Start your scoring in the middle.

Rating 3 – a ‘C’. If they did a good job of adequately answering the question and you believe that they hit all of the salient points – the starting score should be a 3 (or a C).

If you look back and see that they are doing some innovative things that are over and above what you consider to be the industry norm – then bump them up and give them a 4 rating (or a B).

If they are doing something truly unique that is really best in class in the industry – or something that you think is extremely innovative – then rate them a 5 (or an A).

If the innovation and/or data is lacking - or they are doing something that was leading edge many years ago – give them a 2 rating, or a D.

If they really missed the mark, rate them as a 1.

Packaging

Some people are great at packaging an application and making it look really professional. This is important – and it always makes an impact on us – but at the end of the day it’s the content that matters the most. Look at both of those things – if two companies have comparable content but one really has a much tighter story and it is more professionally packaged – ultimately give them a higher rating. But the most important thing to focus on when rating these IS the content.

Feedback

The companies that apply for STAR Awards get a huge amount of value out of the judges’ feedback. Tell them why they got that rating. If things are innovative or a solid best practice – talk about that. If they are not – mention that. For every application (overall) and each of the five (5) criteria scored – there’s a section for comments. Even if it is very brief, please DO PROVIDE FEEDBACK (you actually are required to do so in order for the final save on the STAR Awards platform).

For each application, judging scores and comments are compiled and provided in a report. All judges remain anonymous. Companies don’t know who is on the judging panels (only that members of the advisory boards are the judges).

Judging Calls

Each STAR Awards category will have a scheduled 30 minute judging conference call. In advance of a call – each of the TSIA leaders will have consolidated all of the scores/feedback. To prep for the call – be prepared to do the following:

1. Talk about your top 3 applicants in that category.
2. Talk about your ratings (who came in first, who came in second, who came in third)
3. Talk about the first company and why you rated them number one.
4. Determine if 2-tiers of winners will be awarded based on applicant company sizes

The TSIA lead will capture how the rankings went from tabulating all of the score sheets – and then will give you the overall rankings and scores from all of the judges. The goal of the call is to drive to consensus and assign finalist(s) and the winner. Usually the overall top score becomes the winner – but there is an opportunity here to change other judge’s minds.

Rules and guidelines to determine winners and finalists

The judges must review each individual application in a STAR Awards category and provide numeric scores and feedback comments for each of these individual applications.

Enter numeric scores for each section/criteria with 5 as the highest score and 1 as the lowest score.

The following options are available to the Star Awards judges:

• The judges may declare one clear “winner” in a category. There may or may not be other applications that deserve to be recognized as “finalists”.
• The judges may declare up to two finalists in a given category. To be a finalist – our guidance is that their total score must be within 10% of the winning application score. It is at the discretion of the judges as to whether or not there are any finalists.
• The judges may declare “no winners or finalists” in a category if none of the applications received deserve to be recognized. This situation would only occur if the judges believe that there are no applications that represent the best-in-class in the industry.
• In categories with larger numbers of applicants, judges may elect to award two separate winners: one for the enterprise-sized companies and one for the mid-lower sized revenue companies. Finalists for each level may also be named. This will be determined on a case by case basis and may not occur in all or even most categories.

Finalists can be recognized

It is common to have a situation where there are one or two companies in a category that are very close to the winner(s) in scores and we want to recognize them. For example - if the overall average for the winning company is 4.5, then a runner up would have to be between 4.05 - 4.49. It should be within 10% of the winning score. If the next closest company had a 3.5 average, even though it might be a great application - typically we wouldn’t recognize it because the gap is too far. The judges do have the flexibility to make an exception to this rule in special circumstances if special recognition is warranted - and those exceptions will be decided on the judging call.

Once again, TSIA thanks you for your time and attention on the 2017 STAR Awards competition!