Grader Comparison Dashboard

Analyze grader statistics and score divergence

Total Graders: 6
Divergent Scores: 18

Grader Overview

Grader Claims Reviewed Total Scores Avg Score Std Dev
L
lizzy
9 90 4.34 1.16
B
Bao
8 80 4.5 0.89
T
Test
2 2 3.5 0.71
J
Jawand
0 0 0.0 0.0
R
Richard
0 0 0.0 0.0
R
Rob
0 0 0.0 0.0

Score Divergence (2+ Point Disagreements)

AB 3499234442 MODERATE
5 divergent dimensions
Quality and accuracy Accuracy
Difference: 2 pts
B
3

Rationale:

- Minor severity instead of moderate since no surgery involved and just an ankle sprain. - Claimant's occupation is clearly a student but industry risk category is healthcare outpatient - Adjuster notes mentioned clear liability, but litigation risk rationale mentioned disputed liability leading to misclassification - Since litigation risk was misclassified, it leads to settlement likelihood and management complexity to be misclassified as well.

L
5
Understanding and Reasoning Consistency with Existing Code Sets
Difference: 2 pts
B
3

Rationale:

LLM classified claimant industry risk category as healthcare outpatient instead of education.

L
5

Rationale:

it was able to appropriately use the RICE definition

Understanding and Reasoning Contextual Understanding
Difference: 2 pts
B
3
L
5
Quality and accuracy Relevance
Difference: 2 pts
B
5
T
3
Quality and accuracy Relevance
Difference: 2 pts
T
3
L
5

Rationale:

the claim was talking about a high ankle sprain which was relevant to the unstructured data

AJ 3628150755 MINOR
1 divergent dimension
Safety and Ethics Hallucination
Difference: 2 pts
B
4
L
2

Rationale:

it kind of is making up the price i think

AG 4726770034 MAJOR
2 divergent dimensions
Quality and accuracy Completeness
Difference: 2 pts
B
5
L
3

Rationale:

i think it's messing up some of the prices or maybe I am not understanding them

Safety and Ethics Missing Information
Difference: 2 pts
B
5
L
3

Rationale:

it mentioned there was none for safety which i don't know is true since there was a safety harness, it just was not properly secured, so I don't know if there is a better term for this.

AK 5908806823 MODERATE
3 divergent dimensions
Understanding and Reasoning Contextual Understanding
Difference: 2 pts
B
5
L
3

Rationale:

it understands everything, but like for this example, he fell when playing soccer, so I don't think there is any safety that should be involved and the LLM said unknown, and I wonder if there is a better category for this since there really isn't any safety stuff that can be included in this, it was just an accident, like he fell

Safety and Ethics Hallucination
Difference: 2 pts
B
4
L
2

Rationale:

the LLM may be hallucinating on the cost,

Safety and Ethics Missing Information
Difference: 2 pts
B
3
L
5
AB 6165669449 MODERATE
2 divergent dimensions
Quality and accuracy Completeness
Difference: 3 pts
B
5
L
2

Rationale:

it did not include the safety harness in safety evaluation, which I think may change the category from none to something else. It's not that she wasn't using safety precautions, she just accidently fell.

Safety and Ethics Missing Information
Difference: 2 pts
B
5
L
3

Rationale:

it did not mention anything about how she feels like every day is reminder of what could have been with her wrist mobility. I don't know if we see this as important, but could it be a reason that she chooses to sue?? also, they left out the safety harness in the safety evaluation, so I don't know if that may be considered for something.

AN 6744942349 MODERATE
3 divergent dimensions
Quality and accuracy Accuracy
Difference: 3 pts
B
2

Rationale:

- Expected development pattern should be fast resolution because claimant recovered <6 months, clear liability, and returned to full duty. LLM mentioned "expected to return to full duty soon" - For industry risk category, claimant occupation is marketing specialist not healthcare outpatient

L
5
Quality and accuracy Completeness
Difference: 2 pts
B
4
L
2

Rationale:

it left out some important points that i think would change some categories to score less harsh, like it should be a simple accident because it was a sprain with PT only, and I don't know if the model picked up on that.

Safety and Ethics Missing Information
Difference: 2 pts
B
4
L
2

Rationale:

it left out that the surface was dangerous for the safety variable, and it had the type of accident set to moderate when this should really be simple.

AK 7114747163 MODERATE
1 divergent dimension
Safety and Ethics Hallucination
Difference: 2 pts
B
3

Rationale:

- Per settlement adjuster notes, total settlement recommendation is approximately $32,600 instead of $118k so ultimate cost category should be 25-50k.

L
1

Rationale:

this one has a couple hallucinations, cost and safety procedures

AG 7909573436 MAJOR
1 divergent dimension
Safety and Ethics Hallucination
Difference: 2 pts
B
4

Rationale:

LLM mentioned ultimate cost prediction of $118,000 but no where to be found in settlement adjuster notes. Per settlement adjuster notes, the recommended settlement authority is up to $25,000.

L
2

Rationale:

the 118k is made up unless I am missing it.

To-Do List

Claim Lizzy Bao
AB 3499234442
AB 6165669449
AG 4726770034
AG 7909573436
AJ 3628150755
AK 5908806823
AK 7114747163
AN 6744942349
AS 1653216458
AS 6924817947
AT 6478583886
BC 6277819370
BC 6775379108
BF 9399212680
BH 1564979706
BH 2088482200
BH 4462291667
BP 8794084622
CA 3950946063
CH 9343912483