Grader Comparison Dashboard
Analyze grader statistics and score divergence
Grader Overview
| Grader | Claims Reviewed | Total Scores | Avg Score | Std Dev |
|---|---|---|---|---|
|
L
lizzy
|
19 | 190 | 4.21 | 1.27 |
|
B
Bao
|
15 | 150 | 4.19 | 1.09 |
|
T
Test
|
2 | 2 | 3.5 | 0.71 |
|
J
Jawand
|
0 | 0 | 0.0 | 0.0 |
|
R
Richard
|
0 | 0 | 0.0 | 0.0 |
|
R
Rob
|
0 | 0 | 0.0 | 0.0 |
Inter-Rater Reliability (Lizzy vs Bao)
Score Divergence (2+ Point Disagreements)
BH
1564979706
MODERATE
1 divergent dimension
Rationale:
it does not understand this should be a simple, routine injury
AB
3499234442
MODERATE
5 divergent dimensions
Rationale:
- Minor severity instead of moderate since no surgery involved and just an ankle sprain. - Claimant's occupation is clearly a student but industry risk category is healthcare outpatient - Adjuster notes mentioned clear liability, but litigation risk rationale mentioned disputed liability leading to misclassification - Since litigation risk was misclassified, it leads to settlement likelihood and management complexity to be misclassified as well.
Rationale:
LLM classified claimant industry risk category as healthcare outpatient instead of education.
Rationale:
it was able to appropriately use the RICE definition
Rationale:
the claim was talking about a high ankle sprain which was relevant to the unstructured data
AJ
3628150755
MINOR
1 divergent dimension
Rationale:
it kind of is making up the price i think
AG
4726770034
MAJOR
2 divergent dimensions
Rationale:
i think it's messing up some of the prices or maybe I am not understanding them
Rationale:
it mentioned there was none for safety which i don't know is true since there was a safety harness, it just was not properly secured, so I don't know if there is a better term for this.
AK
5908806823
MODERATE
3 divergent dimensions
Rationale:
it understands everything, but like for this example, he fell when playing soccer, so I don't think there is any safety that should be involved and the LLM said unknown, and I wonder if there is a better category for this since there really isn't any safety stuff that can be included in this, it was just an accident, like he fell
Rationale:
the LLM may be hallucinating on the cost,
AB
6165669449
MODERATE
2 divergent dimensions
Rationale:
it did not include the safety harness in safety evaluation, which I think may change the category from none to something else. It's not that she wasn't using safety precautions, she just accidently fell.
Rationale:
it did not mention anything about how she feels like every day is reminder of what could have been with her wrist mobility. I don't know if we see this as important, but could it be a reason that she chooses to sue?? also, they left out the safety harness in the safety evaluation, so I don't know if that may be considered for something.
BC
6277819370
MODERATE
6 divergent dimensions
Rationale:
- Injury severity should be minor - Medical complexity should be simple - Medical complexity should be fast resolution - Risk level should be low - Experience modifier impact should be neutral
Rationale:
a lot of the moderate category should be simple and low, so it is not understanding that this is a low and easy injury compared to others so I don't think it understands that
AT
6478583886
MODERATE
1 divergent dimension
Rationale:
it did not leave out anything important, just made up some stuff
AN
6744942349
MODERATE
3 divergent dimensions
Rationale:
- Expected development pattern should be fast resolution because claimant recovered <6 months, clear liability, and returned to full duty. LLM mentioned "expected to return to full duty soon" - For industry risk category, claimant occupation is marketing specialist not healthcare outpatient
Rationale:
it left out some important points that i think would change some categories to score less harsh, like it should be a simple accident because it was a sprain with PT only, and I don't know if the model picked up on that.
Rationale:
it left out that the surface was dangerous for the safety variable, and it had the type of accident set to moderate when this should really be simple.
BC
6775379108
MODERATE
1 divergent dimension
Rationale:
it got the cost wrong a couple of times, so it's not interpreting something correctly because this is a pattern now
AK
7114747163
MODERATE
1 divergent dimension
Rationale:
- Per settlement adjuster notes, total settlement recommendation is approximately $32,600 instead of $118k so ultimate cost category should be 25-50k.
Rationale:
this one has a couple hallucinations, cost and safety procedures
AG
7909573436
MAJOR
1 divergent dimension
Rationale:
LLM mentioned ultimate cost prediction of $118,000 but no where to be found in settlement adjuster notes. Per settlement adjuster notes, the recommended settlement authority is up to $25,000.
Rationale:
the 118k is made up unless I am missing it.
Claims Overview
| Claim | Lizzy | Bao |
|---|---|---|
|
AB
3499234442
|
5
|
4.2
|
|
AB
6165669449
|
4.2
|
4.7
|
|
AG
4726770034
|
4.6
|
4.8
|
|
AG
7909573436
|
4.5
|
4.8
|
|
AJ
3628150755
|
4.5
|
4.6
|
|
AK
5908806823
|
4.1
|
4.4
|
|
AK
7114747163
|
4.1
|
4.2
|
|
AN
6744942349
|
4.3
|
4.3
|
|
AS
1653216458
|
3.8
|
3.8
|
|
AS
6924817947
|
3.8
|
3.8
|
|
AT
6478583886
|
3.8
|
3.4
|
|
BC
6277819370
|
4.4
|
3.3
|
|
BC
6775379108
|
4.2
|
4.3
|
|
BF
9399212680
|
5
|
4.6
|
|
BH
1564979706
|
3.7
|
3.6
|
|
BH
2088482200
|
3.8
|
—
|
|
BH
4462291667
|
3.6
|
—
|
|
BP
8794084622
|
4.8
|
—
|
|
CA
3950946063
|
3.8
|
—
|
|
CH
9343912483
|
—
|
—
|