Clinical Trial Radar

Safety assessed by outcome evaluator on an ordinal scale (1,2,3)

Safety will be assessed using an ordinal scale ranging from "acceptable without further changes", through "acceptable with minor revisions", to not "unacceptable in its current form" ( a combination of either absence of important information or presence of incorrect/hallucinated information)

Assessment at one time point - hospital discharge (up to 5 days)

Safety of the edited and unedited LLM generated course summaries evaluated on an ordinal scale (1,2,3)

Safety of LLM-generated, resident-edited and un-edited hospital course summaries compared with standard specialist-written documentation. Safety, will be assessed using an 3 categories ordinal scale ranging from "acceptable without further changes" (best outcome), through "acceptable with minor revisions" (uncertain outcome), to "not unacceptable in its current form" (absence of important information or presence of incorrect/hallucinated information; worst outcome)

At hospital discharge (up to 5 days)

Secondary outcome: Content validity across the following predefined domains (5C).

Secondary outcomes measures: detailed evaluation of quality across the following predefined domains (5C): completeness, conciseness, cohesiveness, absence of critical errors (including hallucinations), and cultural and linguistic fidelity. Individual domains will be separately assessed using a 5-point Likert scale, (1 - complete absence of agreement, 2 - partial absence of agreement, 3 - neutral agreement, 4- partial agreement, 5- complete agreement)

At hospital discharge (up to 5 days)

Time to complete hospital summary (seconds)

Workflow-related secondary outcomes include time efficiency of hospital course documentation when using the LLM-generated workflow compared with standard practice. The outcome measure will be the time taken to complete the hospital course summary (seconds), from initiating discharge to "ready to signing".

At hospital discharge (up to 5 days)

Generation stability assessed on an ordinal scale (1,2,3)

Generation stability will assess the consistency of LLM generated unedited hospital course summaries when the model is applied repeatedly to identical source clinical documentation. Generation stability will be evaluated by the same adjudicator on the scale of 1 - stable, 2 - acceptable variation, 3 - unacceptable variation

From hospital discharge to 30 days after hospital discharge

Adoption. Percentage of hospital discharges where the LLM generated course summary was utilized.

After completing the randomized controlled trial, the adoption of the LLM-generated workflow will be evaluated over a one-month period. The outcome measure will be defined as the proportion of eligible discharges for which LLM-derived summary generation was utilized.

From hospital discharge to 30 days after hospital discharge

Inter-rater reliability. Percentage agreement between two adjudicators of a hospital course summary on safety and content validity ratings.

Reliability-related outcomes will assess the consistency of expert evaluations of hospital course summaries. Percentage will quantify the agreement between evaluators on safety and content validity ratings.

From hospital discharge to 30 days after hospital discharge

Intra-rater reliability. Concordance of safety and content validity ratings between two evaluations of the same hospital course and same adjudicator.

Intra-rater reliability will quantify the consistency of repeated evaluations by the same evaluator based on the concordance of safety and content validity ratings (percentage of agreement)

At hospital discharge (up to 5 days)

Temporal stability (test-retest reliability) - concordance between the ratings of the same adjudicator after washout period.

Temporal stability of evaluator judgments will be assessed through repeated evaluation of identical hospital course summaries by the same evaluator after a predefined 30 day washout period (test-retest reliability). Outcome measures will be based on the concordance of safety and content validity ratings (percentage)

30 days after hospital discharge