The aim of speech and language therapy interventions is to impact communication skills in everyday interactions. In order to measure this, rating scales (both self-rating and consultee-rating) are commonly employed in clinical practice settings. Many rating scales have been shown to have high validity when measured against language assessments, but less when compared to family or clinician ratings. This study aimed to compare the self-ratings of people with aphasia and their family members to performance in everyday conversation. Four people with aphasia and four non-brain injured participants collected naturalistic audio samples from their daily lives at home. These audio recordings were analysed for word count and number of turns used by the participant. All participants also completed an aphasia-friendly communication rating scale designed for the study. There was a great deal of variability evident when comparing ratings with data from naturalistic audio samples, across all participants in this study, whether diagnosed with aphasia or no brain injury. This calls into questions the reliability of using subjective rating scales as an accurate representation of everyday language use. However, given healthcare is delivered within a person-centred framework, a measure of client satisfaction remains undeniably valuable.