The Scottish Government

« Previous | Contents | Next »

Listen

The Independent Evaluation of 'Starting Well' Final Report

Part 2. Outcomes from Starting Well

2. Assessing Impact: the Quasi-Experimental Study

2.1 Introduction

An established literature suggests that home-based interventions delivered by trained health professionals can improve a range of outcomes for vulnerable pre-school children (Olds and Kitzman 1993; Brooks-Gunn et al 2000; Elkan et al, 2000; Bull et al, 2004). Most studies do not have sufficient numbers of participants to detect direct improvements in child health (e.g. lower morbidity and/or mortality rates) but show improvements in related factors, for example, quality of the home environment (Davis and Spurr, 1998), detection and management of postnatal depression (Holden et al, 1989) and improved rates of breastfeeding (Kitzman et al, 1997). In addition, long-term cohort studies show the benefits of home-visiting to be diverse and enduring for both mother and child (Olds et al,1997; 1998).

One acknowledged limitation of this largely American-based literature is its lack of direct applicability to the British context ¹⁶, specifically, its focus on specialist programmes that exist in the absence of a universal health-visiting service. The purpose of this evaluation component is not to establish the impact of a service relative to its absence but to explore the impact of an enhanced service ('Starting Well') on a group of families over the first eighteen months of the child's life relative to a group receiving the established (generic) service.

In this section, we first describe key features of the study design and characteristics of the recruited sample. This section (2.2) includes an exploration of sample representativeness using aggregate-level routine data on opt-ins and opt-outs. In section 2.3, we present the results of multivariate analysis of three health-related outcomes (quality of the home environment, extent of maternal depressive symptoms, child dental registration rates) and one measure of user-satisfaction (with levels of health visitor support) at both six and eighteen month assessments. In section 2.4 we discuss the implications of these findings.

2.2 Methods

2.2.1 Target populations

The intervention population was defined as all births visited by Starting Well health visitors between 01/06/01 and 31/06/02 within the project's strict geographical boundaries ¹⁷. Statistics on the total number of births were generated by the project team's interrogation of the study database. The comparison population was defined as all births ¹⁸ assigned to health-visiting teams working in the Northern Local Health Care Co-operative over the same time period. Comparison 'area' health visitors are attached to particular GP practices and, as they only visit families that are registered at that surgery, there is a strong but not defining geographic focus to their work ¹⁹. Consent to evaluation was lower than expected in the comparison area and consequently recruitment was extended to a further two health visiting teams in the west of Glasgow (Drumchapel and Clydebank) between 01/04/02 and 31/06/02. The total numbers of comparison area births was calculated by the NHS data provider ²⁰ from the interrogation of routine data sources. Table 2.1 shows the total number of births and opt-ins by group ²¹.

Table 2.1: Total number of births and opt-ins.

Just under 50% of eligible families opted into the evaluation, proportionally more so from the intervention area(s). This is perhaps unsurprising given the implicit incentive of receiving the enhanced service.

2.2.2 Procedure

Each opt-in parent/family received a baseline postal survey. This was sent out as soon as the consent form was received from the attending health visitor, in most cases, within two months of the child's birth. The survey covered: background maternal, household and area characteristics; maternal mental health and health behaviour; and attitudes towards parenting and current health-visiting service. Interpreters were made available to assist participants with no or limited English (n=21) and completed surveys were returned using a pre-paid envelope. Participants with overdue surveys were followed up by letter and phone. Further surveys were sent out to each participant when their child was six and then eighteen months old. Content focussed on mother-reported child outcomes and updates of maternal health, support and attitudes to the health visiting service.

In addition, each participant that could be contacted at six and eighteen months received a home visit from one of three trained research nurses who administered the HOME Inventory. The average interview lasted around one hour. At the end of the interview, the nurse administered several additional survey instruments (e.g. the Edinburgh Postnatal Depression Scale) and retrieved any incomplete or unsent postal surveys. Again, interpreters were available for non-English speakers.

Table 2.2: Returns per assessment by group

Note: COMP = comparison group; INT = intervention group

Table 2.2 shows the number of returns by assessment and group with totals expressed as percentages of sample and population. In data not shown, similar fieldwork completion rates were observed at both assessments across cohorts. Contact by research nurses at six months produced a better survey return rate than baseline postal methods, although this advantage was offset at eighteen months by attrition of subjects. The populations from which the cohorts are drawn are residentially mobile and at the time of writing, 73 (11.6%) of the original sample had either voluntarily withdrawn from the study (n=26) or moved house without leaving a forwarding address (n=47).

Table 2.3: 'Rich' six and eighteen months datasets as a proportion of sample and population

All data were collated and stored securely at the study offices, before being coded and entered. Table 2.3 shows the percentage of participants who completed all instruments at six months (n=359) and eighteen months (n=294). A further 93 participants were not seen at baseline but completed a brief retrospective survey at six months, covering basic child, maternal and household characteristics. For the purposes of this report, richness of data is considered to be more important than maximising the number of subjects; analysis therefore concentrates on the two smaller datasets ²².

2.2.3 Sample characteristics and representativeness

The preceding section showed that not all eligible families opted into the study and, of those who did, not all could be contacted at all assessments. This introduces two potential sources of bias, both of which must be understood in order to be able to generalise confidently from results. The first source of bias relates to recruitment and the possibility that there are systematic outcome-related differences between those who opted into the study and those who opted out. By definition, little or no information is available on opt-outs, however, by gaining ethical permission and signed consent to collect individual-level routine data on opt-ins (e.g. from the Child Health Surveillance system), the NHS data provider was able to 'subtract' these children from the known population of eligible births and generate limited aggregate-level comparative statistics on the remainder (i.e. the opt-outs). Tables 2.4a and b show the results of these comparisons, together with p values for associated comparisons of proportions and means respectively. Note that variable (and occasionally substantial) numbers of missing values are a feature of the data.

Table 2.4a: Characteristics of opt-outs and opt-ins (frequencies)

Table 2.4b: Characteristics of opt-outs and opt-ins (means)

The tables show that compared to opt-outs, opt-in mothers were less likely to be from minority ethnic backgrounds and more likely to have smoked during pregnancy and to have lower birthweight children. However, Howevcomparison of proportions and means suggested no statistically significant differences at the 95% level on any of the above measures. Hence, there is no obvious evidence of bias associated with recruitment.

A second potential source of bias relates to attrition and/or participation in assessment; the possibility that participants included in analysis (i.e. the 'rich' datasets of people completing all assessments) are systematically different from opt-ins as a whole. In analyses not shown, we have compared all opt-ins and sub-groups on a number of measures using both routine and survey data and find that persistence in the study is very moderately associated with greater relative affluence and maternal age. Whilst some differences approach significance at the p=0.05 level, the large numbers of comparisons involved greatly increases the risk of a type one (false positive) error and we cannot confidently conclude that there is any bias associated with persistence in the study.

Finally, in this sub-section, we shift the comparative focus away from opt-ins and opt-outs to describe some salient characteristics of the groups of participants included in analysis. Table 2.5 shows a series of survey items relating to both ethnicity and disadvantage for participants included in the six-month analysis (n=359). Similar figures are obtained for the eighteen month dataset (see above). The table makes two principal points: first, that all minority ethnic participants are in the intervention group ²³; and second, that on a number of measures, the comparison group are relatively more affluent in what may be regarded as a generally socio-economically disadvantaged cohort ²⁴.

Table 2.5:Comparison of selected baseline characteristics by group (n=359 dataset)

2.2.4 Measures

Outcome variables

Three measures relating to child health and one measuring user-satisfaction were chosen as outcomes at six and eighteen months. The first two health-related measures - quality of the home environment and extent of maternal depressive symptoms - are chosen firstly, because of their proven association with later child cognitive and emotional development (Bradley, 1993; Murray and Cooper, 1997) and secondly, because well-validated instruments exist to measure them (Bradley and Caldwell, 1988; Cox, Holden and Sagovsky, 1987). The third outcome - the child's dental registration status as reported by the mother - was chosen as an indicator of oral health, an area in which the programme is trying to promote positive change. The final outcome - satisfaction with the levels of health visitor support - is chosen as a key comparative indicator of user-satisfaction ²⁵.

Turning to each outcome in detail, the first is derived from the Infant/Toddler version of the HOME Inventory (Bradley and Caldwell, 1979b; Caldwell and Bradley, 1984). The HOME (Home Observation and Measurement of the Environment) is a standardised interview-and-observation tool that assesses the quantity and quality of stimulation available to a child in its home environment. Administered by trained researchers (usually health professionals), the assessment takes the form of a home interview with the caregiver and index child present. The interviewer asks a set of questions about the child's 'typical' day and in conjunction with more general observation, scores the mother-child dyad on the presence versus absence of 45 key responses and behaviours (for example, 'mother responds to child's vocalisations with a verbal response'). Six sub-scale scores are produced: verbal and emotional responsivity; acceptance of sub-optimal behaviour; degree of organisation of the child's temporal and physical environment; provision of learning materials; active involvement in learning; and inclusion of variety in the child's life. A higher score indicates a 'better' environment, i.e. one that is richer in terms of quality and/or quantity of stimulation. In keeping with many studies, we use the overall total score, i.e. the sum of all sub-scales, as our principal outcome.

The second outcome measure derives from another standardised, validated instrument - the Edinburgh Postnatal Depression Scale (EPDS; Cox, Holden and Sagovsky, 1987. See Appendix III for full instrument). This instrument is widely used as a screening tool for suspected postpartum depression. Participants indicate their strength of agreement with ten mood-related statement (for example, 'I have looked forward with enjoyment to things') and receive a total score ranging from 0-30, where a higher score means 'more depressive symptoms'. In this study, we used a dichotomised measure based on the advisory threshold score for clinical action: a score of 13 or greater was coded as '1'. It is to be stressed that the EPDS is a screening tool and scores exceeding this threshold do not equate to a formal diagnosis of depression. However, validation studies suggest that a threshold set at this value correctly identifies around two-thirds of depressed women (Murray and Carothers, 1990).

The third outcome used here is a dichotomised mother-report measure derived from a survey item. Dental registration indicates whether the child is registered with a community dentist at the time of assessment ('yes' = 1; 'no' or 'don't know' = 0). As this is a mother-reported measure, it awaits validation from routine data sources.

Finally, responses to the item 'how satisfied are you with the general level of support you have been receiving from your health visitor?' ('very satisfied'; 'fairly satisfied'; 'not very satisfied'; 'not satisfied at all') were dichotomised so that 'very satisfied' was coded as '1'.

Baseline predictor variables

A general aim of the quasi-experimental study is to explore the effects of the intervention on child health whilst controlling for the many individual- and household-level confounding factors. Accordingly, many such variables were included in the baseline survey, a sub-sample of which with a priori associations to outcomes were used in exploratory multivariate analysis. A full list of measures are available on request but the main sets of baseline predictors include: survey items relating to basic maternal and child data (child's gender, gestation, birth weight, feeding and behaviour, parity, mother's age, etc); items covering maternal health and health behaviour adapted from the 1998 Scottish Health Survey (Scottish Executive, 1998); self-control and alienation sub-scales from the Multidimensional Personality Questionnaire (MPQ: Tellegen et al, 1982); the Rosenberg Self-Esteem Scale (Rosenberg, 1965); the DUKE-UNC Functional Support Scale (Broadhead, 1988); questions on attitudes to parenting and health-visiting; and items taken from the 2001 UK Decennial Census, for example highest maternal qualification, household employment, tenure and car ownership.

Baseline socio-economic status was constructed from employment-related survey items using the reduced 2001 National Statistics Socio-Economic Classification system (NS-SEC; The Stationery Office, 2002). Some analytic classes were merged due to low counts, resulting in the following four dichotomised variables: NS-SEC class 1 and 2 (professional, managerial and higher technical occupations); NS-SEC classes 3, 4 and 5 (intermediate, lower supervisory and technical occupations); NS-SEC 6 and 7 (routine and semi-routine occupations); NS-SEC class 8 (never worked and long-term unemployed).

Material circumstances were indexed via a self-report measure of household income (after tax). This ordinal measure (participants ticked one of nine income bands, e.g. '200-299') was recoded into three dichotomised variables: lower income (<400/month); medium income (400 - 999/month) and higher income (>1000/month).

Service input was measured by collecting individual-level routine data on the number and type of contacts (including failed contacts) with health visitors and associated professionals. These data were collated from an operational database in the intervention area and from health visitor notes in the comparison area. Whilst a number of measures could be constructed, problems of comparability across cohorts meant that only two - the total number of recorded home visits by health visitors between 0-6 and 6-18 months respectively - were used in analysis ²⁶. These measures are likely to be reliable as health visitors are more likely to recall and record face-to-face contact in the client's home than brief phone contacts or opportunistic encounters at a clinic or in the street.

Table 2.6 shows the mean number of visits of this type at six and eighteen months, for both n=359 and n=294 datasets. All between-group comparisons were highly significant (t-values and dfs available on request). Although these figures confirm and emphasise the more intensive and home-based nature of the intervention, reference to both standard deviations and the range suggests considerable variation in the number of visits; evidently both types of service have the capacity to be flexible.

Table 2.6: Description of health visitor home visits at each assessment

Finally, in order to test for intervention effects, intervention status was entered into analysis as a property of each individual family (i.e., intervention family coded as '1', comparison family as '0').

2.2.5 Statistical approach.

Stepwise ordinary least squares (OLS) regression was performed on the HOME total score and logistic regression on the three dichotomous outcomes. Our general approach was iterative and cumulative and involved initial stepwise experimentation with reduced models containing similar variables (e.g. relating to material circumstances) whilst always retaining five key control variables: mother's age; parity; ethnicity; child's gender and group (intervention vs. comparison). We entered the strongest predictors in the reduced models into a full model whilst again retaining key variables. Overall, we have tried to produce robust models that satisfy three modelling criteria: a priori reasoning; statistical significance; and parsimony in the number of variables retained in the final equation.

2.3 Outcomes at six and eighteen months

2.3.1 Notes on the interpretation of statistics in this section

The meaning of 'significance' in descriptive and inferential statistics sub-sections:

If we had adopted an 'experimental' (e.g. randomised control trial) design with random allocation of births to different treatment conditions, we would not expect there to be any meaningful differences between groups apart from the treatment/service they received. In these circumstances, 'simple' comparisons of proportions and means like those in the descriptive statistics sub-sections of this chapter would be sufficient to demonstrate the presence/absence of 'real' (i.e. unbiased and statistically significant) group differences. However, the area-based nature of 'Starting Well' necessitated the adoption of a quasi-experimental design where births were non-randomly assigned to groups based on where their parents lived. This means that comparisons of the intervention and control groups may be biased by differences other than the service they receive. The regression analyses enable us to adjust for these differences to provide a less biased estimate of the intervention effect. However, it can only be used to adjust for observed differences between the groups, and unobserved differences may still bias the comparison in either direction.

Regression statistics:

Regression statistics express the size and direction of an association between a predictor variable and an outcome, whilst statistically controlling for the influence of other predictor variables in the model. In OLS regression (the analysis carried out on the HOME score), attention should be directed to the 'standardised beta coefficient' where a positive value indicates a positive independent association between that predictor and the outcome, whilst a negative value indicates a negative association. Within any given model, a higher standardised beta indicates a stronger relative effect ²⁷. Similarly, the key statistic in logistic regression is the odds-ratio (OR). An OR of less than 1 indicates a negative independent relationship between that predictor variable and the outcome whilst an OR of greater than 1 indicates a positive relationship. All predictor variables retained in models have an associated probability statistic. We use the conventional statistical significance criterion of p<0.05.

2.3.2 Health-related outcomes

Descriptive statistics

Table 2.7 shows between-group descriptive statistics for three health-related outcomes at six and eighteen months. The comparison area cohort has a non-significantly higher total HOME score at both assessments although the magnitude of the difference is less at eighteen months. HOME scores tend to increase over time. In contrast, identical proportions of women score above EPDS threshold at six months but there are significantly fewer comparison area women in this group at the later assessment (c ²=3.89, dfs=1). Finally, the intervention group had higher rates of dental registration at both assessments, although this is only statistically significant at six months (c ²=13.43, dfs=1).

Table 2.7 descriptive statistics for three health-related outcomes at six- and eighteen months

Inferential statistics

Table 2.8 shows the final models obtained at six and eighteen months for OLS regression of the total HOME score. Five variables were significantly and independently associated with outcome at both assessments, indicating the basic similarity of the models. Other predictors appear only once, either because they did not fit the model or were novel variables entered at eighteen months (e.g. mother's self-control). No statistically significant intervention effect was found at the p=0.05 level although the association was positive at both assessments and at eighteen months was significant at the p=0.10 level. Elsewhere, mother's age and ethnicity were strongly associated with the total HOME score (although in opposite directions) whilst first-time mothers scored more highly at the first assessment but did not show this advantage at eighteen months. Other maternal characteristics, indexing dimensions of personality and (perhaps) levels of personal resources, also predicted outcome: mothers with high self-esteem scored more highly at six months, but impulsive mothers, single mothers and those with more resident children tended to achieve lower scores. Finally, there is a negative association between household income and HOME score at both assessments.

Table 2.8 OLS regression of HOME total score at six-months and eighteen months

This association was supported by the retention in the model of number of cognate variables that express either personal-disadvantage (no maternal qualifications) or household-level material advantage/disadvantage (birthweight and 'lower class' respectively). The models predict, respectively, 42% and 47% of the total variation in HOME scores.

Table 2.9 Logistic regression of EPDS 'caseness' (score is 13 or greater) at 6 and 18 months.

Table 2.9 shows the results of logistic regression modelling with EPDS 'caseness' (the outcome expressing the proportion of women scoring above the potentially clinically significant threshold of 12/13). Minority ethnic women were more likely to post high EPDS scores at both assessments as were women with low levels of self-esteem. These variables, however, are the only real instances of similarity in what are plausible but quite distinct models. In contrast to the preceding section, a substantial intervention effect emerged at six months: after controlling for other relevant variables, mothers receiving the 'Starting Well' service were less likely to report high levels of depressive symptoms than those receiving the generic service. This advantage was not repeated at eighteen months however, and even showed signs of reversal with intervention women now more likely to score above threshold. At six months, mothers whose newborn had either spent time in intensive care or who presented with difficult behaviour were more likely to show depressive symptoms, as were women with a history of mental health problems. This latter association with vulnerability may also underlie the significant positive association between caseness and number of home visits; health visitors adjusted to a high score at routine assessment by increasing the amount of contact. At eighteen months, a more parsimonious model emerged with maternal age, low self-control and the experience of significant life-events (bereavement, pregnancy, job-loss, etc) predicting caseness.

Finally in this section, table 2.10 shows the results for logistic regression with mother-reported dental registration status. A statistically significant intervention effect was observed at both six and eighteen months: more 'Starting Well' mothers reported that they had registered their baby than those receiving the generic service. This advantage was particularly marked at six months. Few other similarities existed across models, although indices of unemployment ²⁸ predicted lower levels of registration at both assessments.

Table 2.10 Logistic regression of dental registration status ('yes') at six and eighteen months.

2.3.3 User-satisfaction

The proportion of 'very satisfied' responses in each group is shown in table 2.11. The table displays two additional statistics: the proportion of all women at eighteen months who reported the service was 'better than expected'; and the proportion of multiparous mothers who reported their current experience of health visiting was 'better then before'.

Table 2.11: descriptive statistics for measures of user-satisfaction

On the strength of these unadjusted comparisons, receiving 'Starting Well' is associated with higher perceived levels of health-visitor support than generic health visiting and the service exceeds expectations for a large proportion of women. This intervention advantage is particularly strong for multiparous women comparing current experiences of the service to their last.

Table 2.12 confirms the finding for satisfaction with health visitor support using logistic regression ³⁰: a statistically significant intervention effect was observed at six months which became more significant at eighteen months. Considerable stability in levels of satisfaction was shown by the fact the baseline measure (collected at around 2-3 months postpartum) was a significant predictor of later satisfaction. Positive attitudes towards health visitors at baseline also predicted satisfaction at six months, though whether these attitudes were pre-existing or had already been changed by the time of the baseline assessment is not clear. Finally, smoking during pregnancy positively predicted satisfaction at both assessments although there is no obvious a priori reason for this to be the case.

Table 2.12 Logistic regression of satisfaction with health visiting service ('very satisfied')

2.4 Discussion and conclusions

2.4.1 Evidence for intervention effects

After first establishing the basic representativeness of the recruited sample and confirming the more intensive, home-based nature of 'Starting Well', we assessed the evidence for intervention effects on a range of health indicators over the first eighteen months of the child's life. By standard scientific criteria, the project was not successful in improving the quality of the home environment relative to generic health visiting, although the association was positive at both assessments and approached levels of statistical significance at eighteen months. A number of clearer intervention effects were observed: more 'Starting Well' children were registered with a dentist at six and eighteen months; their mothers were more satisfied with levels of health visitor support at both assessments and were also less likely to be at risk of postnatal depression at six months. At face value, these findings are undoubtedly encouraging and suggest that diverse outcomes relating to the home environment, psychological functioning, health-related behaviour and service-related attitudes can be modified by an enhanced home-visiting service over a relatively short period of time. We will now briefly discuss each set of findings in turn.

HOME score

A recent and authoritative systematic review (including a meta-analysis of twelve studies) concluded that there was strong evidence for the positive effects of home visitation on the quality of the home environment as indexed by the HOME and related measures (Elkan et al, 2000; Bull et al, 2004). Given this evidence-base, why did we not observe more clear intervention effects? Three sets of points are relevant: basic problems of comparability with other studies; the possible dilution of effects due to service heterogeneity; and statistical power considerations. Taking the comparability point first, Elkan and colleagues' work is unquestionably rigorous but their conclusions are based largely on North American randomised control trials of diverse interventions delivered to particular high-risk groups, often in the absence of routine health visiting. Variation also exists in the timing of HOME assessments and the exact measure used. Moreover, only seven out of seventeen studies reviewed provided positive and statistically verifiable evidence of intervention effects ³¹. Given these basic problems of comparability and the equivocacy of the literature in general, it may be concluded that expectations of very marked intervention effects were unrealistic. Secondly, the intervention itself developed rapidly over time and introduced new components (e.g. the 'skill mix' of auxiliary health professionals and paraprofessionals) more than halfway through its initial phase. Given that participants were recruited over a thirteen month period, it may well be that later recruits were receiving a qualitatively different service to earlier recruits. If true, but not well-captured in analysis, this service variation may serve to dilute intervention effects. Finally, our ability to detect group effects may have been further compromised by the decision to use smaller, 'richer' datasets for regression analysis which may have unfavourably traded power (in terms of number of cases) for comprehensiveness (in terms of the range of available predictors).

Given the above limitations, it is perhaps all the more notable that we observed a 'borderline' intervention effect (p=0.088) at eighteen months. Whilst we do not suggest a shifting of the 'statistical goalposts' to accommodate this finding, borderline results are a feature of the HOME literature ³². Given both the complexity of the project and the limitations of evaluation, it is probably wise to avoid a simple success/failure conclusion based solely on significance level (Sterne and Davey-Smith, 2001) and tentatively explore the possibility that our findings point to a cumulative and/or delayed impact of the intervention on home environment. If this is true, and the small relative advantage continues to grow beyond eighteen months, the well-described associations of quality of early stimulation to later cognitive and behavioural development (Bradley, 1993; Shonkoff and Phillips, 2000) might suggest real future advantages for 'Starting Well' children.

Postnatal depression

Results at six-months postpartum support a number of studies showing the positive impact of home-based interventions delivered by trained health visitors (Holden, Sagovsky and Cox, 1989; Gerrard, Holden, Elliot, et al, 1993; Seeley, Murray and Cooper, 1996; Cooper and Murray, 1997). In this study, the fact that an intervention effect emerged despite the cohorts having an apparently identical proportion of 'at-risk' women underlines the importance of including relevant statistical controls in analysis; there were fewer 'at risk' women in the intervention group than would be predicted from their background characteristics. Findings at eighteen months are harder to interpret. Original work by the primary developer of the EPDS (Cox, 1986) suggested that postnatal depression either occurred in the first few months after birth or not at all. At eighteen months then, it is doubtful whether one can talk meaningfully of women having postnatal depression but the instrument may still have some validity as an indicator of depressive symptoms. If this is true, it is clear that the early mental health benefits afforded to 'Starting Well' mothers fade over time, with prevalence 'returning' to levels predicted by other key sample characteristics (e.g. deprivation) at eighteen months.

Taking the six-month finding in isolation, the evidence-base linking postnatal mood disorders to impaired child cognitive and emotional development (Murray, 1992; Murray, Fiori-Cowley, Hooper and Cooper, 1996) strongly suggest that 'Starting Well' could deliver immediate benefits to the depressed mother and more enduring benefits to the child. This may be particularly salient for this socio-economically disadvantaged cohort as children of poor depressed mothers are at substantially greater risk of impaired development (Murray & Cooper, 1997). Two qualifying points may be made, however. First, developmental impairment is more strongly predicted by disturbed maternal interactional style than by simple exposure to depressed mood; at this stage, however, we have only demonstrated an intervention effect in relation to reduced exposure. A pertinent question might then be 'was there an intervention effect on the interactional style (indexed for example by HOME Inventory sub-items) of depressed mothers?' Unfortunately, both the relatively small absolute numbers of EPDS high-scorers and the fact that only cross-sectional analyses are possible on six-month data ³³ mean that we are poorly equipped to test this hypothesis. Second, the practical, service-development implications of the positive six-month result are not straightforward as is it not clear what aspect(s) of the intervention produced the observed effect. Most intervention studies focus exclusively on postnatal depression and involve dedicated training in non-directive counselling techniques whereas 'Starting Well' has a much broader health focus and improved maternal mood without comparable training. At this stage then, findings are only supportive of a 'whole package' effect of the intervention on depressive symptoms but one candidate for the 'active' ingredient may be the quality of the mother-health visitor contacts (Korfmacher, Kitzman and Olds, 1998; Elkan et al, 2000 ). This point will be pursued in later sections.

Finally, it is clear from our longitudinal findings that women who have previously scored below threshold at six months, can go on to score highly at eighteen months. These families may constitute an important vulnerable group who would not ordinarily be identified as 'at risk' from earlier assessments ³⁴.

Satisfaction with health visitor support

Very few evaluations of health-visiting interventions make explicit efforts to assess client-satisfaction and those that do use methods that are notably inconsistent (Elkan et al, 2000; Bull et al, 2004). This may be due to a widespread perception that survey responses are meaningless when the overwhelming majority of respondents express general satisfaction with all health services due to the desire to give a socially-acceptable response (Avis, Bond and Arthur, 1995). Whilst it is true that the vast majority of participants in this study were either 'very' or 'fairly' satisfied with levels of health visitor support, responses were submitted via a confidential postal questionnaire and to a researcher as opposed to a health practitioner. In these perhaps less-biased circumstances, the distinction between the two terms (the former emphatic, the latter 'lukewarm') may be more valid and when the size of the group differences are considered, alongside those for the other measures expressed in table 2.11, it may be concluded that a clear preference is being expressed for the enhanced service. It is notable too, that contrary to other studies that show client satisfaction diminishing over time (Graham, 1979), our findings show that it actually strengthened from six to eighteen months.

Dental registration

Scottish child dental health statistics are stark: only one-third of children aged 0-2 are registered with a dentist (Information and Statistics Division, NHS Scotland, 2003) and by age five, 55% of children have dental disease (Scottish Executive, 2003). Marked social gradients are observed for both registration and outcomes (Davies, 1999). Against this backdrop, we have demonstrated strong evidence of project impact on dental registration rates over the first eighteen months of life. It is clear, however, that the substantive implications of this finding depend crucially on the extent to which registration is translated into actual attendance and/or better dental health (for example, fewer dental caries or tooth loss). Whilst positive cross-sectional associations between children's' dental registration status and dental health have been demonstrated (Pitts, 1995), the causal nature of this relationship remains unclear. Conclusions from this study should, therefore, be guarded until comparative practice data can be collected ³⁵.

2.4.2 Associations between outcomes and other predictors

Two sets of variables especially merit further comment: material (dis)advantage and ethnicity.

Material (dis)advantage

The last two decades have witnessed an explosion of multi-disciplinary research into health inequalities and their relationship to income and social status. Our findings support this literature in two ways. Firstly, both cohorts are disadvantaged in terms of their absolute material resources and display correspondingly high levels of adverse health and other behaviour (e.g. maternal smoking). Secondly, families' relative position within this cohort, in terms of available material resources, is robustly and consistently associated with each outcome. These findings, whilst unsurprising, point to the relatively limited potential of health services to impact on the health of deprived populations when unaccompanied by improved material circumstances and firmly point to the need to link vulnerable low-income families to both local and national poverty-reduction initiatives.

Ethnicity

Minority ethnic status was strongly associated with lower HOME scores, higher EPDS scores and lower rates of dental registration and, at face value, these findings offer a comparatively poor prognosis for these children. It is likely, however, that systematic measurement error is responsible for at least some of this apparent difference in outcome. The HOME, though used internationally in a variety of cultural settings, has never been validated on a British Asian cohort and the authors of the instrument admit that additional work is necessary in order to establish its wider validity (Bradley, 1993 ). Similarly, there are well-recognised problems with the translation of concepts relating to depression (Launguni, 1997; 2000) and with the EPDS in particular (Elliott, 1996; Gerrard, 2000). Measurement error is, perhaps, a less convincing explanation of lower dental registration rates; not only is this item a much simpler concept constructed from a response to a single survey item, the finding also supports other studies of infant feeding and dental health amongst British Asian populations (see Watt, 2000).

In summary, the findings most probably reflect both measurement issues and real health-relevant cultural differences and illustrate the need for culturally sensitive assessment tools and culturally-competent health workers of the type being piloted in 'Starting Well'.

2.4.3 Next steps

A number of points may be made regarding instrumentation and measures. Firstly, our use of the number of home visits by health visitors is a useful but limited measure of service input that tends to understate the amount of contact (e.g., from health support workers) in the intervention group. More sensitive and comparable indicators are required, including perhaps, contacts with health staff that are not recorded by the index health visitor. It would also be advantageous to develop an equivalised measure of input that ascribed different weightings to different types of contact. This would mean, for example, that a ninety-minute face-to-face home visit from a health visitor is accorded considerably more importance than a five-minute phone contact from a lay worker. Secondly, more work is required on research nurse inter-rater reliability in order to account for the observed investigator effect on HOME score. Thirdly, the cultural specificity of both the HOME and EPDS necessitates careful interpretation of results, and may benefit from separate analyses, although numbers are small. These and other points will be pursued in further analysis in due course.

Finally, we have, by necessity, focussed on a narrow range of indirect/intermediary outcomes. In order to demonstrate a 'step-change' in child health, we would need to show that Starting Well had a direct influence on actual child-centred outcomes such as school readiness or cognitive development. Whether or not this will be possible with the existing cohort remains to be seen, but we are optimistic. We have made strenuous efforts to put in place mechanisms for retaining contact with existing respondents and to maximise the availability of essential baseline information. Our current assumption is that by the beginning of 2005, when the first of the study children will be 42 months old, we might reasonably expect to be able to contact approximately 500 families and that about 70 per cent of these (N=350) would respond positively to a further round of data collection. On this basis there is a strong case to be made for further follow-up.

2.4.4 Conclusions

In conception, the project was influenced by both a general (review-based) and specific (Olds and colleagues) evidence-base that suggested a home-visiting service with key characteristics could result in demonstrable improvements in child-health-relevant outcomes. The exact correspondence of the project plan to this evidence-base and the fidelity with which it was carried out are important empirical questions. Basic problems in transferring findings from the north American to British contexts, however, cast doubt on the extent to which marked expectations of impact were realistic. In addition, methodological problems associated variously with project-area size, short recruitment and assessment timescales, imperfect participation, and attrition, have limited both the range of outcomes used and the number of participants available for analysis. All of these factors have limited the capacity of this evaluation component to detect impact.

Despite these impediments, we have recruited a not-obviously biased sample of families and uncovered evidence that is suggestive of project impact on earlier maternal psychological health and later quality of the home environment. When considered alongside findings relating to client-satisfaction, we can begin to build a picture of a service that is delivering generally higher levels of support than generic health visiting. In doing so, it may have the potential to impact on families in ways that are not only beneficial to the mother in the short-term but may also have more enduring benefits for the child.

« Previous | Contents | Next »