Evaluating legal literacy programmes – aims , challenges , models and a call to action

Generally speaking, if one asks most people whether improving public understanding of the law and legal system is a good thing, most hands will go up. This is perhaps no surprise. An improved level of legal literacy should not only raise awareness of rights and responsibilities but may well provide people with more of an informed choice about what to do if they encounter legal issues. It might give those concerned the tools and confidence to address some of these problems themselves through self-help and it may, address, at least to some degree, inequalities that otherwise exist. Overall, improving levels of legal literacy could enhance access to justice more generally. The generic term often ascribed to raising legal awareness amongst the wider population is public legal education (PLE).


Evaluating legal literacy programmes -aims, challenges, models and a call to action
Richard Grimes1 Building 'best practice' 2 Generally speaking, if one asks most people whether improving public understanding of the law and legal system is a good thing, most hands will go up.This is perhaps no surprise.An improved level of legal literacy should not only raise awareness of rights and responsibilities but may well provide people with more of an informed choice about what to do if they encounter legal issues.It might give those concerned the tools and confidence to address some of these problems themselves through self-help and it may, address, at least to some degree, inequalities that otherwise exist.Overall, improving levels of legal literacy could enhance access to justice more generally.The generic term often ascribed to raising legal awareness amongst the wider population is public legal education (PLE).
There is a wealth of anecdotal material suggesting that all of the above is highly relevant and impactful3 but there is little by way of clear empirical evidence to substantiate such claims. 4is article will look at the need for, and means of, developing such an evidence base and ends with a plea for more research, sharing of ideas and collaboration in terms of evaluating PLE.The article will be presented in the format of an introductory section looking at the background and history of PLE (in particular Street Law) in terms of aims, challenges and models from the perspective of evaluating impact.
This is followed by a 'lesson plan' in which a sample set of possible PLE options are set out with the means of evaluating impact incorporated in the model used.It is hoped that this might provide a guide for those wishing to devise (or revise) their legal literacy programmes from an evaluative viewpoint.
As will be seen below the starting point is that those responsible for design and delivery must identify what it is they are expecting to achieve from any planned session and whether those outcomes are in fact reached.This type of evaluation, formative evaluation, measures whether the objectives of any particular lesson are achieved and provides sign posts for how to revise instruction to ensure objectives are achieved.
The template should enable those using it to replicate and, as necessary, adapt an evaluable model for use in the field.
This paper then has a sample form appended as an example of how PLE sessions might be evaluated.

Aims
Before turning to the models and impact measurement what specifically are the purposes of evaluation?Evaluation is important for several related reasons.First and foremost if planned outcomes and actual achievements are not clear how can the value of what is being carried out be assessed -with a view to monitoring progress, awarding possible credit and making improvements in future delivery?More strategically perhaps, and as will be seen, many PLE initiatives rely on either public funding, the backing of foundations and other charitable bodies and/or the input of voluntary and not for profit personnel and initiatives.The relative lack of impact evidence is surely an obstacle to securing financial and other resource backing?The aim therefore of this article is to raise the importance of evaluation and to identify the means by which any assessment may be carried out.
In a nutshell, can we identify: • whether outcomes set for a particular PLE session or event have been achieved?
• if future planning and delivery can be improved?
• whether there is impact -what difference does an improvement in legal awareness actually make in the immediate, short-term or longer-term contexts?
• if and how the evidence can be used for development purposes -for example funding, curriculum design, policy change?
• if findings of an empirical nature match what instinct and anecdote suggeststhus building a body of evidence that has consistency and resonance.5

Challenges
First, what is meant here by PLE?At its most general, it is the raising of awareness of law and the legal system through a variety of techniques and methods including the provision of information (hard copy, electronic and face to face) as well as through a more formal education interface such as one-off presentations or structured courses and programmes.This takes into account the Street Law approach where target audiences are introduced to a range of legal rights and responsibility issues through interactive learning and teaching techniques often lead by lawyers, judges, and trained law students under professional supervision.6 There seems to be a degree of consensus that PLE should include, but not be limited to, the dissemination of information.A point often made is that in addition to the acquisition of knowledge, PLE involves fostering understanding and the development of skills.It is also argued that PLE should aim to influence attitudes, and build confidence.Distinctions need to be drawn between information per se and education more generally.The latter commonly involves custom made and subject specific material, delivered more than likely in an interactive way, whereas the content of the former tends to be generic, with the direction of flow largely being one way, from 'expert' to recipient.It also is inclined to treat the audience as a passive receiver of that information.Street Law is probably the best example of the education model and one that shapes part of the evaluation template examples given below.
The quest for robust evaluative evidence is, however problematic.This is on a number of levels.First, what is being measured and secondly, how might that measurement be reliably and consistently done?It should also be remembered that studies involving sensitive data may need ethics approval from professional or other institutional bodies (such as universities whose staff or students carry out research) and that there may be legislative requirements over the handling of such data.
One way to measure social impact is to study widespread changes over time.
However, the literature suggests that neither the measures nor the tools appear to exist are or necessarily make, good teachers, but their use in many PLE programmes is a model that can have multiple beneficiaries including the students themselves.
in the PLE context (although there have been a number of evaluative studies in other disciplines using defined methodologies).7 In UK Ministry-led task force on PLE in 2007, PLE presented the following challenges that affect both planning and evaluation: • PLE sessions or materials are typically part of some larger curricula In addition, the lack of a 'like with like' comparison and the nature of learning as process rather than product make evaluation challenging to say the least. 9By way of contrast few ask for the learning legitimacy of other forms of education to be proven.
Judging by the number of students reportedly disinterested in lectures, a similar study on impact and the value-added component of learning passively might be usefully called for!10 All of these factors or characteristics have measurable dimensions and may be interrelated.The measurement of impact is therefore complex and difficult.
Measuring how and when improved awareness rises to the level of impacting a participant's ability to implement the newly acquired knowledge or skill is also difficult to establish -particularly in trying to identify cause and effect.This has previously been shown to be the case in legal self-help situations. 11 Despite these challenges, it is suggested here that impact can be measured at different moments in time and with different techniques and approaches.A combination of methods and, where relevant, an amalgamation of findings coupled with subsequent analysis can reveal valuable insights on impact and outcomes.Let us turn to this now.

Models
Space in this article does not permit a detailed discussion of evaluative research techniques.For those interested there is a wealth of material available elsewhere. 12ffice it to say for present purposes that useful evaluation might consist of a mix of quantitative and qualitative methods.These might start with a simple record of the number of those attending PLE sessions or accessing PLE material.Entry and exit questionnaires might be used to see what the target audience expected and then made of the materials and/or presentations.If a particular PLE project is targeted at a specific issue it may be possible to monitor impact in terms of the resolution of disputes, take up of benefits or incidence of unwarranted activity such as domestic violence, unlawful eviction or anti-social behaviour (in each instance in terms of both number and participant perception).Focus groups could be used to prompt feedback and discussion.Evidence could be gathered from PLE participants (the audience, the presenters and other stakeholders) before, at, after and following the event of release of material in question.The appendix to this paper contains an example of how session delivery might be evaluated.Of course any evaluation will hinge on what is expected and that might vary considerably, for example if a PLE session was a one-off event looking at a particular issue or was more overtly educational in an incremental sense such as a short course or semester-long module.
Hard evidence may be relatively easily obtained when impact can be measured at the time of or soon after the PLE input.The much more difficult question as to lasting or longer-term significance of an increase in legal awareness inevitably requires longitudinal studies which are, by their nature, time-consuming to carry out, relatively expensive to administer and difficult to firmly establish cause and effect.
Some notable examples of highly successful studies in the legal awareness field (rather than impact of PLE as such) can be found and make for interesting reading. 13The Pleasance et al Causes of Action studies, based on a rolling programme, for instance show the very high price paid for unresolved legal disputes (some GBP £3.5 million a year and not including the human cost).It begs the question of the extent to which a greater awareness of legal rights and responsibilities might mitigate against such waste and the negative impact on personal well-being.For illustrative purposes we will work with two examples here following a similar template; first, there is a plan for delivery of a typical Street Law session and secondly a plan for the use of hard copy legal information (which could also/instead be delivered as part of a web-based PLE project).

Example 1 -work plan for a 'typical' Street Law session on stop and search provisions under domestic law 14
Institution: a law school running a credit bearing module called 'Law in the community' Cambridge University Press, 2009.There are also important studies in other subject fields that attempt to evaluate impact -for example in youth justice, numeracy and financial literacy and in consumer protection -see: J. Kenrick, Young People's Social Welfare Need and the Impact of Good Advice, 12See for example: S. Halliday and P. Schmidt, (Eds.).Conducting Law and Society Research: Reflections on Methods and Practices, Foundation for Educational Research, DFE, 2010.For an interesting account of PLE programmes and evaluative methods see: M. Sefton, Public Legal Education Strategy (PLES) Task Force Scoping Report, Paper 2/03a, Research Unit, Department for Constitutional Affair, 2006 (from which the citations to some of the reports referred to in this footnote were taken).

time The stop and search lesson is one of a series of 'know your rights' presentations to be delivered at the school on a set day per week for one hour. 60 minutes plus travel time. Timings will need to be agreed with the school staff and normally fit into an allotted timetable space. Preparation/delivery/ supervision (who?) A team of 5 law students will research the law and applicable procedures and will deliver the session at the school. One staff member to supervise preparation and delivery. The content and delivery format will be checked by the supervisor (an ex-practising lawyer in the law school). Content 1. Police powers to stop an individual/group 2. Police powers to consequently search persons/property 3. Consequences of refusing to stop or be searched 4. Possible sources of help if arrested 5. Possible action/ route for complaint if police powers exceeded or person otherwise 15
A separate set out learning outcomes are required if the law students are expected to achieve certain educational goals.These will vary according to the module studied and the level at which it is offered e.g.first year or final year of study.For reasons of space limitation these details are not included in the template here.