Background and Rationale
For too long the tradition in Canadian
universities has been to give pre-eminent importance
to knowledge generation—the creation of new ideas
and scholarly findings. Only recently has the academy
begun to realize that it has another important role,
namely, to transfer knowledge to the community of policy-makers,
practitioners, and the general public. The time has
come for social scientists not only to dedicate themselves
to knowledge generation but also to knowledge utilization
The tradition among policy-makers
and practitioners, especially at the local level, has
been to rely on unsystematic and eclectic sources of
evidence to make decisions. A
host of factors explains this tradition, ranging from
lack of appropriate evidence, conflicting and confusing
or weak evidence, lack of time and expertise to sift
through evidence, and many other influences on decision-making.
Canadian policy-makers, practitioners, and the general
public should be better informed about the state of research
evidence so they can join the worldwide movement, especially
evident in the United Kingdom, to use evidence to inform
decision-making and improve practice. Important and costly
decisions about programs and initiatives, especially
in emerging areas, are not always based on complete information
about impact, cost-effectiveness, scalability, and sustainability.
Improving Canadian society means creating a better link
between research and practice.
The Canadian Government recognizes the
important steps required to address the challenges of
the future and the concerns of the knowledge-based economy.
For example, in Achieving Excellence: Investing in
People, Knowledge, and Opportunity (2002) the government
stated: “Our
supply of highly qualified people is far from assured
in the medium term. Canada will have great difficulty
becoming more competitive without a greater number of
highly qualified people to drive the innovation process
and apply innovations, including new technologies” (p.19). The
Conference Board of Canada also understands the challenges
Canada faces in terms of innovation and skills acquisition
if we are to maintain and enhance our status as a leading
nation in the world.
Systematic Reviews: In
1976, Gene Glass sparked interest in systematic reviews
when he developed and popularized techniques for the
quantitative synthesis or meta-analysis of empirical
evidence. Chief among his contributions were three ideas. First,
that the accumulated evidence of many studies in an area
is more trustworthy than the evidence supplied by a single
study or selected studies. Second, that reviews of research
should be undertaken with the same care and thoroughness
as primary investigations, and described in sufficient
detail so that the review can be readily replicated.
Third, that researchers and practitioners need to know
both whether and under what conditions interventions
work, and the magnitude of their effects.
Since this pioneering work, others have contributed
to the methodological and statistical tools employed
in the integration of evidence. But there are important methodological
challenges that remain. One concerns whether
and how to include both policy or position papers and
qualitative research in the review process. A second
challenge is evident in the debate about the nature and
type of quantitative evidence to be included in a systematic
review, especially in emerging areas of interest. A third
challenge focuses on cost and timeliness considerations.
Organizing, executing and analyzing comprehensive and
complex quantitative reviews of large literatures often
requires large budgets and years of work, but policy-makers
need authoritative answers quickly and at low cost.
There are also important conceptual
and dissemination challenges that currently limit the
extent to which knowledge from primary research and
research reviews is utilized by policy-makers and practitioners. One
concern is relevance. How do scholars
identify questions of interest to multiple constituencies
(e.g., academics, policy-makers, practitioners, students,
government agencies, the public, etc.)? How are systematic
reviews informed by, but not biased or compromised
by, those with practical experience and strong beliefs
or vested interests? How does one maintain the
integrity of independent, third party assessment, and
actively partner with the educational community?
Another concern is meaningfulness.
How do scholars communicate effectively to policy-makers,
practitioners, and the general public? What form
of feedback and type of evidence impact on improvement?
How does one factor in cost effectiveness and timeliness?
What are the optimal conditions for implementation efficacy?
What individual skills and attitudes about data and what
contextual factors are important for using evidence for
improvement? How does one overcome longstanding
beliefs and practical experience (i.e., craft knowledge)
and encourage the community to accept the value of research
evidence? What are the best ways to translate research
evidence into effective success plans? That is, how does
one move from using research evidence for diagnostic
purposes to using research evidence to structure a remediation
effort/success plan? How do data change attitudes
and behaviors toward success plans? What outcomes are
more amenable to change based on evidence?
Types of Reviews:CanKnow
will produce different types of reviews based on funding
and timeliness considerations. It will use Argument Catalogues,
which are systematic reviews of public, policy-making,
and scientific opinions and positions on topics. The Argument
Catalogues will help inform subsequent systematic
reviews.
Systematic reviews are especially useful for answering
questions about what works, about how much things work,
and the contexts and conditions under which things work.
Certain types of systematic reviews may also be useful
for exploring the underlying processes by which treatments
and interventions work. We will be developing three types of systematic reviews
that vary in completeness and comprehensiveness. Stage
One reviews focus on identifying the overall
impact of a treatment or approach on one or two key outcomes. Stage
Two reviews are more complete accounts of the
evidence with some attempt to explain why findings vary
across studies. Stage Three reviews
are comprehensive, full-scale reviews that attempt to
review all known evidence on a topic exhaustively, and
thoroughly explain why findings differ across investigations. These
stages imply different degrees of completeness to accomplish
different purposes and for different audiences, but not
different degrees of correctness.
Knowledge Transfer:Disseminating
what is learned from systematic reviews is key if evidence
is to be used wisely to inform practice. CanKnow will
develop strategies for knowledge dissemination and use
its Steering Committee to help guide and inform these
dissemination plans. These activities may include
reports, and interactive websites, as well as presentations
to key policy-makers, practitioners, and the general
public, as well as training in understanding evidence
and ways to use evidence to improve practice.
Therefore, CanKnow will be dedicated to identifying,
synthesizing, and disseminating the best evidence in
order to meet the following goals:
- Accuracy
- Comprehensiveness
- Meaningfulness
- Utility
- Timeliness
CanKnow recognizes that policy-makers and practitioners
face a myriad of influences in decision-making. By being
sensitive to community needs, by ensuring the highest
methodological standards, and by providing timely and
usable feedback, CanKnow will encourage important societal
change.
Challenges and Issues: As CanKnow begins
we face a number of important challenges and issues.
These include:
- What is the scope of CanKnow interests, broadly within
the Social Sciences or limited to Education? What are
some key questions to explore at the outset?
- How should CanKnow be funded? What are sources to
request funding?
- What is the appropriate governance structure for
CanKnow? Are there other investigators and Steering
Group members who should be included?
- Should CanKnow centralize the conduct of reviews
or distribute them nationally?
- What form of affiliations should CanKnow maintain
with the Campbell & Cochrane Collaboration?
- How should CanKnow best respond
to the interests, timelines, cost issues, and dissemination
needs of policy-makers and practitioners?
|
Our Experience with Performing Systematic Reviews
One way that the CSLP has tried to communicate to other scholars, practitioners, and policy-makers is by conducting systematic reviews of important educational literatures (see list of papers, presentations and publications below). We have just completed a major review of the literature on distance education that appears in the Review of Educational Research. The paper won the 2004 Award for Excellence in Research from the Canadian Association of Distance Education. We are currently in the midst of another review on the impact of instruction on students’ critical thinking. And we plan to undertake other reviews, for example, on postsecondary education and technology integration.
Phil Abrami and Bob Bernard are co-chairs
of the International Campbell Collaboration Education
Group that is dedicated to improving the quality of
systematic reviews and their dissemination to, and
use by, policy-makers and practitioners. Abrami is
also a member of C2’s Steering Committee.
Anne Wade is a member of the Campbell Collaboration’s
Information Retrieval Method Group and acts as the liaison
between the Education and the Information Retrieval Method
groups. She is also an Associate of the Evidence Network
in the UK.
Our Papers, Presentations and Publications
Related to Systematic Reviews (click on the pdf icon
for abstracts)
Abrami, P.C. & Bernard, R.M. (in press). Research on distance education: In defense of field experiments. Distance Education.
Abrami, P.C., Bernard, R.M., Borokhovski, E., Surkes, M., Wade, A. & Zhang, D. A. (under review). Meta-analysis of instructional interventions affecting critical thinking skills and dispositions: Preliminary results. Paper submitted for presentation at the Annual Meeting of the American Educational Research Association, San Francisco, April 2006.
Abrami, P.C., Bernard, R.M., Wade, A. Canadian Network for Knowledge Utilization: Evidence-based policy and practice. (under review). Paper submitted for presentation at the Annual Colloquium of the Canadian Cochrane Collaboration, Montreal, December, 2005.
Abrami, P.C., Cohen, P.A., & d'Apollonia,
S. (1988). Implementation problems in meta-analysis. Review
of Educational Research, 58, 151-179. 
Abrami, P.C., d'Apollonia, S., & Rosenfield,
S. (1996). The dimensionality of student ratings of
instruction: What we know and what we do not. Invited
article for Higher
Education: Handbook of Theory and Research, XI, 213-264.
Abrami, P.C., d'Apollonia, S., & Rosenfield, S.
(1997). The dimensionality of student ratings of instruction:
What we know and what we do not. In Perry, R.P. & Smart,
J.C. (Eds.). Effective teaching in higher education:
Research and practice. Bronx, N.Y.: Agathon Press,
pp. 321- 367.
Abrami, P.C., Leventhal, L., & Perry,
R.P. (1982). Educational seduction. Review of Educational
Research, 52, 446-464.
Abrami, P.C., Lou, Y., Chambers,
B., Poulsen, C., & Spence,
J. (1999). Within-class grouping: evidence vs. conjecture. National
Institute Economic Review, 169, 105-108.
Abrami,
P.C., Lou, Y., Chambers, B., Poulsen, C., & Spence,
J. (2000). Why should we group students within-class
for learning? Educational Research and Evaluation,
6(2), 158-179.
Abrami, P.C., & Spence J (1993).
Meta-analysis for explanation: A casebook for caution. Educational
Researcher, 22(5), 31-33.
Abrami, P.C., Wade, A. & Bernard,
R.M. (2005). Evidenced-based Practice: Bridging
the gap between educational research and classroom
teaching. Canadian
Association of Principals Journal, 13(1), 16-17.
Abrami, P.C., Wade, C.A, & Bernard, R.M. (2005, Sept.). Models for reviewing and disseminating research evidence to aid practice and policymaking. Poster presented at the Concordia University Librarians Forum. Montreal.
Azevedo,
R., & Bernard, R. M. (1995). A meta-analysis of the
effects of feedback in computer-based instruction. Journal
of Educational Computing Research, 13(2), 109-125.
Bernard, R.M. (2005, September). Different models and roles of brokerage agencies in evidenced-based policy and practice. Invited speaker, OECD International Conference on Linking Evidence to Practice, The Hague, Netherlands.
Bernard,
R.M., Abrami, P.C., Lou, Y., & Borokhovski, E. (2004).
A methodological morass: How can we improve the quality
of quantitative research in distance education? Distance
Education, 25(2), 176-198.
Bernard, R.M., Abrami, P.C., Lou,
Y., Borokhovski, E., Wade, A., Wozney, L.,Wallet, P.A.,
Fiset, M. & Huang,
B. (2004). How does distance education compare to classroom
instruction? A meta-analysis of the empirical literature. Review
of Educational Research. Recipient of the
2004 Award of Excellence in Research from the Canadian
Association for Distance Education.
Bernard,
R.M., Abrami, P.C., Lou, Y., Wade, A., & Borokhovski,
E. (under review). The effects of synchronous and asynchronous
distance education: A meta-analytical assessment of Simonson’s “equivalency
theory”. Proceedings: Instructional Technology
and Distance Education.
Bernard, R.M., Abrami, P.C., & Wade, A. (under review). Canadian Network for Knowledge Utilization: Evidence-based practice and policymaking in the Canadian context. Roundtable discussion proposed for the Campbell Collaboration Colloquium,(Los Angeles, Feb. 23-25, 2006).
Bernard, R.M., Abrami, P.C., & Wade, A. (2004, October). Assessing methodological quality in a large research literature: The case of distance education comparison studies. Poster presented at the12th Cochrane Collaboration Annual Meeting, Ottawa, ON.
Bernard,
R.M., Abrami, P.C. & Wade, A. (2004, February). Distance
education compared to classroom instruction. Presented
at the Fourth Annual Campbell Collaboration, Washington,
DC.
Bernard, R.M., Abrami, P.C., & Wade,
A. (2005, February). Assessing methodological quality in a large research
literature: The case of distance education comparison
studies. Poster presented at the Campbell Collaboration
Colloquium, Lisbon, Portugal.
Bernard,
R.M., Lou, Y., & Abrami, P.C., Wozney, L., Borokhovski,
E., Wallet, P., Wade, A, & Fiset, M. (2003, April). Is
distance education equivalent to classroom instruction?
A meta-analysis of the empirical literature. Symposium
presented at the annual meeting of the American Educational
Research Association, Chicago, IL. (Invited address at
AECT, Anaheim, California, October 2003).
Bernard,
R. M., & Naidu, S. (1990). Integrating research into
practice: The use and abuse of meta-analysis. Canadian
Journal of Educational Communication, 19(3), 171-195.
Bernard, R.M, Zhang, D., Abrami, P.C., Sicoly, F., Borokhovski, E.& Surkes, M. (under review). Exploring the Structure of the Watson-Glaser Critical Thinking Appraisal: One Scale or Many Subscales? Paper submitted for presentation at the Annual Meeting of the American Educational Research Association, San Francisco, April 2006.
Cobb,
B., Abrami, P.C., Bernard, R.M., Lauver, S., Ritter,
G., Scher, L., Turner, H., Wade, A., & Goelich Zief,
S. (2005, April). Producing C2 Systematic Reviews
to Inform Evidence-Based Policy and Practice in Education. Workshop
presented at the American Educational Research Association,
Montreal, QC.
Cobb, B., Abrami, P.C., Bernard, R.M., Turner, H., Wade, A., & Nye, C. (under review). Producing C2 Systematic Reviews to Inform Evidence-Based Policy and Practice in Education. Workshop proposed for presentation at the Campbell Collaboration Annual Colloquim (2006, February).
Lou,
Y., Abrami, P.C., & d’Apollonia, S. (2001).
Small group and individual learning with technology:
A meta-analysis. Review of Educational Research,
71(3), 449-521.
Lou,
Y. Abrami, P.C., & Spence, J. (2000). Effects of
within-class grouping on studentachievement: An exploratory
model. Journal of Educational Research, 94(2),
101-112.
Lou,
Y., Abrami, P.C., Spence, J., Chambers, B., Poulsen,
C. & d'Apollonia, S. (1996). Within-class grouping:
A meta-analysis. Review of Educational Research,
66, 423-458.
Lou,
Y., Barrington, J.M. Bernard, R.M., & Abrami, P.C.(2002,
April). Third generation distance education: A review
of the research. Paper presented at the annual meeting
of the American Educational Research Association, New
Orleans, LA.
Lou,
Y., Bernard, R.M., & Abrami, P.C. (2005, April). Distance
education research and practice in higher education:
A theory-based synthesis. Paper presented at the
American Educational Research Association, Montreal,
QC.
Lou, Y., Bernard, R.M., & Abrami,
P.C. (under review). Distance education research and
practice in higher education: A theory-based meta-analysis
of the empirical literature. Educational
Technology Research &Development.
Turner, H., Abrami, P.C., Bernard, R.M., Cobb, B., Nye, C. & Wade, A. (under review). Producing C2 Systematic Reviews to Inform Eivdence-based Policy and Practice in Education. Workshop submitted for presentation at the American Educational Research Association Annual Meeting, Los Angeles, CA. (2006, April).
Wade, A., Abrami, P.C., Bernard,
R., Turner, H.M., & Peretiatkowicz,
A. (2005, February) Performing the literature search:
Beyond the basics. Workshop presented at the Campbell
Collaboration Colloquium, Lisbon, Portugal.
Wade, A. & Rothstein, H. (2006). Systematic reviews and information retrieval: Searching the databases. Workshop submitted for presentation at the Campbell Collaboration Colloquium (Los Angeles; February 23-25, 2006).
Wade, A., Turner, H., Rothstein, H., & Lavenberg, J. (under review). Information retrieval and the role of the information specialist in producing high quality systematic reviews in the social, behavioral, and education sciences. Evidence and Policy.
Also:
Wade,
A., Abrami, P.C., Poulsen, C. & Chambers, B. (1995). Current
resources in cooperative learning. Lanham, MD.:
University Press of America.
 See also Knowledge Links
|
CSLP
