Sunday, June 1, 2008

Designing Tools for Evaluation

Back in 2000-2002 during a research project called MIRA Multimedia Interface for Research and Authoring I created a tool for reflective practitioners in school called Multimedia Clipboard. Later on, most of the features of this tool have been integrated into the OLEFA IMS-CMS http://www.olefa.com which offers a powerful web based environment including a database module that can be designed and personalized for multiple uses.

Multimedia Clipboard permitted the collecting and grouping of multiple types of data in order to create layered and multi-perspective descriptions and interpretations of learning situations.
With this kind of tools teachers can evaluate students learning products and processes by creating chunks of information, and include background information of situations and concepts in which these products, processes and their interpretation are embedded.

Once an item is created by importing for example a video clip, the interpretation can be broadened by adding descriptions from different perspectives (comments by students, peers, teachers, parents etc.) and deepened or "thickened" by adding layers of information and interpretation (see the concept of "thick descriptions" by Clifford Geertz and Gilbert Ryle).
This approach is a lot different from a quantitative evaluation, but also from a so called evaluation by competencies in which learning is split up into modules, key stages and key competencies and in which students learning outcomes are, in the end, always compared to normative clusters of knowledge.

The greatest difference however lies in the fact that evaluation by competencies is proclaimed to be formative but neither sets the learner nor the teacher in the focus of an inquiry but the implicit norms.

In opposition to this, a descriptive and interpretive (ethnographic) approach puts the learner and the teacher(s) in the focus because their judgments, ethical considerations and theories become part of the evaluation process. Here, teachers are not observers, but actors in a learning process, and an evaluation is created or co-constructed through collaboration and negotiation. And, students are not objects of evaluation and inquiry but co-evolving subjects. A tool like Multimedia Clipboard is as much a instrument for evaluation as a means to gain insight in the functioning of evaluation and the nature and power of the hidden theories involved - including those which gave birth to the tool itself. Furthermore, and because of the non-linearity of the tool, a small change in the tool has the potential to open a high variety of new possibilities to interconnect information. A small change in a check list of competences - which I consider to be a linear system - will never have such an impact.

By adopting open instruments and ethnographic approaches, evaluation will take the shape of a narrative based on self-organization and reflection instead of a diagnosis conforming to a given structure. We will also reach a high level of unpredictability, variety, redundancy and even messiness, while simultaneously reducing centralized control and normative comparability between individual learning outcomes. On the other hand we will open to all actors involved a space for self-reflection, creativity and double-loop learning.

As there is a growing demand for alternative forms to evaluate students learning, I think that evaluation by normative competencies cannot be but a transitional stage before adopting open tools like Multimedia Clipboard which will extend the possibilities and concepts of so called learning records and portfolios and which will at the same time force teachers in a position of self-reflection.

3 comments:

  1. When we talk about evaluation, it is very important to put the student and the teacher in the focus, because it will give them a tool to become reflective praticioners.

    The problem of all quantitative evaluation is that teachers believe that it comes after the fact. Items are not real problems. Current tests focus on aptitudes. They yield results that are analyzed on the national level rather than used to improve the performance of students. They want students to plug-in what was “learned” out of context. If you learn to drive, you won’t do it by written tests.

    Evaluation by competencies will still be decontextualized and output-measuring, say it is measuring the past. Those tests aren't enabling, they aren't forward looking. Current assessment practices place too much emphasis on the assessing content (various skills who had been put under drill instruction) and aren't linked to learning, neither in time nor in space.
    Above all those tests allow us to make placement decisions. Don’t forget that student scores can be used to evaluate schools and teachers.

    By using comparative test scores to assess the relative “quality” of teachers and schools, it is no wonder that “teaching to the test” is a common practice.
    Tests are built by and for educational systems to produce “the needs for its own services and products, i.e. the labour and instruments of correction” (Bourdieu).

    Complex problem solving, divergent thinking and collaborative efforts of learners aren't taken into account. As people don't increase their competency (notice the word) by building a tower of new facts and skills, but by transforming their knowledge structures, this reconfiguration should be observed and taped, filmed, noticed,...
    New strategies and models can't be measured by tests or by evaluating by competences (this learner has got 7 out of 10 - so does he know or not, that is here the question).
    But as evaluation isn't quite so romantic, nor should it be so dramatic or let's say shakespearian, we should focus on process recording in real learning situations, a more “authentic” assessment, or challenging performance assessment.
    A tool like the one who was set up by Pino would be a good one to record ongoing processes.

    So if we have to assess, (and I don’t feel the need to assess everything that is happening in classroom, because I don’t think that a quantitative evaluation provides us with an oppurtunity to focus on learning processes), we have to link assessment, curriculum and instruction.
    As we have adapted the curriculum to the needs of the students (at least we try so), and our “instruction”, we have to adapt assessment.
    We should plan assessment (if it has to be) together with instruction, with the learning process, so that assessment can give us an opportunity to push instruction forward.
    There we will have to take risks and to learn by mistakes, just as we expect our students to do.
    As we try to not disturb the community of practice that the classroom makes up, we should involve peer assessment, which will relieve us of some grading responsibilities.
    Students will be responsable by constructing their responses when we use a more challenging assessment.

    ReplyDelete
  2. When we talk about evaluation, it is very important to put the student and the teacher in the focus, because it will give them a tool to become reflective praticioners.

    The problem of all quantitative evaluation is that teachers believe that it comes after the fact. Items are not real problems. Current tests focus on aptitudes. They yield results that are analyzed on the national level rather than used to improve the performance of students. They want students to plug-in what was “learned” out of context. If you learn to drive, you won’t do it by written tests.

    Evaluation by competencies will still be decontextualized and output-measuring, say it is measuring the past. Those tests aren't enabling, they aren't forward looking. Current assessment practices place too much emphasis on the assessing content (various skills who had been put under drill instruction) and aren't linked to learning, neither in time nor in space.
    Above all those tests allow us to make placement decisions. Don’t forget that student scores can be used to evaluate schools and teachers.

    By using comparative test scores to assess the relative “quality” of teachers and schools, it is no wonder that “teaching to the test” is a common practice.
    Tests are built by and for educational systems to produce “the needs for its own services and products, i.e. the labour and instruments of correction” (Bourdieu).

    Complex problem solving, divergent thinking and collaborative efforts of learners aren't taken into account. As people don't increase their competency (notice the word) by building a tower of new facts and skills, but by transforming their knowledge structures, this reconfiguration should be observed and taped, filmed, noticed,...
    New strategies and models can't be measured by tests or by evaluating by competences (this learner has got 7 out of 10 - so does he know or not, that is here the question).
    But as evaluation isn't quite so romantic, nor should it be so dramatic or let's say shakespearian, we should focus on process recording in real learning situations, a more “authentic” assessment, or challenging performance assessment.

    A tool like the one who was set up by Pino would be a good one to record ongoing processes. Isn’t evaluation a certain search for proof that tends to set the focus on the visual aspect of learning? A classroom is a complex environment and common evaluation standards aren’t the best way towards understanding. Complex systems tend to be very unstable and unpredictable. This level of unpredictability is very important. Students come to class every day with their developmental stages but the trajectories in which development occurs can be partly unpredictable. This unpredictability gives us a good opportunity for self-reflection (for students and teachers). I think we have to focus on it.
    When you flip a coin, what can you predict?
    Well you can predict with 50% that it will land heads up. But what about how many times it will flip and roll? Where will it land? Will we find it or will it land in the grass and it will be out of sight for a moment?


    So if we have to assess, (and I don’t feel the need to assess everything that is happening in classroom, because I don’t think that a quantitative evaluation provides us with an oppurtunity to focus on learning processes), we have to link assessment, curriculum and instruction.
    As we have adapted the curriculum to the needs of the students (at least we try so), and our “instruction”, we have to adapt assessment.
    We should plan assessment (if it has to be) together with instruction, with the learning process, so that assessment can give us an opportunity to push instruction forward.
    There we will have to take risks and to learn by mistakes, just as we expect our students to do.
    As we try to not disturb the community of practice that the classroom makes up, we should involve peer assessment, which will relieve us of some grading responsibilities.
    Students will be responsable by constructing their responses when we use a more challenging assessment.

    ReplyDelete
  3. I remember the Multimedia Clipboard Pino presented us during the MIRA project, and I wondered some times ago, if it still exists. During the project I was really excited about this tool, and the first “try outs” one of the project members showed to us were really convincing. If you consider now the ongoing portfolio discussion, I would say that this tool could be used as an “electronic portfolio”. One of the advantages would be the linking-up of videos, texts, background information, products, interviews, and different points of viewing (student, parents, teacher and others…). Thus the information about the student and his/her learning will be “thickened”. Moreover, the evaluation will not be “second hand”, as the reader will be able to consider the product and the background information him – or herself. On the other hand the traditional evaluation leaves the interested observer with an interpretation of an achievement expressed through points and notes.

    As Pino and Joël Mischaux already pointed out, the traditional evaluation is more or less summative. And changing to an evaluation by competencies will not have much of an affect on this tradition, if we stick to the tests or the standardised tests, the ministries all around Europe are keen to present as the answer to the educational “misery”.

    Anyhow I consider this change of focus on competencies as a step into the right direction. It is true that the evaluation by competencies includes implicit norms to which the students are compared. (Until now they were compared to norms the single teacher put up in his tests.) On the other hand, any teacher acting seriously according to an approach by competencies will have to reflect on his concept of evaluation. Every student will have “a different level” of attainment for any competence. It would not be credible if the teacher would put all the students through the same test or to any test at all! Thus she/he will have to reflect on her/his evaluation methods. This will be the time to introduce new ways of assessment including cooperative reflections between students, teachers and the other actors in school-life. And may be the Multimedia Clipboard could be one tool enabling all the actors to participate in evaluation.

    ReplyDelete

Some Reading Milestones

  • Towards reflexive method in archaeology : the example at Çatalhöyük (edited by Ian Hodder) 2000

  • The Book of Learning and Forgetting (Frank Smith) 1998

  • Points of Viewing Children's Thinking: A Digital Ethnographer's Journey (Ricki Goldman-Segall) 1997

  • Verstehen lehren (Martin Wagenschein) 1997

  • Computer im Schreibatelier (Gérard Gretsch) 1992

  • The Boy Who Would Be a Helicopter. Uses of Storytelling in the Classroom (Vivian Gussin Paley) 1991

  • La cause des adolescents (Françoise Dolto) 1988

  • Scuola di Barbiana. Die Schülerschule. Brief an eine Lehrerin. (Edition of 1980) / read in German 1982
    Letter to Teacher by the Schoolboys of Barbiana (1970)
    Lettre à une maîtresse de'école, par les enfants de Barbiana (1968)
    Lettera à una professoressa (Original Edition) 1967


  • Vers une pédagogie institutionnelle (Aïda Vasquez, Fernand Oury) 1967



Documentary Films on Education

  • Eine Schule, die gelingt (by Reinhard Kahl) 2008

  • Les temps des enfants (Jacques Duez) 2007

  • Klassenleben (by Bernd Friedmann und Hubertus Siegert) 2006

  • Lernen - Die Entdeckung des Selbstverständlichen
    (Ein Vortrag von Manfred Spitzer) 2006

  • Die Entdeckung der frühen Jahre
    Die Initiative "McKinsey bildet" zur frühkindlichen Bildung (by Reinhard Kahl) 2006

  • Treibhäuser der Zukunft - Wie in Deutschland Schulen gelingen (by Reinhard Kahl) 2004

  • Treibhäuser der Zukunft / Incubators of the future / Les serres de l'avenir; International Edition (by Reinhard Kahl) 2004

  • Journal de classe, 1ères audaces (1), Les échappés (2), Sexe, amour et vidéo (3), L'enfant nomade (4), Remue-méninges (5) (by Wilbur Leguebe, Jacques Duez, Agnès Lejeune) 2004

  • Spitze - Schulen am Wendekreis der Pädagogik (by Reinhard Kahl) 2003

  • Journal de classe, (by Wilbur Leguebe and Agnès Lejeune; Jacques Duez) 2002

  • Etre et Avoir (by Nicolas Philibert) 2002

  • The Stolen Eye (by Jane Elliott) 2002

  • The Angry Eye (by Jane Elliott) 2001

  • A l'école de la providence (by Gérard Preszow) 2000

  • Blue-Eyed (by Jane Elliott) 1996

  • A Class Divided (by Jane Elliott) 1984

  • Eye of The Storm (with Jane Elliott) 1970

Past quotes of the day

For every problem, there is one solution which is simple, neat and wrong. Henry Louis Mencken

Traveler, there is no path. Paths are made by walking.
Antonio Machado

The best way to predict the future is to invent it. Immanuel Kant

The significant problems we face cannot be solved at the same level of thinking we were at when we created them. Albert Einstein

To paraphrase a famous quotation, all that is necessary for the triumph of damaging educational policies is that good educators keep silent. Alfie Kohn

We used to have lots of questions to which there were no answers. Now, with the computer, there are lots of answers to which we haven't thought up the questions. Peter Ustinov

I had a terrible education. I attended a school for emotionally disturbed teachers. Woody Allen

A professor is someone who talks in someone else's sleep. W. H. Auden

When I was an inspector of schools I visited one classroom and looked at a boys book. He'd written, 'Yesterday, Yesterday, Yesterday, Sorrow, Sorrow, Sorrow, Tomorrow, Tomorrow, Tomorrow, Love, Love, Love.' I said, 'That's a lovely poem.' He said, 'Those are my spelling corrections.' Gervase Phinn

Real thinking never starts until the learner fails. Roger Schank

If what is wanted is a reexamination of schooling in terms of purpose, structure and process, then testing programmes are the wrong vehicle (...) Caroline V. Gipps

Not everything that counts can be counted, and not everything that can be counted counts. Albert Einstein

Act always so as to increase the number of choices. Heinz von Foerster

Another way of avoiding teaching is by relying exclusively on a textbook, workbooks, and other commercially packaged learning materials. Teaching is reduced to administering a set curriculum without giving any thought to the substance of what the students area learning or to their particular needs. H. Kohl

The right to ignore anything that doesn't make sense is a crucial element of any child's learning - and the first right children are likely to lose when they get to the controlled learning environment of school. F. Smith

Learning is the human activity which least needs manipulation by others. Most learning is not the result of instruction. It is rather the result of unhampered participation in a meaningful activity. - Ivan Illich

Too often we give our children answers to remember rather than problems to solve. - Roger Lewin

I have never let my schooling interfere with my education. - Mark Twain

About Education on Youtube

Loading...