|
|
|
Federal Claims on the Educational Agenda
The control of curriculums in TAFEs has become appealing to the Federal establishment. This is part of a long term general trend favouring Federal political management. The rhetorics of privatization, economic rationalism and micro economic reform are not really about weak central government. There is an unspoken belief that relinquishing the ownership of production leaves the governing class actually better able to manage social and economic outcomes. "The planning of outcomes for the free market is becoming increasingly centralized at a national level" (Dwyer 1995: 97). The rationalization of training has a natural place in this effort.
Here is the genesis of ANTA and the National Reporting System. True to the new philosophies, teaching institutions like TAFEs now have "clients" who a not students but paymasters like DEET. To get paid, a provider (i.e. a TAFE etc.) must meet detailed contract specifications by getting students to perform a range of defined tasks, now spelled out in the NRS. The educational provider no longer has as first priority coping with the individual needs, abilities and interests of students. There is a sense in which the ethical base of teaching itself is undermined since ".. anything is acceptable at the delivery point provided that it complies with budgetary constraints, yet leads to the achievement of the outcomes and targets predetermined at the centre" (Dwyer 1995: 98). Adult educators in the field are acutely aware of this switch in priorities from students to the bureaucratic demands of institutional "clients", notably DEET.
Why is the political climate so receptive to a National Reporting System? There is great populist political appeal to the claim that diffuse educational programs in the recent past have produced an inferior educational product. There is some nostalgia for the lock-step centralized programs of earlier regimes. For such proposals, protagonists on both sides can produce any number of examples. Genuine evidence of a decline in, say, literacy or second language acquisition is hard to come by. However, what is significant for funding in the end is the moving tide of social and political practice. Having lost control of teaching methodology and even content to teaching professionals, the political and bureaucratic elites have moved to ensure their control of educational outcomes. Their tool is to be competency based training.
Competency based programs are a natural derivative of mass political organization, and of mass production industries where every task must be atomised. CBT found early expression (like so much of our culture and technology) in the military-industrial complex of mid-20th Century America. The immense technical systems of modern military commands are, on the whole, maintained by individuals who have not been able to operate very successfully in the messy, ambiguous cultures of civilian societies. They tend not to be the best and the brightest (with honourable exceptions), yet what they manage is critical to the survival of nations. The military educational response in USA to clever machines + unclever operators was to combine behavioural conditioning (after the Skinnerian model) with an extreme atomization of learning tasks. The steps to, say, dismantling a radar unit, were analysed in minute detail, and then described in the simplest possible way. The educator was reduced to a trainer, whose task was to condition trainees to follow a set of procedures until they were response-perfect. The trainee was then "competent".
The appeal of CBT (competency based training) is its apparent simplicity and measurability. There are indeed many discreet tasks, like riding a bicycle or making a bed, which we can do or not do. There are many complex industrial processes, like making a car, which have been analysed to the minutest degree and whose practitioners can be measured against something like the ISO9000 standards. Certainly it is natural for an accountant, who assesses productivity by measuring dollar inputs against dollar outputs, to suppose that all human learning can be usefully corralled by the CBT method.
To this language educator, the goal of clear, unambiguous CBT outcomes for much human activity seems foolish. One can imagine a useful CBT program for packing bread rolls in a cardboard box, or perhaps for training airline pilots to exchange information with international flight control centers. CBT programs for being prime minister, writing a novel or practicing speech therapy would have to invite satire.
The attempt by the NRS Document to define ambiguous, notional CBT outcomes from language learning (especially) seems guaranteed to generate a medieval religious hypocrisy in educational practice. The document is a collection of idealized descriptions which are impossible to measure objectively. Take a statement such as ".. draws on a repertoire of strategies to maintain understanding through lexically dense or extended texts.." (NRS Level Five). My own first reaction to such Level Five descriptions is that few TAFE students or teachers would meet the language standard. In practice, the descriptions will be solemnly decoded by moderation meetings of teachers and shoe-horned to fit whatever group of students is at hand. A general referendum on the meaning of those words would yield a biblical range of understandings. Is this what modern education is about? The curriculum guide which needs a theologian or a lawyer to rule on its interpretation is a throwback to the Middle Ages.
So what is the future of a CBT-based document like the NRS? At the moment (1996) there are a small number of educators who are sure that CBT is doomed as a useful tool in their line of work. There are a large number of teachers and curriculum writers who are uneasy to varying degrees, somewhat confused, but resigned to keeping a low profile for the sake of professional survival. There are a small number of educational managers, plus large sections of governmental bureaucracies, and the general public who find the superficial idea of CBT for everything very sensible.
What is(n't) the National Reporting System ?
The introduction to the NRS document specifies what the system is and is not. We will examine the substance of these statements presently. However it has to be said at the outset that the realization of a manifesto (and the NRS document is a manifesto) is never entirely what its principles proclaim. Most of us could agree heartily with some stated objectives in the Christian Testaments, the Communist Manifesto (of Marx and Engels) and the Tax Assessment Act while remaining dismayed at outcomes from their dogmatic application. All such manifestos create and are interpreted within a political context. We have to turn to certain constants in human behaviour to sensibly decode their likely impact on events. Here is a list of NRS disclaimers:
1. The NRS "is not an assessment system" but assessments will be "mapped against the NRS" (1995, p2). In other words, assessments which are not fashioned to NRS premises, scope and language will be unsuitable. This is a bit like Henry Ford telling A-model Ford customers that they could have any colour so long as it was black.
2. The NRS "is not a curriculum". It looks like a curriculum, it uses the language of curricula, it has been synthesized from "over 30 ALBE and ESL curriculum documents" and is "intended to promote good curriculum practice". Who are the NRS people kidding? Deng Xiaoping is not president of China either, but a PRC citizen lives or dies by his dogma.
3. The NRS "is not a model of language acquisition" but "it is both data driven and inclusive of different theoretical perspectives and curriculum approaches". The NRS concedes that nobody really knows how a language is learned, but is confident that the knowledge once acquired can be measured. Hence the "data driven" specification. This paper will argue later that the data which drives the model is thoroughly unreliable. The claim to be "inclusive of different ... perspectives etc." can safely be ignored as rhetoric. The NRS is solidly behaviourist. At its heart is a black box called "student" with input and output terminals attached. Yet the teacher as assessor must make assumptions about, for example, what a student "depends on" to generate certain language behaviours. B.F. Skinner (the father of behaviourism) would be appalled.
4. The NRS "is not a means for categorising students by a simple `level', nor is it a set of broad competency statements". Readers are referred to the National Framework of Adult English Literacy and Numeracy Competence for such categorisation. Students "should not be assumed to be at any given general level". Again the NRS writers are being disingenuous. The Framework or whatever other curriculum document is being used will be mapped against the NRS and students will be effectively assigned a set of NRS levels. Supposed student competency will be matched to NRS competency concepts.
5. The NRS "is not a recruitment instrument for employers" and it "is inappropriate to use the NRS as an assessment tool for recruitment or appraisal purposes". Of all the disclaimers, this one must be the most naïve or the most cynical. It is unclear whether the document is referring to the recruitment of ex-students or of teachers. The marketplace value of basic CBT literacy and numeracy certificates (whether NRS structured or not) is very doubtful. However, let us take the matter of teacher employment. Do the NRS people really understand what they are creating, and what goes on in the minds of educational managers? This writer must intrude a little personal reality here. Before this article was written his Head of Department vehemently insisted that his employing institution should not be identified in any way. Funding might be jeopardized she thought. Perhaps it was silly, but the psychology is widespread. An earlier employer, the Adult Migrant Education Service, excluded the writer from sessional contracts after he wrote a critique of their Certificate in Spoken & Written English, leading to over $10,000 loss of income (they comfortably deny this of course). These are the facts of life on the ground out in teacherland.
Earlier this paper discussed the origins of CBT in the simple mastery of tasks which had been highly atomized. The authors of the NRS do recognize that simple criteria of mastery cannot be applied to most language and communicative tasks. Typically such tasks can be accomplished well, or not so well (but still accomplished) in a wide variety of ways. Indeed, superb linguistic performance in one sociolinguistic context may be a dismal failure in another. The response of the NRS to this dilemma is to implicitly substitute for mastery an equation of two criteria: complexity and autonomy. The idea seems to be that an increase in both complexity and autonomy (ratio unspecified) corresponds with the NRS gradation from level 1 to level 5. Unfortunately both criteria are approached in an anecdotal and thoroughly inconsistent manner.
Complexity is a science in itself now. There are many books and scientific papers on the subject, some to do with linguistic complexity, and a range of definitions of the concept which have radically different consequences. The NRS team seems blissfully unaware of these niceties. The NRS specifications (like those of most so-called competency curriculums) float hazily between lay notions of syntactic complexity and levels of general intellectual or social sophistication. They provide no principled instrument, for example, to distinguish between the progress of a highly intelligent, sophisticated English language beginner, and a cheerfully fluent, dim, long term resident NESB speaker who will never, ever "communicate ideas, arguments and conclusions logically, clearly and concisely", let alone "write organizational procedures and time-frames to take account of different roles and perspectives" (NRS level 4). As every TAFE teacher knows, both characters frequently turn up in the same class. Are we to assess linguistic achievement, or that much more amorphous quantum related to general intelligence?
The matters of linguistic and social autonomy are similarly confused in the NRS document. Again, autonomy remains undefined, a token of shifting assumptions. Teachers (let alone curriculum writers) see students in a narrow controlled setting. After twenty years in the field, this writer continues to be astonished by the confidence or timidity of students accidentally encountered in other settings. Their behaviour is often radically at variance with classroom impressions.
The NRS cannot be viewed simply as an instrument to take a snapshot of student populations at a given point in time. Its principals will not use it in that way. The funding priority of DEET will be to track students diachronically to see if they are yielding value for money in terms of the program model. With this in mind, the logic of grading from level 1 to level 5 requires students to progress through competency levels, yet these levels claim to calibrate language + cognitive sophistication. Language mastery is not directly related to intelligence. The skills of language fluency and, say, analytic analysis will be acquired at different rates. In the latter case they may not be acquired at all. In summary, given its confused criteria and objectives, the NRS is not suitable as an index of language skills acquisition in the micro environment of individual courses.
A DEET or ANTA staff member reading this discourse must of course feel a certain exasperation with the kinds of doubts which have been expressed. Firstly, fixing on the NRS as some dark political design for mastery of the universe is daft from the normal perspective of a public servant's work. Secondly, the NRS, in spite of methodological flaws, may well be defended as the best available tool in an imperfect world. Yet surely it is possible to conceive of better, much better tools for reporting language achievement than the NRS. A couple of suggestions are made at the end of this piece. More detailed proposals are really the topic for another paper.
The political problem is no so much in the personal designs of administrators as in the independent life and power which institutions accrue in their own right. This perceived power has real and often detrimental effects on individuals at a distance from its source (e.g. the heads of department etc. mentioned earlier). Against this, it is certainly true that much teaching was, is and perhaps always will be poorly focused. This reflects the ratio of human competence to incompetence found in all trades and professions (competence here is used in its original, non-devalued meaning). Efforts to mitigate such incompetence are necessary, ongoing and part of professional life. The problem with the NRS is that it will scarcely help, and may hinder this task.
The Quality of Data in a National Reporting System
From the driving seat in DEET or ANTA, there will always be a powerful tendency to consolidate the myriad activities of a nation of people into the ordered, predictable sequences and outcomes of a computing system. The greater the quantity and variety of data available back in head office, the greater the scope for modeling this illusion. However, the data must be reliable, meaning consistent and constrained within suitable parameters. Reality must be manufactured to fit the data requirements. Whatever teachers are doing, for example, must be made to fit the data requirements. The best way to arrange this is to have them spend increasing amounts of time generating the data rather than teaching.
What ANTA is attempting through the instrument of the NRS is just one variation on a whole recent philosophy of government that could be summed up as power without responsibility. Dwyer (1995: 101) puts it into a proper context:
" The game of truth (Foucault 1988:15) currently being played out with regard to post-compulsory education both in Australia and internationally, has a make-believe quality to it. Policies are presented in decidedly positivist terms - projections are made, models are developed, targets are set, computer runs of data are output - and the outcomes, though lacking in substantive or qualitative content, are pre-ordained in those same positivist terms and are programmed in a self-fulfilling way."
The make-believe quality of CBT outcomes comes to ground at the employment coal-face. As Foucault noticed, truth is a moveable feast. Teachers, in their own idiosyncratic ways, also have agendas. Amongst those agendas is a fairly universal wish to survive and to pay the rent. CBT outcomes are certainly being bent to that end.
If there is one constant in human behaviour, it is that many people will tell you (often with conviction) what you want to hear if the incentives are right. Mass conversions to Islam in the 7th Century A.D. came with hefty tax breaks for the faithful, just as conversions to Christianity came with political preferment in Rome. Was there ever a Soviet factory manager who did not report exceeding the 5 year plan by 500% rather than vanish into a gulag? Was there ever a workplace where the demands of management did not lead to small, daily betrayals of principle?
When a teacher has fifteen low literacy students, ten of whom have made no visible "progress" over a semester in terms of some competency description, what will she mark on their "statements of attainment"? It is odds on that she will tick half a dozen boxes. She will indulge in a little hypocrisy, for in the end she is being assessed here, not the students. This is not speculation. I have had long term unemployed students with a sheaf of certificates and up to 2,000 hours of classroom attendance who could barely read or write. Perhaps their teaching had not always been brilliant, but no teacher was going to admit that. The students' problems came from a blend of demoralization, ignorance, particular aptitudes, social pressures and plain bad luck. Occasionally, with a change in motivation or for reasons invisible to the teacher, progress could be spectacular. Language, and language competence is made inside people's heads, often incubating below the sight line of any CBT program.
Putting aside all the political agendas, there remain technical questions about how effectively CBT programs (and their derivatives like the NRS) can be assessed. Here we will just note the issue, which has been discussed elsewhere. Brindley (1995: 148) for example comments on the very weak construct validity of many CBT assessments: "how adequate is the assessment as a measure of what it is supposed to assess?" Part of the assessment problem, again noted by Brindley, is a shortage of empirical evidence needed to support assumptions of CBT contributions to language acquisition.
There is a great need for professional, in-depth linguistic case studies on what is really happening in ESL and literacy teaching. There is also an urgent need for sociolinguistic surveys on the meaning which students, teachers, employers and the general public attach to the statements which appear on a host of training certificates. My own experience with institutions, and off-the-record attitudes of teachers and students leaves an impression which is worrying. The feeling is that Australia is now awash with statements of attainment, competency certificates and the like which are effectively attendance certificates. If so, they are not worth the paper they are written on. Certainly, by the standards of common speech, many of their owners cannot do what the certificates claim that they can do.
"Rubbish in, rubbish out" was the cliché which nourished the first generation of computer programmers. On present trends, ANTA and the NRS team are about to collect mountains of rubbish for their nation-modeling game. The figures will improve year by year. The funding will be "justified". And nobody will have honestly proved that a single student learned the English language more quickly or retained it more thoroughly than his predecessors from the dark ages 30 years ago.
Is There Really A Role For A National Reporting System?
No. Not as a Big Brother check on global language skill targets which teachers must attain. The history of all attempts at total central control, from the earliest empires to theocratic regimes to intellectual and economic ideologies is that they generate oppression, hypocrisy and stagnation. There are already signs of this in the Australian context.
Other models of central facilitation are more promising. One of the best for educational development might be that of a national language research program and information clearing house. The NCELTR already makes some claims in this direction, but any financial stake in curriculum production, or theoretical bias (e.g. to systemics) would need careful separation from the gathering and empirical analysis of data. The need for factual linguistic information on language teaching outcomes, and sociolinguistic attitudes to attainment statements, has already been mentioned. This kind of information, suitably communicated to teachers would be genuinely useful. If they could be given some "ownership" in its production, with no hint of threat, then their professional development would be truly enhanced. Teacher-linguists are already widely used in aboriginal language programs. They would be a welcome addition to the metropolitan society.
A prerequisite to any developments like the above is a special need to take the ideology out of CBT and other curriculums. A notional statement in a curriculum is, at bottom, an ideological statement. That is, the following (as an example) requires an ideological interpretation: ".. depends on prior knowledge of context and personal experience to make predictions when reading/listening." A teacher or other party might guess that a student was exercising such dependence, and make a value judgement about whether it was an appropriate level of dependence for NRS Level 1. No one could devise a repeatable, independently verifiable experiment to prove/disprove that a student met the above curriculum criterion. Yet teachers not trained in experimental method will not generally question the value of such a curriculum goal. This is where dispassionate help from a national research centre and information clearing house could help.
Imagine the research centre that collected reading material used by teachers all over the country. Imagine that its own linguists checked the understanding of many students against these texts. It would take a lot of footwork. What if they were able to tell teachers, based on systematic investigation (not vague guesstimates), what a typical Level 1 student was able to recognize in sample texts, tell them that is with clear reference to each relevant feature in a specific text?
By using appropriate sample texts, teachers would have a clear basis for judgement. This would be vastly preferable to the woolly meta-language of competency curriculums. The same principles could be applied to speaking, listening and writing. These researchers would not be in the business of telling teachers what to teach or how to teach it. Yet, as linguists in the research centre continued to collect materials year after year, and test them personally against samples from the student population, they could begin to develop some realistic ideas about what kinds of language teaching were actually working best.
Averbach, Elsa 1986 "Competency Based ESL: One Step Forward or Two Steps Back?" TESOL Quarterly Vol. 20, No.3, Sept. 1986
Brindley, Geoff (ed.) 1995 Language Assessment in Action, pub. NCELTR, Macquarie University NSW, 324pp. Note: A number of papers in this collection are pertinent to issues discussed in the present article.
Brindley, Geoff 1995 "Competency Based Assessment in Second Language Programs: Some Issues and Questions", in Brindley, Geoff (ed.) 1995 Language Assessment in Action, pub. NCELTR, Macquarie University NSW, pp.145-164
Coates, Sharon et. al. 1995 National Reporting System, a mechanism for reporting adult English language, literacy and numeracy indicators of competence, prepared at the Office of Technical & Further Education (OTFE) Victoria and jointly published by the Commonwealth Department of Education, Employment & Training (DEET) and the Australian National Training Authority (ANTA)
Cohen, Jack & Ian Stewart 1994 The Collapse of Chaos, discovering simplicity in a complex world, pub. NY: Penguin, 495pp. Note: This is included as one of several available popularized introductions to the general science of complexity
Dwyer, Peter 1995 "Post Compulsory Education in Australia", Journal of Education Policy Vol.10, No.1, January 1995, pp. 95-103
Foucault, M 1988 "The Ethic of Care for the Self as a Practice of Freedom" in J. Bernauer & D. Rasmussen (eds.) The Final Foucault, pub. Cambridge MA: MIT Press
May, Thor 1987 "Evaluating Linguistic Difficulty", TESOL News Vol.8 No.3, pp.24-33, pub. Curriculum Development Centre, Commonwealth Schools Commission, Woden A.C.T.
Storey, Haddon 1995 "The Development of the National Vocational Education and Training System: the Victorian Perspective" in Training Fax Vol.3 No.1 March 1995, pp1-4, published by the Victorian Office of Training & Further Education (OTFE)
Tollefson, James 1986 "Functional Competencies in the US Refugee Program: Theoretical and Practical Problems", TESOL Quarterly Vol. 20, No.4, Dec. 1986.
_____________
The writer: Thor May has taught ESL and lectured linguistics for twenty years in Australia, New Zealand, PNG and Fiji. He is writing a PhD in cognitive linguistics (January 1996). His current employer is a TAFE (which wishes to remain unidentified) somewhere in Melbourne.