Category Archives: methodology

Thoughts on: Teaching online (in Teaching thinking: Beliefs and knowledge in Higher Education) (Goodyear, P. 2002)

Writing about work by your supervisor feels a little strange but, as adults and scholars, it really shouldn’t. Obviously there is a power dynamic and a question for me of what to do if I disagree with him. Putting aside the matter that Peter Goodyear has worked and researched in this field forever and is highly regarded internationally while I am essentially a neophyte, I’m almost certain that his worst reaction would be the slightest brow-crinkling and a kindly, interested “ok, so tell me why”. He even made the point that the research may now be dated but it could be worth following the citation trail. Fortunately none of this is an issue because, as you’d hope from your supervisor, it’s pretty great and there is much to draw from it.

In summary, this chapter focuses on understanding what and how teachers think when they are teaching online. Sadly perhaps, little has changed in the nature of online teaching in the 14 years since this was written – the online teaching activities described are largely related to students reading papers and participating in discussions on forums. This gives the chapter a degree of currency in terms of the technology (although a few questions emerged for me in terms of the impact of social media) and I imagine that little has changed in teacher thought processes in this time related to assessing and trying to engage students online.

In some ways it’s the methodology used in the study that is the most exciting part of this – it steers away from the sometimes problematic reliance on transcript analysis used often (at the time?) in research on online learning and makes more use of the opportunities for observation. Observing a teacher reading, processing and replying to discussion forum posts offers opportunities for insight into their thoughts that a far richer than one might get in observing face to face teaching. By using a combination of concurrent and retrospective verbalisation and interview, a rich picture emerges.

Concurrent verbalisation involves getting the tutor to keep up a kind of stream of consciousness dialogue as they work on the discussion posts, with the researcher prompting them if they fall silent for more than 10 seconds. This can prove difficult for the teacher at times as they need to stop speaking at times to concentrate on the replies that they write but a balance is generally found. The session is also videotaped and the researcher and teacher watch it back together, (‘stimulated recall’),  which gives the teacher the opportunity to discuss what they were thinking in the quiet moments as well as enabling them to expand on their recorded comments. In terms of understanding the things that are important to teachers and how they work with the students, I find this method really exciting. I’m not at all sure how or if it will align with my own research when I come to it but this rich insight seems invaluable.

The author opens the chapter by thoroughly going through the motivations for researching teaching – ranging from an abstracted interest in it as a good area for study to a more action research oriented focus on improving specific aspects of teaching practice. He explores the existing literature in the field – particularly in relation to online learning and finds that (at the time) there were a number of significant gaps in research relating to practice and he proceeds to set out six high level research questions relating to online teaching. It seems worthwhile sharing them here

  1. What are the essential characteristics of online teaching? What tasks are met? What actions need to be taken? Are there distinct task genres that further differentiate the space of online teaching?

  2. How do these practices and task genres vary across different educational settings (e.g between disciplines, or in undergraduate vs postgraduate teaching, or in campus based vs distance learning) and across individuals?

  3. For each significant kind of online teaching, what knowledge resources are drawn upon by effective teachers? How can we understand and represent the cognitive and other resources and processes implicated in their teaching?

  4. How do novice online teachers differ from expert and experienced online teachers? How do they make the transition? How does their thinking change? How does the knowledge on which they draw change? How closely does this resemble ‘the knowledge growth in teaching’ about which we know from studies of teaching in other, more conventional, areas?…

  5. What do teachers say about their experiences of online learning? How do they account for their intentions and actions? How do their accounts situation action in relation to hierarchies of belief about teaching and learning (generally) and about teaching and learning online?

  6. How do learners’ activities and learning outcomes interact with teaching actions? (p.86)

Skipping forward, Goodyear conducted the research with a number of teachers working online and identified several key factors that shape what and how teachers teach online. The focus of their attention – is it on the student, the content, how well the subject is going, whether students are learning, the technology, how students will respond to their feedback etc – can vary wildly from moment to moment. Their knowledge of their students – particularly when they might never meet them in person – can shape the nuance and personalisation of their communications. This also ties to “presentation of self” – also known as presence – which is equally important in forming effective online relationships. Understanding of online pedagogy and attitudes towards it are unsurprisingly a big factor in success in teaching online and this also impacts on their ability to manage communication and conflict in an online space, where normal behaviours can change due to perceived distance.

There were a lot of other noteworthy ideas in this chapter that are worth including here and it also sparked a few of my own ideas that went off on something of a tangent.

Those who foresee an easy substitution of teaching methods too frequently misunderstand the function or underestimate the complexity of that which they would see replaced (p.80)

Teaching is not an undifferentiated activity. What is involved in giving a lecture to 500 students is different from what is involved in a one-to-one, face-to-face, tutorial. Also, interactive, face-to-face, or what might be called ‘live’ teaching is different from (say) planning a course, giving feedback on an essay, designing some learning materials, or reflecting on end-of-course student evaluation reports. (James Calderhead structures his 1996 review of teachers’ cognitions in terms of ‘pre-active’, ‘interactive’ and ‘post-active reflection’ phases to help distinguish the cognitive demands of ‘live’ teaching from its prior preparation and from reflection after the event) (p.82)

The affordances of the user interface are an important factor in understand how online tutors do what they do. This is not simply because online tutors need to understand the (relatively simple) technical procedures involved in searching, reading and writing contributions. Rather the interface helps structure the tutors’ tasks and also takes some of the cognitive load off the tutor (P.87)

Studies of ‘live’ classroom teaching in schools have tended towards the conclusion that conscious decision-making is relatively rare – much of what happens is through the following of well-tested routines (Calderhead, 1984). While swift routine action can be found in online tutoring, its curiously asynchronous nature does allow more considered problem solving to take place (p.97)

Many of these ideas crystallise thoughts that I’ve come to over recent years and which I’ve shared with Peter in our supervision meetings. I’m going to choose to believe that his inner voice is saying at these points, ‘good, you’re on track’ rather than ‘well, obviously and I wrote about this a decade and a half ago’. This is why we go with this apprenticeship model I guess.

As for the other random thought that emerged from reading this paper was that as we get more comfortable with using video and asking/allowing students to submit videos as assessments, we’ll need new ways to ‘read’ videos. Clearly these will already exist in the scholarhood but they may not be as widely known as we need.

Quick reading: Five papers on Academic Development (Hannon, 2008; Hicks, 2005; Boud & Brew, 2013; Lee & McWilliam, 2008; Bath & Smith, 2004)

Academic development refers to the professional development of academics – which makes sense when you think about it. Evidently I hadn’t thought about that a lot because until I skim read these five papers, I had put academic developers in the same broad (and perhaps vague) category as education designers and learning technologists. People working with teachers/academics to support teaching and learning and developing resources.

These are the papers:

I had just assumed that given that the terminology hasn’t really been settled yet (consider blended/flexible/online/technology-enhanced/e-learning), people have been using the terms that they prefer. (I’ve been toying with Director of Education Innovation as a new title but apparently that will upset the Directors of our schools, so that won’t fly).

Anyway, this was the first of a few realisations that I’ve had in the last week of trying to get my research back on track – ironically enough perhaps while I’ve been in the midst of a major academic development project of my own. (STELLAR – which will get its own post shortly).

Recognising that I need to move on to a new topic of exploration in my holistic overview of the central elements in supporting TELT practices in Higher Ed. but also feeling that I haven’t yet covered Education Support Staff (ESS) adequately, I decided to take the temperature of ESS research via five papers. (I’ve also been concerned that while the deep reading that I’ve been doing has been valuable, I’m spending too long on individual papers and chapters in the process.) I allocated a single 25 min pomodoro period to each of these new papers, including writing notes. Admittedly, four of the five papers I’ve decided that I still need to read in full and may well come back to them in the next topic anyway. (However, I changed my initially planned ‘next topic’ from Universities as Organisations to Teachers as a result of these papers and some other thinking recently, so this still feels like progress)

In a nutshell, as I’ve been looking at research relating to education support staff over the last couple of months, I’ve probably been in my own tribal mindset. I do still believe that there are significant cultural factors at play in higher ed. that mean that knowledge and experience aren’t always appropriately used or recognised if you’re not in the academic tribe and this is an area to work on. There are also an incredibly diverse range of reasons for this, some more understandable than others. I have to admit that I’ve not been as open to the more understandable (and valid) ones as I should’ve and that empathy is always an important part of communication and collaboration.

So after this post on the matter, I’m going to take a first pass at my lit review relating to ESSes and focus on the academic/teacher side. (Ultimately people that teach are teachers and this is the side of the academics’ work that I’m looking at – it’s also a more meaningful term in this context – but I realise that terminology is perhaps more important than I thought.

These are my quick responses to the papers that I skimmed

This is a particularly insightful paper that uses “the discourse analytic method of “interpretative repertoires (Potter & Wetherall, 1987)” (p.15) to consider issues in academic development with a particular focus on education technology and changing teaching practices.

Hannon essentially distills the approaches into ‘enabling’ and ‘guiding’ and interviews 25 individuals working with education technology (including academics and ESSes) about their experiences in one university in this space.

He identifies four main differences in the ways that practice is organised:

  • Developing staff or developing courses (p.19)
  • Implementing or adapting institutional strategy (p.20)
  • Drawing together – systems or community (p.22)
  • Reframing technology or reframing the user (p.23)

Ultimately, Hannon finds that:

it is neither institutional strategy nor learning technologies that impose these constraints, rather the discourse or repertoires associated with their operationalisation (p.27)

I’ll certainly be coming back to this paper in the future.

Hicks looked at issues more in relation to the role of Academic Developers – and people working in Education Support units – as ‘change agents’, caught between the strategic requirements and priorities of the university executive and the needs of teachers and learners.

She felt that the voice of academic developers is seldom heard in research in this field and takes time to address this within a Bourdieuian framework emphasising social systems by inviting developers to participate in a number of focus groups.

Hicks’ paper sits well alongside most of the other papers that I have looked at already, with a focus on the tensions between academic and professional staff as well as academic staff and ‘management’ – with the ESSes torn between the two and underutilised.

This paper may be a useful source of additional supporting quotes and could also be worth reviewing when I get to university as an organisation.

David Boud is a major figure in research into Higher Education in Australia, (Angela Brew presumably is as well but it’s Boud that I’ve heard more about to date), so I was keen to read this one.

The idea of practice theory (Kemmis) is something that I keep coming across (and has also been suggested by my supervisor) and it’s at the heart of this paper. In a nutshell, it’s about framing academic work as practice and considering three key foci

practice development, fostering learning-conducive work and deliberately locating activity within practice. It also suggests that academic development be viewed as a practice (p.208)

Given that my new area of exploration is teachers/teaching/academics, this is a timely examination of academic practice that I will absolutely be delving into in far greater depth. It also offers a nice bridge between these two areas and I think it will also help me to inform my other (professional) work.

This paper presents a solid overview of tribalism in academia and the emergence of Higher Education as a field of study in its own right that needs to be claimed by academic developers. (I’d wonder whether an idea of “academy developers” is more fitting here).

One thing that I’ve come to realise in this sector is that trying to take on organisational cultural issues directly is unproductive, so while I’d prefer tribalism to be replaced with the embrace of a broader notion of being part of a collaborative community of scholars, I realise that it won’t happen any time soon. I guess the real questions are; do the members of a tribe respect the knowledge of another tribe and is teaching and learning in Higher Education something that can be owned by one tribe? Perhaps something more along the lines of tribal elders – strictly in the H.E T&L discipline area, never the ‘academy’ itself – could work?

When it comes to the role of ESS, I note that the authors quote Rowland et al (1998), which has popped up in most of these papers and is high on my list of future reading. It’s a fairly brutal quote however.

[a]t best, they [i.e. academics] view those in these [academic development] units as providing a service to help them teach. At worst, they ignore them as lacking academic credibility and being irrelevant to the real intellectual tasks of academic life. (Rowland, Byron, Furedi, Padfield & Smyth, 1998, p.134) (p.10)

This is certainly another paper to read in full as I explore the idea of academic work and teaching.

This final paper by Lee and McWilliam leans heavily on Foucault and “games of truth and error” and a fairly specific idea of irony. It again explores the tensions that academic developers encounter in the space between executive/management priorities and teacher needs. As someone that hasn’t yet explored Foucault, I imagine it might be of value if this is theoretical direction that I choose but for the most part I just felt that I didn’t get the joke.

Ok, so hopefully this give me a decent starting point for writing something about the literature as it relates to education support staff (obviously there is always more to explore but the best writing is the writing that you’ve actually done and having something to show will make it easier to find the gaps – both in ideas covered in the research as well as in what I’ve been reading and not reading.

Onwards to teachers.

 

Thoughts on: ‘Sleeping with the enemy’: how far are you prepared to go to make a difference? A look at the divide between academic and allied staff (Wohlmuther, 2008)

At this stage of looking at the matter of professional staff and academic staff in Higher Education, I feel that I’m somewhat flogging a dead horse and everything that needs to be said, has been said. So why am I still looking at this paper? Initially I was concerned that it grated on me because it doesn’t fit with my current narrative that there are significant cultural factors in universities that make it unnecessarily difficult for professional staff – particularly those in education support roles – to be heard when it comes to discussing teaching and learning.

If this was the case, I’d clearly not being doing my best work as a scholar – open to new information and willing to reconsider my world view in the face of it. Having looked over the paper a few times now though, I have to say that I think it’s just not that great a piece of research. A number of assertions are made that simply aren’t supported by the evidence presented and some of the reasoning seems specious. Events from four years prior to the publication date are referred to in the future tense but there is no discussion of whether they happened or what the consequences were.

Assuming that this is poor research – or perhaps poor analysis – it makes me happy that I’ve reached a point where I can identify bad work but also a little concerned that I’m wrong or I’m missing something because this was still published in a peer reviewed journal that I’ve found a lot of good work in previously. (Then again, I assume that most journals have their own favoured perspectives and maybe this was well aligned with it). I searched in vain to find other writing by the author but she appears to be a ghost, with no publications or notable online presence since the paper came out.

In a nutshell, based on an anonymous online survey of 29% of all staff – academic and professional at her institution, which included questions about demographics, perceptions of the nature of their roles, the ‘divide’ and the value of different types of staff in relation to strategic priorities, the author concludes that there is minimal dissension between academic and “allied” staff and most of what little there is, is felt by the allied staff.

Now it’s entirely reasonable that this may well be the case but there are a few elements of the paper that seem to undermine the authors argument. Wohlmuther asks survey participants about their perceptions of a divide but doesn’t dig directly into attitudes towards other kinds of staff, which McInnis (1998), Dobson (2000) and Szekeres (2004) all identified as central factors. She looks at the perceptions of contributions of academic and allied staff members to the strategic goals of the organisation which obliquely explores their ‘value’ within the organisation but it seems limited. Given the ambiguous value of some higher level strategic goals (Winslett, 2016), this would seem to tell an incomplete story.

The greatest weakness of the paper to my mind is that ‘allied’ and ‘academic’ work roles are unclear.

Survey respondents were asked what percentage of their time they spent on allied work and what percentage of their time they should spend on allied work. The term ‘allied work’ was not defined. It was left to the respondent to interpret what they meant by allied work (p.330)

With no further examination of the responses via focus groups or interviews, this alone (to me anyway) seems to make the findings murky.

She found that only 29% of staff – all staff? that is unclear – felt that there was “good understanding and respect for the significance of each others roles and all staff work well together” (p.331) across the institute, however doesn’t take this to be an indicator of division.

Looking over the paper again, these are probably my main quibbles and perhaps they aren’t so dramatic. This tells me that I still have a way to go before I can truly ‘read’ a paper properly but I’m on the way

 

Making a PhD plan

One of the greatest challenges of my chosen research topic (how universities can support tech enhanced learning and teaching practices) is the sheer breadth of it. Obviously this focus will narrow in time but at the moment it can sometimes feel a little overwhelming, even when I get into an interesting sub-branch.

As a result, I’ve been pretty good at gathering resources to read later and dipping into a few papers that seem particularly relevant to my work-work in the here and now. Overall however it is a very scattered approach and while some projects at work-work are helping to give me a hands on view of some of the real issues and factors that act as barriers to TELT practices in Higher Ed (so many barriers, so many dumb, needless barriers), I am keenly aware that I’m drifting a little and need to be looking more through the frame of the research proposal in the first instance.

So I spent the weekend organising my thoughts, setting some priorities and starting a plan. (I do really hope that this isn’t just a more advanced form of procrastination – it feels more than that at the very least.) I’ve been drawing my approach partially from David Allen’s Getting Things Done methodology, which is in some ways a fancy process for making to do lists but with a little more forethought. It encourages a very large scale gathering process to begin with, capturing all of the small tasks (and larger projects which are made of many small tasks) and ideas and putting them into categories in an active storage system that you trust. (I’m using Wunderlist which I like for the interface, cross-platformness, the ability to create collections of lists and integration with browsers.

wunderlist screenshot

I’ve got tasks organised by to do today/this weekend/this week/this month as well as in a range of contextual categories. I’ve tried to estimate the time each task will take to give me a sense of how long I need to allow to get it done. I’ve ordered them to set priorities (and because some tasks are contingent on other ones).

Now it’s just a matter of doing them.

Probably the biggest thing I’d like to do is to identify 6 (or so) key topic areas to spend a month each focussing on. These may well change over time but at least it will give some much needed direction.

Writing more regular blog posts about my progress is also another must – trying to keep myself honest.

Thoughts on: Two guides from Ako Aotearoa on education projects and researching learners

It was always my intention that researching in the area that I work in would help me to shape my professional practice (and it is) but I’ve been surprised lately at how much things are flowing in the other direction. I’ve been thinking a lot lately about what is needed to make an educational project successful and how we know that learners have actually benefitted.

This is partially coming from the big picture work that I’m doing with my peers at the university looking at what we’re doing and why and partially from my own college, which has recently launched a Teaching and Learning Eminence Committee/project to look into what we’re doing with teaching and learning. I wasn’t initially invited onto the committee, (it’s all academics), which speaks to some of the ideas that have been emerging in some of my recent posts (the academic/professional divide) as well as the fact that I need to work on raising the profile of my team* and understanding of our* capacity and activities in the college.

Anyway, while trawling through the tweetstream of the recent (and alas final) OLT – Office of Learning and Teaching – conference at #OLTConf2016, I came across a couple of guides published recently by Ako Aotearoa, the New Zealand National Centre for Tertiary Teaching Excellence, that fit the bill perfectly.

logo

One focusses on running effective projects in teaching and learning in tertiary education, it’s kind of project managementy, which isn’t always the most exciting area for me but it offers a comprehensive and particularly thoughtful overview of what we need to do to take an idea (which should always be driven by enhancing learning) through three key phases identified by Fullan (2007 – as cited in Akelma et al, 2011) in the process of driving educational change – initiation, implementation and institutionalisation. The guide – Creating sustainable change to improve outcomes for tertiary learners  is freely available on the Ako Aotearoa website, which is nice.

I took pages and pages of notes and my mind wandered off into other thoughts about immediate and longer term things to do at work and in my research but the key themes running through the guide were treating change as a process rather than an event, being realistic, working collectively, being honest and communicating well. It breaks down each phases into a number of steps (informed by case studies) and prompts the reader with many pertinent questions to ask of themselves and the project along the way.

The focus of the guide is very much on innovation and change – I’m still thinking about what we do with the practices that are currently working well and how we can integrate the new with the old.

The second guide – A Tertiary practitioners guide to collecting evidence of learner benefit – drills down into useful research methodologies for ensuring that our projects and teaching practices are actually serving the learners’ needs. Again, these are informed by helpful case studies and showcase the many places and ways that we can collect data from and about our students throughout the teaching period and beyond.

It did make me wonder whether the research mindset of academics might conventionally be drawn from their discipline. Coming from an organisation with an education and social science orientation, one might expect an emphasis on the qualitative (and there are a lot of surveys suggested – which I wonder about as I have a feeling that students might be a little over-surveyed already) but the guide actually encourages a mixture of methodologies and makes a number of suggestions for merging data, as well as deciding how much is enough.

Definitely some great work from our colleagues across the ditch and well worth checking out.

(* The team is me – but one day…)

Thoughts on: “Doing Insider Research in Universities” (Trowler, 2012) Part 4 – Social practice theory

Ok, so one more post and I can put this book (Is it still a book on a Kindle?) down.

Social Practice theory is a direction that I’ve been encouraged to explore by my supervisor and I can see that if offers some promise if I do end up travelling down “how things work in organisations” type path.

I have to be honest, I understand that theory is important in underpinning the shape of one’s research and in being able to make a contribution to the scholarly world but what I’ve seen of SPT so far feels a little simplistic. Again, this is kind of the point of theory in that “making the familiar strange” type way and I have no doubt that the further into it I go, the more complex it will seem. On the plus side, there’s nothing in it yet that I strongly disagree with.

To paraphrase (horribly) my current understanding – the things that people do are influenced by their contexts and particularly the physical/material aspects of the things that they interact with. (The actions also influence the form that the material things take). People have a set of things that they do regularly and routinely. The contexts and external actors on these practices are important and should not be overlooked – these include the locations and groups where the actions take place.

Some interesting quotes from Trowler:

Practices have an evolving trajectory, rarely a revolutionary one

The process of context generation… is a very significant factor in changing organisations

Change initiatives work best when they are grounded in current sets of practices and build on them

(That has been a fairly common constant in much of what I have been reading of late – we want to scaffold “innovation” where possible. That and benchmarking really)

Trowler identifies what he calls eight “moments” in teaching and learning regimes (TLRs):

 which shape contextual concerns and which depict the social practices in place.
1. Recurrent practices
2. Tacit assumptions
3. Implicit theories of teaching and learning
4. Discursive repertoires
5. Conventions of appropriateness
6. Power relations
7. Subjectivities in interaction
8. Codes of signification

I’m not yet sure how I’ll start to unpack these but they appear to merit further consideration.

He wraps the book up with a discussion of further resources and some of the gaps in the current literature. He notes that there has been much less research looking into the organisational and management sides of higher education than other areas.

He goes on to examine some of the more common methodologies used in insider research in higher education, which gives me a few more leads to follow.

Common research methods or methodologies used in higher education research generally are: documentary analysis; comparative analysis; interviews; surveys; multivariate analyses; conceptual analyses, phenomenography; critical/feminist perspectives; and auto/biographical and observational studies

As an entry-way to this particular type of research, I found this book invaluable to setting the scene, systematically laying out the pros and cons of a range of approaches and providing some theoretical frameworks to wrap it up in. In fairness, I’m still at the stage where I don’t entirely know what I don’t know but there is an evenhandedness and accessibility to his writing that inspires some confidence.

I feel relatively confident that, given enough thought, it will be possible to conduct some interesting and worthwhile research in my institution.

 

Thoughts on: “Doing Insider Research in Universities” (Trowler, 2012) Part 3 – Good research design and ethics/politics

I’m not sure that this is how I’ll process all of the books that I read – in fact I’m almost certain that it isn’t – but I’ll continue this series of posts about Trowler – Doing Insider Research in Universities because I have found it to be a great way to dip my toe into the many issues that I will face in my research.

The next two chapters look at value and robustness in insider research (which, again, I take to be about being able to defend your methodology – and choosing a good one) and then the ethics and politics of insider research in your university, which is pretty much unavoidable.

He opens with a discussion of some of the criticisms that case studies face as well as some of the responses to these. I’ll leave it here in full as it sums it up well.

Case study researchers may find their work subject to the following criticisms (Flyvbjerg, 2006): it only yields concrete, practical knowledge rather than the supposedly more valuable general, theoretical and propositional knowledge;  generalization from one case is not possible and so the research does not contribute to the development of knowledge; the research can generate hypotheses, but other methods are necessary to test them and build theory; case study design contains a bias toward verification, i.e., a tendency to confirm the researcher’s preconceived notions. Both Flyvbjerg and Yin (2009) refute these criticisms, but those who research their own universities need to be clear about precisely what their research questions are, what the rationale behind the research design is, and what the truth claims are. This advice holds for any kind of research, but other designs tend to draw less critical fire.

(He also highlights Gomm et al (2000), Simons (2009) and Yin (2009) as great starting points for further investigation of case study methodology)

Trowler dips back into some of the ontological questions that he touched on earlier in the book, comparing the merits of the true vs the useful. (I may be oversimplifying this). This draws on Sayer’s notion of “practical adequacy” in prioritising the usefulness of information. I kind of get it but think I’ll need to dig a little deeper. I can see how some true things mightn’t necessarily be valuable but as for things that are kind of true…?

This is echoed in further discussions of Bassey’s idea of “fuzzy generalizations”. In short, this is about acknowledging that life is complex and theory won’t always accommodate the range of factors at play. So that rather than saying that in situation A, if B happens then C will follow, we might say in situation A, if B happens then C will generally follow between D and E% of the time. It’s not as neat and arguably not as helpful but no doubt more realistic.

In terms of the design of research, Trowler posits twelve questions for researchers to consider to test the rigour and quality of their proposed methods.

1. In designing the research, how do I know that my planned approach will answer the questions I have more fully than any other?
2. How do I design the research to take best advantage of the benefits of insider research while avoiding its pitfalls as far as possible?
3. Conceptually, how do I represent my organization, its culture and its practices? (And how does this representation shape my design?)
4. How and from whom will I secure access to the data I need? (Why them and why not others?)
5. Whom should I inform about the project, and how should I describe it, when I seek ‘informed’ consent? (And how might this affect my data?)
6. How will I ensure that the project is run ethically so that all participants and institutional bodies are protected? (While at the same time being as transparent as possible to readers so they can judge the robustness of my approach and conclusions?)
7. If I am using participant observation, what are the ethical issues in relation to the people I observe in natural settings? (And how might my answer to that question affect my data?)
8. If using interviews, what measures should I take to deal with interview bias? (And will they assure a sufficient degree of robustness?)
9. What should the balance be between collecting naturalistic data and formally ‘collected’ data? (And how can I offer assurances of robustness about conclusions drawn from both?)
10. How should I analyse the different forms of data I have, given that there will almost certainly be a large amount of various sorts? (And how do I ensure that sufficient and appropriate weight is given to each form of data in generating conclusions?)
11. How, and how much, will I communicate my findings to participants to ensure that they are happy with what I intend to make public? (And will this affect the way I present my conclusions to other audiences?)
12. Generally, in what other ways can I satisfy the reader about the robustness of my research and its findings?

(I was a little hesitant to just paste this in holus bolus but it all seems particularly valuable).

In very pragmatic terms, there will also inevitably be a number of ethical and political issues to consider when undertaking insider research in one’s own institution. The question of whether to anonymise the institution and the research participants is a live one – though at this stage, to me, it seems impractical and counterproductive, particularly when it could mean reducing the number of sources of data for fear of not being able to correctly reference them. The lack of transparency could also arguably lessen a reader’s view of the robustness of the research.

I also think that the question of to what extent people change their behaviour under observation is a valid one, however there are no doubt ways to mitigate this.

Politically, senior leaders in the organisation will imaginably want to feel confident that the research won’t damage the reputation of the university before granting access. Does this then lead to self-censorship and selective reporting if there are areas where there is room for improvement?

At a higher level of ethical debate, the selection of standpoints from which to pose questions and to begin observations and investigations can raise some concerns. If I fail to incorporate the perspective of voices that are less often heard, am I guilty of perpetuating the status quo?

Lots to think about and to be perfectly blunt, I would be naive not to factor in the question of what asking the wrong person the wrong question might mean for my career prospects in the organisation. Fortunately I have a little time to think about some of these things.

 

Thoughts on: “Doing Insider Research in Universities” (Trowler, 2012) Part 2 (Being an insider and research methodologies)

I just realised that Part 1 of these posts had been sitting in my drafts folder – waiting for ??? – so if it seems like I’ve had an incredibly productive burst of writing, yeah nah. (Which isn’t to say that I don’t have a fair bit to say about this book now, as I have found it particularly helpful)

One thing I have learned, sadly, is that the Kindle doesn’t sort “highlights” (selections that I’ve made in the text) by chapter, so my notes aren’t as useful to me as I might’ve hoped. I’ll know to add notes to the highlights as well next time, listing the chapter. (It is still pretty handy being able to collate them in the kindle.amazon.com/your_highlights page though)

Chapter 2: Researching Universities. 

Trowler looks here at some of the knowledge and data issues that we face when researching in universities. Primarily the nature of the different types of knowledge present – implicit and explicit.

Implicit relates to practice knowledge (a.k.a phronesis) that has been developed through experience and which can be harder to express in words. Things like how to deliver a rigorous and yet engaging lecture. Because it is lived knowledge, it can be argued that it carries more legitimacy than that which has come from theory. At the same time, it will be very important to remember that there can often be a gap between people’s perspectives of their actions and the actions themselves.

On the other side we find propositional or technical knowledge, which is more like your old fashioned book learning. Easier to express and generally more objective but if we take into consideration the fact that unis exist in different contexts and can have their own quirks, this knowledge is likely to be useful in different applications.

Trowler references Blackler (1995), who

distinguishes between five types of knowledge identified in the organizational literature: embodied; embedded; embrained; encultured and encoded. These terms describe, respectively, knowledge which lies in muscle memory and is seen in skilled physical performances (performative knowledge); that which is located in systematic routines; knowledge which lies in the brain (cognition); that which is located in shared meaning systems; and finally knowledge which is encoded in text of various sorts.

For the research that I’m considering – which currently seems to be drifting toward the role of the university as a whole (or as an “ecology” if you will), I can see that there may be value in most of these kinds of knowledge. I get the feeling that there is definitely going to be a need to consider some of the theories relating to organisations and management.

In a nutshell, I need to be mindful of the ontological and epistemological questions that come up relating to the nature of reality and of knowledge and have an answer ready for questions about how I have done so.

Chapter 3: Research design, data collection and theory

This is a major section for me as I’ve been out of the formal research loop for a long time. If you don’t read anything else in this book (and methodology is important to you), read this. This chapter runs through major methodological approaches, offering up pros and cons for all of them. Ultimately Trowler’s own research took an ethnographic case study approach in a single site and drew from four sources of data: “interviews, observant participation, documents generated within and without the institution and other studies of it”.

Having stepped us through the alternatives, Trowler’s case for his approach in his research appears sound. (Of course, at this stage I’m going off intuition rather than experience but my understanding is that as long as you can make a strong/valid enough argument for your methodology, you can pretty much do what you like)

He anchors this discussion with a comparison of two ontological perspectives (whether they exist as a dichotomy or as points on a spectrum is debatable, of course). There is the “realist” approach (more quantitative, tied to “correlations of a generalisable nature”) and the “social constructivist” (more qualitative, reality is more subjective, more specific but less generalisable).

A key question raised – and certainly one that I’ll need to consider – is that of single site vs multi site research. Beyond the basic logistical issues, there is also (for me) the matter of one site being insider research and another being outsider research. (Even though if I was to go with a second and even a third site, I have connections and relationships in both).

To be clearer, at this stage I’m considering primarily researching within my own university (ANU) and possibly the neighbouring University of Canberra. A third option could be my former employer, the Canberra Institute of Technology, a Vocational Education and Training provider. On the one hand, this would introduce a lot more variables to the things being examined – well-resourced, research “elite” university vs emerging, “standard” university vs resource-challenged teaching focused adult education provider. All of these variables will necessarily shape the data collected and the questions asked. On the other hand, being able to compare approaches to similar questions (how can we better support TELT practices) could well be more illuminating.

Trowler identifies 6 type of comparative projects of interest:

1. The factors influencing the success or otherwise of an innovation (for example around virtual learning environments’ deployment and use)
2. Approaches to management and leadership and their effectiveness
3. The implementation of a national policy
4. Compliance (or otherwise) with national quality (or other) guidelines
5. Professional practices in a discipline or field of study
6. Student responses to an innovation

To be honest, the added complexity feels like an overreach at this juncture – I’d rather do something simpler well. It does open the door for further research down the track of course.

Of the methodological approaches discussed, I’d summarise them as follows:

Ethnographic – Participant observation
Researcher ‘lives with the natives’ for prolonged periods of time, fly on the wall type observation, often starts froma particular standpoint, “should be for people and not just about them”, language and textual objects vital,

Action research
Focused on solving particular problems, researcher is often a practitioner, iterative processes

Evaluative research
“Evaluative research in higher education aims to attribute value and worth to individual, group, institutional or sectoral activities happening there”. Care needs to be taken to ensure that it makes a larger contribution (theoretical/methodological/professional) to knowledge in the academic world

Hypothesis testing
Often trying to replicate findings of other studies, may try to improve on methodological flaws in prior studies, applying and extending theory

From here, Trowler moves on to discussing the importance of connecting theory to research. From what I can see, it seems to be partially as a way of finding a standpoint and a set of questions and partially as a way of testing some of the hypotheses.

He offers a solid overview of the characteristics of theory:

1. It uses a set of interconnected concepts to classify the components of a system and how they are related.
2. This set is deployed to develop a set of systematically and logically related propositions that depict some aspect of the operation of the world.
3. These claim to provide an explanation for a range of phenomena by illuminating causal connections.
4. Theory should provide predictions which reduce uncertainty about the outcome of a specific set of conditions. These may be rough probabilistic or fuzzy predictions, and they should be corrigible – it should be possible to disconfirm or jeopardize them through observations of the world. In the hypothetico-deductive tradition, from which this viewpoint comes, theory offers statements of the form ‘in Z conditions, if X happens then Y will follow’.
5. Theory helps locate local social processes in wider structures, because it is these which lend predictability to the social world.
6. Finally, theory guides research interventions, helping to define research problems and appropriate research designs to investigate them.

One of the goals of theory seems to relate to being able to “render the normal strange”. A challenge to face as an insider researcher – and particularly in the face of information sources that will come with their own biases and preconceptions – is to maintain that objectivity.

One way of addressing this is by “the application of Stenhouse’s (1979) notion of the ‘second record’ (the use of a detailed understanding of meaning systems to ‘read’ interview data) to this kind of secondary data about the institution.”

Trowler mentions that he probably collected too much data which caused problems in the data analysis phase – no idea how to deal with that but this is something that I will look for in my further reading about research practices.

This chapter also includes a solid description of the methodology that he used in his own PhD research – I think what I’m going to need to do is to create a spreadsheet or database outlining different methodologies – that should be particularly helpful. (Already partially discussed above).