Talk:Module 7-b: Reference Services
From DL Curriculum Project
Instructions
1. Click an 'edit' link to type in your comments. Your evaluation may cover general issues concerning the module or a section of it, or you may make more fine-grained comments (e.g., on a particular section by referring to the section number or on a particular point by referring to the page and line number).
2. Add four '~' (without quotations) at the end of each comment. Your id, date and time will show up later.
3. Click the 'Save page' button on the bottom of the 'edit' page.
Module Objectives: Are the objectives appropriate for the topic?
Are the objectives observable? Will students be able to achieve the objectives, given the content in the body of knowledge?
The objectives of the module focus on the basic knowledge of reference services and how they can be incorporated in different types of DLs. They are clearly stated and I can foresee several ways to measure the achievement of these objectives; so these objectives are observable.
The content in the body of knowledge covers various formats of reference services (human-intermediated and automated) and the intersection of reference services and DL; students should be able to develop a solid knowledge of reference service itself as well as its connection to DL, and hence achieve the three objectives in this module.
I only have a question about the terminology -- does the phrase “question answering (QA) services” in the second objective mean the same thing as QA systems (the automated QA systems discussed in page 4)? In page 9, services like Yahoo! Answers are also referred to as “QA systems” with no associated DL. --Lili Luol 14:08, 14 January 2008 (EST)
By QA services I meant TREC-style automated QA. But this is a good point, this module should differentiate between library-based human-intermediated reference services and community-based services. I will make changes to the module to include services like Yahoo Answers. Pomerantz 16:01, 14 January 2008 (EST)
The objectives seem reasonable to me; you use "varieties of service" in (a) and "forms of service" in (c), which I'm guessing is just for variety and not a meaningful difference. Janesj 17:46, 21 January 2008 (EST)
---
The objectives seem both clear and measurable. To help the students accomplish objectives A & C, you may want to include a section on determining who the audience(s) might be for a particular DL in order to determine the best mode of service. I would opt for 'mode of service' versus 'varieties'. Thanks for clearing up the QA question; that was also confusing to me. Perhaps there is a descriptive adjective that might clear that up? --Vondracekr 14:14, 23 January 2008 (EST)
Body of Knowledge: Does the module address all areas of the topic that need to be addressed?
Will the body of knowledge enable students to achieve the objectives? Are there any topics that you think are critical to add to the body of knowledge? Are there any topics on that you would remove from the body of knowledge?
This module’s body of knowledge is well-rounded and responds well to the objectives. I only have one suggestion: it might help students better understand the organization of DLs with reference services if there’s a discussion about how the reference service component is run in different DLs – is the service provided by experts affiliated with the DL, or by professionally trained volunteers; how are the policies set for the reference component in terms of what kind of questions get answered; etc.
Other than that, I think the body of knowledge has covered the key topics for this module and will enable students to master the major themes and achieve the objectives. -- Lili Luol 19:50, 14 January 2008 (EST)
I concur that by and large the body of knowledge is reasonable. I have a few specific comments:
p3 line 31; also allows the user not to have to wait around for a response
p3 line 34; worth mentioning environments such as Second Life?
p3 line 30s--lots of challenges here, in both domains (can you really do an interview asychronously, people hanging up online, feeling of impatience in the user and therefore rush on the librarian, etc
p5 line 8ff; perhaps worth exploring any evidence whether such knowledge-based systems really work or are worth the trouble? (viz QuestionPoint, IPL, etc)
Janesj 17:51, 21 January 2008 (EST)
I agree with Lili and Joe's comments, and think Joe's comment about evidence on whether or not knowledge-based systems work is a critical one. Again I would add in information on developing a profile on the intended audience(s) and determining the appropriate mode of service for that audience. Also assessment and usability may be covered elsewhere in the modules, but having something in place to measure the reference service would be appropriate. --Vondracekr 14:14, 23 January 2008 (EST)
Readings: Are the readings the best and most appropriate for the topic?
Are there any readings that you think are critical to add to the list? Are there any readings on the list that you would remove?
The readings are well-selected and they are the core readings for each topic included in this module. I don't have anything to add to the list; it's quite sufficient as it is now. -- Lili Luol 20:16, 14 January 2008 (EST)
They look fine (tho many are 3+ years old...pity there aren't more recent things). Readings on p7 on QA systems are particularly old; maybe include some of Oren Etzioni's work? Or developments at ask.com, Google, Yahoo, Windows Live, etc? Janesj 17:53, 21 January 2008 (EST)
Looks sufficient with Joe's additions. If something is added to the module about assessing user satisfaction or assessing the service in general, you may want to add to the list: "E-mail and chat reference: assessing patron satisfaction" Stoffel, Bruce and Toni Tucker. Reference Services Review v. 32 no. 2 (2004) p. 120-40., or Laurie Probst and Michael Pelikan report on the use of a “Tell Us What You Think” button to gather user feedback in Assessing Reference and User Services in a Digital Age, Hawaorth Press, 2007. --Vondracekr 13:12, 24 January 2008 (EST)[User:Vondracekr|Vondracekr]] 14:14, 23 January 2008 (EST)
Learning Activities: Are the activities appropriate for the topic?
Will students be able to accomplish the activities, given the content in the body of knowledge? Will the activities enable students to achieve the objectives? Can you think of any other class activities appropriate for this module?
I like the learning activities and assignments designed for students in this module. Completing these exercises and assignments, students will be able to reinforce their mastery of the major themes of this module and therefore achieve the learning objectives. I especially like the task of analyzing data from a DL associated reference service like IPL – this will help them have a concrete sense of what role reference services can play in DLs. If students with a CS background have difficulty conducting such an analysis, maybe this article will help “Carter, D. S., & Janes, J. (2000). Unobtrusive Data Analysis of Digital Reference Questions and Service at the Internet Public Library: An Exploratory Study. Library Trends, 49(2), 251-265.”.
Under the “Evaluation of Learning Outcomes”, the evaluation measure seems to focus on measuring the second learning objective "Articulate the differences between human-mediated reference and automated information retrieval (IR) and question answering (QA) services". Maybe more measures can be added for the other two objectives. For example, when provided with different DL scenarios, students are expected to assess a variety of factors and determine whether reference services will enhance the use of each DL, and if so, what type of reference services should be incorporated.
Also the Virtual Reference Desk’s AskA+ Locator's URL http://www.vrd.org/locator/subject.shtml no longer works. VRD is on webjunction now; I checked the webjunction site but couldn't find the AskA Locator. --Lili Luol 01:08, 15 January 2008 (EST)
I agree that the activities and evaluation mechanisms don't match up with objectives. In particular, there's little or no discussion (say p9 lines 20s) of users: who are the users of various systems, who might prefer what kinds of services (synch/asynch), and who might prefer different kinds of services for different questions (I use email to our library when I don't need an urgent or quick answer, chat when I do, phone when I can, for example). Which trumps which, and why?
The objectives could be measured by something more nuanced in writing, posing effectively those questions in some way, perhaps by letting students choose an environment and sketching answers to the questions for that setting? Janesj 17:56, 21 January 2008 (EST)
Yes, I agree with the above comments. --Vondracekr 13:12, 24 January 2008 (EST)
Level of Effort and Prerequisites: Is it feasible to teach the module as it is currently constructed?
Is the level of effort required in class appropriate to the scope of the body of knowledge? Is the level of effort required prior to class appropriate? Is the prerequisite knowledge required sufficient for students to comprehend the body of knowledge?
Yes I think the level of effort required prior to and in class is appropriate to the scope of this topic. An hour and a half is proper to cover the three major themes in this module, and an hour should be sufficient for students to try out three digital reference services as well as analyzing a few digital reference questions/answers.
The prerequisites cover the basic knowledge about information seeking, needs and technology that will prepare students for this module, so I think the prerequisite knowledge required is sufficient for students to comprehend the body of knowledge. -- Lili Luol 01:15, 15 January 2008 (EST)
90 minutes seems pretty thin to me, with this much ground to cover and topics that seem similar (mediated services vs QA vs IR) but are in fact quite different. If some of this ground has been previously plowed in other parts of the curriculum, it may be less of a concern, but I'd think at least 2 hours if not more would be necessary, especially with students with no "reference" background. Janesj 17:58, 21 January 2008 (EST)
This does seem like a lot to cover and comprehend in 1 1/2 hours. Two hours perhaps, possibly more. --Vondracekr 13:12, 24 January 2008 (EST)
Overall Structure of the Module: Is the current module well structured?
Can the topics and their corresponding resources be easily divided? Is there a clear mapping between the objectives and the content of the body of knowledge section? If not, how could the objectives be mapped to the body of knowledge more clearly?
The reference module is well structured, and the mapping between the objectives and the content is quite clear. I can easily which part of the content serves to render students’ achievement of which objective. No further clarification is needed. -- Lili Luol 01:18, 15 January 2008 (EST)
Additional Comments
It's been a pleasure to read through this module and provide my feedback on it. I'm glad to see that reference has a role in the DL curriculum. :-) This module is well-developed and well-organized, and it has given me some good ideas of teaching my own reference class. Great work! -- Lili Luol 01:32, 15 January 2008 (EST)
I'm concerned about the remedial material (section 8, beginning page 2). It is largely cast as being about the reference "interview" but much of what's in that section ii isn't the interview, it's the entire process. There's also a very heavy emphasis throughout on instruction, which of course is an important part of reference work but spotty at best (virtually none in special libraries, precious little in publics, academics when you can get people to sit still for it). No real discussion of the search process, presentation/synthesis of results, or evaluation of sources/decision making about them, all of which are crucial aspects.
Also, section iii (page 3, line 4) says it's about location but points talk only about instruction again.
This material as presented here would give a kind of erratic portrait of reference work; maybe use another Bopp & Smith chapter or so to further orient to the overall process?
Janesj 18:03, 21 January 2008 (EST)
--- I haven't seen the content in the prerequisite modules, so it's difficult to say if that provides enough background to understand what is different about reference within this context. Having a basic reference course does seem as if it would prepare the library students; computer science students may need something deeper than having used a service themselves, especially around the reference interview and the philosophies of reference and public service. All in all, I believe it is a thorough module and logical in it's approach. Like LiLi I'm very pleased to see a module on reference included in the DL curriculum. --Vondracekr 13:12, 24 January 2008 (EST)
