External Regents Review
External Regents Review of the Department of Instructional Technology Utah State University
2001 External Regents Review (full version) .pdf
Submitted by:
Michael Hannafin - University of Georgia
Tom Hilton - Utah State University
Tom Schwen - Indiana University
May 18, 2001
Introduction & Background
Utah State University's Provost, in accordance with administrative policies requiring periodic external program review, commissioned this report. The review involved an analysis of several documents provided by the Department of Instructional Technology (IT), numerous interviews with USU students, staff, faculty, and administrators, as well as interviews with recent graduates and current employers of program graduates and interns. This report is organized according to the guidelines provided by the Provost.
Overview of Major Programs of Study
USU offers varied graduate IT degree programs at the masters (M.S. and M.Ed.), specialist (Ed.S.), and doctoral (Ph.D.) degree levels. The emphases and job placement history in these programs is mainly in training in business and industry, school media, classroom teaching, and higher education. In addition, the department offers an undergraduate minor in multimedia development, and has engaged in discussions related to both a collaborative graduate program with the School of Business as well as a distance-based graduate program in support of the training and related needs of regional business and industry.
Students, Faculty, and Staff
The department has identified nine FTE faculty members plus five adjunct faculty and two faculty who have administrative appointments at Utah State University. Two staff members support the academic and administrative operations of the department. Approximately 150 students are enrolled in the various degree and certificate programs offered-roughly half work in public schools and half are interested in entering the corporate world. In addition, the department employs several individuals in support of computer laboratories and facilities it manages: one full-time manager, four part-time employees, and three graduate assistants.
Role and History of the Department
According to the self-study report, The Instructional Technology Department was initiated in 1966 as the Department of Instructional Media and Library Science. Faculty were gradually added on full and part-time bases and shared with other programs in the College of Education and University. In the late 1960's-early 1970's, the program focused mainly on the preparation of public school library/media specialists. Starting in the mid-1970's and continuing to the present, the emphasis was broadened to include applications in business, industry, and government. Increases in the number of both faculty and students, as well as increased national visibility and prominence in the IT field, accompanied these shifts under their long-time department head, Dr. Don Smellie. With his retirement in 1999, the transition from quarters to semesters, and the recent appointment of a new permanent department head, the timing is good to gauge progress, assess needs, and act strategically to ensure the program's continued excellence.
Special Facilities or Activities Managed by the Department
The department manages ITRS which consists of two computer teaching labs, several computer servers, and specialized multimedia production and development technology. The Department also maintains a technology-equipped classroom. A "real life" approach is incorporated into project based class assignments, where class members serve as a production team to area business clients. Company representatives hold conference calls with teams and occasionally even visits a class. Internships-required for the M.S. degree-are another means through which the department stresses the development of authentic, real-world instruction and training products and programs. Another noteworthy enterprise is the annual Summer Institute, which has for the past 12 years attracted scholars and major industry leaders in the field to Logan.
Review Activities
The review committee included 2 external members (Dr. Michael Hannafin, Professor of Instructional Technology at the University of Georgia and Dr. Tom Schwen, Professor of Instructional Technology at Indiana University) and one USU faculty member from a department outside the College of Education (Dr. Tom Hilton, Professor of Business Information Systems and Education).
The Department of Instructional Technology's Self-Study Report and detailed Appendix were provided roughly one month prior to the committee's site visit. The Report provided a summary of the department's self-assessment, program offerings, strengths, and needs and served as the primary source material for the review prior to and during the site visit. These documents were reviewed independently by committee members.
The Office of the Provost provided the Review Committee's charge roughly one week prior to the site visit. The site visit was held during April 2-3 in Logan, Utah. The itinerary included interviews with administrators at the University level (President, Provost, Assistant Provost, Dean of Graduate Studies) and in the College of Education (Dean, Associate Dean for Research), as well as numerous interviews with current IT faculty, staff, and students. Additionally, IT facilities were toured and several program alumni and employers were interviewed either in person or via teleconference. Several individual interviews were also scheduled to provide greater focus to specific questions or issues as they emerged.
Finally, following an opportunity for the review committee to exchange perspectives and impressions, exit interviews were held with the IT faculty, the Dean of the College of Education, and the Provost and Assistant Provost.
Evaluation Criteria
As noted previously, evidence presented in the Self-Study Report and Appendix documents provided the most significant sources of data before, during and after the site visit. The interviews provided both the opportunities to confirm or question the observations and conclusions summarized in the Report as well as to identify questions and issues not addressed in the Report. The criteria used reflected the specific committee charge. Both formal and informal criteria are summarized below.
Mission & Goals
The department's mission and goals were examined both with regard to USU's mission and peer state-supported, land grant institutions, as well as with regard to their implications for attaining and sustaining program excellence and a national-international reputation for scholarship and intellectual leadership described in the Report. This was especially important in assessing the balance of contributions and in examining the potential tradeoffs involved in each.
Program Effectiveness
Several indicators of program effectiveness were considered. We began with the department's self-assessment as perhaps the most relevant immediate indicator of effectiveness. In addition, we obtained informal assessments of program effectiveness from College and University administrators, including perceptions based on word of mouth, program involvement with key initiatives, student end-of-course ratings, "customer" perceptions, new internal initiatives undertaken during the past 5 years, external grant activity, and reputation of individual faculty and of the overall program. We also queried current students and recent graduates as well as employers of program graduates for perceived strengths and weaknesses of IT students. Finally, we compared the three review committee members' peer perspectives of program quality.
Students
Indicators of student quality included both traditional test scores and performance indicators: GRE scores and undergraduate (or graduate) grade point average. In addition, the demographics of current enrollees were examined to establish diversity of the incoming students. The placement and job performance of program graduates was also considered. The review reflects the students as a group; no attempt was made to associate data or findings to individual students.
Faculty
Faculty indicators included regular teaching load and credit hour production per FTE, end-of-course evaluations, overload teaching and/or project work, committee and service assignments, advising load, graduation rate across advisees, scholarly productivity in terms of refereed journal articles and conference presentations, grant dollars generated, and local, regional and/or national role and recognition. As with students, no attempt was made to relate data or findings to individual faculty members.
Administration
The influence of administration was examined in two primary ways: 1) internal to the department (i.e., IT department head); and 2) external to the department (i.e., college and university level administration impact on the program). Though important in the review, this was mainly of a self-report and anecdotal nature derived through interviews.
Resources and Facilities
Evidence of the quality of resources and facilities was derived based on tours and demonstrations, reports by faculty and students most directly effected, and an informal comparison between USU and the comparable peer programs represented in the review.
Departmental Strengths
This part of the evaluation report contains a summary of the strengths of the Instructional Technology Department as identified by the review committee. Note that areas for improvement and recommendations for specific action are in subsequent sections of the report.
Committee members regard the Instructional Technology Department as very strong, a department others would do well to emulate. Its strengths far outweigh its shortcomings. Following is a summary of strengths we found noteworthy:
Faculty
The Instructional Technology Department faculty is impressive. We found them as a whole connected with their students, the University, and their field. We found them productive in teaching, with a large number of both masters and doctoral students. The number is respectable for any department and more impressive in light of the fact that there is no undergraduate program in the Department. A few Instructional Technology faculty are also highly productive and nationally prominent researchers. We found several large grants funding their research that totaled several million dollars. We found them presenting regularly at relevant conferences and publishing regularly in high-quality peer-reviewed journals. Finally, the faculty is also conscientious in giving service to the University and to the field of instructional technology. We found them involved in and contributing to an extraordinary number of campus instructional initiatives. We also found them involved in organizing conferences and representing the field to private industry and the public.
Students
The students in the Instructional Technology Department appear to be excellent. The average GRE score of students admitted to Department programs is about 20 percentile points above the minimum mandated by the USU School of Graduate Studies. The students we talked to impressed us as articulate, motivated, and friendly. They appeared well informed about their department, the University, and the instructional technology field. Virtually all reported significant involvement in research or instructional product development.
Mentoring
The Instructional Technology Department has developed a strong culture of mentoring their students. Every student to whom we spoke had an assistantship. Every student had access to faculty for help with class work, thesis or dissertation research, and other publication efforts. We heard several reports that IT faculty routinely help their students attend conferences, sharing room and transportation costs; the department head reported devoting department funds to this activity as well. Indeed, the only complaint we heard in this area was that students wanted more; some students expressed the sentiment that recent changes to streamline the masters curriculum make it too short to learn, deeply, things they considered important. Faculty and students both reported strong placement support by the Department as well; we understand that virtually every graduate who seeks a job gets one.
Curriculum
In connection with the points above, the review committee found a relevant, state-of-the-art curriculum in place in the Instructional Technology Department. We found several well-equipped media labs, Web-enabled classrooms, and computer labs. We noted some discussion of strengthening the presence of the department in the University's undergraduate curriculum, with some consideration even being given to inaugurating an undergraduate major (although one motive seems to be to make courses for the graduate students to teach). We found a debate among both faculty and students weighing the relative merits of instructional theory and instructional applications, but we find such give and take stimulating to the progress of the department and the health of the curriculum.
Physical Plant
The Instructional Technology Department is housed on the second floor of the Emma Eccles Jones Education building, one of the newest buildings on campus. All faculty offices are on the same floor as the department office, and there is space available for graduate student offices and research projects as well as classrooms, labs, and faculty offices. We find that such physical arrangements promote collegiality, productivity, and good public relations.
Campus Citizenship
The Instructional Technology Department is both well known and well regarded throughout the Utah State University community. The Department offers a masters degree off campus via electronic distance education. Instructional Technology faculty provide expertise to the Faculty Assistance Center for Teaching (FACT), a center that supports any University instructor in developing instructional products of all sorts. In contrast to many peer programs nationwide, USU's IT program seems to have established important and valued roles in support of teacher education. IT faculty recently completed in-depth collaboration with three other USU departments in the development of an inter-departmental information technology major, and the Department head is conducting similar discussions for collaborative programs with other academic units on campus.
Support of USU Administrators
Every administrator the committee interviewed praised the Instructional Technology Department. The University president reported that the Instructional Technology Department is central to the future of the University as a whole, given the general movement to online and mediated instruction. The provost hailed the IT faculty's excellent record of research and teaching. The assistant provost noted the Department's recently renewed emphasis on grantsmanship as an important contribution. The dean of the School of Graduate Studies complimented Department administrators on their administrative efficiency. The dean of the College of Education told us he regards the Department as a role model for the rest of the College in admission procedures and ability to attract, educate, and place top-quality students.
Areas for Improvement
The review committee found areas in the Instructional Technology Department where improvement might be considered. We identify the following areas for possible improvement as ways to help maintain and enhance the position of the Department as a leader in the field of instructional technology.
Fragmented faculty vision
The review committee found, mainly in private interviews with faculty, diverse views on the mission, strategies, and goals of the Instructional Technology Department. While diversity of opinion is healthy, we noted reticence among the faculty to share their views with each other. We regard the primary benefit of such diversity as the cross-fertilizing and hybridizing of ideas that occurs in lively academic debate, but we noticed what appears to be an overly careful avoidance of disagreement. We believe that such scholarly diffidence can lead to a paucity of discussion and a stagnating of academic progress if left unchallenged.
Overextended commitments and competing priorities
While the Instructional Technology Department is highly regarded throughout the University community, that regard has come with a high price. We found several examples of programs implemented without sufficient budgeted faculty time and other resources. We found graduate students working in assistantships that bore little relation to their area of emphasis and instances of on-campus service opportunities that compete with teaching and research opportunities. Such conflicts are hardly unique to the Instructional Technology Department, but we believe the large number of conflicts found in this case to be problematic in the long-term.
Regionality
The review committee found that, for a nationally and internationally known department, the Instructional Technology Department was surprisingly regional in its activities. Most students come from the Mountain West, most job placements are in the Mountain West, and most faculty were recruited from the western United States. This is worrisome since the Department risks developing a parochial perspective and reputation if it does not build on its history of accepting students from and placing graduates with other instructional technology programs and markets around the nation and the world. This is particularly true for their doctoral students.
Transition Issues
The review committee found the Instructional Technology Department to be in a state of transition on multiple fronts. It is developing a new administrative structure and style as a result of the long-time department head's recent retirement. Faculty, staff, and students are still learning to function comfortably in the recently reworked semester-based curriculum. The doctoral program has just finished moving out of the College of Education interdepartmental doctoral program and begun operating as an independent program. Finally, several influential faculty who have long histories in the Department will likely retire within the next few years (although none of them has been forthcoming with details of their plans).
Support for Students
The review committee found that Instructional Technology graduate students have assistantships, which is a good thing. However, some doctoral assistantships contribute little to the graduate student's career development, hence reducing opportunities to teach, collaborate with advisors on research, and develop the student's own research interests. Some students reported that, at the time of their acceptance into the program, assistantship opportunities were offered in terms so vague that students were put in the position of having to develop their financial support plans without needed details such as remuneration level, time commitment, or even the general nature of the experience. Some students expressed concern that most assistantship opportunities came without the prospect of medical insurance; they were specifically critical of the student medical insurance offered by the University as being overly expensive, lacking needed benefits, and being extremely poorly administered. Finally, the review committee found that students were generally concerned with a dearth of teaching opportunities. This was particularly worrisome to doctoral students planning academic careers.
Recommendations for Improvement
Focus on National/International Prominence The review committee lauds the Instructional Technology Department for its accomplishments and for maintaining a top tier program for the past two decades or so. We strongly encourage the Department to continue this focus on national-international prominence. The faculty is encouraged to work together in an open, collegial fashion to forge a consensus on specific means for maintaining and enhancing this prominence. The review committee believes the faculty could elect to continue past practices, or they could agree to move in new directions; the important point in this recommendation is that they develop a united focus on cooperating to enhance their national prominence.
Address Contextual Issues
The review committee sees at least three general contextual issues that frame the other issues facing the Instructional Technology Department. We recommend that these be addressed as part of all other initiatives and as a prerequisite to developing the focus recommended in the prior paragraph.
Explicitly discuss tacit taboos
The review committee recommends that the issues presently inhibiting the free and frank exchange of ideas among the faculty be identified and explicitly discussed. The review committee perceives that some of these issues could be differing religious convictions, differing perceptions of various faculty members' relative success or importance, and differing perceptions of various faculty members' administrative influence in the department and across campus. Doubtless, individual faculty will have different views on these issues, so this list should not be taken as particularly accurate or complete. Rather we offer it as a starting point for discussion that can lead to the identification and discussion of these hitherto subliminal influences.
Recognize and plan for transition
The review committee recommends that the faculty, staff, and students cooperate in open discussion of the issues of transition facing the Instructional Technology Department. Particular recommendations are that the Department engage in detailed planning of the new administrative structure, continued adaptation of the curriculum to the semester schedule, continued refinement of their independent doctoral program, and upcoming retirements.
Reallocate and obtain resources
The review committee recommends that the Instructional Technology Department establish explicit priorities for its various programs and reallocate department resources to reflect those priorities. For instance, in line with the review committee's recommendation to focus on national prominence, the Department might consider allocating fewer resources to on-campus services. If all present commitments are to be maintained, the review committee strongly recommends allocating additional resources to the Department from the College or University. For instance, distance education programs and FACT Center participation should likely be entirely funded extra-departmentally.
Aggressively Recruit Top PhD Students Nationwide
The review committee recommends that the Instructional Technology Department begin as soon as possible an active program of national recruiting into their PhD program. One strategy is to target master's degree students from other top IT programs. This will go far toward reducing the regionalism we noted above.
Recruit Senior Faculty
The review committee recommends that the Instructional Technology Department start cultivating contacts now in anticipation of upcoming retirements. There should be no illusions that the right people to occupy the faculty position of M. David Merrill, J. Steven Soulier, or other retirees will naturally respond to a call for applications without any other effort. A coordinated effort is recommended to procure adequate funding to offer an attractive salary while simultaneously nurturing interest among potential applicants. This effort should in no way be constructed or construed as an exercise in dancing on the professional graves of the departing faculty; rather it must be seen for what it is: prudent-indeed necessary-planning for a smooth transition and continued success.
Strengthen Program's Identity Beyond Mountain West Region
The review committee recommends that, as part of the discussion of Departmental focus and priorities, the Instructional Technology faculty come to consensus on opportunities for leadership in which to excel. Such opportunities might be in distance education, business training, preparation of public educators, a particular research stream (e.g., ID2, learning objects), a funded center for instructional development, or preparation of PhDs for the higher education academic community. The review committee suggests that, although several of these efforts may possibly coexist, it is important that a clear focus and identity be established and cultivated that reflects the vision and focal point of department accomplishment.
Develop Programmatic Student Support
The review committee endorses and supports the Department's goal of preparing PhD graduates for careers in higher education. Increasingly, opportunities for teaching, research, project work, grant writing, and the like are important to prospective university faculty. Such support also infuses additional resources into the ongoing scholarly and teaching activities of the department. The review committee recommends that the assistantships for graduate students be actively managed so that they can be allocated to complement student interests. Some kind of deliberate management body (e.g., all graduate committee chairs, all doctoral faculty, a designated department committee, etc.) might convene regularly to project needed opportunities for students, identify potential resources to address these needs such as externally funded research projects, and evaluate and match assistantships with students.
Additional Information
Several additional questions were posed by the Provost's Office:
What is the State of the Discipline, and how well is it reflected in the department? How do the goals of the department compare to the rest of the profession?
Instructional technology is a field (similar to artificial intelligence for example) where relatively few scholars have a disproportionate influence given the size of the field. There are relatively few first-rate doctoral degree granting institutions in the US and significantly fewer worldwide. The practitioners of instructional technology range from first time users of PowerPoint to sophisticated designers of performance support tools. The community of scholars in IT is quite small and the opportunities for contributing to the discipline and society have never been better. USU's IT Department has achieved substantial national recognition largely on its excellent regional service and a few nationally prominent scholars. The potential is clearly there for national and international prominence, but the modus operandi would need to change. While the goals of the department reflect a desire for national prominence, a regional orientation is still central in recruiting students and faculty, institutional support for research, extensive service to the region, placement of graduates, and related matters. We admire what has been accomplished; we merely note a discrepancy between goals and methods. One or the other would need to change to achieve organizational consistency.
What should be the roles of this department in the university? Is it successfully fulfilling these roles?
As we noted, the department has an excellent record of service to the region, including service within the university. Often university services by members of the department or alumni have extensive regional significance. The teaching support service, the appointments of alumni in other academic departments, the excellent relationships with employers and alumni all contribute to an exceptional public image. The risk to the department and the university is that this exceptional regional service orientation (and resulting demand) may detract from its national and international potential. From our standpoint, thoughtful planning that focuses service to a few well-chosen goals would make more sense than addressing a multitude of requests that can only be met nominally. As in other high tech fields, there will be many more opportunities for service than can be reasonably fulfilled. The very savvy programs of the future will select arenas of service that are mutually beneficial. The potential of this department to contribute is immense, the service orientation is admirable, and the most important risk is from over-commitment.
How does the quality of this department compare with departments elsewhere in areas of teaching, scholarship, students and outreach?
The department is clearly a top tier program. It is easily in the top ten and probably in the top five programs in national reputation and recognition. Its strengths would be in teaching and increasing scholarly productivity. The students as a group, while capable and talented, rarely leave the region upon graduation. This is a long-term concern to the program's national and international aspirations. The program's reputation is divided between a few well-known national-international scholars and others dedicated to excellent regional service. The tension in reconciling an integrated mission and vision across potentially competing initiatives can be difficult to overcome. Some long-standing programs, such as school library media certification, are seriously understaffed while new areas of research show great promise but will also require resources. The goal is to achieve synergy among initiatives whenever possible since the resources required to sustain and nurture multiple initiatives can overtax even large, well-supported programs.
Given the resources at hand, are they being used effectively used?
The faculty, staff and students show remarkable creativity in supporting a high tech curriculum. The department addresses a wide range of needs with comparatively modest support. Given the resources at hand, the program faculty is over-extended. The program trains a significant number of students, supports large numbers of students in a variety of assistantships across campus and the community, and graduates and places students in the region. By these indicators, the program is highly productive and successful. However, the consequence may be declining national and regional impact. Available resources are not only consumed in addressing the regional mission, but opportunities for external resources and impacts may be forfeited in the process. For example while many students are aided in finding support outside of the department, the outside commitment frequently militates against close programmatic mentoring between faculty and students-a hallmark among programs whose graduates influence the intellectual direction of their field. The consequence of resource allocations on leadership aspirations-local, regional, national, and international-needs to be weighed carefully.
Provide recommendations concerning the direction the department should take if…
Resources remain constant. If resources remain constant, the program should consider becoming more focused and selective in their allocation. Several low- or no-cost actions could be considered, such as alternatives for recruiting diverse students and faculty. The program could deliberately focus the Ph.D. efforts on select intellectual issues such as preparing graduates for faculty placements in well-established institutions, or preparing training leaders in the corporate world. The masters program could remain regional in focus but could profit by recruiting students from and placing them in a broader range of geographic locations. The balance between service goals in the curriculum and classic academic pursuits would need to be resolved more clearly. If resources remain constant, the viability of a simultaneously world-class and regionally responsive graduate program would likely need to be reconsidered.
Additional resources become available. If additional resources become available the program could carefully target its service so as to bring synergy to the goals of the program; recruit students and faculty with a greater emphasis on intellectual, gender, and ethnic diversity; promote a few well selected arenas of scholarship where they are clearly world class. These can be achieved with careful attention to USU's mission and culture. The program as noted above is in transition with key senior faculty on, potentially, short terms of service. The action plans for the department need to be addressed in serious, candid, prolonged discussions where the next generation of department faculty plays a significant role in defining its future. It is important to balance the perspectives and interests of those charged with ensuring the program's future with those who played significant roles in establishing the program to date. The deference of the new timers to old timers often accompanies-and complicates-such transitions. At some point those who will inherit the future of the program need to be encouraged to plan now for that eventuality. One key agenda item in such discussions should be the nature and form of senior faculty replacements. Clearly it would be important to recruit a nationally prominent senior faculty member. Salary and other traditional barriers to such recruitments would need to be seriously planned for and addressed. In addition there are several interesting possibilities for future development including: joint programs with other departments and undergraduate major etc. These possibilities need to be considered in light of our advice about targeting the department's efforts on a few well-selected goals.
Resources are decreased. If resources are decreased then serious attention should be given to cutting back on aspects of the program. The faculty is close to being over committed on several fronts. They could consider reducing service to the campus, reducing the variety of specialties in the graduate program. The goal of a sensible reduction in resources would be to find areas of current department commitment that can be optimized, and areas that can be reduced or eliminated. The program is currently maintaining unusual levels of quality with relatively modest resources. The department does more with less than other programs nationally. A serious cut in resources could create an unfortunate degradation of program quality and service. To survive such a cut and maintain quality, some services/ program elements might need to be eliminated entirely. We believe these decisions should be faculty decisions. An external review team may provide a reasonable assessment of USU's program reputation, contributions, and image to the broader IT community, but cannot judge well the importance of a program's internal contributions. Options that could be considered are: substantially reducing campus service, consolidating and simplifying the master's program, pursue external funds that focus on a regular source of income, e.g. routine R&D service to industry, convert some hard money faculty lines to clinical faculty lines etc.
Printer-friendly version
