Study Design:
This
study compares and contrasts the web sites of the fourteen Pennsylvania
State System of Higher Education schools to identify those that
provide efficient, accurate, and effective access to a standardized
list of information resources. The study takes a two-fold approach
to achieve this goal. By examining the content provided and further
analyzing the presentation of that content, an evaluation can be
made, and a comparison can be drawn between institutional web sites.
The researcher first visited each school’s web site in August of
1999 and used the same instrument to revisit the web sites in May
2000 and March 2001 looking for twenty specific pieces of information
common to all state system schools.
Each
of these twenty items was evaluated based on five criteria:
1) Does
the school’s site provide the requested information?
2) How
difficult is the information to locate?
3) How
current is the information?
4) Does
the page use appropriate design skills and contain necessary navigational
elements (as defined by Patrick Lynch’s Yale
Web Style Manual [Lynch & Horton 1999]
5) Is
the page professional in appearance?
A point value
from one to five was used for each of the above questions for a
total of 25 points. These 25 points could be accumulated for each
of the 20 information items that were researched.
A total of 500 points were possible.
Assessment criteria assumptions:
As the Internet
continues to grow, a school’s web site is increasing becoming marketing,
recruitment, and public relations tools that educational institutions
need to closely monitor. A recent survey by the Slippery Rock University
Admissions office indicated that the school’s web site was second
only to family and friends in influencing their decision on applying
to the institution [Bracco 2000]. The importance
of a school web site outlining accurate and detailed information
for prospective students is clear. A well-designed web site providing
information that matches the audiences’ needs will result in higher
enrollment, increased alumni donations, and a greater level of credibility
for the institution [McCollum 1999].
This
study makes the following assumptions in the collection and analysis
of data:
·
University sites should all provide a baseline
of information resources
·
University site visitors should be able
to easily locate information without searching deep into a site
hierarchy
·
University site pages should provide current
information
·
University site pages should have a professional
appearance (good grammar, accurate spelling, consistent layout)
Identification of common information
University web
sites are designed to serve a variety of target audiences. These
audiences include but are not limited to: prospective students,
current students, alumni, institutional employees, community members,
researchers, and colleagues. Since university web pages are expected
to serve a variety of users, a list of information that these groups
might request was generated. That list was refined to twenty items
that would be typical of the SSHE institutions. Those twenty items
include:
1) Search engine
2) Site Map
3) Tuition and fee schedule
4) Academic Calendar
5) Distance education
course offerings
6) Library resources
7) Address of the institution
8) Registrar’s name and
address
9) How to order transcripts
10) Department listings
11) Academic department
chair (English)
12) Mission Statement
13) List of degree and certificate
programs
14) Admissions information
15) Name of the President
16) Name of the Provost
17) Calendar of Events
18) Link to State System
of Higher Education
19) Faculty directory
20) Alumni page
Data Collection
All data for each of the fourteen schools was collected during
the first week of August 1999, the second week of May 2000, and
the second week of March 2001. The researcher accessed each web
site attempting to locate the information from the list. Search
engines, directories, and browsing techniques were used to locate
information. The following point scale was used for evaluating question
#1; “Does the school’s site provide the researched information item?”
5 points – requested information
is provided
3 points – some of requested
information is provided
1 point – requested information
is not located
If
the information was not located, the institution received zero points
for the remaining four questions. The minimum score any school could receive
on any of the items was one point. Therefore, the lowest possible
score was 20, and the highest possible score was 500.
Once the information was located the shortest path to accessing
the information was determined. Significantly if the information
was located via a search engine, the researcher needed to determine
how this information was linked to the main page (on what level
of the hierarchy did it reside). The assumption for evaluation is
that if the user has to review fewer pages to find the requested
item, it is easier for them to locate the information. The following
point scale was used for question #2, “How difficult is the information
to locate?”
5 points – required access
of three or fewer pages
4 points – required access
of four pages
3 points – required access
of five pages
2 points – required access
of six pages
1 point – required access
of seven or more pages
The web page that contained the requested information was
evaluated to determine if the information provided was current.
Many pages list a last date of revision. Other dates could be determined
by the content on the page. These dates were used for the evaluation.
If no date of revision is provided, the researcher assumed that
information was current unless there were overt indications otherwise
(i.e. calendar of events for a previous year). The following point
scale was used for question #3, “How current is the information?”
5 points – requested information
is less than 6 months old
3 points – requested information
is less than 1 year old
1 point – requested information
is more than 1 year
Question #4 evaluated the web pages use of good design skills.
Patrick Lynch’s Yale Web Style Manual provided the standard
for evaluation. The researcher looked critically at only the most
elementary of design skills. Did links on the page work? Were graphics
used on the page able to load? Were consistent navigational tools
employed? Were text colors and size appropriate and readable? The
researcher limited the evaluation to three responses to question
#4, “Does the page use appropriate design skills and contain necessary
navigational elements?”
5 points – page uses excellent
design skills
3 points – page uses average
design skills
1 point – page uses poor
design skills
The final question looked at the page’s professional appearance.
Does the page use proper grammar, punctuation, and spelling? Is
the information organized and presented in a logical format? The
researcher will again limit the responses to question #5, “Is the
page professional in appearance?”
5 points – page is professional
in appearance
3 points – page has one
or more spelling, grammar, or punctuation errors; or page is poorly
formatted
1 point – page has multiple
errors and/or is poorly formatted
Data Analysis
Previous reports have discussed the finding of the first
two rounds of data collection. This analysis reviews the data collection
that took place during the second week of March 2001. The researcher
visited each of the fourteen SSHE school websites looking for the
informational items. Internet Explorer 5.0 was used with a screen
resolution set to 600 x 800. All sites were accesses via a 56k modem
via an independent Pennsylvania Internet service provider. Scores
were totaled for all of the twenty information items and a cumulative
score was tabulated for each institution. The individual scores
for each of the twenty elements for each school can be found in
Appendix A. The following table is a summary of the cumulative scores
for each of the three past years.
Table 1
Cumulative scores and rankings for the past three years for
each of the fourteen Pennsylvania State System of Higher Education
schools.
The average total score has increased each year, and this has
had an impact on schools’ rankings. West Chester, who ranked 1st
after the year of data collection, is now ranked 11th
because their score decreased by 18 points over the three year period.
Shippensburg, who has seen an improvement in their score each year,
has fallen from 4th to 8th position. The lesson
to be learned is that if you are not actively working to improve
your site, you are falling behind the competition. A web site is
a constantly evolving product that needs to be actively molded,
revamped, and updated. A stagnant site reflects poorly on an institution
and is not accomplishing what it is intended to do.
These scores reflect a snapshot of institutional web pages during
the collection period. They are a single perspective on usability
and information availability.
Summary
Users are frequently relying on more advanced navigational aids
to find information. The need for a search engine and a site or
index map has become increasingly more important. Because of the
quantity of information that is being delivered at most of the institutional
sites, the location or organization of information in the hierarchy
is not always intuitive. Some links fall under department's representative
areas as a result of institutional history rather than traditional
form. The number of schools that provide search engines jumped from
five in 2000 to eleven in 2001. The three schools that do not provide
a search engine on their site recorded the three lowest scores.
Only six of the schools currently provide a site map or site
index. The schools that did provide a site map provided an outline
view of the site hierarchy. A few schools used the site index approach,
which alphabetized all the major pages located on the site. The
researcher found the alphabetized approach easier to locate specific
information.
Several of the information items
that were searched for are well represented at all the institutions.
All schools recognize the importance of providing admissions information,
library links, an academic calendar, and the address of the institution.
All but one of the schools provided tuition and fees information,
registrar’s name, transcript information, department listings, the
chair of the English department, a link to the State System web
page, faculty directory and an alumni page. There is confusion at
some sites concerning the distinction between department listings
and degree programs. One site provided identical links to both listings.
The listing of degree programs offered by an institution is a critical
element to be linked on the admission or prospective student areas.
By forcing students to search through the department listings, schools
risk alienating the student because the information is difficult
to locate or may not be available at the department level.
The required element that generated
the highest mean score was the academic calendar. In March 2000, the department listings provided the highest
mean score. The academic calendar provides critical information
to those on campus, but also to prospective students, off-campus
organization, and those at sister institutions. It is no surprise
that the administration sees the academic calendar as an imperative
in constructing a web site.
There is little argument
that the Internet, web design, and institutional sites are in the
early stage of their evolutionary process. The playing field continues
to change, and standards are modified as hardware, software, and
user behaviors change. Designers must attempt to create pages that
reflect current usage statistics for their audience.
A recent
study by Mycomputer.com revealed that 49.5% of users are reviewing
web pages with a screen resolution of 600 x 800 and that less than
10% of users still use 640 x 480 as their preferred resolution size
[Lake 2001]. While these types of hardware and software profiles
change over time, current designers need to create screens that
meet the profiles of the majority of their visitors. Therefore as
pages were analyzed for this study, they were reviewed using a screen
resolution setting of 600 x 800. If the schools’ pages did not format
correctly to this screen size, points were deducted related to graphic
design. A few schools had home pages that were designed for 1024
x 768 screen sizes. This caused the content to be cropped and forced
the user to scroll down and to the right to review all the information.
For the first time this study collected data on how long
it takes for an institutions web page to load. A recent Jupiter
Research survey showed that 84% of users indicated that they left
sites because of slow or broken links [Lake 2001]. Optimizations
of images and being cognizant of the total file size for the university’s
main home page is an imperative factor in web site design. The researcher
used a 56k modem on a 550mhz PC connected to a Western Pennsylvania
Internet service provider. The data was collected between 11 a.m.
and 12 p.m. on a Friday. This time was selected to test the schools’
sites during a period when it would normally be receiving heavy
traffic. Given that 88% of households connect to the Internet at
56kps or less, it is easy for designers at institutions with T3
line access to forget that download times can be critical [Lake
2001].
The data collected on download times was not factored into
the final score for each school. Future studies may include a factor
related to download times. The collection of this information came
from the researcher’s frustration of slow download times while reviewing
the individual sites. One of the SSHE school’s home page took over
six minutes to load, while the average for the schools was over
three minutes. Perhaps some of the slow response time can be attributed
to SSHEnet, the State System of Higher Education Network. If the
bottleneck for delivery is the network, SSHE schools still need
to take this into account as they design pages, particularly pages
for prospective students, alumni, or other outside groups. Many
of the schools that reported low times had extensive graphics files
(some with over 60 files) or rotating color slide shows on their
home page. Edward Guttman, lead designer at Viant, a web development
firm, notes, “Only if you can prove that users will get added value
through enhanced site features are they worthwhile” [Lake, 2001].
While these slideshows may provide wonderful images from the admissions
view book, their slow load time outweighs any aesthetic value they
may add to the page.
Shippensburg and a few other schools have begun to address
the multi-audience design considerations through the use of an Intranet,
a portion of their web site designed primarily for on-campus personnel
and students. While some off-campus access takes place on this Intranet,
the design considerations can reflect that most of the users will
have high speed access. This strategy may be appropriate for some
of the schools.
One school still uses a cover page on their site. Cover pages
are graphical pages that load as a preamble to a site. They provide
minimal content and usually only one or two links to proceed into
a site. Their effectiveness verses their annoyance has been debated
[Lynch & Horton 1999]. The majority of site critics feel they
simply create an additional barrier between the user and the content.
However, they can provide some functionality by preloading images for graphically intensive sites.
After the completion of the data collection, a follow-up
question was sent to each of the schools asking, “Do you employ
a full-time webmaster?” and “What division in the institution is
responsible for maintaining the school’s web site?” The following
table provides the results of these questions for each of the schools.
Ten of the fourteen schools indicate that they have full-time
webmasters, although Shippensburg splits this responsibility between
two people. The top five schools in the study analysis all have
full time webmasters.
The responsibility of the web site broke into two distinct
areas, although sometimes this was a shared effort. Seven schools
indicated that the Advancement or Public Relations (normally a division
of Advancement) areas were primarily responsible for the content
and updating the schools’ pages. Four schools indicated that an
Information Technology area held that responsibility. Three schools
have joint responsibility between the IT and Advancement/Public
Relations areas. The top four schools in the instrument analysis
either had Advancement/Public Relations or joint responsibility.
Table 2. “Does the institution
employ a full-time webmaster?”
and “What division at the institution is responsible for maintaining
the school’s web site?”
A few years ago it was easy to determine the area of responsibility
of a school’s web page by looking at the images and information
organization. Schools that attempted to replicate their view book
in the first few pages of the hierarchy were normally created by
public relations areas. School’s web pages that pushed items like
e-mail instructions or ftp links in the upper levels of the site
were probably developed by the IT divisions. While these distinctions
are not as clear today, there are design issues that are addressed
differently by both areas. The IT area may sometimes choose solutions
that are technologically efficient and require less maintenance.
For instance, the choice of searchable databases for faculty and
staff directories may be determined by software availability and
efficiency rather than usability. In contrast, sites developed by
advancement areas may become too graphically intensive and slow
to load in an effort to accomplish the “branding” requirement.
In the brief period of time that this study has been conducted,
a maturation of many of the SSHE school web sites has taken place.
Early sites relied heavily on organizational structure to dictate
the organization of information on institution’s web site hierarchy.
Next came the revelation of designing for the audience and sites
grouped all their content around audience expectations. Many sites
have now moved to a hybrid model melding the two with a heavy focus
on the audience but some additional organizational structure from
the institution included. The past three years has seen a consistent
improvement of most SSHE school web sites, however the work is never
complete. Schools need to be constantly, reevaluating their audience,
reviewing their content organization, and revising their pages to
reflect the changing dynamics.
Suggestions for Further Research
Several follow-up or study reviews may be incorporated
in future reviews. The researcher felt that the top rated site was
not necessary the most aesthetic of the sites. The instrument does
a much better job of evaluating the information architecture of
a site rather than the visual design. Perhaps a second phase of
this study could review just the graphic design.
One way to provide credibility
to this study would be to identify a correlation between these ranking
and next year’s applications and enrollment. In theory, schools
with better web sites should see an increase in the number of applications,
particularly electronic applications.
After three years of using the same information items, it
is probably time to generate a new list for review. A review of
the SSHE imperatives may provide ideas for new items to include.
In addition, some schools may become savvy of the study and specifically
improve their page to score higher.
Bibliography
Bracco,
Amy. “Electronic Application Survey Report.” Survey completed by
Slippery Rock University Admissions Office, Slippery Rock, PA. April
2000.
Gardner,
Elizabeth. “Demanding Clients Are Driving Motivation.” Internet World. November 1, 1999, pp 54-56.
Lake, David. “Quick and Easy” The Industry Standard Magazine.
http://www.thestandard.com/article/display/0,1151,22342,00.html.
March 05, 2001.
Leibowitz,
Wendy R. “Colleges’ Web Sites Should Heed Users’ Needs, EDUCAUSE
Speaker Says.” Chronicle of
Higher Education. October 27, 1999, p A45.