• Implementing a programme/course management system is a large, complex project.  Before embarking on it, an institution must ensure it has the time, people and funding available to make it a success.  Stakeholders at senior management team level must actively support and drive the project.
• The development of curricula within HEIs is a primary responsibility of academic staff. The implementation of a course management system should therefore include academics as key stakeholders.
• The time needed to populate a new, structured system from existing unstructured information should not be underestimated.  It is likely that it will have to be done manually, and will take considerable time and effort.
• Extensive and properly structured data about programmes can lead to more innovative uses of the information elsewhere.  Without a common, consistent structure and means of access, re-use (internal, public and third-party) is almost impossible.
• The provision of an XCRI-CAP feed was, correctly, a minimum requirement for the JISC programme of funding. However, institutions should recognise the value of their full programme data on a much broader level, understanding that it offers insights into one of its core activities (teaching and learning) and can inform other change management programmes across the campus.[1]
• Application development around the use of course data has so far tended to be marketing-focused. However through the use of visualisations and interactive tools, it could be put to much wider use. The HE community should consider how the use of course data might fundamentally change the design of curricula and provide the basis for discussion between staff and students about the re-production of academic life. Improving the level of oversight and insight into the core activity of teaching and learning can reveal and help question existing constraints in the curriculum design and QA processes, for example.[2]
• Course data, alongside other datasets (e.g. space utilisation, research activity, energy use, facilities, etc.) provides unprecedented insight into the nature of our organisations. Institutional managers should approach their organisations as objects of research and development. The knowledge and skills exist within HEIs, among students and staff, to better understand, critique and develop the form of the organisation.
• Data visualisation techniques provide a new and novel oversight of large and rich course datasets. However, due to the complexity of this data, the use of static visualisations quickly reaches their limit. Software applications are needed that help staff and students interact with the data and model new pathways to learning, reveal greater opportunities for collaborative and cross-disciplinary teaching, and critically evaluate the history and future of the institution.

Firstly, we’ve introduced paging to the APIs, so now results can be easily broken down into pages, with the amount of results per page set to a reasonable amount, depending on the size of the objects being returned. The pagination object returned with each page of results shows useful information, including the size of the page, the amount of results the page contains, if there is a previous page available or a next page available. This will make the process of programatically searching through all of the results much easier.

Whilst using the APIs, Tony also highlighted several typographical errors, particularly with the URLs of child objects that were returned in the results, we’ve since corrected these.

These are only relatively minor changes, but should improve the usability of the course data APIs!

• Allows university staff to login
• User can select a module to view (currently limited to 10 modules, purely for testing purposes.)
  
• Application retrieves module information from Nucleus and presents the key points to the user – assessment strategy, learning outcomes etc.
  
• Assignment details are retrieved from Nucleus, user can select which assignment to create documentation for
  
• User is prompted for the necessary details to create a full assignment brief document.
  
• When the user has completed all of the required fields, they can generate a PDF document, based on existing assignment documentation.
  
• Now that an assignment document has been created, the user can create a Criterion Reference Grid document, which details the various criteria used when marking the corresponding assignment.
  
• The learning outcomes associated with the assignment can be grouped into multiple rows, so that one learning outcome can be addressed by multiple criteria in the CRG. Each learning outcome has to be assigned to at least one row of the grid. Each row then requires a descriptor and a description of the requirement for meeting each grade for that particular criterion.
  
• Once these have been completed for each row, a PDF can be created. This PDF is versioned and stored, so that older versions of the marking grid can be referred to, if required.
  

As I mentioned previously, this isn’t intended to be a fully functioning application at this stage, further development and refinement would be required in order to make it into a fully usable application. However, this proof of concept application shows how existing sources of course-related data at the university can be re-purposed and reused in order to (potentially) improve the efficiency and effectiveness of the assignment writing processes within the university.

An earlier blog post describing (briefly) the process / logic behind the application can be found here.

First off, a quick tour through the application.

Search parameters are broken down into 3 categories: subjects studied, subjects interested in and general keywords. The first two are based on benchmark subjects used in QAA validation of programmes. This makes it easier to tie subjects to programmes offered by the university, but does present a problem in that some words aren’t recognized that a potential user may expect. For instance, ‘religion’ is not recognized, instead the QAA benchmark subject is ‘theology’. The latter input field is based on keywords identified by Open Calais (explain in a previous post). This offers a wider range of keywords, but these keywords often link to quite a large range of programmes offered by the university. Obviously some kind of middle ground would be ideal, but that is not explored in this initial application.

Entering the following parameters – studied : education, interested in: psychology, keywords – teacher. Generates the results as shown above. The results are broken down by how many of the specified criteria they meet. In this example, the PGCE course meets two of the specified criteria, which seems reasonable. Selecting the result from the list takes the user through to the screen shown below.

This screen should contain whatever data is considered pertinent to the user discovering if this course is suitable for them. At the moment this screen just contains the aims and objectives of the programme, but can easily be extended (and should be, should the application be developed and used properly). The screen also allows the user to recommend this course as being relevant to their search parameters, as well as showing similar courses. Similar courses are (at the moment) based on the keywords as identified by Open Calais, with suitable limits and restrictions put in place. These restrictions are discussed in an earlier blog post, but there seems to be a fine line between having every course linked through keywords such as education (everything identifies with this keyword as they’re university courses) and having very few courses identified as being similar, even when they obviously are. Further work would have to be done on this to improve it.

Recommending the course as matching your criteria improves the search process for anyone else that may search for the criteria you specified. If entering the same criteria in a new search, the results look slightly different – recommended results have a star next to them, indicating that they have been recommended.

As mentioned previously, this application (at it currently stands) is merely a proof of concept, and would require further development before being used properly. Points to consider for further development include:

• How should we allow users to enter search criteria? A free-form text field would allow users to search for exactly what they want, but searching all of the relevant data sources would prove challenging and would be very time consuming. Using high-level JACS codes means that far too many potential matches are returned, with only a few courses being truly relevant, surrounded by a large amount of, essentially, noise.
• If the OpenCalais keywords are to be used, more work would have to be done on determining the correct level of filtering to be carried out. At the moment courses that are obviously similar are not being shown as so. This is, however, an improvement over a previous point at which a ridiculously high amount of courses were being identified as similar, when they obviously were not.
• A lot of data is being stored relating to click-throughs for search results. This *could* be integrated into the search rankings, so that courses that are often clicked on are moved further up the rankings.

There are many ways in which this application could, and should, be improved, but as a proof of concept it demonstrates what can be done with course data to potentially improve the student experience.

Firstly it was necessary to design a database structure that would allow the effective storage of a large amount of course data. The Nucleus Database  stores a lot of institutional data, as shown in the database diagram.

By taking the data from the APMS APIs and breaking it down into its constituent objects, the data is much easier to work with. For example, in the APIS from the APMS, learning outcomes can only be accessed as follows:

• Get all modules.
• Find module in array of results.
• Follow API URL for this module to the module endpoint (uses internal ID number)
• Access learning outcomes

Obviously, if we’re wanting to repeat this process multiple times, for multiple modules; or to carry out some form of processing on JUST the learning outcomes, the entire process would be very heavy handed and overly complicated. By breaking the data down on the import process into Nucleus, then re-exposing the objects in their smallest form, the data becomes much more usable.

At the time of writing, Nucleus (which is documented here) contains API endpoints for the following course-related objects:

• Academic Years
• Accrediting Bodies
• Assessments
• Awards
• Benchmark Statements
• Benchmark Statement Levels
• Benchmark Subjects
• Benchmark Subject Types
• Contact Time
• Contact Type
• Contact Type Category
• Course Codes
• Curriculum Mappings
• Delivery Modes
• Marketing Records
• Learning Outcomes
• Levels
• Module Code
• Module Link
• Modules
• Programme Outcomes
• Programme Outcome Categories
• Programmes
• Subjects

Obviously these can be combined with other data from within Nucleus, such as organizational structure, staff details, location data etc to provide useful and interesting applications.

Each API endpoint is documented at docs.n2.online.lincoln.ac.uk, each having appropriate query parameters to narrow down the search results returned. For example, assessments has many parameters that can be used to query the database, allowing the results to be narrowed down depending on the week the assessments (should) be due, the weighting of the assessment, the module the assessments are related to, if the assessment contains group work or is the final assessment for that module.

Using Nucleus APIs, you could get details of  all assessments that are the final assessment for the module, that contain group work at: https://n2.online.lincoln.ac.uk/assessments?access_token=ACCESSTOKEN&group_work=1&final_assessment=1. To get the same data using the original APIs would require the user to retrieve all of the module records, loop through each module, loop through each assessment, check against the criteria and assign the matching assessments to some form of output. The advantages of absorbing the data from the APMS, storing it in our own Nucleus database and re-exposing the data through our own hand-crafted APIs are obvious : easier, faster, more efficient.

After meeting with a member of staff from the School of Computer Science, we developed an idea around an ‘assignment wizard’, that would make use of the course data already available, such as awards, modules, assessments and staff.

The purpose of this application is to make the process of writing assignment documentation quicker, easier and more accurate. By tying the application in with assessment data, the assessment strategy delivered within the module will be identical to the strategy as defined in the validated module documents.

The expected flow of the applications is :

As well as reducing the amount of data that has to be entered by academics (such as learning outcomes, module details etc), the versioning and PDF generation will make the writing process more efficient. Further to this, it allows one lecturer to write a part of the assignment brief, and another to log in and complete the assignment.

A follow-up blog post will show the completed application, and start to evaluate it.

The (short) abstract is included below :

Large amounts of data are collected and stored within universities. This paper discusses the use of data visualisation and visual analytics as methods of making sense of the collected data, analysing it to assess the affects of historical institutional decisions and discusses the use of such techniques to aid decision making processes.

The fundamental principles of programme management have not changed as a result of the project, and nor was it intended to change them.  What has changed, and in my view at least has a big impact on the way we work with programme information, is how we go about recording information and the mechanisms that lie behind the various processes.  Process maps make things like modifying a programme still appear fairly convoluted when it comes to the approvals, whereas in fact the workflows in APMS mean that it is fairly simple to either click the button to decline approval (sending it back for further work) or click the button to approve the changes (forwarding it to the next stage).

Here, then, are a few examples:

Stand Alone Credit Creation

To create a new Stand Alone Credit module, there was a Microsoft Word form to complete containing information about the module selected, type of activity, academic school, and so on.  There are sections to confirm approval at the end.

To accomplish the same in APMS, the module is selected from the list of existing modules, and the other information entered (many items as list selections).  The academic confirms College approval and submits it to the Quality team who, having checked it, click the approved button and that’s it.

Short Course Creation

To create a new short course there was a Microsoft Word Short Course Application form to complete with fields including the course title, level, credit points, course leader, school and confirmation of approvals.  Accompanying the application form would be a the Short Course Specification, based on a Microsoft Word template, containing the course title, level, credit points, school, delivery mode, rationale, learning outcomes, learning and teaching strategies, module specification(s) and so on.

To accomplish the same in APMS a proposal for a new short course is created and items selected from lists where available (e.g. school, course leader) or information typed in.  The modules are selected from the module list (or new ones created) and the proposal submitted through an approvals and validation process.  There is no duplication of information needed and each person involved can see all the information related to it.

Programme Modification

To modify a programme an academic would complete a Programme Modification Form in Microsoft Word.  A series of tick boxes would indicate whether all evidence had been provided, external examiner approval gained, revised module specifications provided and more.  Details about the school, campus, affected programmes and modified modules would be provided, along with the rationale and summary of the change.  New programme and modules specifications would be written and attached, which themselves would also contain details about the school, campus and other duplicative information.

A programme modification in APMS starts with a discussion between the academic and a Quality Officer to determine whether the magnitude makes it a revalidation or a modification, and the Quality Officer starts the appropriate workflow.  For a modification, certain fields are then available for change by the academic whilst others (that would require a revalidation) are not.  Once the changes have been made to the programme and/or modules, the academic submits the modification proposal for approval, and it goes through the workflow getting the appropriate approvals until validated.  There is no duplication of information, and everyone can see where in the process the modification is and all information relating to it.

Benchmark Statements

As one final example we briefly look at benchmark statements.  The format of benchmark statements vary from subject to subject – in some cases they can be presented as numbered items whereas in others they are paragraphs of text.  In APMS each item for a subject is presented as an individual item, having been painstakingly extracted from documentation by our project intern, Louise.  Statements are presented automatically once the benchmark subject is selected, and the system presents them in a matrix with programme outcomes so that the programme leader can map where each is covered.  Previously, each programme leader would need to conduct this exercise themselves and duplicate information throughout the document to make sure it was presented in the appropriate way.