General Software Design and Implementation

1. Django Framework
2. Server Decision
3. Unit Tests
4. Feedback Functionality
5. Registration
6. Password Reset
7. Notification Framework
8. Logging

Django Framework

Based on Django’s official website, Django is defined as a “high-level Python web framework that encourages rapid development and clean, pragmatic design” [6]. SmartUni is built based on Django. In this section, we will outline the general Django framework and describe how SmartUni uses it.

Django Project Structure

A Django project consists of many applications. Django applications are portable and can be shared among different Django projects. The question of how many applications are needed in a Django project, and whether all the logic for the Django project can be combined into one application, may come to mind. There are many approaches to application design. However, best practice is to let each Django application handle a particular task. Django applications should be designed following the “loose coupling” design philosophy [7]. In SmartUni, we followed this philosophy and split up the project functionalities into three separate - the Core or base of SmartUni, the SmartPlanner, and StudyBuddyMatch.

In a Django project, there are 3 types of applications:

• built-in applications
• new applications
• third party packages

We will briefly describe these in the following sections.

Django Built-in Applications

By default, a Django project has six built-in applications created automatically after creating a Django project:

• django.contrib.admin: this application is used to add the admin site to the project [8]
• django.contrib.auth: this application adds a user authentication system to the project [9]
• django.contrib.contenttypes: this application enables the contenttypes framework [10]
• django.contrib.sessions: this application enables the session framework [11]
• django.contrib.messages: this application implements the messaging system [12]
• django.contrib.staticfiles: this application enables the framework for managing static files [13]

These built-in Django applications are usually needed in most websites. Thus, Django provides these ready-made applications so developers can add them easily to their websites.

SmartUni New Applications

As mentioned before, a Django project consists of applications. The SmartUni project has three new applications in addition to the six built-in Django applications we have addressed before:

• SPC: Core Module
• SPM: Smart Planner Module
• SBM: StudyBuddyMatch Module

These three new applications are described in detail in our documentation.

SmartUni Third Party Packages

Third party packages have been designed to be pluggable into any Django project [14]. The SmartUni project has imported 11 third party packages:

• debug_toolbar: this application has configurable panels to debug requests and responses [15]
• bootstrap5: the bootstrap5 application blends Django and Bootstrap 5 [16]
• crispy_forms: this application contains DRY (Don’t Repeat Yourself) Django forms to build reusable layouts [17]
• django_filters: this application allows users to add dynamic QuerySet filtering from URL parameters [18]
• django_extensions: this application contains a collection of custom extensions for the Django Framework [19]
• crispy_bootstrap5: this application works with the crispy_forms application [20]
• django_feather: this application is used to add feather icons to the project [21]
• imagekit: this application is used to process images [22]
• django_q: this application enables multiprocessing distributed task queues in a Django project [23]
• notifications-hq: this application implements a notification system in the project [24]
• [django.forms] (https://docs.djangoproject.com/en/4.1/ref/forms/renderers/): this application enables using built-in Django templates [25]

Here we can see the power of Django, separating the functionalities into seperated applications and reusing them in different Django projects.

An important application we want to mention here is the Django admin documentation generator application. This application generates a documentation automatically from a Django project and all its applications.

MVT Design Pattern

Django is based on the “Model View Template” (MVT) architecture. MVT is a software design pattern and its structure has the following three parts:

- Model: The model returns data from the database. It is a data access layer that handles the database. The models are usually located in a file called models.py.

- View: A view is a function that receives HTTP requests and returns the final result. The views are usually located in a file called views.py.

- Template: The Template is the presentation layer. It describes how the results are rendered and displayed in the user interface. The templates are located in a folder called templates.

This diagram explains how MVT parts work and interact [26]:

Django File Structure

One of the most important things when creating a new Django project is to organize projects’ directories and files. This helps the developers to search and browse paths and files easily. The Django framework has a basic file structure that developers can start with and later add new files and directories to. This structure can help keep the project DRY (Don’t Repeat Yourself), which is one of the Django design philosophies [27]. After creating a new Django project, Django automatically creates a directory with the project name and adds starter files to that directory. One of those starter files is settings.py. The settings.py file has a configuration called INSTALLED_APPS, which is a list of the applications included in the Django project. When creating a new application, a new directory with the application name is also created. The application directory has a few starter files. This chart explains the Django file structure [28]:

SmartUni File Structure

Since we created a new Django project with three new applications for SmartUni, our Django framework included a main directory with the name “smartuni” and three subdirectories with the names spc (for the Core module), spm (for the SmartPlanner module) , and sbm (for the StudyBuddyMatch module).

These directories contain the default starter files as shown in the image below:

Each starter file has a specific role in the Django framework. They are structured in a way that helps to apply the Model View Template (MVT) software design pattern in Django projects. Additional details about them can be found in the tutorial on techvidvan.com.

Server Decision

The SmartUni server, which hosts the production system, is a virtual machine in the Rechenzentrum (computing center) of the University of Osnabrück. The server has 4 gigabytes of random access memory (RAM) and 4 virtual processor cores. The machine runs Ubuntu 20.04 Server Edition. The server edition of Ubuntu runs headless, meaning there is no graphical user interface available. The server can instead be accessed and operated via Secure Shell Protocol (SSH) terminals, the standard command line application to access remote computers. The faculty supervisor and the project management team had user access rights to the machine. These access right allowed login to the server via an SSH connection and allowed necessary changes to be made. The webserver used was Apache2. The database was a MariaDB (MySQL) database also hosted by the Rechenzentrum. The server was set up for HTTPS with a certificate by Let’s Encrypt, an initiative providing an easy way to add HTTPS certificates to websites.

Unit Tests

Unit tests are developed to automatically test the smallest components of an application, also called units. This involves writing test functions that call the desired function and compare the result with an expected outcome via assertion. In test-based development, the expected outcome is defined first, and only then is a function written that results in it. Although this is a useful approach for writing code that exactly provides the functionalities necessary for the user, there is still a lot of effort involved in full code coverage. Therefore, we decided to only partially automatically test the code and to forgo a test-based approach. Nevertheless, all important Django-based code elements, namely views, models, and forms, are checked with automated test functions within the modules.

Before each merge into the Core’s main branch, it was a prerequisite that the appropriate test functions were added and a test run was performed. Thus, the unit tests played an important role in quality assurance. The effort for the development and the use of the tests were worth it, because one or the other error was always found, such as an unsafe redirect or incorrect insertion into the database.

For writing the tests, a very good knowledge of Django, and partly even of the source code itself, was necessary. For this, the Mozilla Django tutorial was very useful as it gave us a better understanding of unit testing at the beginning of development.

Feedback Functionality

For every service and website, it is crucial to include user feedback for improvement purposes. Every user platform depends on the user base and without user satisfaction, no service can survive. Thus, we implemented the possibility for each user to submit feedback to us at any time they want. The Feedback Submission page (Fig. 1) can be accessed via the navigation bar on the left side of the website (Fig. 2). The Feedback Submission page shows a simple form that can be filled out by the user.

The Feedback Submission page allows the user to target their feedback at one of the main SmartUni pages. These include the Welcome page, the Settings page, the SmartPlanner pages, and the StudyBuddyMatch pages. An Other category is also provided for general feedback. The user can rate the page they have selected by chosing from a scale ranging from “Not at all” to “Extremely!” satisfied. A large text box allows for textual input of the actual feedback and ideas. After submitting the feedback, the user will be notified whether the feedback was successfully submitted or if some input is still missing. The page selection field and the textual input is mandatory for submission, but the rating is optional. After submission, the feedback will show up in the feedback list (Fig. 3) that is only accessible for staff users on the left navigation bar.

The feedback list contains basic information on the service (the page the user has submitted the feedback on), status (explained later), the user who submitted the feedback, date of submission and an excerpt from the feedback description. Clicking on the Details button will redirect the staff user to the Feedback Detail page.

The Detail page (Fig. 4) includes the same information as in the list view, but shows the rating (if it was selected during submission, otherwise it will be None) and the full text of the feedback description. The rating here is translated into numerical values of the aforementioned scale (1 = “Not at All”, 2 = “Bit of a Let Down”, 3 = “I´m Neutral”, 4 = “A Lot”, 5 = “Extremely!”). Available states for any feedback submission are open (when just submitted and nobody is working on it yet), in work (somebody is working on it), verification (work is done and is waiting for pending verification), and closed (feedback is resolved). The feedback ID can be seen in the heading.

Registration

The Registration page can be accessed in multiple ways. The simplest way is to access it via the sign-up button on the front page of the website (accessible by clicking the “SmartUni” font on the top left). Optionally, the Registration page can be accessed via the left navigation bar. The SmartUni registration is similar to any other website registration process. The registration form (Fig. 5) requires input of the user’s first and last name, their e-mail address, their chosen username, and a password. The password is required to be at least eight characters long and the e=mail has to include either the “uni-osnabrueck.de” or the “uos.de” suffix. Upon succesful registration, the user can log in immediately and use all the services.

Password Reset

In case the user has forgotten their password, Django offers a built-in password reset function that we customized. Now we have a custom Password Reset page with custom error messages. The Password Reset page can be accessed via the “Lost password?” link on the Sign-In page (Fig. 6). The password reset function requires the e-mail of the account to be recovered. A recovery link will be sent to the e-mail shortly after. The user can then reset their password following the same password constraint as during the registration (Fig. 7). Their password will then be updated immediately.

Notification Framework

The Core implemented a notification framework to allow for e-mail and in-website (also called in-app) notifications. The SmartPlanner and StudyBuddyMatch modules make use of this framework to notify the user about new events or matches. The functionality of the framework is accessable via the CoreNotificationHelper. The notifications-hq package was used for the majority of the notification functions. A wrapper function was implemented which helped the modules implement their notifications by hiding most of the complexity of the full notification system. When a module sends a notification to the user, the user sees a new red ping alert on the top right of the navigation bar, showing the number of unread notifications (see Fig. 1). We also implemented the live-updater functionality of the notification package such that new notifications are shown without the need to refresh the browser.

By clicking on the bell button, the user can open a dropdown list of the recent notifications, and clicking on each of the notifications will redirect them to a page that is associated with that notfication.

Every notification that is stored within the notification table of the database has a URL assigned for redirection. Depending on which module has sent the notification, a corresponding colored label will indicate the domain. The user can also access the notification overview page via the dropdown list by clicking on the “all notifications” button. The overview page (Fig. 2) contains the history of all the notifications a user has received and some information on them, including module, time, description, notification level (success, info, warning, or error), read/unread state, e-mail status, and the corresponding linked page.

The user can specify their notification mode (either none, e-mail, in-app, or both) within the general settings page (see Fig. 3). The default settings for each user consist of e-mail notifications about password changes, in-app notifications for SmartPlanner reminders, and e-mail and in-app notifications for StudyBuddyMatch events.

The kinds of notifications in the example settings shown are a set of default settings and can be expanded if desired in the future.

In addition, each notification that is handled by the framework can be given additional data. This is done in the case of e-mail notifications where special event-specific templates were prepared which need this addtional data (e.g. user name, dates,etc.) to fill out their specific fields. These filled-out templates are then sent to the e-mail address the user has specified in the sign-in process. For sending out e-mails, we used university e-mails for testing purposes; this will be replaced with a proper SMTP server on the release server. Technically, since the additional data is handled as a Python dictionary, any kind of primitive or complex data can be sent with a notification, making data transfers via notifications possible. Additionally, the notification framework is currently intended to send notifications to a single recipient. This means that group-based notifications are to be handled by the corresponding modules, such that multi-recipient notifications need to be looped to send out a notification for each user separately. Even though the underlying notification package used is technically able to handle group-based notifications, a concept for Django user groups needs to be developed. If Django notification groups exist, sending group-based notifications is as easy as substituting the recipient user ID with the corresponding Django group name. The CoreNotificationHelper currently already has the functionality to create Django groups as well as add users to these groups.

Logging

Before discussing how we implemented and configured logging in our project, we will first discuss logging in the Django framework in general.

What is Logging?

Logs are an essential part of troubleshooting applications. Logging is an important technique for developers because it helps them to track events, or things that happen within a software system, in order to keep an overview of the software’s functioning and track down any errors that arise. Developers are not only responsible for making software, but also for maintaining them, so the ability to see what has been happening in the software they are responsible for is key.

Log management

Log management is the practice of gathering, storing, and processing data from running applications in order to optimize system performance and detect technical issues.

Logging in Django

The logging module in Django is a built-in Python module. It is preinstalled with Python3. Python programmers will often use the print function in their code as a quick debugging tool as this provides a crude but effective means of creating a record of what has occurred while running a program(s). With the logging framework, this debugging becomes much more structured and flexible. The logging framework can also provide developers with more and better-structured information about the state of an application.

Django uses Python’s built-in logging module to perform system logging. The Python standard library and Django come with an integrated logging module that provides logging features.

There are 4 main parts of the logging module:

1. Loggers
2. Handlers
3. Filters
4. Formatters

Loggers

The loggers are the objects that developers are concerned with. They are used in the project files and are the functions which will be invoked when they are called. When the function is invoked, we get a detailed report of what has been happening in the project application.

A logger can generate multiple levels of responses, as detailed below.

Logging Levels

A logger is configured to have a log level. This log level describes the severity of the messages, which can be any of the following:

1. DEBUG: Provides detailed information and is most commonly used when diagnosing problems.
2. INFO: Used as a confirmation of things working as expected.
3. WARNING: Indication of something unexpected.
4. ERROR: Indicates a serious problem which may mean that the program is/was unable to perform a function.
5. CRITICAL: Indicates an extremely serious error which could potentially mean that the entire program cannot continue executing.

Handlers

The handler is the engine that determines what happens to each message in a logger. It describes a particular logging behavior, such as:

1. writing a message to the screen.
2. writing a message to a file.
3. writing a message to a network socket.

For example, ERROR messages can be sent in real time to the developer while WARNING level logs might be stored in a system file only.

Filters

A filter is used to provide additional control over the logs. They indicate which log records should be passed from the logger to the handler. By default, any log message that meets the log level requirements will be handled. By using a filter, additional criteria for the logging process can be applied. For example, we might install a filter that only allows WARNING messages from a particular source to be emitted, disabling these messages from other sources.

Formatters

Formatters describe the exact format of a log record’s text.

For example, when a log “The program starts” is sent through the following log formatter:

%(asctime)s — %(name)s — %(levelname)s — %(funcName)s:%(lineno)d — %(message)s,

it will become the following:

2018-02-07 19:47:41,864 - a.b.c - WARNING - <module>:1 - The program starts

Security Implications

The logging system often works with sensitive information about an application and that application’s data. To ensure the security of this sensitive information, the following aspects should always be carefully considered:

• What information is collected.
• Where information is stored.
• How information will be transferred.
• Who might have access to the information.

References

[1] https://data-flair.training/blogs/django-logging/

[2] https://docs.djangoproject.com/en/4.1/topics/logging/

[3] https://linuxhint.com/how-to-use-django-logging/

[4] https://www.freecodecamp.org/news/logging-in-python-debug-your-django-projects/

[5] https://betterstack.com/community/guides/logging/how-to-start-logging-with-django/

[6] Django Project https://www.djangoproject.com/ [last accessed: 05.10.2022]

[7] Django design philosophies - Loose coupling https://docs.djangoproject.com/en/dev/misc/design-philosophies/#loose-coupling [last accessed: 06.10.2022]

[8] The Django admin site https://docs.djangoproject.com/en/1.8/ref/contrib/admin/#module-django.contrib.admin [last accessed: 06.10.2022]

[9] User authentication framework https://docs.djangoproject.com/en/1.8/topics/auth/#module-django.contrib.auth [last accessed: 05.10.2022]

[10] The contenttypes framework https://docs.djangoproject.com/en/1.8/ref/contrib/contenttypes/#module-django.contrib.contenttypes [last accessed: 05.10.2022]

[11] The session framework https://docs.djangoproject.com/en/1.8/topics/http/sessions/#module-django.contrib.sessions [last accessed: 06.10.2022]

[12] The messages framework https://docs.djangoproject.com/en/1.8/ref/contrib/messages/#module-django.contrib.messages [last accessed: 05.10.2022]

[13] The staticfiles app https://docs.djangoproject.com/en/1.8/ref/contrib/staticfiles/#module-django.contrib.staticfiles [last accessed: 05.10.2022]

[14] Third-Party Packages https://github.com/wsvincent/awesome-django/blob/main/README.md#third-party-packages [last accessed: 06.10.2022]

[15] Django debug toolbar https://github.com/jazzband/django-debug-toolbar/ [last accessed: 06.10.2022]

[16] Django bootstrap5 https://pypi.org/project/django-bootstrap5/ [last accessed: 05.10.2022]

[17] Django crispy forms https://github.com/django-crispy-forms/django-crispy-forms/ [last accessed: 05.10.2022]

[18] Django filter https://github.com/carltongibson/django-filter [last accessed: 05.10.2022]

[19] Django extensions https://github.com/django-extensions/django-extensions/ [last accessed: 05.10.2022]

[20] Crispy bootstrap5 https://pypi.org/project/crispy-bootstrap5/ [last accessed: 05.10.2022]

[21] Django feather https://pypi.org/project/django-feather/ [last accessed: 05.10.2022]

[22] Django imagekit https://github.com/matthewwithanm/django-imagekit [last accessed: 05.10.2022]

[23] A multiprocessing task queue for Django https://django-q.readthedocs.io/en/latest/ [last accessed: 05.10.2022]

[24] Django notifications https://github.com/django-notifications/django-notifications [last accessed: 05.10.2022]

[25] Django forms https://docs.djangoproject.com/en/4.1/ref/forms/renderers/#templatessetting [last accessed: 10.10.2022]

[26] The MVT Design Pattern of Django https://python.plainenglish.io/the-mvt-design-pattern-of-django-8fd47c61f582 [last accessed: 05.10.2022]

[27] Django Design philosophies - Don’t repeat yourself (DRY) https://docs.djangoproject.com/en/dev/misc/design-philosophies/#don-t-repeat-yourself-dry [last accessed: 05.10.2022]

[28] Django Project Structure Best Practice 2022 https://studygyaan.com/django/best-practice-to-structure-django-project-directories-and-files [last accessed: 05.10.2022]