WIXLIB

Chapter 2Background and Literature StudyThis chapter gives a background to the metaphor technical debt and the recentresearch done on managing technical debt. Common causes of technical debt havebeen gathered from other papers and talks and are presented in a section below.The last section lists the different activities done in technical debt management.2.1The technical debt metaphorThe term and metaphor technical debt was first coined by Ward Cunningham in1992 [1]. He described a situation where the long-term goal of software quality wastraded for short-term gain, creating technical debt.“Shipping first time code is like going into debt. A little debt speedsdevelopment so long it is paid back promptly with a rewrite. [.] Thedanger occurs when the debt is not repaid. Every minute spent onnot-quite-right code count as interest on that debt. Entire engineeringorganizations can be brought to a stand-still under the debt load of anunconsolidated implementation.”The metaphor helps us think about problems that are prevalent in softwareprojects, and it says that if we do things the “quick and dirty” way we will gettechnical debt. An example might be technical debt from a poor software architecture, that developers have implemented as a compromise to meet a deadline. Likefinancial debt, we need to pay interest on technical debt. In our example it would bethe extra time needed to implement a new feature because of the poor architecture.Either the developers can continue to pay interest, or they can pay back the wholedebt by refactoring the architecture.When the term was coined in 1992 it saw little use, but around 2000 the termstarted to gain traction. It was mostly being used in blogs and essays as a term forall software problems. [3] Software engineers often advocate for investments in codequality and documentation, which might not generate short-term revenue, but are3

CHAPTER 2. BACKGROUND AND LITERATURE STUDYcrucial to systems lasting quality. The term is then attractive to use for engineers,as a tool to communicate to managers the importance of software qualities.2.1.1Technical debt quadrantThe term was further defined when Martin Fowler in 2009 [2] proposed the technicaldebt quadrant. It came out of the argument if technical debt should be reserved forconsidered decisions that gives short-term gains but create debt in the long-term,while messy code created by programmers ignorant of good practices should notbe considered technical debt. Fowler suggested that instead of thinking in terms ofnon-debt and debt, that we differentiate between prudent and reckless debt.Another dimension can then be added, with the difference between deliberateand inadvertent debt. Reckless deliberate debt comes from when developers dothings the “quick and dirty” way, while reckless inadvertent debt comes from lack ofknowledge. Prudent deliberate debt comes from when developers take on technicaldebt for a good reason, for example it might not be worth the time to implementthe perfect code design in a system that is rarely touched. The last type is thenprudent inadvertent debt that comes when developer realize after they have createda system what the perfect design should have been.Table 2.1. Technical debt quadrant2.1.2RecklessPrudentDeliberateWe don’t have time fordesignWe must ship now anddeal with theconsequencesInadvertentWhat’s layering?Now we know how weshould have done itManaging technical debt researchThe topic of technical debt got further attention in 2010 when several researchersfrom different institutes and universities met at the Carnegie Mellon Software Engineering Institute. They published a paper [4] that sets the agenda for futureresearch in the field of managing technical debt, by establishing the workshop managing technical debt.It says that research on managing technical debt can make the intuitive understanding of technical debt into a more rigorous definition. Research can also help thetechnical community by clarifying the strengths and weaknesses of different types4

2.1. THE TECHNICAL DEBT METAPHORof technical debt management. The paper suggest several open research questionsthat are listed below. The first research question is how we find refactoring opportunities. We mostlydepend on developers’ experience to detect debt, but there has been toolsdeveloped to detect symptoms of poor design in code. However such toolsare difficult evaluate quantitatively, as it is hard to estimate the impact ofre-factorizations. While small local re-factorizations can improve a code base, system-widearchitectural issues often need larger efforts. It can become a subjective matter to evaluate if and how to re-architect a system. Having tools to measureand manage technical debt in architecture could help developers make moreinformed decisions. Identifying dominant sources of debt in different contexts from field studies,can give directions on what research should focus on. Measurement of technical debt has been researched and there are many issueswith it. To make the measurements useful they need to be combined andcreated to help decision-making. The measurements should then be put intoseveral specific concepts. The first one is principal which is cost of eliminating the technical debt. The second one is interest probability, which “is theprobability that a particular type of technical debt will in fact have visibleconsequences”[4]. The third one is interest amount which is the added cost ofperforming maintenance because of the technical debt. Non-code artifacts can also be subject to technical debt research, for examplehow technical debt occurs when test plans are not followed. Monitoring and visualization tools of technical debt could help provide insightinto when there is “too much” debt. To know if there is too much debt, thereneed to be some acceptability thresholds to be researched. To ensure that the technical debt metaphor leads to good practices, researchis needed to evaluate projects that have been launched to remove technicaldebt. To make techniques of removing technical debt explicit, current defined development processes should then be adapted to include tracking andmanagement of technical debt.2.1.3Types of technical debtA systematic literature review from 2014 [6] mapped 94 studies about managingtechnical debt. From the studies the paper collected what different types of technicaldebt that had been identified, and it resulted in a classification of 10 types.5

CHAPTER 2. BACKGROUND AND LITERATURE STUDY1. Requirements technical debt is the difference between the optimal requirements’ specification and what is actually implemented.2. Architectural technical debt is the limitations of an architecture that for example impair maintainability.3. Design technical debt is defects in the code design, for example having overlycomplex classes.4. Code technical debt is poorly written code that does not follow the best practices and standards.5. Test technical debt refers to shortcuts taken when creating tests, for examplelow test coverage or inaccurate test cases.6. Build technical debt refers to problems in the build system, for example thatit is hard to deploy code.7. Documentation technical debt refers to insufficient or outdated documentation.8. Infrastructure technical debt refers to sub-optimal configuration of systems orproblems with development-related supporting technologies.9. Versioning technical debt refers to problem with the version control system.10. Defect technical debt refers to defects or bugs in a software system.The paper [6] also tries to define what is not technical debt. It lists exampleslike unimplemented features and functionalities. Some types above are not strictlydefined, for example the problem of overly complex code or duplicated code can beidentified as both code and design debt.2.2Causes of technical debtThere are many causes of technical debt in software projects. Listed below are themain identified causes of technical debt, that have been chosen after review of whatother studies [6, 9] reported about technical debt.This thesis puts the technical debt metaphor in the context of Django application, and it mostly discusses code and design debt. Other causes of debt likeversioning or infrastructure technical debt, are not specific to Django applications,and therefore such types are out of scope for this thesis.2.2.1Architectural issuesSoftware architecture refers to the high levels structures of a program. For examplehow different modules communicate or what framework and programming language6

2.2. CAUSES OF TECHNICAL DEBTthat is used. There are many types of architectural issues that one can get. Examples are over abstraction or lack of layering (instead of using for example theModel-View-Controller pattern). Architectural debt often comes from architecturaldecisions that made sense when the system first was built, but after the system hasgrown it now requires other patterns to solve its problem as best as possible.2.2.2Lack of adhering to coding style and conventionsCode is more often read than written, therefore it is important to make sure thatcode is readable. By following a style guide when coding, the readability can improve, as the code is consistent within the code base (e.g., using the same indentationpattern). Python has a de-facto style guide called PEP8, which should be followedfor projects to increase readability [7]. Lack of adhering to this code standard, orthe one decided upon in a project, is a form of technical debt.2.2.3Code duplicationDuplicated code occurs frequently in code bases, but it should be avoided. It mightoccur from copying and pasting code, as it is faster than writing code from scratch.It is however considered a bad practice. When developers need to fix a bug or makea change in the code, they will need to check all possible duplications for that fixor change. Code duplication often indicate that there are design problems in thecode, and that abstractions and re-factorizations can be done.2.2.4Dead and obsolete codeDead, obsolete, or unreachable code is code that is never executed in a program orcode that does not do anything useful. This creates technical debt as it can confusethe reader of the source code, and make the code less clear. In large programmingprojects, it can become difficult to recognize and delete dead code, as whole classescan be unused. The dead code might still have tests, but is actually never used inproduction.2.2.5Overly complex codeTo make code maintainable developers should write their code as readable androbust as possible. This means that they should avoid creating overly complexcode, that might look elegant, but will be hard to understand by other developers.As the computer scientist Brian Kernighan wrote [11]: “Debugging is twice as hardas writing the code in the first place. Therefore, if you write the code as cleverly aspossible, you are, by definition, not smart enough to debug it.”7

CHAPTER 2. BACKGROUND AND LITERATURE STUDY2.2.6Legacy software package dependenciesDevelopers often use external software packages in order to save time and resources.Making sure the external packages are of the latest version is important, as newreleases might include bug fixes, but also new features. When upgrading packages,each dependency is a potential problem or obstacle. Updating one package mightrequire other packages to be updated as well. Having to update all packages atonce can create many problems (e.g. APIs changing). Therefore, it is important tokeep all external packages up to date, to avoid any technical debt.2.2.7Lack of testingTesting is an important part of software development. There are many types oftests like acceptance, unit, performance, load, integration or regression tests. Having extensive testing suites ensures that software changes do not break existingfunctionality. This eliminates bugs before they reach production. Having extensivetests will also make developers confident in that their program do what it is expected to do, and does not do what it is not supposed to do. It is often impossible(and not economical) to get full test coverage, but having no tests is a cause fortechnical debt.2.2.8Poor testsTests for software programs are expected to be deterministic. A test should alwayspass or fail for the same code. In practice there are usually several flaky tests (dueto for example test order dependency) in large software projects [8]. To have anon-deterministic test mean that you can get false positive or false negative resultsfor the same test after running it several times. Such tests makes it difficult to trusttest suites and is considered technical debt.As code and requirements change, tests needs to be updated as well. If the testsare not maintained, they do only cause more problem than they help developers.Therefore, any poor and not up to date test is technical debt.2.2.9Lack of or outdated documentationMany developers lack the motivation to document code, as they perceive it as a secondary task compared to writing code. Some software development techniques evensay that code should be self-explanatory and that code comments are unnecessary.However, studies show that lack of or outdated documentation increases maintenance costs [10]. So wherever code is created without needed documentation, thereis technical debt. The documentation also needs to be up-to-date, as any outdatedor out-of-sync documentation only creates confusion.8

2.3. PYTHON AND DJANGO APPLICATIONS2.3Python and Django applicationsPython is a dynamic typed programming language, that supports both objectoriented and functional programming. Its syntax is intended to be highly readable, and often uses English keywords rather than special characters. It also useswhitespace indentation, instead of curly braces which is common in other languages.According to the TIOBE Programming Community index, which measures the usage of programming languages, Python is the 5th most popular language [12].Django is an open-sourced web framework written in Python. It follows the architectural Model-View-Controller pattern. It also includes its own object-relationalmanager, internationalization system, template system, form serialization and validation system and many other components. It also comes bundled with an authentication system and an admin interface, which is often one of the main selling pointsof the Django framework.2.4Django-specific causes of technical debtWhen talking about causes of technical debt in the context of Django web applications, there are several Django-specific causes of technical debt. The main causesof technical debt in Django applications that have been identified are lised below,they were collected from talks [13, 14], books [15, 16, 17] and blog posts by Djangodevelopers.2.4.1Monolithic Django appsIn the Django framework small libraries are called apps. A Django project oftenconsist of several apps that each has their own responsibility. They should bekept small so that they are potentially shareable between different projects. JamesBennett, one of the core developers of Django, said in a talk [18] that one shouldwrite apps that “do one thing and do it well”.Some Django projects consists of one large monolithic app, that contain most ofthe features for the whole site. Large monolithic apps are often hard to understand,in comparison to small well-defined apps. These large apps often encourage andenable tightly coupled code, which makes it hard to break out code or make changes.2.4.2Misuse of signal receiversSignals are used in Django to notify when an action occurs, for example when amodel is created or when a HTTP response is finished. A receiver can be set up totake some action upon that signal. This concept is not unique to Django, it existsin many other languages (for example Java has listeners and events).Below is an example of a signal receiver, that is executed after the model MyModel is saved.9

CHAPTER 2. BACKGROUND AND LITERATURE STUDY123from django.db.models.signals import post savefrom django.dispatch import receiverfrom myapp.models import MyModel4567@receiver(post save, sender MyModel)def my handler(sender, **kwargs):passSignals are often unnecessary and can create code obfuscation. For example thepost save code example above, could have been put in the model itself, where othermodel operations exist. One of the core Django developers Aymeric Augustin says[15]:“I advise not to use signals as soon as a regular function call will do. Signalsobfuscate control flow through inversion of control. They make it difficult to discoverwhat code will actually run.”Many developers [15] also make the mistake to think that signals are asynchronous, when they in fact are synchronous and blocking.2.4.3Overuse of middleware and context processorsMiddlewares in Django are used to hook into the process of every incoming requestand outgoing response that reaches the application. It is common to use middlewares to log requests, compress response content or cache responses. These taskscould be done in each individual view, but that would require a lot of boilerplatecode that the middleware class removes.Figure 2.1. Shows the flow from request to response.Context processors are similar to middlewares, but these processors are run over10

2.4. DJANGO-SPECIFIC CAUSES OF TECHNICAL DEBTthe context that is passed to templates. They provide common information for alltemplates, for example data that is kept in the footer and header of a website.While both middlewares and context processor are useful to avoid repeatingoneself in the code, it can be misused. If heavy operations or large database queriesare executed in middlewares or context processors, they can impair performance.As they are executed for every response, request or template it is important they arekept minimal and well performing. If business logic is hidden in context processoror middlewares, it might also confuse the reader of the code, as it is not the obviousplace for such code to exist.2.4.4Heavy queries from Django ORMDjango comes with an ORM (obj

teknisk skuld i kontexten utav Django webbapplikationer. En enkätundersökning gjordes och de främsta orsaker-na till teknisk skuld i Django applikationer visade sig vara arkitektoniska problem och brist utav testning. Detta ligger i linje med andra studier utav orsakerna av teknisk skuld. Verktyg och metoder som används i Django utveckling ut-