Designing software as open source

Hello, this is Youngjae Kim from ABC Studio. I've directed and developed over 10 open-source projects, and whenever we start designing software in our team, we try to keep the possibility of transitioning to open source in mind. In this post, I want to share what I focus on during this process.

Most online articles related to open-source software talk about the philosophy of open source, how to write a README for beginners, or explain the differences between licenses. However, in this post, I want to talk about the structure of the project. In addition, I plan to mention the areas where you should value while participating in open-source development activities and naming. Just to clarify, these are just from my perspective and are not absolute guidelines.

Just to note, I will often use the term "technical user" in this post. This refers not to the end-user of the software, but to the engineer who installs and operates the technology that has been made public as open source, or the person who is considering adopting the technology.

Mindset when developing software as open source

When you first make your work public as open source, you may feel excited just at the thought of your work being in the public eye. There are many articles explaining the meaning of open source with a noble philosophy of contributing knowledge to the world. However, I think it's better to take a more casual perspective, and think of it as just one of many distribution methods. This is because there are two side effects if you attribute too much philosophical meaning to open-source development. 

First, you may become too absorbed in the purity of the code. Software evolves through trial and error, messy debugging, and experiencing failures in real situations. Especially for software used in operating services, it can often be made into an undesirable structure or have many branches due to unavoidable requirements and constraints.

When developing as open source, you may not want to show all these flaws. Most people would hate the idea of explaining or putting in unwanted structures or branching points in code that anyone can see.

However, if you think of open source activity as just one of many distribution methods, you can feel more at ease. Also, if you are too absorbed in the purity of the code, it can be difficult to appreciate or accept feedback from others, and your perspective may become narrow as you obsess over whether the feedback fits your perspective.

Secondly, the meaning of the activity can be distorted. If you view open source as a promotional tool and look at the number of people who starred your project on GitHub as a "like" on other social media, you may end up only talking about grand goals or only dealing with popular topics, wasting time that should be spent on improving the software.

In the end, strong software faithfully implements the features introduced in the open-source project, is easy for technical users to install and use, and is a software that continuously fixes bugs and updates according to dependencies on libraries or environments. Therefore, instead of focusing on external numbers, you need to focus on how faithfully you implemented the README you wrote and how much you meet the expectations of the users. 

Advantages of open-source distribution

As I mentioned earlier, I often say "open-source distribution" rather than "open-source activity". To illustrate, it's a development and distribution process centered around the blue Public Repository seen in the diagram below.

Typically, software developed in companies don't require the Public Repository stage shown in the picture. So, what are the benefits of adding this stage? The advantages you can gain by adopting open-source distribution are as follows.

1. You can ensure the independence of the architecture.
   I call this "architecture forced by the environment". When you create it as open source, you are forced to clearly distinguish it from the business logic, and as a result, you end up making it loosely coupled with the outside as much as possible. For example, let's say you have an SMS transmission server in your company. If you make a system that uses this as open source, you'll think it's better to configure the API address of the SMS server not by hard-coding it in the code, but by setting it as an environment variable, wrapping it with an interface, or making it a separate library and injecting it.
2. You can adopt an open-source license to increase the freedom of use. 
   When you developed software with the funds provided by the company you work for, the software becomes an asset of the company. Currently, our company has a very complex corporate relationship with over 100 affiliates. If you set a general license like Apache for the open source you've distributed, you can step back from the complex asset handling relationship and usability discussions between corporations and increase the agility of the product.
3. You can pay more attention to security from the start.
   In many companies' private repositories, secret keys are often carelessly included. Every month around the world, there are issues where a privacy breach occurs due to a secret key in the commit history when trying to make a repository created within the company public as open source. If you make it with open source in mind from the start, you can define the environment variable file from the first commit and specify it in the .gitignore file to reduce the risk from the start. If you have to comply with it from the start due to the distribution conditions, you can also save the effort of persuading people who find it bothersome to create separate environment variables.

I think the "free participation" often mentioned as an advantage of open source is of secondary importance. It's meaningless if there are no external participants. Consistently attracting voluntary participation from outside is quite difficult. So, it's better to focus on developing a good foundation for your software by yourself from the start, with the expectation that there won't be any external help.

The first step to creating a good open-source structure

Even though I said not to expect participation, one of the greatest joys you can get from operating open source is the experience of technical users naturally transforming into participants who give opinions, and even further into contributors who help with the code. Generally, the way to design good software can be summarized as following the SOLID principles well. In this section, I'll look at how to make it into "software you want to cultivate more".

The image on the left is Robert C. Martin's Clean Architecture diagram, widely quoted by many. Since the reason for the existence of software is to solve real-world problems, it seems natural that the business situation where the software is used is at the center.

Nonetheless, the flexibility of open source increases when it's designed independently from business logic, making it advantageous to minimize the business logic as much as possible. This is arguably the most significant distinction in architecture when considering open source. Consequently, a simple, dry set of features takes precedence. Features that may not appear interconnected often exist under the premise that they "could be useful to someone".

As a result, the technology to solve the problem defined by open source is gradually gathered at the center when distributing software as open source. Therefore, if you look at the repository of an organization that releases a lot of open source, it often looks like a toolbox. A notable example is Netflix OSS.

Based on this, I summarized the direction of open source development into three axes as shown on the right side of the above image. Each axis is enhanced with the themes of consistency, extendability, and maintainability. Healthy software also manages about ten more quality attributes including testability, availability, and ease of training. When starting open source, I chose consistency, extendability, and maintainability as particularly meaningful themes, and I will explain the representative items of each attribute with examples.

For quality attributes, it would be good to refer to "5.2 Define the Quality Attributes" in Design It! by Michael Keeling. The book was a great inspiration for me, and I've had the pleasure of publishing the Korean translation of it.

Consistency - Establishing a clear worldview

A worldview can be described as the value and potential that can be gained by using this software and technology. It makes technical users imagine what value they can get and what service scenarios are possible. Excellent software clearly defines its worldview in the first sentence. Below are a few examples, all extracted from the first sentence of a GitHub repository's README.

• Example of desktop software: Microsoft PowerToys is a set of utilities for power users to tune and streamline their Windows experience for greater productivity.
• Example of SDKs: Flutter is Google's SDK for crafting beautiful, fast user experiences for mobile, web, and desktop from a single codebase.
• Example of databases: OrientDB is an Open Source Multi-Model NoSQL DBMS with the support of Native Graphs, Documents, Full-Text search, Reactivity, Geo-Spatial and Object Oriented concepts.

If you look at each sentence one by one, you can see that they are meticulously written sentences that could have their whole meanings altered if even a single word is removed. Like this, it's necessary to clearly define the purpose that a single project is aiming for. Once you set a purpose, the essential features are organized, and once the essential features are organized, you can set up the development phase.

Often, some people have such a rich imagination that it takes a long time and is difficult to organize the purpose, features, and their order. In this case, it's good to refer to the BANEX template that matches the eye level of all stakeholders from business to development (available in Korean only) that I published on the previous LINE Engineering blog.

When making software, features that have nothing to do with the worldview defined at the beginning often gradually get included. This means it's time to redefine the role at the repository level and redesign it. Until then, it's good to gather a minimum of features and define a worldview with a sentence that can clearly explain it. 

Scalability - External interface for participation

Even if technical users clone the repository, most of them will open the repo in an IDE and only look at the code immediately significant to them. However, a few curious technical users may think, "Can I use this the way I want with just a little modification here?" and we need to be prepared to answer this. To stimulate the desire to participate, you need not only reader-friendly and reliable documentation, but also consider interfaces that consider extensions such as plugins, extensions, APIs, webhooks from the beginning.

At this point, a benchmarking process is needed for similar software. It's good to study how software with a similar worldview provides scalability and secure the possibility for technical users to participate in their software from the design stage.

The extension function varies depending on the language and technology of the software. For example, if it's a web application, focusing on a web API is one way. Web APIs typically exchange data in the following relationships. Depending on the role of the program, you can choose an appropriate communication method with an external system.

If there's an important trend in program development these days, it's considering external connectivity from the start. This is probably because it's the first task of creating a technology ecosystem. The simplest method to try in the above diagram would be the push method at number 2. By allowing the program to register a URL and simply sending it as an HTTP client according to a specific event or schedule, the development burden is low and it has little effect on program stability.

Maintainability - Program operations that can be read from a configuration file

The first impression of open source is the README document. However, for technical users, how easy an open-source project can be adopted can also be considered a first impression. From this perspective, there are two major first experiences that are important for technical users who want to adopt an open-source project. They are installation and configuration experiences.

For installation, for example, if you've created a node.js package, you need to set up npm connection to consider the installation convenience for technical users.

For easier configuration, one of the recommended development methods is to start by creating a configuration file. If you start by creating a configuration file without writing any program logic at all, you can create a new text file, define an arbitrary name, put in a value, and think about how well it can be read by technical users and what potential for future development there might be.

The code example below is a YAML configuration file for a Slack bot server that runs some automation scripts. This bot receives webhooks issued when a pull request is made on GitHub, processes the content appropriately, and sends a notification to Slack. Just by looking at the configuration file below, you can anticipate the operation to some extent. Also, if the authentication token changes or the URL of the server to be called changes, the next person in line (future me?) can respond without having to understand the program.

github-pr-webhook-to-slack-notification:
    github:
        url: https://sample.github.com
        username: sample-name
        authtoken: sample-key
    slack:
        url: sample-webhook.slack.com
        authtoken: sample-slack-key
    template:
        default: sample-template.txt

Training to create a configuration file first has two advantages.

1. The lifespan of the program is extended.
   Many of the automation scripts I've seen in various companies had all values hard-coded in the logic without any room for external configuration, so nobody dared to touch them. However, the physical time it would take to write code is similar whether a configuration file exists or not. This is because it's now very easy to write code that reads configuration values and makes them into variables in most languages. Programs can operate and be active somewhere forever once they leave the hands of the developer. Therefore, by separating the configuration function from the logic and making it clear and easy to understand, someone other than me can change the settings and use it for a longer period of time.
2. You can clearly indicate the potential for scalability.
   When you start creating a configuration file, there are many chances to think, "I can easily respond in the future if I just modify this part". Accordingly, you can create a software structure that can be changed as much as possible by configuration, and by separating the parts that can be changed by configuration from other code, the cohesion of the core logic becomes stronger. If you look at the example code above, there's an item called template. Perhaps you can make various formats of Slack messages (as technical users can infer). At first, when the script function is simple, you can be satisfied with one message expression, but over time, users may request various expression methods. For this, if you define the template item in the configuration file in advance and create a structure accordingly, even if there is only one sample-template.txt file as an example for now, you can expand it with less effort in the future.  

Structure by software type

Open source doesn't necessarily have to be executable software. Technical specifications like the HeadVer versioning rules can be open source, and there are open source projects like Microsoft Docs that operate documentation for rapidly changing cloud services on GitHub. When thinking about open source these days, it often doesn't just mean source code. However, in this section, I will focus on software with functions and explain the points to note, and the experiences that can be gained for each type of software.

Here, I'm explaining using repositories or directories as units of structure, but just because they are beautifully composed doesn't mean they can be called architecture or design methods. Architecture includes abstract concepts as well as physical ones, and beyond the physical and abstract meanings, there's also a process perspective (work order, collaboration method). Nevertheless, the reason I'm talking about the physical repository unit in this article is that, apart from the structure and goals of all software, the physical repository is the unit that is first distinguished and handled by the technical users of the open-source project.

In the above picture, a Module can be a single class file, a directory, or a repository. It simply means a collection of features gathered under a common concept. Depending on the definition and scale of the problem that the open source project is trying to solve, it can be one of the above structures or a combination of several through enhancement. Although you might think that the "Scalable" type at the end is the ultimate structure because abstraction is a virtue in software development, it's not the only answer. Even simple functions don't need to declare all interfaces.

If you start with a specific structure and nurture it, based on the characteristics of the software you're creating, you'll experience gradual growth in software development without major disruptions. This helps technical users understand the overall structure more quickly when they skim through the open source. It provides them with a hint to get a faster grasp.

Now let's look at some considerations for a few types of open source.

Utility library

This is the smallest distribution unit of a repository. Common examples are libraries for text processing and time conversion.

If it's a library that is close to a collection of functions with little external dependency, it's good to try for the purpose of practicing implementing unit tests. At ABC Studio, we also released a few features commonly used in iOS and Android as lightweight integrated interfaces while developing based on Kotlin Multiplatform.

• abc-kmm-h3: H3 geo-index library
• abc-kmm-analytics-tools: Event, log tracker for development convenience
• abc-kmm-shared-storage: Integrated interface for managing local storage
• abc-kmm-location: Integrated interface for managing location permissions and information
• abc-kmm-notifications: Integrated interface for managing push notifications

Distributing a library as open source has the advantage of easy maintenance. In fact, you need to define the function and structure in a way that's not difficult, which often leads to making it as general as possible for ease of use by developers. A well-organized library clearly defines inputs, expected operations, and outputs according to the function definition. So, aside from developing additional functions, you only need to update external dependencies from time to time.

If it's a utility consisting of a few functions, it's generally composed as a monolithic. Among this kind, a notable source to refer to is the Apache Commons Text library, which is a venerable and large-scale project. 

Data-dependent library

A library is simply a package that organizes and collects a part of software. However, if it's designed to perform complex roles, it can become infinitely complex. The complexity of software is somewhat proportional to the complexity of the data it handles. If the library needs to directly connect to a database and process it, or if it needs to temporarily store data, it requires a much broader perspective than the previously mentioned utility.

The moment you handle data or a database, you need to check the following items.

1. Whether it handles personal information
2. Whether obfuscation, such as hashing, is needed
3. Requirements of the supported database
4. Audit response: support for access rights, integrity checks, data retention period settings, etc.
5. Support for backup and restoration

I'll tell you about a few trial and errors I've experienced. 

First, in the case of database requirements, it's difficult to test with various databases when working on a small scale, so inevitably you can only mention one or two databases in the development environment. In this case, you need to separate and explain the databases you've tested and the databases that are theoretically supported in the README.

As for data retention periods, integrity checks, backups, and restores, it's difficult to implement them perfectly at the level of commercial software, but it's good to keep them in mind to some extent. What it means to support here is, for example, if a database administrator arbitrarily deletes a record in the database, the logic checking data integrity will generate an appropriate exception.

For backups and restores, there can be issues where it doesn't work properly after the restore because certain unique values don't match. Failures in minor authentication can occur even if the timestamp when the database was first created is set as the hash seed for encryption. Therefore, you need to test with the assumption that the database can be modified or changed from the outside, not just based on the logic within the library.

API or CLI application

In this article, I'm defining an application as software that can run independently. So, API or CLI applications are considered software with a UI in the form of commands and responses, even if they don't have a GUI. So, what's the most common UI for them? For instance, what would be the most common response code when a requested resource isn't available via HTTP? Would it be 200, 403, 404, or 410? In this case, it's helpful to refer to the GitHub.com API, which I consider to be the most common API design of this era. Here, by "common", I don't mean the correct answer. I mean it's widely accepted without resistance in the current era.

For CLI, it's good to refer to Docker's commands and their responses. Basic Linux commands can seem unfriendly because they don't always provide a response according to the command. Docker, a relatively new software (launched in 2013), is more independent of language and technology. Since it's an application, not an operating system, I think it's more appropriate to refer to Docker in this context.

One tip is that making this type of application more independent from the operating system will improve the response. When you're a minority in a team, there may be cases where technical users with no stake find it hard to run it normally due to the use of environment settings that are only common within the team. In fact, there are quite a few such cases. A simple example is the case below where an OS-dependent file path is referenced.

# Python
filename = "~/example/file.txt"  # File path that only works on my computer
filename = os.path.join('example', 'file.txt')  # Safer file path less dependent on the environment

Software that is large and mature has an ecosystem in place, so various reference materials can help overcome obstacles during installation or execution. However, for small, up and coming software, you need to carefully set the table rather than having a do-it-yourself attitude. For instance, if you provide a Docker configuration file (Dockerfile), some developers might find it easier to understand the execution environment than by reading the Getting Started document.

A good way to test reproducibility in web-based software or CLI applications is to set up various operating systems or environment combinations different from the development environment, and then run the distributed code. Setting up tests in various environments could be part of the continuous integration (CI) processes. This is because paths and basic requirements are often written without consideration in bash scripts, makefiles, or application code.

Application with end users

So far, utilities and API applications haven't required open source developers to directly deal with end users. However, if the software clearly has end users, more consideration is needed. This is because open source developers have to cater to both technical users, who want to implement the technology in services, and end users, who will use the service after implementation.

For example, there's software like ABC User Feedback developed by ABC Studio, which comes with a beautiful web management screen as standard. ABC User Feedback is a web server program that collects feedback entered by end users in text fields through an API. However, the management screen that collects and tags this feedback is intended for someone in the customer service center or operations team.

As mentioned, open source forums, blogs, shopping malls, and programs often used by engineers, like Jenkins and Redmine, come with a web management screen that's ready to use as soon as they're installed. Accordingly, you need to pay extra attention to at least the following things.

• Profile: If the system can identify the end user (like ID, login), you need a storage space that considers time zone display, language, and various locale correspondences based on their usage environment. Similar to the setting experience mentioned earlier, it's better to configure multi-language resources from the start.
• UX: Creating good user experience (UX) is very challenging. This is because what seems common sense to me often differs from the end user's perspective. The reason it's hard to create open source software with good UX is that the motivation for improvement gradually decreases. It's tough to confirm improvement numerically or receive clear feedback even if the UX is enhanced.
• Contact: As mentioned earlier, since it's software with end users, it's beneficial to include a contact point where you can hear the opinions of the end users. Since we can't assume they all have a GitHub account, providing more accessible contact points, like an email address or a link to your Discord server, might be a better choice.

While it might be burdensome to cater to the end users in this way, increasing ways for end-users to come in contact with your project will in turn increase the likelihood of benefiting from word-of-mouth.

Others

Software is extremely diverse, including firmware, game engines, AI/ML models, and so on. Unfortunately, this article can only cover some of the representative types. For other types, it would be very helpful to learn by studying existing mature projects.

When looking at existing projects, it's good to look from the first commit, not just the current structure. It would be good training to read the developer's release notes at each milestone and understand it in the context of consistency, scalability, and maintainability that I mentioned earlier.

Bonus - The virtue of boring but practical naming

Naming has a significant impact on the direction of the software. The reason we put a lot of thought into even one variable name is because the responsibility and role are defined by one variable. Similarly, the range of roles for software, which is made up of variables gathered, can also be determined by its name.

The more you consider open source, the more effort you need to put into naming the repository with a clear, singular purpose. Sometimes, while working, your imagination can expand infinitely, and you might feel confident that you can create at least one spaceship with your own hands. However, if you name the repository Sputnik (a Russian spaceship) or Insight (an American Mars rover), the product can also lose its direction.

When naming, it's a good idea to make the function the name. Giving it a peculiar name can be considered technical branding. The value of this is only properly conveyed after enough persuasive power is built up through technology. To put it simply, you can only bring it up in conversation after giving meaning to technical users through code and results. Even famous repositories like Armeria gained persuasive power in their naming only after a long time of experience and trial and error, and trust in Netty was built.

Until then, it's good to give a name that's faithful to the function. For example, if it's a data conversion library, it's good to include the word "convert", or if it's a library that handles messaging, it's good to include "messaging" in the name. Rather than prematurely harboring expectations of becoming a common Apache library like Kafka or Zeppelin, thinking of just developing one function at a time according to the development roadmap and creating software that solves the problem defined in the repository with high completeness is ultimately the fastest branding method.

Conclusion

We've looked at the meaning and appropriate structure when developing with open source as a distribution method, and checklists for each type of software. Open source can be a core element of software design and development strategy, giving it more meaning than just making your code public. Whether or not you actually release your code, I hope it is used meaningfully as one way to design and develop software.

Finally, I would like to thank Heesung Lee (trustin), the founder of Armeria and Netty, for reviewing this article.