For home WiFi routers and access points, the OpenWrt project is perhaps the most well-known Linux distribution; it originated from the source code of the now-famous Linksys WRT54G router twelve years ago. In early May 2016, when a group of core OpenWrt developers announced that they would begin working on a derivative of OpenWrt (or possibly a branch) called the Linux Embedded Development Environment (LEDE), the OpenWrt user community was thrown into chaos. The reasons for the split were not clear to the public, and the LEDE announcement shocked some other OpenWrt developers, hinting at internal conflicts within the team.

The LEDE announcement was sent by Jo-Philipp Wich on May 3rd to all OpenWrt developer lists and the new LEDE developer list. It described LEDE as a “restart of the OpenWrt community” and “a derivative of the OpenWrt project,” hoping to create a Linux embedded development community that emphasizes transparency, cooperation, and decentralization.

The reasons given for the restart were that OpenWrt had long suffered from unresolved internal issues— in other words, about the internal handling and policies. For example, the announcement states that the number of developers was decreasing, yet there was no way to onboard new developers (and seemingly no way to delegate access to new developers). The announcement mentioned that the project’s infrastructure was unreliable (for instance, last year’s server outages caused considerable strife within the project), but internal discord and single points of failure prevented it from being fixed. There was also a general lack of “communication, transparency, and cooperation” both internally and externally from the project. Finally, some technical flaws were cited: insufficient testing, lack of regular maintenance, and precarious stability and documentation.

The announcement continued to describe how the LEDE restart would address these issues. All communication channels would be opened to the public, decisions would be made by voting across the project, and merging policies would be relaxed, among other things. More detailed explanations can be found on the rules page of the LEDE site. Other details included that contributors would only have one class (i.e., there would be no group of “core developers” with extra rights), decisions would be made by simple majority voting, and any infrastructure managed by the project must have more than three administrator accounts. In the LEDE mailing list, Hauke Mehrtens added that the project would strive to submit patches to upstream projects— a point that OpenWrt had been criticized for in the past, particularly concerning the Linux kernel.

Besides Wich, the announcement was co-signed by OpenWrt contributors John Crispin, Daniel Golle, Felix Fietkau, Mehrtens, Matthias Schiffer, and Steven Barth, and concluded with an invitation for others interested in participating to access the LEDE site.

Responses and Questions

One might speculate that the LEDE organizers anticipated their announcement would receive either positive or negative reactions. After all, a close reading of the criticisms of the OpenWrt project in the announcement implied that the LEDE camp found some OpenWrt project members difficult to work with (for instance, “single points of failure” or “internal discord” prevented the infrastructure from being fixed).

Indeed, there were many negative responses. One of the founders of OpenWrt, Mike Baker responded with some warnings, refuting all the conclusions in the LEDE announcement and stating that terms like “restart” are vague and misleading, and that the LEDE project failed to disclose its true nature. Meanwhile, some closed the @openwrt.org email access for those who signed the LEDE announcement; when Fietkau objected, Baker replied that the account “was temporarily disabled” because “it was still uncertain whether LEDE could represent OpenWrt.” Another core OpenWrt member, Imre Kaloz, wrote that most of the issues they were now complaining about with OpenWrt were actually created by the LEDE team.

However, most responses on the OpenWrt mailing list expressed confusion over the announcement. Members of the mailing list were unclear whether the LEDE team would continue to contribute to OpenWrt or what the exact nature of the architectural and internal issues that led to this split was. Baker’s initial reaction was to express sadness over the lack of open discussion regarding the issues cited in the announcement: “We recognize that the current OpenWrt project suffers from many problems,” but “we hope to have the opportunity to discuss and try to resolve” them. Baker concluded:

We want to emphasize that we do want to have open discussions and resolve the issues at hand. Our goal is to work with all participants who can and wish to contribute to OpenWrt, including the LEDE team.

In addition to questions about the new project’s intentions, some mailing list subscribers raised doubts about whether LEDE would serve the same use cases as OpenWrt, particularly regarding the naming of the new project. Many, like Roman Yeryomin, expressed confusion over why the LEDE team needed to leave (to solve) these issues, especially since the LEDE team was composed mainly of active core OpenWrt developers. Some mailing list subscribers, like Michael Richardson, were even unclear about who would continue to develop OpenWrt.

Clarifications

The LEDE team attempted to elaborate on their circumstances. In Fietkau’s reply to Baker, he stated that discussions about purposeful changes within OpenWrt quickly became “toxic,” leading to no progress. Moreover:

The crux of these discussions is that those who control key parts of the infrastructure have limited energy and refuse to allow others to join and help, even in the face of significant problems that cannot be solved in a timely manner.

This sort of single point of failure has persisted for many years, with no meaningful progress made to resolve it.

Wich and Fietkau did not explicitly point to specific individuals, although others on the list might think that the infrastructure and internal decision-making issues of OpenWrt could be blamed on certain people. Daniel Dickinson stated:

My impression is that Kaloz (at least) holds the infrastructure hostage to maintain control, and the fundamental issue is that OpenWrt is not democratic and ignores what those who actually work on OpenWrt want, disregarding their wishes because he/they control the critical parts.

On the other hand, Luka Perkov pointed out that many OpenWrt developers wanted to migrate from Subversion to Git, but Fietkau blocked such a change.

It seems that OpenWrt’s management structure did not function as expected, resulting in personal conflicts, and because there was no well-defined process, certain individuals could simply ignore or block proposed changes. Clearly, this is not a sustainable pattern.

On May 6th, Crispin posted a new message to the OpenWrt list, attempting to reframe the LEDE project announcement. He stated that this was not meant to imply “hostility or division” but rather to clearly delineate from the structurally imbalanced OpenWrt and to start anew. The issue was that “no one should blame a single event, person, or flame war,” he said, “we want to separate ourselves from the mistakes we’ve made and the poor management decisions we’ve repeatedly made.” Crispin also acknowledged that the announcement was poorly executed, stating that the LEDE team “screwed up the launch agenda.”

Crispin’s email seemed to fail to satisfy Kaloz, who insisted that Crispin (as release manager) and Fietkau (as lead developer) could easily make the desired changes within OpenWrt. However, the discussions subsequently became quiet; what would happen next with either LEDE or OpenWrt remains to be seen.

Purpose

For OpenWrt members who want to explore more details about what LEDE believes to be problematic, there are additional sources of information that can provide clues to this issue. Before the public announcement, the LEDE organization spent several weeks discussing their plans, and the IRC logs of those meetings have now been published. Notably, the meeting on March 30 included detailed discussions of the project’s goals.

These included some complaints about OpenWrt’s infrastructure, such as the shortcomings of the project’s Trac ticket tracker. It was filled with incomplete bug reports and “me too” comments, Wich said, resulting in almost no contributors using it. Additionally, they were also tracking bugs on Github, which confused people regarding where tickets should be discussed.

These IRC discussions also set the development process itself. The LEDE team wanted to make changes to use a staged development branch that would merge into the main trunk, differing from the “directly committing to the trunk” method used by OpenWrt. The project would also provide time-based releases and encourage user testing by only releasing binary modules that had been successfully tested, with testing done by the community rather than core developers on actual hardware.

Finally, these IRC discussions also determined that the LEDE team’s purpose was not to intimidate OpenWrt with its announcement. Crispin mentioned that LEDE was initially “semi-public” and gradually becoming more public. Wich explained that he hoped LEDE would be “neutral, professional, and open the door to welcome OpenWrt for future merging.” Unfortunately, the initial launch work was not done well.

In an email, Fietkau added that core OpenWrt developers did indeed encounter bottlenecks in tasks such as patch reviews and infrastructure maintenance, which hindered them from completing other work like configuring download images and improving the build system. Within just a few days after the LEDE announcement, he said, the team successfully resolved the tasks related to images and build systems that had been pending for years.

Much of what we are doing in LEDE is based on the experience of decentralized package development by moving to Github and relinquishing much control over how packages should be maintained. This ultimately effectively reduced our workload, and we gained many more active developers.

We genuinely hope to do something similar for core development, but based on our experience of wanting to make larger changes, we feel we cannot do that within the OpenWrt project.

Fixing the infrastructure will also yield other benefits, he said, such as improving the system used to manage the passwords for signing release versions. The team is considering rules for non-upstream patches in certain cases, such as needing a description of the patch and an explanation of why it was not sent upstream. He also mentioned that many remaining OpenWrt developers expressed interest in joining LEDE, and the relevant parties are trying to determine if they will re-merge the project.

Some hope that LEDE’s flatter management model and more transparent division of labor will succeed in addressing the issues that have plagued OpenWrt. Resolving the communication issues criticized in the initial announcement would be the biggest hurdle. If that process is handled well, then in the future, LEDE and OpenWrt may be able to find common ground and collaborate. Otherwise, both teams may be forced to evolve in directions with even fewer resources than before, which may not be what developers or users want to see.

via: https://lwn.net/Articles/686767/

Author: Nathan Willis[20] Translator: XYenChi Proofreader: wxy

This article is originally compiled by LCTT and published with honor by Linux China.