jGrants: Grant application and management system for the Ministry of Economy, Trade and Industry of Japan

Launched in December 2019, the Ministry of Economy, Trade, and Industry's (METI) jGrants is a Drupal-based web application which allows businesses to apply for grants, as well as to streamline the management of applications and business information by the grants office. 

ANNAI worked closely with the METI for the last three years, conducting fundamental research and developing the system using an agile development methodology.

In this blog post, we will describe the following:

• the unique challenges of agile development involving Japanese ministry
• how we came to develop jGrants using an agile methodology
• some of the benefits of agile that are not usually talked about

The challenge: Highly complex grant application process and operational tasks

Due to the current complexity of grants application and management processes, substantial time and effort were required to arrive at the point where we can commence the actual development. For example, there are two types of subsidies; direct subsidies and indirect subsidies. The former is managed and administered directly by a ministry; the latter is managed and administered by a variety of agencies and organisations, including subordinate organisations and auxiliary organisations directly commissioned by the ministry and even subcontracted by these organisations. In addition, the application, adoption, and operation of subsidies are not standardized, but each organisation sets their own rules and processes based on the Act on Appropriate Budget Execution for Subsidies, etc. (hereinafter referred to as the "Appropriation Act"). As a result, developing a system that covers all of the differences was considered impractical. ANNAI, therefore, proposed to first conduct a Business Process Re-engineering (BPR) based on observation and analysis of existing application and adoption processes, as well as management methods of major institutions, as this would lay a foundation for simplification of the business processes which in turn significantly reduces the complexity of the system to be developed.

Agile Project Management

Since the simplified business processes were yet to emerge, it was not possible to define requirements/technical specifications of the system at the beginning of the project. ANNAI proposed that the project be managed in an agile fashion, with an iterative approach to discovery, development and improvement. While this was accepted by the client, this naturally meant that the final product to be delivered could not be defined at the beginning of the project. This was highly unusual for METI, as adoption of the waterfall model is still the norm in government software projects in Japan and it was their first project to be managed by agile. ANNAI helped METI to familiarize themselves with an agile development method during the course of the project.

In the planning phase of the project, we were considering making the APIs publicly accessible, as well as decoupling the front- and backend. In the process of assessing the feasibility of the project, we conducted prototyping to determine the feasibility of these plans. Such an exploratory and flexible approach was only possible with an agile approach.

The project management was based on Scrum with some adjustments to accommodate the client’s peculiar needs and yet the actual management of the project was fraught with many difficulties. Since most of METI’s project team members had never used agile management methods before, we made sure that they understood the philosophy of agile and trained them accordingly.

Practising agile

The key to agile project management is to ensure that the client - especially the person who takes on the role of the product owner in Scrum - understands the values of agile and adheres to the rules set by the framework, as the product owner is essentially the director of the product. By concentrating the power to make key decisions as well as responsibilities to a specific role, rather than decentralizing them, the time it takes to make a decision can be reduced and the project can progress more quickly. However, a product owner has no prior experience in agile, they often struggle to move a project forward with a completely different mindset. ANNAI supported the product throughout the project owner to move forward with the project in accordance with the values of the agile methodology, based on their authority and responsibility.

Changing requirements / technical specifications and communication

In waterfall development, the client often communicates their requirements to the developers in an early phase of a project, from which the specifications are derived and the system developed. In agile development, however, specifications are defined gradually throughout the course of the project. This is because, unlike waterfall, it is assumed that not all the requirements are to be implemented. As circumstances change, technical issues are encountered and the client's understanding of the system deepens, requirements and specifications are redefined, amended and/or dropped during regular reviews.

Prioritisation of requirements and tasks

One of the fundamental differences between agile and waterfall is the notion that time and resources are finite. Within this constraint, the decision on what to / what not to implement must be made in order to maximize the value of the product while adhering to the delivery schedule.

As the development team works out the details of the specifications for each milestone or sprint, the developer estimates the efforts required to implement a particular feature. The product owner then decides whether to implement the feature as per the original specification, change the specification, or forego the implementation itself. Such decisions must be made while taking into account the requirements of the feature, the overall situation at the time, as well as the priority of other requirements. This is fundamentally different from the mindset of waterfall development, where features are implemented without regard to their value because "it's been agreed". An agile coach, therefore, needs to help product owners to adopt this mindset and continually ensure that they practice it consistently.

"In order to implement the feature A in this milestone, features B, C or D must be dropped from the milestone” - such a negotiation often takes place in agile development. For the client who was accustomed to the waterfall approach, such a statement was difficult to accept until the concept of prioritisation and finiteness of time became ingrained.

Managing the client's expectations regarding the schedule

The challenge on the client side

Another challenge was how to manage the increase in the client’s expectations for the pace of the development as the specifications gradually emerged.

Specifications usually start to emerge as the project progresses, but the details will not become clear or get defined until the last minute. It is therefore crucial that the product owner be always available for the developers to answer questions. When different features are developed in parallel, however, the amount of work for the product owner increases and becomes a bottleneck as a result. For this reason, helping to enable the product owner becomes crucial in order for the project to run smoothly. As mentioned earlier, it takes time for a newly-appointed product owner to grow into their role. The client may suggest increasing the budget by ten-fold in order to develop the system 10 times faster, the client - in particular the product owner - would not be able to manage the increase in demand in the first place. This would also require fundamental changes to the way the project and teams are managed, such as introducing Disciplined Agile Delivery.

The challenge on the developer side

On the development side, it is well known that adding new developers to a project temporarily slows down the pace of the project due to the time required for onboarding and increased communication overhead. To put it simply, even if you estimate the remaining development work to be done in 100 man-days, you cannot complete the work in a single day by adding 100 developers to the team. Also, the practice of agile scaling is not yet popular in Japan and remains a challenge for ANNAI.

Without having the client understand these points, ANNAI would have been deemed ‘too small for the project’. It was therefore crucial that we help the client understand the implications of adopting agile methodology as well as the nature of software development.

Managing the client's expectations regarding the deliverables

As mentioned at the beginning of this article, since the operation of grants in the real world is highly complex and developing a system to cover all the use-cases was impractical, the approach we took was to find the common denominator through BPR.

Because expressing complex business processes that involve a large number of steps in text or diagrams is ineffective, it was difficult to discuss detailed specifications. Based on this constraint, we agreed to build a prototype based on which we discuss the details. However, some of the stakeholders mistook the prototype as the final product and complained about “the lack of indispensable features”.

We also had to overcome several abrupt changes, which are often derided as "meteorite-fall approach" among Japanese software developers. For example, a high-level stakeholder suddenly decided to move up the production schedule, forcing us to review our entire prototype-based development approach.

While this kind of critical situation is often overcome by developers working overtime, it is not a sustainable solution in the long run. Nor is it a problem that can be solved in a short period of time by increasing the number of developers, as in the scaling discussion above. "Which features should be prioritised given the current needs, as well as resource and time constraints?" was therefore a very real question which we had to ask ourselves and discuss. By sharing with the client the mindset of prioritising the requirements based on the available time and resources at hand, we were able to make the right decision to overcome such crises.

The (hidden) benefit of adopting agile

By using the Agile development methodology, we were able to launch jGrants on schedule while implementing the requirements in a way that maximized the value of the system. Through the journey, the way the client was involved in the project had also changed considerably.

In waterfall development, once the requirements are defined, generally clients do not get involved in the project during the development phase. However, agile brings fundamental changes to the relationship between the client and the developer. 

As the client committed themselves to the project and respected the philosophy of agile, they started to take ownership of the project and consistently got involved in the project. This allowed the team to communicate closely to develop an optimal system, while taking into consideration the changing needs and circumstances.

In addition, the product owner, who was integral to the success of the project, has come to best understand the specifications of the system through this development process. An individual with such a deep level of understanding on the project is irreplaceable as we continue to develop jGrants further.

When talking about the benefits of adopting agile, the focus of the discussion often seems to be on the speed of development, short Time To Market and flexibility in decision making. However, it is worth noting that they also have a significant impact on the people involved in the project, as described above, resulting in a positive long-term impact on the project, or even on the organisation.

Challenges specific to government projects

Achieving government-standard security requirements

Unlike waterfall development, agile development does not involve detailed discovery and specification design at the initial phase of a project. Test cases can therefore only be prepared after the details of the requirements and specifications are finalized as the project progresses. Before the jGrants project was initiated, all the IT systems whose development was outsourced to the private sector were closed systems that were accessed only within government offices. Because jGrants was the first to allow users’ access to administrative and editorial functions, the client was not able to present the requirements until the last minute. In order to pass the enormous number of security tests in order to meet the security requirements, a large number of test cases had to be prepared as well as external resources to execute them had to be secured in a relatively short period of time.

Institutional and structural problems in the Japanese government that hinder the practice of agile values

The budget for the Japanese government’s IT projects must be acquired at the beginning of each fiscal year. This alone has a great influence on how IT projects for the government are run, as the contractors are required to adhere to the specifications and master schedule envisioned at the time of bidding, regardless of whether the project is managed in an agile manner. This clearly conflicts with how agile projects are meant to run.

Also, because of the yearly budget allocation, it is not possible to run IT projects that span across multiple fiscal years. The government’s desire to launch a service on an annual basis adds difficulty to the government’s IT projects.

These fundamental problems, unfortunately, prevent IT projects for the public sector from benefiting from values that agile methodologies provide.

The UK government has made it clear that all government IT projects must use agile methodology, and is providing a high-quality service manual on agile delivery (https://www.gov.uk/service-manual/agile-delivery). In order for the Japanese government to effectively use agile development to build high-value IT systems in the future, a number of reforms are needed within the government.

Conclusion

Starting with the task of deriving the lowest common denominator of the different processes at thousands of secretariats, it was a daunting task to solidify the specification from the vague outline of requirements to the level where it could actually be implemented. METI and ANNAI worked as a team to overcome the constraints and difficulties, and jGrants was able to launch on schedule, despite the drastic change to the schedule and various other challenges. By leveraging the flexibility and reliability of Drupal and effectively applying agile development methodologies, ANNAI was able to successfully complete this challenging project. It is also worth noting that METI benefited not only from the system that meets their requirements, but also the experience of leading a project to success using the methodology which was entirely new to them, as well as the fostering of their human resource as a result.

(Note: jGrants is a service that is completely independent of the infrastructure and systems of the Employment Adjustment Grant. jGrants is also in no way related to the Sustainment Benefit, and the Service Design Engineering Council is not involved, nor is it affiliated with any of the council's subcontractors).