How did we reach the festive season already? December snuck up on us this year, and so did a million milestones in the Generative AI space. I’m especially excited to discuss Engine Room’s latest innovation. It’s a work-planning tool directly connected to our last blog post about Atlassian’s Rovo AI, and it’s designed to net you maximum savings on your work time and effort!
What we’ve recently begun working on is an AI-fueled tool specializing in providing reasonable time estimates, which can be kept or tweaked, for each of a team’s work items. It does this by searching the team’s Jira history for issues similar to an issue that it’s been asked about — analyzing all their original estimates and actual time-tracking data to inform a predicted estimate for the issue in question. This Issue Time AI Estimator is an app we’re building as an AI agent of Rovo by Atlassian — meaning that it’s sure to be powered by the finest in LLM capabilities, and that it operates directly inside the Atlassian Jira workspace. This exciting product is also planned to coexist with the main ongoing project we’ve told you all about: the AI-powered Test Case and Test Script Generator we’re building for the world to use. Together, these two tools can transform the most time-consuming portions of your end-to-end software development and quality assurance workflows!
We’re so excited about Issue Time AI Estimator for a number of reasons. Whenever a new task comes up in a project, developers in teams like ours can’t even begin work until they decide on and report a time estimate so bookkeeping can run smoothly. This is far from the most exciting part of a dev’s job – in fact, it can be quite stressful to come up with an estimate that won’t backfire. That time and stress are reduced greatly by allowing a fine-tuned AI agent to provide an informed initial estimate that the dev can then respond to and tweak as needed. It’s also quite easy — when devs have a ton on their plates — to forget or miss small details about a task that can dramatically impact the time needed to complete it. AI agents are much less prone to missing the nitty-gritty, providing extra assurance of coverage in your estimates.
And think about it: if an AI agent throws a number out first — even if inaccurate — that number can simply elicit a reaction from the developer (i.e. “That might be too low/high...”). Just like that, the dev has found a direction in which to begin estimating; more effectively than they could from staring at a blank field. What’s more, there may be dozens or hundreds of work issues in the team’s history, with only a few of them serving as relevant examples to compare with a specific new work item. Developers shouldn’t be expected to remember what those exact two-year-old issues were, or how far over/under they all ended up — and browsing to find out can be even more cumbersome! With an agent handling all that, the developer is able to quickly get back to their “flow”, continuing to tackle the more interesting challenges of the project.
How exactly are we going about this? Last blog, we discussed plenty about Rovo and its use cases, but not so much of Forge, Atlassian’s serverless app development platform that facilitates the creation of custom apps and Rovo agents like AI Estimator. The Forge process is quite friendly for new developers, though it does favor having some JavaScript experience. It all begins in Atlassian’s Forge CLI, where you create a project for your new app using your secure Atlassian API Token. Several files are then generated for you to use as templates. Most of your time will be spent in the manifest. This is a YAML file that comes with metadata related to your app, such as pre-filled app identification info, an editable list of app permissions in Jira or Confluence, a list of actions the app can take, and — for Rovo apps like the AI Estimator — your AI agent’s core prompt! Writing up an effective prompt, testing the agent, tweaking the prompt, and re-testing is a lengthy but valuable cycle. Like any LLM-powered tool, Rovo AI agents do their best work when they’re prompted as clearly and effectively as possible.
We’ve faced a couple of challenges in developing Issue Time AI Estimator — the most prominent one being hallucinations. Even with clear guidelines to follow when fetching comparison issues to fuel estimates, the agent sometimes gets so enthusiastic that it makes up issues or projects. These hallucinated issues are actually relevant to the task being estimated, so the agent’s comprehension of what it’s looking for is solid, but we just need to ground it further so as to continue applying that comprehension strictly to the material within the team’s projects. Other difficulties, more on the logistical side, include finding clean issues to test the agent on — for example, those that don’t already have listed estimates that it could just copy from, or partial time tracking records that give it “clues” that it wouldn’t have with an end user. Since my team members’ project work is ongoing, cleaning up this data creates interference concerns with their tracking, so some maneuvering has been needed for me to reliably test the agent’s performance.
We’re also keeping some follow-up ideas on the board to add to Issue Time AI Estimator in the future. These include chunking down each task into clear sub-task estimates, establishing baseline time values for recurring task types (with situational tweaks), and more.
Of course, the AI Estimator agent is just one step for us in our long-term deep dive into Atlassian’s Forge and Rovo. It’s still our ultimate goal to complete Engine Room’s very own AI Test Case and Test Script Generator QA app! AI Estimator presents a great opportunity for us to simplify some tedious duties for your teams and our own, while we also gather knowledge about ways we can put together the perfect QA agent for you all. Both agents have us at the edge of our seats, and we hope you feel the same!
If you’re as excited as I am about harnessing the current AI explosion (and seeing tenfold reductions in your team’s expenses and workload), be on watch for my next post in this Engine Room blog series. And if you missed the previous post, take a glance at my friendly breakdown of Atlassian’s AI — or go back to the beginning to see my big reveal of our efforts to create an AI-fueled Test Case and Test Script Generator!
Happy holidays from us all here at Engine Room!
- Shahpur