November 7 2011

Migrating JIRA to Bugzilla

by Brian Dunagan

Recently, my team needed to migrate from Atlassian’s JIRA to Mozilla’s Bugzilla. I expected this process to be uncommon but not unheard of. Not so. Atlassian has a nice tool for migrating Bugzilla data into its JIRA system, but the one googleable conversation I could find about migrating JIRA to Bugzilla was in 2006.

I’ll describe the process I used to migrate around 6K bugs.

Tools

Bugzilla’s importxml.pl: Bugzilla does ship with an import script. However, it’s specifically designed for moving bugs from one Bugzilla instance to another. As input, it takes an XML file of one or more bugs, assuming you just used the XML export built into Bugzilla. Makes sense. It meant that I just extract all the issues from JIRA and reformat them into Bugzilla’s DTD. I found it very helpful to add the –verbose flag for better errors.

./importxml.pl --verbose bug.xml
# The log below represents a successful import.
OK: Bug http://example.com/bugzilla/show_bug.cgi?id=1 imported as bug 1.
http://example.com/bugzilla/show_bug.cgi?id=1

Bugzilla’s checksetup.pl: Bugzilla will check your setup with this script and let you know if you are missing any Perl modules. In particular, for importing, you will need XML::Twig. Oddly, when I started the import process, I found that I was still missing XML::Parser, so I had to install that from CPAN as well. Also, the script changed the ownership and permissions of the bugzilla directory such that it was no longer accessible on the web; I had to recursively revert those like so:

chmod -R 754 bugzilla
chown -R nobody:apache bugzilla

nokogiri: This handy Ruby gem allowed me to use XPath to search through the JIRA XML files and extract specific fields and values. It’s extremely useful. It installs effortlessly on Lion (10.7) with a simple gem install nokogiri. Unfortunately, I was using Snow Leopard (10.6). Installing the gem on that OS was a small battle in itself. Finally, I used this gist and then followed steps here. That worked for me. Here are some handy snippets of Nokogiri in action:

# Element contains
doc.xpath("//h1[contains(.,'Could not find issue with issue key')]")
# Element value
doc.xpath("//item/updated")[0].text
# Attribute value
doc.xpath("//item/reporter/@username").text
# Value of subelement for element
doc.xpath("//attachment").each do |attachment|
  attachment_id = attachment.attributes["id"].value
end

JIRA Issues: There are a couple ways to export issues from JIRA. One obvious way is to simply search for the issues you want and click “View” at the top to select a different format, like XML. While straight-forward, this method has two significant downsides. First, data loss: comments are not included. Second, size: thousands of issues take a long time to process as a single XML file, and neither JIRA nor Chrome seemed happy about the size.

An alternative to this export is using curl to export issues individually. This process includes comments and involves lots of quick JIRA queries. As an added bonus, I can avoid any XML SAX state logic that a single large XML file would have needed, so I can focus on transforming issues into bugs in isolation. Sweet.

# Extract JIRA issues. (Figure out how many issues to include.)
(1..1000).each do |i|
  # Use curl with credentials to extract each JIRA issue's XML.
  xml = %x[curl -u username:password https://jira.example.com/si/jira.issueviews:issue-xml/BUG-#{i}/BUG-#{i}.xml]
  # Save the XML to a file.
  f=File.new("BUG-#{i}.xml","w")
  f.write(xml)
  f.close
end

JIRA Attachments: In addition to the issues, I wanted to extract all of the attachments. Luckily, JIRA provides a standard HTTP API for getting these files. We just need all of the attachment IDs from each extracted JIRA issue to access its attachments. We’ll save those to an attachments directory. Bugzilla actually imports attachments as embedded base64 strings in the XML files, but we’ll address that later. For now, we just want to save the attachments out of JIRA.

# Download all the JIRA attachments. We'll need to convert them to base64 and provide them inline to Bugzilla.
Dir.mkdir("attachments") if !Dir.exists?("attachments")
(1..1000).each do |i|
  puts i
  # Open the JIRA issue.
  doc = Nokogiri::XML(open(File.expand_path("BUG-#{i}.xml")))
  # Find all the attachment references.
  doc.xpath("//attachment").each do |attachment|
    # Get the attachment ID and file name.
    attachment_id = attachment.attributes["id"].value
    file_name = attachment.attributes["name"].value
    # Get the attachment contents using curl.
    contents = %x[curl -u username:password https://jira.example.com/secure/attachment/#{attachment_id}/#{file_name}]
    # Save the attachment.
    Dir.mkdir("attachments/BUG-#{i}") if !Dir.exists?("attachments/BUG-#{i}")
    f=File.new("attachments/BUG-#{i}/#{attachment_id}_#{file_name}","w")
    f.write(contents)
    f.close
  end
end

Bugzilla Field Values: Bugzilla will not automatically create people, products, versions, components, or milestones during the import process. Those need to already exist. Otherwise, Bugzilla will use the default product and component. In my case, they did already exist, but the names had been changed. To handle these changes, I map JIRA strings to Bugzilla strings in my main script.

jira2bugzilla.rb

At this point, we have all the tools and the data we need to tranform JIRA issues into Bugzilla bugs. I committed the full jira2bugzilla.rb Ruby script to a GitHub repo. It doesn’t work out of the box, as there are quite a few instance-specific variables, but the script provides a nice base. I’ll touch on a couple points:

Attachments: Bugzilla expects attachments embedded in the XML file, so we need to convert our binary files into base64 strings and then include them inline. Below is Ruby code to convert to base64. Keep in mind that the import process takes far longer when attachments are included.

# Get the attachment ID and name.
attachment_id = attachment.attributes["id"].value
file_name = attachment.attributes["name"].value
# Figure out where we saved it.
attachment_path = "BUG/#{bug_id}/#{attachment_id}_#{file_name}"
# Encode the file as base64.
encoded_file = Base64.encode64(IO.read("#{attachment_path}"))

Description: While JIRA gives the bug description its own element (description), Bugzilla considers it the first comment on the bug. When rewriting the JIRA issues into a Bugzilla bug, I needed to migrate the description into the first comment.

Severity: JIRA doesn’t seem to have a notion of severity, so I assigned a default severity to all the bugs.

QA Contact: JIRA doesn’t seem to have a notion of QA contact, so when setting up the components, be sure to assign the default QA contact correctly. The import script will assign each bug to the correct person.

Again, see this GitHub repo for the full jira2bugzilla.rb script. I simply run ruby jira2bugzilla.rb in the directory with all of the JIRA XML issues.

Waiting for Import

At this point, we have converted all the JIRA issues into Bugzilla bugs. Next, we transfer them onto the Bugzilla server into a subdirectory of the bugzilla installation, like bugzilla/bugs/. To import the bugs, I ran the following one-liner. It finds all the XML files, sorts them alphabetically, and feeds each one by one into the Bugzilla import script.

find . -name "BUG-*.xml" | sort -n | xargs -n 1 ./importxml.pl --verbose &> bug-import.log

The script imported 6K bugs in two hours. However, I didn’t include attachments. When I tried to import a single bug with a 5MB attachment, it took around thirty minutes. With multiple gigabytes of attachments, I opted to not include them.

Most importantly, check the log. The above one-liner pipes all of the log information to bug-import.log. Run tail -f bug-import.log while it’s running to make sure the process is working. Grep through the output afterwards for terms like “Bad” or “Error” to ensure all the bugs were imported. With 6K bugs, most imported correctly, but a few did not.

Atlassian

I just wanted to highlight that migrating away from Atlassian’s JIRA is not a reflection of their product. I was simply tasked with making it happen. Frankly, Atlassian has been doing an amazing job at steadily growing into a vertically integrated company. They received $60M in VC funding in 2010 to grow their business, and they’ve scooped up businesses like BitBucket and SourceTree. Just a couple weeks ago, they announced that their main products will now be available as cloud services. Atlassian seems to have a long-term strategy that they’re executing very well.