Haiku QA team: looking for volunteers!

1

This post is a followup to post: http://www.haiku-os.org/community/forum/dedicated_haiku_qa_testing_team

So I am currently trying to assemble a QA team for the Haiku project which will be tasked with attempting to implement a QA process that would hopefully drive quality standards within the Haiku project. Currently, there is no dedicated QA team to oversee the testing process that would include a formal QA process for Haiku development.

The misson is simple: to establish a formal QA process (and test cycle) which is complimentary to the Development process without impeding on the efficiency and effectiveness of the current and future development and release cycles.

Things to consider getting done (in no particular order)…

  • implement a test regression suite/framework (in short: write test cases for current features and functionality)
  • write test cases for upcoming features and functionaility
  • establish protocols and procedures for test cycles for any and all release candidates prior to releases (including alphas, beta and final releases)
  • help the Dev team triage reported defects
  • generate testing and defect reports for all interested parties (stake holders, product owners, etc...)
  • automate as many manual test cases when/where possible
  • manage beta testing with Haiku end-users
  • of course: test, test, test and test some more

    • Most of the things listed above will take time to get done and mature. And some of the things listed are probably not the most fun to do as it can be a lot of digital paperwork (so-to-speak).

    I can assume some folks interested in being a part of the Haiku QA team already have some QA/testing experience, do QA/testing for a living or just like to break things (with QA/testing) and so the best team in my mind is one that has folks with different talents and strengths that harmonize well. Now for me, I’m no QA lead for this endeavour, I plan to be in the trenches with everyone else and I’d like to bring together people who simply desire to push and advocate for quality in everything that is Haiku and we help each other manage the whole QA process that we have established and agreed on. We all want Haiku to be the best operating system on the planet and the Haiku Dev team can only do so much (when not coding); so for us folks who can’t help out with coding can certainly drive the QA/testing aspect of the project and maybe be the loud and proud voice of the Haiku end-user when it comes to driving quality.

    If anyone is interested, please give a shout out here and maybe this thread will be a starting point going forward.

    2

    I don’t have much time or any QA experience but I’m happy to help as much as I can. Perhaps a good start would be to simply create a Haiku “bug squad” to go through old bugs and see if they still exist.

    3

    I have some time to put towards this. I have professional experience doing project management and some QA.

    4

    I think your list sounds great (though ambitious) and I would be happy to act as a Haiku developer liaison with the QA team, if such a thing is necessary. While I should probably spend most of my time developing, I do have a great ability to find bugs, so I could also play some QA role.

    One thing which I may work on soon is an automated test tool, with the first initial goal of having an automatic test suite of the Haiku menu system. Right now our “menu tracking” code leaves a lot to be desired, and it needs to be replaced, but it has years of subtle bugs fixes in it. Plus menus are a pain to test manually, and our current code doesn’t even fully emulate the BeOS menu behavior, so having a full menu test suite would greatly increase the speed at which a replacement could be made, and also ensure it works how we like (emulating BeOS or not.) I know I sure have more confidence in refactoring my day job Ruby code when there are good RSpecs behind it.

    Given that there is now renewed discussion of the alpha4 release, and I may be the release coordinator, I may not have time for the above soon, but it is definitely on my list.

    Regards,
    Ryan Leavengood

    5

    @leavengood

    Developers probably make some of the best testers since they know their (and others?) code well and probably have an inkling on the weak areas where QA folks usually make wild guesses as to where bugs might be found (like finding a pinata while blind folded to beat with a stick in hopes to get candy and other goodies). I currently work in an agile development environment, so requirements are either in code, some random wiki (in Trac for example) or word of mouth, the rest of it is common sense and guess work.

    I think the best goal for assembling a QA team is to maybe get one together to start work towards either supporting a formal test cycle prior to the beta release (at best) or the final release (at worst) of R1.

    I work in a Rails shop myself as a QA member and there is some use of RSpec by some of the dev teams (it all depends on the team and the devs and what they think is the best test tool), but I am not involved at that level with the exception of reviewing tests they commit to better understand the functional aspects of the code and figure out what I may need to cover with testing. I do come in on functional and integration testing whereas the Rails developers mainly focus on unit testing with a smattering of functional or integration testing using such testing tools: Ruby Test-Unit, RSpec, Capybara, Selenium, etc… QA does get involved (independently) with Cabybara and Selenium (I use Selenium a lot myself). Of course I totally understand the Rails philosophy of “test driven development” and the team I’m on does a pretty good job practicing it and Rails provides a good test framework to support it, so why not?!

    Oh, back to the subject here… yeah, if you and other devs would like to serve as liaisons to QA, that is great since it would hopefully be a mutual relationship that would benefit all and QA definitely depends on the devs in multiple ways. I’d be willing to learn the test automation capabilities of Haiku and attempt to write some of the automated tests that could support (and replace?) manual tests we generate.

    I guess I’d like to get a feel for what the devs would like from QA; how do they see QA fitting in and making a valuable contribution to Haiku development, etc…?

    -Dennis

    6

    Working off of #1 from DCATT’s list, the next step is to focus on what specifically needs a regression test. Ryan, what would you recommend as high value targets for testing? I would suggest that we make a manual first pass. This is gets us acquainted with the tasks at hand, and also gets us started a bit quicker, while we think about automation as a phase 2.

    7

    I agree with your assessment… the manual test cases certainly inspire the automated equivalents (like a recipe of sorts).

    8

    Hi,

    I am eager to join any testing effort for Haiku. Please see my bio for details. I would like to help out with automation or exploratory testing, although doing both always seem to produce the best results.

    Wouldn’t it be great if Haiku was so good it became a standard for some apps? I’d like to see it be a (the?) standard for cross-platform development work with a very wide selection of languages and APIs all working as seemlessly as possible. I like to work in many different languages (perl, python, groovy, etc.)

    The future may be running multiple OSes, each on its own virtual machine. We will be going to 8 and 16 processors on the desktop, it would be nice to be able to exploit all this power using VMs. And it would be really great to see Haiku on a VM. Even better would be if Haiku was the VM, but who knows.

    James Goss
    Pasadena, CA

    9

    @krazykat

    Looking at your bio, seems like you’re overqualified (not that that’s a bad thing). Anyways, I’m not in any way dictating how things should run concerning the QA team or its process. I’d rather brainstorm with those who want join the effort and come up with the best, most rewarding and fun way to test Haiku with some discipline so that the mission is continuously successful. Of course there will be some less than thrilling work to get the essential pieces in place (like a regression test suite/framework, etc…), but once that stuff is out of the way, it’s just keeping it maintained and updated and the rest is the more exciting stuff.

    I’m currently trying to get any of the Haiku devs to help us update the QA mailing list, which I guess I need to followup on that (maybe they expect me to do it?).

    10

    Well, I guess “overqualified” is better than nothing … a bottle in front of me is better than a frontal lobotomy, Tom Waits might still say.

    It is, after all, an operating system and there is lots that can go wrong - showstoppers, non-reproducibles, performance issues, usability, stability, boot times, APIs and compatibility, drivers, and so on. Some things with operating systems you don’t really test, you just verify that after the changes are made, stuff still works about the same.

    Anyway, I’d love to help out and it would give me a chance (excuse?) to really get into Haiku. I used to lead projects (as technical lead at Sun Microsystems, and as project lead sometimes, too) like this, so I feel right at home.

    When can we start? :slight_smile:

    James

    11

    If you have some idiot-proof scripts to run, I would be delighted to help.
    Mark

    12

    The sooner the better… I have been really trying to gauge interest and go from there. Awesome that you worked for Sun, I loved just about everything they did and I worked for a few technology companies that used Sun hardware and software for mission critical applications. Sounds like to me you’d like to take the lead on this effort, which is great if have the experience already. I think the first order of business is getting the QA mailing list fixed as it has not been maintained and needs to be updated.

    Mailing Lists | Haiku Project (look for Quality Assurance and Testing)

    I’m guessing that we can take care of it, but I thought maybe the Haiku project has a sysadmin that sort of manages that stuff for the project.

    13

    A new mailing list will be created in the next 24 hours at http://www.freelists.org/list/haiku-qa

    14

    Thanx a bunch!

    15

    All,

    The Haiku QA mailing list is now configured and working. So anyone interested in joining the Haiku QA team should subscribe to the mailing list so we can start sending and receiving official communications through there.

    http://www.haiku-os.org/community/ml

    Maybe we can setup an IRC channel as well for more immediate communication when doing testing events, etc…

    16

    In case everyone has not subscribed to the QA list yet (I just did myself), I figured I’d post a last message here, but from now on I’d suggest we discuss things on the mailing list.

    Also someone should feel free to create a #haiku-qa IRC channel on irc.freenode.net.

    Anyhow, we are starting the plans for an Alpha 4 release, which is currently scheduled for September 1, but that may be overly optimistic and certainly is not set in stone. Obviously I seriously doubt we can get a full QA process going for the Alpha 4 release, but if those of you who want to participate in the QA team could play a role in testing the release and see how we do that, I think it will be a great experience for all of us.

    I am the release coordinator for Alpha 4, which mainly means being the developer who manages the release branch, with merging and cherry-picking relevant bug fixes from the master branch into the release branch. We have not yet done a release while using Git, but I’m reasonably experienced with Git and I expect the branch management to be much, much, much more pleasant with Git than it was with Subversion.

    Right now fortunately Haiku is in pretty decent shape, and while plenty of bugs remain, overall things run pretty well. WebPositive was recently upgraded to a newer WebKit though, and there may be various regressions there, beyond all the bugs and lacking features it had (or didn’t have) before :wink:

    But there is no way all of those can be fixed in the next few weeks, so we will just triage and do the best we can, and I’ll just have to put some work into WebPositive and WebKit after the release.

    Also one major feature which will be released as part of Alpha 4 is better wireless support, in particular support for WPA security, which was a pretty major lack in Alpha 3. Unfortunately the GUI side of wireless management is just not there at all really, and again I don’t think we can pull that off for the release. But frankly we have gone too long without a release so it really just needs to get done.

    While there definitely are quite a few changes from Alpha 3, I don’t know if the release notes will be quite as impressive as they have been in the past. In a sense I consider this a maintenance release, if such a thing makes sense for something called Alpha (but as has been said our alphas are like some people’s released product.)

    Anyhow, I’m probably rambling a bit. Back on point: let’s start with a basic manual testing process, where the various QA members can divide and conquer testing Haiku itself and the various included applications. As a quick example, we need to make sure ShowImage can:

    1. Open images when they are double-clicked in Tracker.
    2. Open images from the menu.
    3. Open all images formats supported by the Translation Kit.
    4. All menu items work as expected.
    5. Drag and drop works.
    6. Selection mode works.
    7. And much more...

    If the tests are all documented they can later serve as a template for automated or semi-automated tests. The Haiku User Guide would be a great place to start to better understand some of the major features of the included Haiku applications (even I learn stuff when reading it.) Everything in the User Guide should work as expected (unless the User Guide is out of date, which is something else which should be found and fixed for the release.)

    OK, I think that is enough for now, let’s try to talk more on the mailing list.

    Regards,
    Ryan Leavengood

    17

    It would be helpful to make sure that the optionalpackages work as expected, don’t introduce any unexpected dependencies, and workhybrid GCC 2 hybrid.

    18

    I would be happy to join a QA team as I have been on many in the past (I also love breaking things and trying to fix them) I have subscribed to the mailing list so I would be happy to start straight away.

    -Matt

    19

    Do not get lazy at this point. If you have done the first part of their procurement process successfully, you should now be enjoying the fruits of their efforts - the receipt and sale of products that suits your needs.
    First Article Inspection In China

    20

    Hi!

    Well… its a great opportunity to do some work in the field. I have no practical experience, but I can do some simple work and I can study things I dont know.

    P.S. forgot to write my bio… will do it tomorrow

    Best regards,
    Andrew Sushko