Improving our development process

Posted By: Team XBMC on Dec 07, 2010 in Site News

At this year’s DevCon we took a long hard look our development process; we decided that while it works, it is starting to show a few cracks and we know we can do better.

We have a lot of issues with the current development model, mostly due to a lack of communication between developers prior to committing to mainline. This makes mainline more unstable than it needs to be; as churning of commits (initial commit, fix, re-fix, fix builds etc.) takes place. Instead, development in separate branches prior to commit, with sign-off and peer review from the developers who are active in the particular code area will reduce this churn and increase code quality of each commit. Currently we’re reviewing after the fact, rather than before, which clearly is not good practice. Git makes it easier to do review beforehand, so hopefully better tools will lead to better practices.

For those interested in how XBMC is developed, please read on.

Maintainer Model

The basic idea is that maintainers will be identified for various key areas in XBMC – people that have the most experience in particular code areas. There is already an understood “owner” of most parts of the code, but this will make that ownership more official. We will establish a simple rule that changes should be run past the maintainers prior to committing to mainline. Again, it is hoped by introducing peer review and accountability for commits, it will lead to better code.

Unit Testing

Many areas and functions in XBMC could be improved with the help of unit tests. Currently we have very few of them. From now on when implementing a new featuring or when bug fixing, we plan to start implementing unit testing where it is appropriate.  This will aid in reducing regressions and will allow us to more easily replicate issues and reliably conduct testing. This will also allow the community to provide more accurate and useful testcases.

The library scanner is a perfect example of something that would benefit from unit testing. When scanning (not scraping!) files into the library there are so many edge-cases for naming conventions and file locations that changing the behavior even slightly for one will likely break another. If we had a clear set of tests, and a dummy file-system layout to match, we could say with certainty that each case is verified working.

Moving to Git

We want to move away from using SVN as our version control system and move to Git. It was decided that github seems to be the most likely candidate for hosting our git repository, though we’d still have a clone on sourceforge.

We will initially use git the same way as we use SVN – i.e. all developers have commit access, but in time, will look to move to a hierarchical model i.e. several “key area” branches/trees that are managed by maintainers that merge into mainline when features are complete. This process will evolve in time, and should lead to much more maintainable code.

The git migration is technically done. We are only waiting for the Dharma release to flip the switch. This means that there may be a small downtime for things that depend on svn (feeds, automated builds, trac, etc.) but they’re obviously a priority and we will have them working again asap. If there is a significant interest in the move , what it entails, and what it will change, we can post more information.

Making sense of the mess

The build system currently leaves a lot to be desired. The folder structure in particular needs cleaning up; we have external libraries in places they should not be (i.e. code not written by us polluting the code written by us). It is also agreed that we move from a “default to internal libraries” model to a “default to external libraries” model. This will allow natural attrition to get rid of the internal libraries we currently have as we get the external libraries set up. It is easier to do this using git, so moving to git will be a priority.

We also want to get reduce the number of dependencies to minimize the difficulty of building XBMC.  By removing the number of internal dependencies that we currently have, it should make it easier for us to be included in distro repositories and make the code much easier to maintain.

Share on reddit
Share on StumbleUpon


Discussion - 55 Comments

  • Atamido Dec 07, 2010 

    Excellent post. It looks like you’ve identified the primary issues, the best changes to resolve those issues, and a rough time table for implementing those changes. I know that changes like this can be difficult for people to adjust to, but I also know the benefits that will be had once adjustments are made.

    Good luck!

  • Gareth Oakes Dec 08, 2010 

    Fun times! It is great to see this project maturing so well. What you say makes a lot of sense.

    Team XBMC kicks ass and you should be proud of your achievements :)

  • Derek Dec 08, 2010 

    Sounds fantastic that you guys are improving your development processes!

    Might want to take a look at your bug tracking tool too, it makes it difficult for anyone really to get a handle of managing all those bugs, gathering reports or having to coordinate, organize and consolidate everything when there is so many bugs everywhere that get reported! I would STRONGLY recommend switching from whatever bug tracking tool you’re using to something much more robust like JIRA, HP Quality Center, or BugZilla…. Have you thought about having someone become a full-time defect coordinator?

    Also, in regards to your library scanner, I have already put together some test sets quite a long time ago…
    http://forum.xbmc.org/showthread.php?t=66189

    or here for other stuff that helps with file scrape testing:
    http://forum.xbmc.org/showthread.php?t=66191

  • Ralob Dec 08, 2010 

    I am definitely excited to see the move to Git. It is so much faster and more manageable than SVN.

    I am also willing to help in anyway I can in regards to writing articles concerning weekly advancements in XBMC.

  • James McPhee Dec 08, 2010 

    LOL You’ve had users telling you the development process sucks for years… Aloof developers (you know who I mean…) doesn’t help matters.

    Hopefully these measure will make XBMC even better (love the product still!!)

  • T-Man Dec 08, 2010 

    I wish my company would be run more like the XBMC Project (or by managers with skills like prae5). Just like Atamido said: Identifiying main issues, then working out ways to solve them, and finally clearly communicate the planned changes. Instead, people here run around like a flock of sheep, reorganize the company every half year without thinking, without a specific goal and without communicating anything worthwhile.

  • XIYL Dec 08, 2010 

    XBMC has been my player of choice since 2005 and seeing it evolve instead of die has made me very happy. I think with these changes we can definitely say that post Dharma we’re going to see the next stage in that evolution. This project wouldn’t be what it is today if it weren’t for the passionate developers it has. Keep it up and thank you.

  • berny Dec 08, 2010 

    I’m using XBMC since its early XBOX days. – The improvements over the years have been really great (especially the WAF has been very high lately ;) ).

    It’s greatly appreciated that you’re reworking your development process. A stable core with the new add-on system as well as shorter release-cycles help to bind users by releasing features early. Anyway, speaking of latter I would like to see an automatic upgrade system for the XBMC core. Maybe this feature could make it into Eden?

    Keep up the good work!

  • Johnny Dec 08, 2010 

    XBMC Dharma is shaping up to be a great release. Its good to see that the XBMC project team are concentrating on the work detailed here. I vote for making it as bullet-proof as possible vs new features.

  • Henrik Dec 08, 2010 

    I really recommend that you adopt the gitflow branching model. See more at http://nvie.com/posts/a-successful-git-branching-model/ and https://github.com/nvie/gitflow

  • Tom4Surfing Dec 08, 2010 

    Sounds like a solid plan. Sorry to see you’re leaving sourceforge, but your reasoning sounds solid. Love the new release. It demonstrates that you guys know what you’re doing!

  • Godee Dec 08, 2010 

    Loving the recent news updates. Can’t wait till Dharma is final.
    I think moving to GIT is a great decision.

  • jmarshall Dec 08, 2010 

    @Derek: Any chance you’re volunteering for that role? :)

  • Anonymous Dec 08, 2010 

    Yay, unit tests! Having them will make bug reporting so much easier, and being able to create new ones to show deficiencies, etc will be great!

  • WiSo Dec 08, 2010 

    I hope you’re all right and this won’t stop creativity. For me XBMC reminds me more and more of my company which it shouldn’t as it is still my hobby (maybe therefore the lack of commits from me ;)
    It was more fun in the past …

  • Rodalpho Dec 08, 2010 

    @Derek
    Huge bugs with massive consumer impact sit around in the tracker fully documented and proven for literally years due to XBMC’s “hobby” development model. Development is 100% focused upon features and fixes that each developer personally wants and uses. A defect coordinator would certainly improve application quality, if he had teeth, some way to get devs to work on things they don’t personally care about.

    Before someone reads this as complaining, I’m really not– all of the XBMC devs work for absolutely free, and I’ve used the app since it was xbox media center on a chipped xbox1. Nothing else even approaches XBMC’s quality, as shown by the success of the boxee box as a local media player (the internet stuff was boxee, not XBMC). I’m just sayin’, it would be nice if we could seek through WMV files.

    http://trac.xbmc.org/ticket/5089

  • topfs2 Dec 08, 2010 

    @WiSo
    Well its meant to make it more fun, you spend less time fixing regressions and less time watching the timeline like a hawk for stuff that should _not_ go in or should have gone in differently, at least I am tired of that and I am one of those that does that very little even :)

  • Rodalpho Dec 08, 2010 

    @Rodalpho
    Actually, I’ve used it since it was xbox media PLAYER on a hacked xbox1. Man, that was a long time ago!

  • WiSo Dec 08, 2010 

    @topfs2: Thats the theory and therefore companies does it as well. But I didn’t saw a company where it actually works and currently I have the feeling our company just put more work in going along the processes than in the product.

  • coolcal Dec 08, 2010 

    I suggest a set of regression test cases for each functionality should be created – an automated run of different cases will help reduce the bugs

  • Steve Dec 09, 2010 

    I can’t recommend enough setting up some sort of continuous integration – that goes hand in hand with your desire to expand your unit test suite. Getting something like CruiseControl, Hudson, or the like going is key for your peace of mind as you move forward. It’s a positive feedback loop – the more tests you write, the more ground the CI covers each time someone commits to the repo, and the more confidence you can have that your product is solid.

  • oo_void Dec 09, 2010 

    I’d highly recommend looking into JIRA Studio. It’s still SVN based at the core, but provides a number of integrated tools to address some of the pain points you’ve mentioned. Though not publicized, hosting for most OSS projects is free. That said, good luck with the transition. Having done this professionally a number of times, the payoff of a well honed SDLC is well worth the pain and suffering you’ll undoubtedly experience at the beginning.

  • HP User Dec 09, 2010 

    @Henrik
    I haven’t read the article in details, but I’m missing a very important step in the git branch model pictures, and that is merge-and-test before commit to the main.

    That is: Before you should commit anything to the main, you should merge the main to the development branch and handle all conflicts in the development branch. Thereafter you should run your unit tests and automatic regression tests. When all are GO, you are allowed to commit to main.

    This works, and yes it is “boring”, but it relay improves quality!

  • HP User Dec 09, 2010 

    Unit test and regression test – Yes!

    It is a very good practice to write tests for you code. You have to more deeply figure out what you like the code to do, to be able to write good tests. Often you realize that you have to re-code during the development of the tests, since you are thinking on the effect/function instead on how to code during the test-development-phase.

    I’d also like to stress the importance of assertions. I think you should fill your code with assertions. This of two reasons.
    1) You documents (for you and all other coders) what you expect and within what boundaries your code are designed for.
    2) You get a 3:rd line of test, besides unit and regression tests, “code-level” tests.

    The C / C++ language pre-compiler is a great tool if you use it right!

  • HP User Dec 09, 2010 

    @WiSo
    Who are you coding for? If you – generally speaking – change the focus from your screen to the users, you’ll see the need for routines, processes and tools. And yes, it is not always “fun”, but right implemented you can actually increase your coding time.

    Just make sure your routines and processes are not time-consuming by them selfs. You should not add lot’s of meetings and document that no one reads etc.

    The creativity, that you are concerned about, should not be limited within a development branch, hack away! And now you can code even more! You can code test-code too! At first that sound boring, but you can make that part interesting and creative.

    What I’m trying to say is that; routines and processes right implemented with the right mindset does not decrease creativity or steal time and yet it improves quality. Just think of all hours spent on reading code-files, log-list, figure out scenarios to re-produce a reported bug, etc, figuring out that a that a bug fix unintentionally crashed some code – That is time consuming, boring and drains energy and creativity.

  • Voyager-xbmc Dec 09, 2010 

    The downside of the hierarchical model is that we will not see the changes in the main branch (trunk) as quickly as they’re coming in the old model, making the community interaction “less interactive”. I agree the final quality is going to improve, but the concept of a nightly build may no longer be relevant on the mainline, but developer enthousiasts (like me) will have to choose which sub-branch they like to follow to make their own builds. So I agree with WiSo that the “fun” aspect will go down.

  • HP User Dec 09, 2010 

    @Voyager-xbmc
    It is still possible to run “nightly builds” and experimental builds. Create a “experimental branch” and a script that automatically merges all or pointed out development branches and run nightly builds on that branch.

    Done it and in works, and it does not decrease quality on the main!

  • ecco Dec 09, 2010 

    Hi Guys… First comment is, I love XBMC and showing it to as many people as I can!

    Second comment. At work, I was looking at doing the peer reviewed commits before they made it into the mainline. I opted not to because the project was still in it’s infancy (and it’s a small dev team) and it would have added too much overhead for my needs. Anyways, to make a long story short, I spent a couple of hours looking at ‘gerrit’ to formalize code reviews and approval before code makes it to the main line. It looked like it could be a pretty useful tool and I am thinking of implementing it when the project matures and we’re looking more at bug fixes / improvements vs core product development.

    Hope it helps!

  • topfs2 Dec 09, 2010 

    @Voyager-xbmc
    Well I can’t argue against that features will take longer to arrive in trunk, the question is how much would it really be. And when they arrive they are usable :)
    Most devs already use a local repo and commit when the feature is done, so instead of them being forced to that they can push to git in the new model making it possible for other devs (or users) to look at the development faster. It will be more spread out but it shouldn’t be to much harder.

    Another thing is that the new tool is git, in git its so easy to merge so you could easily merge all interesting branches in a few minutes and you would have the perfect nightly for you :) Not something the average nightly user is likely to do though.

  • Konan Dec 09, 2010 

    Testing, more testing, testing as much as possible, and even more testing! This is a rule of thumb for building reliable, rock solid software. Unit testing is great and it does help improve code quality however it does not usually involve community: only developers or those who build the app will be able to execute unit tests.

    However having such an enthusiastic, large and capable community of users and not utilizing its testing ability does look weird. Please involve us for testing and together we shall make XBMC even better. There are a lot of XBMC users who for any reason don’t write the code but are willing to help testing changes and new features. Just organize users who wants to help testing and enjoy great success.

    One particular XBMC difference is that it’s multi-platform meaning that testing must be done on different platforms. Organize users by platforms, build a list of standard regression tests, document testing techniques, provide with a mechanism of collecting test reports and XBMC is a winner and everyone else with it.

    This is my 2 cents to your brainstorming of improving the development process.

  • topfs2 Dec 09, 2010 

    @Voyager-xbmc
    Sorry my last answer was a bit vauge. Current model for most devs is this:
    Create a feature locally in their own repo (which noone sees). When its done they will either push to trunk or to trac depending on if they want review or not.
    The new model would be:
    Create a feature locally and push to their branch / tree (here they can push sooner and more often if they want). When the feature is done they submit a pull request and the maintainer goes over the code and do the pull.

    All in all the feature should ideally not take more than a few days longer to arrive, so the features will probably arrive somewhat the same speed but with a delay of a few days, which is something I doubt anyone will notice that much :)

  • WiSo Dec 09, 2010 

    I won’t argue against that we need more testing ;)
    You all say my doubts in “.. if you do it right …” and thats the point. Also if we look on our stable releases: what did we gain comparing to our svn releases in the past? Maybe a little bit more stability (really?) but looking at the commit cycle at the end of each release the progress is dramatically decreasing.

  • djdafreund Dec 09, 2010 

    This is all great news. I gotta admit there is one SVN turned GIT project that failed once they changed, as git is not very user friendly at all compared to SVN (no easy method’s like having tortoiseSVN, as it’s all command line instead, and no easy ways to use window’s explorer based that i’ve seen to date.) but XBMC is VERY experienced, and will no doubt ably will use GIT much better. Looking forward to these great changes.

  • jmarshall Dec 09, 2010 

    @Konan: We need volunteers to organise it. In the past, all the testers we’ve chosen have ended up becoming devs one way or another, so we obviously suck at selecting testers ;) Let us know if you want to help out organise this.

  • Alan Ratio Dec 09, 2010 

    @Sebak – Agile methods don’t work very well when the developers are distributed – unless you have a strong communication infrastructure and team discipline.
    Unit tests are essential, though, as are reliable build methods. I’m amazed the devs got this far without them!

  • djdafreund Dec 10, 2010 

    @Theuni
    OMG, That is too cool. Thanks for that info. No disadvantages anymore to GIT now. LOL. I’ll pass that info on BTW. Too many people left the other big scene (“FoFiX”), cause people didn’t like GIT at all. Mostly cause of that reason though. This will ‘hopefully’ bring people back.

  • Klojum Dec 10, 2010 

    As an application tester myself, I’m glad that the XBMC-devvers are (finally, hehe) “seeing the light”. I’m backing Konan here, and unit testing is just one step in the whole testing chain. Any TMap or ISTQB software tester will tell you that testing starts already at the very basic level when designing an application. Starting tests with an already available application is not impossible, but it should be done at as many levels possible. Perhaps very boring, yes… But the amounts of time saved by not having to contiously “fix-rebuild-retest” over and over again will pay off in the end. The same goes for documentation, another one of those ‘yucky’ topics in the IT world.

  • Anonymous Dec 10, 2010 

    @jmarshall
    i’d be more than happy to give it a shot IF you were able to change tracking tools to something else, it’s too hard to nearly impossible with the options that are exposed to me right now in the tracking tool! :)

  • Derek Dec 10, 2010 

    btw, the anonymous comment was from me :)

  • Henrik Dec 10, 2010 

    @HP User: I believe this is handled. Read the article in detail :)

  • jmarshall Dec 10, 2010 

    @Derek – there is likely more tracking tools available to trac admins – what sort of requirements would you be wanting? We have considered switching away from trac at some point, so if you have a set of requirements that you feel is essential, start a forum thread and let us know.

  • Pedro Alves Dec 10, 2010 

    Sounds great. Are there development mailing lists as well? (I can’t get my head around the whole forum thing.)

  • Derek Dec 10, 2010 

    @jmarshall

    I need to do some analysis into the tool and see what it’s capable of, its shortcomings, figure out a process that would work best with the team and when its all ready I’ll post a list of recommendations to help team and speed up its workflow. Good news is, I’ve already got some documentation I can re-use, I’ve done this before :)

    cheers,
    D

  • Rob Dec 12, 2010 

    @Konan
    Yes what would be great would be some sort of testing model and plan a tester or even just a user of XBMC could work through with each revision of Xbmc. I’m not sure how each report could be published though Trac definitely wouldn’t be suitable for that. There must be a developed platform out there that would be suitable for XBMC’s needs. 100′s or even 1000′s of people run nightly builds so some sort of testing platform I think would certainly benefit XBMC.

    I also think a bug/trac/development manager would be a good idea. WordPress I believe have a lady called Jane that does it, on a much larger scale of course due to the size of the wordpress project. I’m not sure of all the details of her role but it might be something to look into.

  • Rob Dec 12, 2010 

    WiSo :
    I hope you’re all right and this won’t stop creativity. For me XBMC reminds me more and more of my company which it shouldn’t as it is still my hobby (maybe therefore the lack of commits from me ;)
    It was more fun in the past …

    WiSo I think this could possibly make it more fun for you. This Dharma release cycle has been a long one especially in the last testing cycles of it meaning a big feature slowdown. If you were working on your own repo of xbmc you could pretty much do what ever fix and add whatever new feature you wanted in your own time and not have to really worry about development cycles. Your new fixes or features could then just be added when you are happy with them.

    ?

    I dunno just my thoughts.

    What happened to crystalPT by the way he seemed to be doing a bang up job of helping out with some of the windows and dvdplayer development.

  • Anon Interface Race Condition Fixer Dec 12, 2010 

    The major problem that I have with how the devs handle this project is the unwillingness to update the current stable package for major distros (Ubuntu) as they are upgraded. This forces far too many users to use the unstable SVN package and causes significant noise (whining).

    I came here today to see if there were stable ppa packages for Ubuntu 10.10 yet.

    Nope.

    Waiting for Dharma to provide a stable package for ubuntu 10.10 (Maverick Meerkat) is counterproductive and puts the devs in the middle of a crowd of crying users who are forced to use unstable, buggy SVN code.

    What’s the point of stable versions if you push everyone onto the nightly builds anyway? The pending ‘golden’ version of Dharma is meaningless as you will force everyone back onto the buggy post-Dharma code shortly after it comes out anyway.

    Rebuilding the 10.04 packages for 10.10 would quiet things down considerably for you guys while making the users’ experience with your application much more solid.

    The periods where stable versions exist for current distro versions are very short. Most of the time, most distros are on an unstable version of xbmc because the packagers do not roll the previous stable version forward.

    Xbmc does not have to be this chaotic of an experience.

  • Derek Dec 13, 2010 

    @jmarshall

    fyi, i’ve done some initial analysis for a replacement for trac, and i’ve come to the conclusion, that based on the upgrade requirements for all defect reporting tools, including checking out hp quality center (which i maintain at work), itracker, trac, bugzilla and jira, JIRA came out on top. Here’s why:

    - Jira is a web-based solution, no software req’d on user’s pc.

    - it is a fully loaded app that makes project reporting very easy to see at a project level all defects very easily compared to the other tools

    - triaging defects is very easy to do, especially since you can build a custom workflow for xbmc to work around the dev team

    -makes it easy for each ‘maintainer’ of each area to customize their dashboard to see only items that they need to see

    -it ALREADY has a way to migrate from trac to JIRA, by importing and using cvs files

    -its FREE for open source projects! All we have to do is get the trial up and running on the website and apply for the free license and they’ll give it to us–unlimited access for a well-supported platform!

    Dont worry i’ll still post a note on the forum to discuss with the xbmc team, i’m on the train right now.

    Cheers, d

  • prae5 Dec 13, 2010 

    @Derek

    For what its worth, this is something I have been looking at for the past few weeks as I have been fighting with trac whilst trying to get something workable out of it to produce a decent changelog for dharma.

    I have a local JIRA install I have been evaluating here and it is much nicer, I’m in the process of testing how this can link up to git. Have come up with a few issues, but think I’ve found workaround for most of them.

  • Derek Dec 14, 2010 

    @prae5
    That’s awesome prae5, let me know what issues you’ve came up with and also how I can help you with trying to integrate it with git. :)

  • Derek Dec 14, 2010 
  • Anonymous Dec 14, 2010 

    @Derek

    Thanks, have it installed on my test machine – just haven’t got round to fully setting it up yet.

    Hopefully will by the end of the week.

  • Derek Dec 15, 2010 

    @prae5
    I was also looking at other alternatives to that git plugin– there’s also another tool called Fisheye from the same company that integrates with git AND jira. This tool is also free for open source projects :)

  • Marisol Perry Dec 21, 2010 

    @prae5 That’s awesome prae5, let me know what issues you’ve came up with and also how I can help you with trying to integrate it with git. :)

About Kodi

Kodi is a free and open source media player application developed by the XBMC Foundation, a non-profit technology consortium. Kodi is available for multiple operating-systems and hardware platforms, featuring a 10-foot user interface for use with televisions and remote controls. It allows users to play and view most videos, music, podcasts, and other digital media files from local and network storage media and the internet.