38 Studios in Rhode Island - A Lesson in Software Risk
This is a guest post by Capers Jones, Vice President and CTO, Namcook Analytics LLC. He is also the founder and former chairman of Software Productivity Research LLC (SPR) where he holds the title of Chief Scientist Emeritus.
In 2010 the Economic Development Commission (EDC) of the State of Rhode Island agreed to loan $75,000,000 to the 38 Studios game company owned by former Red Sox pitcher Curt Schilling. In return the company moved to Providence and began operations with about 250 employees.
As is common with software applications in the $75,000,000 cost range the main product of 38 Studios ran behind schedule. As is also common with start ups, the 38 Studios company itself ran low on funds and fell behind its payments to the State.
In the absence of fresh capital from either film credits, external investors, the State, or other sources, the company missed payrolls, ran out of funds, laid off the entire staff and then declared bankruptcy.
The Rhode Island State Police have launched an investigation which is just starting, so the full story is not known. However from the partial information published in the Providence Journal and discussed on talk radio shows, the 38 Studios situation looks like a typical Rhode Island morass of cronyism, back door deals, secret negotiations, and concealed facts. For example according to newspaper reports the Rhode Island General Assembly did not know that a vote they were asked to cast for funding the EDC was actually aimed specifically at 38 Studios.
As it happens my regular professional work is designing and building software cost estimating and risk analysis tools. At the time of the 38 Studios failure I was working on a new estimating tool called Software Risk Master. One of its purposes is to aid investment groups in ascertaining the risks of start-up software companies.
I have no personal knowledge of the 38 Studios software development methods. I had never met Mr. Schilling or any of the 38 Studios software team. But it is easy to generate a risk analysis for software packages that cost about $75,000,000 to develop. That is a very risky region with many failures.
Without knowing what 38 Studios has for compensation and overhead, I plugged in a generic value of $10,000 per staff month for operating costs. I then ran my Software Risk Master tool in backward-loading mode to match a $75,000,000 outlay.
Three general forms of risks were predicted for 38 Studios:
- Development risks including cost and schedule overruns
- Quality risks including excessive defect rates after delivery
- Maintenance and customer support risks after delivery
Due to lack of information about specific 38 Studios methods and practices I merely used default values or average settings for team skills, methods, and other variable factors. However even with a top-notch team start ups are risky and usually encounter delays.
Using industry default settings my tool showed staffing of about 250 people and a development schedule of 42 months, which would be about 6 months later than typically desired. The size of the application was predicted to be about 2,500,000 lines of source code in mixed languages.
[Read Caper's letter in the Providence Journal and how he predicted the risks.]
It would interesting to know if either 38 Studios itself or the EDC used any of the available commercial software cost and risk analysis tools. If not, why not?
Apparently there was an earlier risk analysis in 2009 before the State made the loan. This risk analysis was for private investors and concluded that 38 Studios was so risky that only investors who could afford to lose their entire investment should consider it. This information was apparently not shared with the State of Rhode Island, which certainly cannot afford to lose the entire investment.
In any case $75,000,000 software start ups are among the most risky business investments in the modern world. It would be useful to know if the EDC knew about the high failure rates and risk levels of start-up software groups. If the EDC did not know, they should have brought in qualified software experts before making the investment. Rhode Island has quite a few.
Quality is not as critical a factor for computer games as for business software, but poor quality can drive away customers and slow down acceptance. My quality risk predictions showed a probable defect rate at delivery of about 1,000 serious bugs. Around $26,000,000 out of the total investment of $75,000,000 would have gone to defect removal costs.
It happens that excessive bug levels during testing is the main source of software schedule delays, so any software risk tool should be able to handle quality predictions and the impact of quality on costs and schedules.
The third risk prediction was for post-release maintenance and support costs. Since 38 Studios has ceased operation, these predictions may be moot. However prior to investing in a major software development project it is obvious that down-stream investments will also be needed to keep the application running, fix bugs, and add enhancements.
It is not clear if the State of Rhode Island planned for post-release investments. So far data on post-release maintenance and enhancement costs have not been published in local papers, but the predictions from my tool showed post-release costs of about $67,000,000 for a five year period.
These costs are over and above the initial investment of $75,000,000. It is not clear whether these post-release costs would be funded by revenues from 38 Studios or would require additional rounds of investment.
Normally for start ups several rounds of investment are needed. Maintenance costs plus initial development bring the financial exposure up to $142,000,000.
The final prediction was for the total amount of venture money that might be needed for development, maintenance, support, marketing, advertising, global sales, and other corporate expenses.
My tool predicted a total investment of about $206,000,000 and three rounds of financing over about a 10 year period when all expense elements were aggregated. It is not clear from newspaper reports whether the State planned for continuing investments after the initial outlay, but no doubt the State investigation will find out.
Even without a formal risk analysis, the success rate of venture-funded start ups is only about 10%. This statement is made in the context of professional venture capitalists who know what they are doing.
The success rate of start ups funded by well-meaning but inexperienced State economic advisors is unknown because most states don’t jump into such risky ventures. It will be interesting to see what the State Police investigation uncovers about the background of the 38 Studios investment.
One of the most troubling aspects of the 38 Studios situation is that once again Rhode Island is getting national attention for government blunders. It only took 7.5 minutes to predict the risks discussed in this blog. It is not difficult to do risk analysis for software.
Normal venture investments are preceded by a careful due diligence process that examines risks and benefits. Apparently Rhode Island ignored due diligence and risks and was blinded by potential benefits.