Subversion to Perforce: Post-Mortem
We Moved to Perforce!
I just migrated the team over from Subversion (svn) to Perforce (p4) at work. We’ve been dying for this change for a while and have been holding off until the timing was right. This is the third p4 setup I’ve done. The first was at Gas Powered Games where we switched from SourceSafe (and had a party to shred the VSS discs when it was done), and the second was at Oberon where we started fresh using p4 and were generally happy with the setup from the start to the end.
In preparation for the Loose Cannon migration, I was using svn2p4.pl from the Perforce Public Depot to keep the server up to date with our Subversion server in parallel, for about a week. The script isn’t perfect, so if you use it, watch out. It doesn’t deal with p4-illegal filenames well. For example we have a certain console manufacturer’s SDK checked in that has filenames like @file@ in it, and those caused errors to the script, making me skip some revisions. P4 also can’t do filename case changes on a Windows server without going through an intermediate step first (so filename.txt -> filename.txt2 -> FileName.TXT) and the script didn’t have a handler for this. We had a few of these in our revision history that had to be skipped as well.
The initial import took a day solid, with me making adjustments to start it up again when it broke, and then just a few minutes here and there to keep it in sync while we counted down to the big migration day. That day (which was Thursday) went pretty well. We decided to take advantage of the chaos to reorg the whole source tree, which I’ll talk about in a later posting on Perforce “generated” clients. Doing the final import, shutting down svn, reorganizing the tree, updating all the build scripts, etc., took about 6 hours. Switching the team over the next morning took a couple hours, and then a couple more to fix up all the little issues that popped up.
In this post I want to do a little post-mortem of our Subversion experience. Overall it was a 4 on a 10 scale. Not fantastically awful, but not great either. I got to know svn pretty well while we were on it, and I still like it a little – I’m continuing to use it at home for some side projects.
Subversion Considered Slightly Harmful
Svn is pretty easy to set up. As long as you use a good client it’s great for bootstrapping a company that has a tight budget. Perforce is insanely expensive and that’s hard to swallow on Day 1. That money does have a lot of buying power – they have some of the best support people in the world (Audiokinetic is the only other company I’ve worked with that has support people of that caliber) but I wish they had a middle ground where you get the server and client licenses but without the support contract, say for half price.
I know so many smaller developers that have wanted to use p4 pretty badly but have been forced to go with something else due to the price. I’d go with svn again to get a team started if I needed to, but not keep it a second longer than necessary. It falls down fairly quickly as you scale up in project size and people, start checking in 50 meg Maya files, etc.
I’ll go over the top three key problems we ran into, in order of importance. Svn and its tools are missing a lot of important features that p4 has, but we could live without them, find workarounds, or create our own tools to get us 80% of the way there. The problems listed below are the simply intolerable ones.
Performance
Svn is slow, slow, slow, due to the client-managed state. We often found our hard drives locking up solid with activity because svn was scanning and running checksums on every single file in the tree. It can’t detect if a file has changed otherwise. This is where p4 gets its speed – the server manages the state of every client. Modern svn clients like SmartSVN do things like hooking notifies from the system when folders are modified, so after the initial scan is done they can very quickly show changes without needing to do a full tree rescan. But restart the app, reboot, etc., and you get to run that insane scan all over again. This drove us nuts, especially when trying to ship a milestone.
The downside is of course that p4 clients have to promise not to make changes without telling the server. This makes working offline a little more of a pain because you have to mark files read/write and then either remember to check out/check in those files on reconnecting to the LAN, or to run one of the “check consistency” tools in P4Win P4V. The consistency check tools are essentially the same as what svn does.
I’ll take the p4 way any day over svn, of course. Just buy a fat-ass server, throw 8 gigs of memory in it, and let it keep all the metadata in its system cache. Full depot syncs on p4 are instant if nothing has changed.
Just to throw it out there, SourceSafe does what svn does on both the client and the server, because there really is no server. All your clients are doing checksums on all their local files, on top of scanning the server’s files to run checksums also. So now your network is saturated with unnecessary file transfers too. At GPG, when trying to ship Dungeon Siege, this was so bad for us that most of our network was unusable in the first few hours of the morning when people would come in and start their daily sync. Switching to p4 totally eliminated that problem, and we could sync the full depot as often as we liked. Then we got Incredibuild, which also saturated the network 100%, but that’s another story.
Lack of Branching
There’s a svnmerge.py that does per-changelist integrations. Supposedly the next release of svn will make this pattern officially supported instead of an add-on, but it’s still not very good. P4 is well-known for its integration support, so it doesn’t surprise me that svn falls down here, as most rcs’s probably do. But I am disappointed they are not shooting higher than svnmerge.py for the future.
I tried really hard to get private branches going with svn and svnmerge.py. There are just too many steps required to set up and maintain a branch, and a lot of stability problems. I almost spent more time maintaining the damn branch than I did working in it. Bouncing changes back and forth across them is poorly supported, and only on a per-revision level. P4 supports per-file integrations which are absolutely key for private branches to work. Integrations must also be quick, easy, and safe, and I got none of that from using svn.
By the way, if you don’t know what I mean by private branches, I’m referring to a special branch made for an individual or very small group, either for full-time work for created on-demand for a specific task. Some companies like Valve use full-time private branches for their engineers, where people always work in their branch and then integrate back and forth to keep in sync with the mainline. Personally I don’t like this idea, and prefer to use private branches only for individual tasks. Even though p4 is fast with integrations, it saves a lot of time to always work on the mainline and only run off to a private branch for big changes. If engineers breaking content development from frequent engine changes is a problem, then I’d recommend splitting the mainline into a “live” engine separate from the codebase and dependent tools. I’ll talk about this in a future post on our p4 setup.
Lack of Stability
This was the source of a lot of loud cursing on our team and throwing things. If svn is slow, you just work on something else while it’s screwing around. But if it fails and causes local state corruption, you have serious problems. One engineer on our team would have to use the svn cleanup every time certain files were updated during a sync, probably due to the whole thing getting wedged if a file was in use during the sync. Another spent hours trying to resolve svn errors that made no sense until he just nuked the whole folder and synced it fresh. We had a number of stories like this. I ran into a lot of these issues myself, especially when attempting to use private branches.
My guess is that the .svn folders are the problem. All that state, distributed over the local client’s project tree. Lots can go wrong there. It’s old school file-system-as-database tricks that nobody does any more. The Microsoft world learned its lesson on file system-based databases with the cthulu known as SourceSafe. I’m shocked that something as new as svn would continue to use stone-age tricks like this instead of a real database. Ever copy-pasted a folder managed by svn on a computer by accident? It copies all those .svn subfolders along with it, which then totally confuses the tools. Awful.
Svn also has big problems using multiple tools at once. If you like Tortoise’s integration with Explorer but you also want SmartSVN for its speed and usability, forget it. They will fight with each other and corrupt the state. And if you have any tools that automatically do things with svn then you have to freeze until it’s done. Even an auto-update on SmartSVN will potentially wedge your tree and require a cleanup. For example, our build script sets the game’s build number by asking svn for the revision number the local client is sync’d to.
Subversion Considered Not Too Bad
Like I said above, I still use svn at home because I do actually like a few things about it. Here are the top couple on my mind as we move to p4 at Loose Cannon.
SmartSVN
I know that a quick search of the web comes up with tons of praise for TortoiseSVN, but I strongly disagree, especially after using SmartSVN. Tortoise is lousy from a usability standpoint, taking far too many actions to get even basic things done. I am far faster and more accurate using SmartSVN. It does a lot of things right, such as:
- Adding a p4-style pending changelist feature to svn, which was sorely lacking. It’s client-only so you can’t see anyone else’s changelists, but it’s 80% of the way there, which is good enough.
- Automatically syncing your state versus the server upon committing a folder. It scans for adds, deletes, and changes, and can make intelligent guesses about when you’ve renamed a file. Very smart. One ‘commit’ is all you need to do. This is even better than p4, and I love it. I never forget to add/remove files this way because SmartSVN always notices.
- Maintaining a constantly updated list of differences from the depot. It does this by scanning your whole tree and then getting notified by the system of changes. So you alt-tab from your editor or export tool or whatever to SmartSVN and a second later its view is updated with all the files on the depot that have changed (that you’d be fetching if you did a sync) and the changes you’ve made locally. It shows new/deleted files, changed files locally or on the server, etc. It has a column for local state and another for remote state so you can see clearly where the differences are. There are filter buttons to show/hide remote changes, go recursive from the current folder, etc. Awesome.
- This is, ironically, exactly what I was complaining about above that costs so much performance on startup with the insane hard drive scanning. It’s not worth the cost, especially as we scale up the project tree, but this is the #1 thing that I will really miss now that we’re on p4. Maybe I have Stockholm syndrome.
- I’m tempted to almost write my own tool as a supplement to p4win/v to do this, one that works quietly in the background instead of burning the hard drive up constantly.
- Automatically requiring exclusive locks on certain file types. This is client-side, again, but a good emulation of what p4 does. Certain files, particularly binary files, should be exclusive-lock. SmartSVN can use wildcards to specify those types, and it marks them read-only until you grab an exclusive lock on them. This is basically the p4 workflow mapped onto svn.
- Normally svn lets you change whatever you want and deal with merge issues at checkin time. This can be really awful when making complex changes to binary file types (such as a Flash file) and then discovering on checkin that someone else has also modified it and having to lose all your changes. Enforcing a required-lock is perfect to work around this.
In short, if you are using svn, you should be using SmartSVN as your client. It’s free! And if you want some of the advanced features like the ones I wrote about above, then it’s very cheap to buy, and absolutely worth it. SmartSVN is written in Java which I’d normally hate, but it’s a really decent native-feeling Windows app. And the Java makes it cross-platform. I’m sure the experience on Mac and Linux is good as well.
Properties
Svn lets you attach arbitrary name/value pairs to folders, which get checked into the server and sync’d on all machines just like files do. Very nice, can be used by tools to store simple state. In fact, svnmerge.py uses properties to keep track of what revisions have been merged and skipped. The alternative is to have specially named files that get checked in. Using properties is a lot cleaner. The ‘p4 counters’ command is incredibly weak in comparison to this.
Properties are also used in ways I don’t really like. For example, if you want to set the file type on your source files to enable keyword substitution. P4 does this through a server-wide typemap that applies default file types using patterns. Svn distributes these properties through its entire tree, and they are not recursive. I definitely prefer server-wide specifications, though this is decidedly less democratic (only superusers can typically modify the typemap).
One very useful property type in svn is “ignore”. This basically tells the client to pretend that files and folders matching that pattern do not exist. Very nice for masking out all those local-only folders that get generated like obj, .cache, or .mayaswatches. This is on top of the client-side “global ignores” that can mask out things like Visual Studio’s .suo, .ncb, .user. P4 has something similar – you can mask out folders using your client spec, or the permissions spec, but that only hides the files from view on P4Win. On P4V they always show (I currently have an outstanding feature request to have this fixed up).
In Summary
I would strongly recommend avoiding svn for game development on teams larger than perhaps 15 people, or with client trees bigger than a few gigs. It’s “free” but there is a significant cost to this free-ness that increases as months go by and the team grows and the database gets larger.
We’re already super happy with the move. Things are so fast now! Plus I can start writing tools that depend on some of p4’s more advanced features. More on this later. 
Hi Scott,
Firstly, nice entry – I enjoy your candid approach to the issues surrounding SVN. I have a quick question in relation to the broad subject of asset management, and that is how does Alienbrain stack up against Perforce/SVN? I have heard you can manage code as well as your typical digital assets – Perhaps you could share some of your knowledge of this area?
Cheers,
Mark
Mark
5 Mar 08 at 4:37 am
I haven’t personally used Alienbrain, so I really can’t comment except about what I’ve heard. It’s been a few years since I knew people using Alienbrain, too, but when I asked about it I either got (engineers) “I hate it” or (artists) “it’s pretty cool…”. I can’t remember why the engineers I knew didn’t like it.
So! Not much of use here from me, ha ha!
Scott
5 Mar 08 at 9:57 am
Thanks so much for this! It contained exactly the info I was looking for: namely which SVN client to use to emulate the bits of Perforce that I can’t live without.
I have a lot of experience with Perforce (and some with VSS and Alienbrain *shudder*) from large projects at Turbine and SOE, but we recently started using SVN for our tiny startup. (We use the two-license Perforce for free at home, but when we added a third person to our team we just couldn’t justify the expense.) My biggest problem has been finding a client that works with my established p4 workflow. My husband is happy with Tortoise, but then he didn’t much mind VSS either.
Regarding Alienbrain: As far as I can tell, the terminology and feel of Alienbrain works well for artists, which is why they like it. But the actual functionality falls far below what a large game team needs. For example, branching with Alienbrain … not so much.
Sandra Powers
15 May 08 at 11:20 am
I’d tried SmartSVN briefly a long time ago and not loved it, and then (as you noted) because of the raving fanboism on the net had switched to Tortoise. Tortoise is awful (and on windows is/has been buggy in some of its most basic functionality
), so I’m really glad you’ve highlighted Smart here – I’m trying it out now. Thanks!
Incidentally, as you mention them too, Properties are IMHO one of the most tragically under-explored features of SCM. A couple of years ago, I realised we could turn any properties-based SCM into a complete workflow system with ease: essentially taking some of the work out of the build system and putting it into the SCM as metadata. Where your SCM and build system specifically exist to work together, and you need to put metadata on files, attaching metadata *inside* the SCM is simply the right approach.
adam
4 Jun 08 at 1:34 am
Thanks for this article. I’ve used P4 for years and years, and CVS (yuck) and VSS before that. While I agree that P4 does handle startup for the client much more quickly, and I really like its changelist-oriented workflow, I dislike it for many more reasons. I’ve been using SVN in my startup for the past two years and, while I’ve also seen the occasional corruption on a specific client (and had to nuke the folder and re-sync), and I’ve also seen the stupidity of SVN clients when updating files that are currently locked as they fail and require a “cleanup pass” to reset its state, those are relatively minor inconveniences. The locking of files was a real problem because we checked in our tools and Maya plugins straight into the depot, and artists used them straight from that location. Convenient. So, we trained them to close Maya before updating that folder, or gave them a batch file to copy the folder to the Maya install and make them use plugins from there only. Either worked.
I agree with SmartSVN being the best client. I would never use SVN without it. I tried Tortoise and command line and even looked for other clients, but since I’d used SmartCVS before, SmartSVN was the best choice for me. It does do most of the things I enjoyed from P4, even though the way it handles changelists is a little weird… they show up in the folder view with the depot, rather than as a separate window that you can dock elsewhere (or hide).
My biggest gripe about SVN is the startup time. And the odd thing is, one dev machine will have a two second startup time while it checks the timestamps and does NOT do an extensive scan. So it’s fine. But on another dev machine, it will do a complete scan of the tree. No explanation why. If that gets fixed (and it can be fixed), I’ll be completely happy.
The thing I dislike about P4 is the client sucks. It’s a jumble of a lot of different concepts, many of which are extremely fiddly… like Jobs. Never used them, have no idea how they would improve my workflow. I’m sure they’re like sliced bread, but it’s so poorly set up that they look like changelists. And the client spec is horrible. Who the hell wants to edit a bunch of text files and randomly map things all over the place? Sure, it is powerful, but they’ve been paid bucketloads of cash, why haven’t they put any effort into a better solution? It always happens that someone forgets to add files, which never happens with SmartSVN. It always happens that when integrating a branch, you get the order backward the first time and completely hobart the integration–which wastes some significant time. And to top it off, P4Win is a nicer client than P4V, but P4Win doesn’t work through VPN or on a multi-IP machine, because you can’t specify which port it should route through. P4V is just crap. Also, P4Win will crash if you attempt to add 20,000 files through the “Add Files…” window. You *have* to resort to the command line for some things. Ouch.
SVN definitely has some weaknesses that I hope they fix, but with SmartSVN, I have had a better experience than with P4. So far.
Maybe when our team grows to the point that yours has, I’ll re-evaluate my decision, but for now, we’re completely happy with SVN.
Again, thanks for the article.
JH
Jason Hughes
19 Dec 08 at 11:39 am
Thanks for your comments, Jason. And I am in total agreement about P4V and P4Win. And the stupid jobs. And the client specs. I plan to write a post about just how angry and disappointed I am in that company. Years ago I was singing their praises but that’s all changed. If only the SmartSVN people would make a P4 client.
Perforce Inc. has been moving at a snail’s pace for years. Considering how much money everyone is pouring into their company with their ridiculously overpriced licenses, they should be way ahead of where they are now.
There’s still no real rename/move. And the fake one they have has awful UI. The tools to manage branches are so painful to use that nobody on our team bothers. Client specs still require an unbelievable amount of nerdery to manage, as you mentioned. In fact we have a tool to manage everyone’s client specs. I’ve had to home-brew all kinds of tools to fill in the enormous gaps in their toolset. Grr, just thinking about this makes me upset.
We tried hard to make SVN work for us but it just fell down too many times. Hard. Serious daily problems. And alternatives like git and mercurial have far worse UI’s than P4Win/V. They’re definitely meant for grognard engineers.
I feel trapped.
Anyway, I’ll be writing about this in the future. I’ve worked with Perforce for 8 years and I’m starting to think they’re the worst RCS out there (except for all the others).
Scott
22 Dec 08 at 7:03 pm
Seeing the complaints about client management, I figured I’d help by posting my solution (it’s the one perforce is peddling these day as well, which is why they changed the terminology to “workspaces”):
Don’t try to keep one “client” per machine; use lots of “workspaces”. Use
the P4CONFIG file to set the client name at the top of the workspace. For simple projects, the map is one line If you need to drag in code from several projects, you can – and that works a lot better than trying to do it in subversion.
And of course, I’ll mention the one thing I love about Perforce that all the other tools I’ve checked on (git, hg, svn, archie, cvs, others as well) don’t do: keeping all the state information serverside means my working code doesn’t have little turds from the SCM scattered through it that screw up trying to use Unix directory tools (find, grep -R, du, …) on the source tree! Sometimes you can work around the problem, but it’s almost always “issue the obvious command, get a bunch of .svn cruft, curse, reissue the workaround command”, wasting time.
MIke
14 Apr 09 at 10:52 am
Not that I’m impressed a lot, but this is more than I expected when I found a link on Digg telling that the info is quite decent. Thanks.
Vince Delmonte
14 Apr 09 at 12:59 pm
I don’t like the idea of using multiple workspaces per project. Many of our guys can barely get a single workspace working right. You’ve got to have a single //depot and a single button to push to sync it. Multiple workspaces loses the all-important atomic sync. All of our processes assume that people have a tree whose state is consistent with a single point in time.
I have a different solution to the client workspace management problem that I keep putting off posting, but I need to get to soon. It’s a tool that generates and manages client workspaces based on user class and traits (artist, tester, engineer, audio guy, remote VPN worker). It allows central management of the primary client that everybody uses and guarantees consistency across users. Will post about this soon.
Scott
16 Apr 09 at 7:33 am
Good blog entry.
Re: Jason Hughes: I sympathize with the user complaining about “svn” update times… brings back bad memories of emails with developers refusing to acknowledge the issue at all… the issue we had was with a client app though not “svn” per se. Not only was it taking a long time to scan it was scanning folders that theoretically weren’t in the scope of the workspace. That is one thing to remember… if you make your workspace the root of the drive then theoreticaly everything on the drive is potentially “files to add” and hence all changes on the drive “justify” monitoring. Always make your workspace in a sub-folder.
Re: alternatives… if you have a choice, you might want to look at accurev. There’s a lot to be admired there. Workflow and restrictions may not be for everyone (but they’ve tried to be more flexible since first starting) but there are some real nice features that provide ROI (like “transparent” branching of your workspace).
Peter
1 May 09 at 2:27 pm
I looked at and discarded accurev a while ago, but can’t remember why. I’ll give it another look.
One SCM that I need to check out soon is Plastic. It looks all glitzy, which concerns me. Seriously dudes – you’re using 3D in your interface?? But they have an apparent hard core focus on the end user client, on real workflow scenarios. As opposed to just deconstructing the server into features and mapping a GUI feature onto each. That’s the unixy way that the Perforce people do things.
Anyway, Plastic. Looking very interesting. And they are obviously targeting Perforce. They’re a little cheaper, and they are working hard on getting performance in the same ballpark. I’m ready to see some real competition.
Scott
1 May 09 at 3:24 pm
Hey Scott. Your post is very nice. Iam also planning to migrate from subversion to P4. You have mentioned that there are some errors in the script(svn2p4.pl). If you don’t mine can you please give the detailed information on migrating from SVN to P4. I wil be glad if you can send me any document on how to migrate to my email.
deep
10 Dec 09 at 10:21 am
Sorry but it’s been too long since I did the migration to remember specifics about problems in the perl script. Fixing was pretty easy though. Just make sure that you are prepared to ditch your depot and start over a few times until you can get a clean import done. It may take a few days to do a migration, fixes included, depending on how big your svn database is.
Good luck!
Scott
10 Dec 09 at 10:26 am
Thanks for the reply Scott. We are planning to migrate it in the next month. Did you documented any of the issues that you came accross? If you have any such kind of documentation, is it possible to share with me?
deep
10 Dec 09 at 10:38 am
Well, all of the issues that I recall are in the second paragraph of this post. I have no other documentation than that, sorry.
Why not do a test migration right now to a local depot and see what happens? You can do them whenever you want. They don’t modify the svn state that I recall (perhaps setting a single metadata property or something but nothing significant). Who knows, it might just work. And if not, then you’ll know what you’re in for.
If I remember right, the time-consuming part of the migration wasn’t messing with the perl script. It was getting 80% of the way through and having it barf, then messing with the script, and starting over, then getting 85%, repeat.. If you did a test migration to get all of those out of the way with a solid perl script, then when you do the real migration it ought to just work the first time.
Saludos,
Scott
Scott
11 Dec 09 at 8:38 am
A single user here–so cost is not a constraint. Most of them seem to be free for a single user.
Any comments on Perforce vs PureCM vs Vault vs Plastic?
My main uses are coding (C, Matlab), Office and LaTeX.
Best
jake
12 Oct 10 at 11:00 am
Vault was built as a replacement for SourceSafe. Last I looked at it I wasn’t impressed. I haven’t heard of PureCM, but I have been following Plastic.
Plastic is doing some interesting things and trying to advance the state of the art on the client side for SCM. I like how they’re supporting fast branching and the git-style distributed model. I also like how they’ve put a lot of effort into their GUI application, which really sets them apart from everybody else.
They are directly targeting Perforce customers with their pricing, feature set, and performance. I don’t know how they would handle the kinds of binary loads a big game development project would involve, of course.
As a single user, though, I’d check out Plastic for sure.
Scott
12 Oct 10 at 1:56 pm
Thanks Scott. Recommend you take a look at PureCM http://www.purecm.com/.
PureCM, Perforce, and Vault are free for single-users. Not sure about Plastic–have a query out. Will update when I hear from them.
The screenshots for Plastic sure look good.
jake
12 Oct 10 at 5:13 pm
Some guys not sure about jobs. They represent a unit of work and can be used to group related changelists. They also provide a natural link between bug trackers and work done on that issue. Typically you configure the bug tracker to create a Perforce job and then associate 1 or more changelists with the job to implement the necessary changes.
There is no need to use jobs if you don’t want/need to.
You think Perforce is expensive: try Clearcase, fat and bloated and without changelists as a feature out of the box is unfit for purpose. We chucked out Clearcase for Perforce and the licences for 350 users were less than a years Clearcase maintenance. It is free for use on freeware projects or 2 users otherwise.
ref: my my comment about changelists above: As a CM engineer I’m need to know all the changes that implement a fix, I’m not interested in individual file changes, and the submission is transactional – all the changes go in or none do in case of failure. In one team in a certain large database company using Clearcase, a guy doing a large checkin would place a soft toy on top of his console to warn the other members of the team that they shouldn’t try building until his checkin was complete because checkin is non-transactional.
The ability to try any command in Perforce with the -n option – say what would be done but don’t do it, has been a life saver on many occasions and also a useful way of checking status in various situations.
Perforce now has a command for rename – move.
I like Perforce, it’s not perfect, but it is rock solid, well engineered (server is still only a couple of meg in size).
Giles
10 Nov 11 at 4:00 am