For context, since a lot of people on HN haven't worked on games - this is not intended to compete with Git for general software development. This is a competitor with Perforce for game development.
Git is fine for text based files like code, but it's really bad at stuff like textures, 3D models, audio files, and other non-text files that game developers need to collaborate on. For example, one artist might need to obtain an exclusive lock on some art assets while editing them, because there is no sane way to merge two artists' async edits.
The SOTA in this area is Perforce (https://www.perforce.com/products/helix-core), a proprietary system. From what my gamedev friends tell me, when Perforce works it's great, but it hits enough snags that you need a tools engineer to manage it and occasionally fix issues manually. Git LFS is an alternative, but my gamedev friends all prefer Perforce especially when working on team projects beyond like 3-4 people.
One thing I don't like about Git LFS is that there is no way to delete very old history. It's the 'git spirit' to not allow deleting history, but in the context of LFS it sounds horrible. Especially if you use Github.
If there is an asset that is updated very frequently in the early stage of development, you'll be charged for all the storage for the rest of the repo's life. That happens a lot in gamedev: most assets go back and forth early on but once it's done no one will touch them ever.
Its important to understand that in Game Dev a ’git clone’, aka ’p4 sync’, can be a terabyte of stuff.
Git is bad at such volumes of binary assets, textures, models, sounds, etc.
Git LFS is a major PITA, and if you use GitHub is even worse since there are quotas and rate limits that are charged separately.
P4 is more "industry standard" than "state of the art"... But it does handle large files and partial checkouts without feeling bolted on.
I am building a small asset heavy game. Ran into a similar problem. Built a storage cost efficient tool for exactly this [1].
I wonder how useful this could be as a generalized version control for regular user systems, as a way to rollback, or scrub through history. Presumably if this is designed to work at Epic and Big Game studio scale, it should work at home computer scale
There's also the virtual streams (branches). A mapped overlay over the actual contents of the repo, which you can check out and get a subset of the contents. Or even can provide different files that have been mapped in the server for each particular virtual branch.
This is used so e.g. an artist gets a repo that contains sources for the art assets, while a programmer gets the same repo but instead of art sources, it downloads the already produced binaries. As a SE, you just want to build the code, and don't care about 800 GB of art asset sources.
P4 is also really well integrated into IDEs and UE Editor so that I don't need to think about it as much as I need, compared to Git. Locking assets, releasing them, merging into streams etc., is overall pretty streamlined. When it works, it's great, but when it doesn't work though, it's pretty hard to diagnose issues.
There's also a new player called diversion (diversion.dev) which I think may be a YC startup? Anyway it takes a different approach of being more like Google drive but bringing in VCS behavior making it more indie and designer friendly.
> Git is fine for text based files like code, but it's really bad at stuff like textures, 3D models, audio files, and other non-text files that game developers need to collaborate on.
How well does Mercurial work in this situation (or even Subversion, given that Perforce is non-distributed like SVN)?
Has nothing to do with Perforce being the Oracle of VCS because it’s baked into the big 3? Riiiight.
Can confirm, we have a team dedicated to the care and feeding of p4.
A significant part of my job, unfortunately, is helping people fix their workspaces when Perforce (p4) goes bad, or creating guardrails and wrappers to stop Perforce doing bad things.
In fairness, p4 predates most of the VCSes we consider "modern", so I empathize with a lot of the underlying architecture decisions. However, it has and continues to utterly fail at improving at a reasonable pace.
For example:
- p4 tracks file metadata of client workspaces on the server (sync'ed locally, opened for edit, file revision, etc) and uses this as the basis to avoid doing unneeded work. If this becomes desync'ed, a reconcile or force sync must be used. A reconcile can take hours, potentially days; it tries do detect file moves by default, so likely at least O(c^n) for some c>1. I have never personally seen a default reconcile operation _complete_ over any modestly large game code base, and in practice, people accumulate a litany of workarounds and scripts to fix this for themselves.
- Scripting p4 is a nightmare. Documentation is poor, schemas do not exist, and all the language-specific libraries are just thin wrappers over its C++ API.
- By default, p4 "helps" you with text files by "correcting" line endings on sync or even converting between encodings. This works until you have a mixed-OS environment, and discover a part of the pipechain that _must_ have a certain style. There are various levers to pull to make this better, but I've yet to find something fool proof.
- By default, p4 keeps flies read-only, only unlocking them when explicitly marked as being edited. This means, to avoid having to do this manually, every tool you use needs to be p4-aware. Or, you can turn this off, and choose to contend reconcile instead. (See above)
- Branching a modest game project, with, say, Unreal source code, can take hours. And this is the quick version where you ask the server to simply create new metadata, with no file transfer to a client.
- p4 is licensed by the user-account. Every user entity in p4 not intended exclusively for performing backups and maintenance operations counts toward this, including users required to integrate with other services. Plus, often times, these integration users must have admin access to be useful. The security posture is horrific.> Git is fine for text based files like code, but it's really bad at stuff like textures, 3D models, audio files, and other non-text files
Git-annex ?
Git LFS causes nothing but pain, suffering, and regret.
Git is certainly not great with binary assets, but calling perforce SOTA ... ouch.
If perforce is the best there is out there for large binary asset management, then there is a blue ocean worth of potential improvement for git.
Perforce is a piece of crap, a relic of the 20th century that must die in a fiery inferno.
Seems to me that this would be great in general software development for just checking in your dependencies as binaries. No more build-time supply chain attacks with vetted binaries.
> this is not intended to compete with Git for general software development. This is a competitor with Perforce for game development.
Well, it is intended to compete with Git for version control. It's just that Git happens to be so bad at some aspects of version control that it isn't used much in those cases.
There's no good reason that Git couldn't be good at versioning binary files, or splitting up large projects. I mean people have tried - there's LFS and submodules. It's just that those both suck balls.
I was kind of hoping Jujitsu or Pijul would take a stab at these major Git deficiencies but unfortunately it seems like they are content to do them as badly as Git does.
That sucks, git is so absolutely horrible. It's crazy to me that nobody has made anything better yet. Although I could start that myself and yet have not.
Jonathan Blow found it convenient to represent all assets in a large number of text files, to enable merging. For instance he'd have one text file per entity on a map. The game and editor could read either this or the compiled binary version.
Git LFS has file locking, and no VCS can provide you with the tools for diffing binary assets. I don't see any meaningful difference between Perforce, Diversion or Lore and git + LFS + file locking. Unless there's a meaningful performance impact for large projects (I only work on small / medium projects), the capabilities are the same. However, I get excellent git support for code in any editor, as opposed to Diversion or Lore which have none.
Something else that git isn't good at: permissions. In gamedev, you might have proprietary work that you want to restrict to certain users. In P4, you can add restrictions to certain directories for only those who have signed the required NDAs. That's not something that you can do in git: it's all or nothing. Maybe you can set something up with submodules, but that's going to upend your repository if you hadn't planned for it.