XNA 4.0 == Me pulling my hair out

XNA 4.0 has been out over a year now, but I've only just recently had the time to use it. Shawn Hargreaves from the XNA Development Team explained how a lot of things would be broken by XNA 4.0 (although these were all good things in their opinion), and boy he wasn't kidding. As a result of the changes, here's what I've gone through in my game engine trying to convert it from XNA 3.1 to XNA 4.0. Here's a link with a summary of the breaking changes.

1.) Clipping planes are no longer supported within XNA. Clipping planes were supported directly by DX9 hardware, but not DX10 hardware, so in an effort to be forward looking, support was removed from XNA 4.0. I had to put all clipping plane calculations into my shaders. This means every single mesh that could be rendered in a scene with water (pretty much everything) had to have clipping plane stuff put into the shaders. Annoyance factor (6 out of 10). Read more here.

3.) Point-sprites are no longer supported, also because they're not supported by DX10. This affects particle systems, which often used point-sprites to represent particles. Using a point sprite meant 1 vertex per particle, but without a point sprite you need at least 4 vertices to represent a square polygon (quad). This meant having to add an index buffer to the particle emitters, and changing the way particles are created and updated. It also has performance implications, non-rotating particles render a bit slower on average than they used to, but rotating particles actually perform better, so it's not a total loss.
Annoyance factor (4 out of 10). Read more here.

3.) RenderTarget2D was changed so that it derives from Texture2D and you can no longer call .GetTexture() on it, additionally the clearing of render targets was changed which forced me to shift around some code. Annoyance factor (2 out of 10). Read more here.

4.) Number of primitives that can be drawn in a single draw call has been limited to 1,045,575. As part of my visual debugging I had a way to render a small line for each vertex of the terrain heightfield, however on a map that is 1024x1024, plus a few other objects on screen, that is enough to push the vertex count past the new limit. This one actually isn't that annoying for me, other than the fact I now have to fix it, but apparently some DX10 cards don't allow more than this per draw call so it's probably best that I fixed this anyway. Annoyance factor (2 out of 10).

5.) The quality of SpriteFonts has been decreased. It wasn't really a performance problem in the past, so I'm assuming this was done for performance reasons with Windows Phone 7. Read more here for a comparison of XNA 3.1, and XNA 4.0. Additionally the linked article provides a font processor you can use if you would like to maintain the high quality sprite fonts. Annoyance factor (1 out of 10), annoying that quality needlessly drops in my engine, but at least I didn't have to do any work to get it running properly.

6.) Here's the one that gets me the most. In an effort to unify hardware requirements for games, XNA now forces all games to run in one of two graphics profiles, called 'Reach' and 'HiDef'. The problem is that 'HiDef' is only DX10 and higher, and while 'Reach' supports DX9, it had to make sure it supported almost all DX9, so all kinds of stuff you could do in DX9 with XNA 3.1 is now gone. So I now have three choices with my game engine (that I can think of).

• I can use the 'Reach' requirement only for the engine, and I lose support for many graphics techniques that have been used for 6+ years, and lose features that I've had working in the engine for years.
• I can use the 'HiDef' requirement only, and drop support for all DX9 hardware (about 40% of the GPUs out there at this time). This seems ridiculous to me knowing that everything in my engine runs fine on about 80% of DX9 GPUs.
• I can try and use both of the requirement types, which means I have to duplicate all of the game engine's assets so it can cover both sets of assets (one in each type of graphics profile). This puts a huge burden on anyone making a game using the engine that wants to also support both formats.

All of these options are pretty bad IMO. Here are the problems I'm having with the new 'Reach' format they've imposed.

• They dropped support for most texture formats that you would need to do self-shadowing (meshes casting shadows on to other meshes). I have to entirely re-write my self-shadowing implementation to not use depth shadow maps if I want to support the 'Reach' graphics profile. Read more here.
• They also dropped support for texture sizes larger than 2048x2048, which many DX9 cards support. This means that shadow quality based on standard depth shadow mapping will be cut to 1/4th of what it could be in XNA 3.1.
• They dropped support for 32-bit index buffers, which means that for large terrains I now have to render it in smaller chunks. The sweet spot I had found for most machines was 512x512 chunks for terrain, which limited how many frustum against terrain axis-aligned bounding box checks I had to do, and limited the amount of draw calls that were being made. The largest that can be done with 16-bit index buffers is 128x128 chunks. That means if my camera previous had four 512x512 chunks in view, I had four draw calls, and now I would have up to 64 128x128 chunks in view. In reality it would probably be less than 64 chunks because the smaller chunks would give more granularity to the frustum checks, but overall I would have many more frustum checks, putting more work on the CPU, and more draw calls but potentially less geometry, which depending on the situation could put more or less work on the GPU.
• They dropped support for non-power-of-two texture sizes, which means the quality of the water reflection and refractions is either going to drop, or it will be higher than it needs to be, hurting performance.

There are likely other restrictions I haven't run into yet, but I'm sure others running into problems are also bothered by this. I'll admit it has the one major upside, and that is no longer having to worry about something working on a developer's PC and not on an end-user's PC. You either meet the requirements of the graphics profile or you don't, that simple. Sadly I believe they set the requirements for the 'Reach' profile as low as they did for Window Phone 7 development, at the expense of killing features for a large percentage of DX9 users. Honestly I'm not surprised by this, it's somewhat standard for Microsoft to make a change that benefits them, and tell everyone else to deal with it. This will make Windows Phone 7 development easier, give people more reasons to upgrade to DX10 or DX11 (both Microsoft proprietary formats), and those newer GPUs give customers more reasons to upgrade to Windows 7 or the upcoming Windows 8. I do feel that Microsoft is slowly trying to pull XNA away from PC development, they're continuing to restrict the ability of developers to use XNA to create a PC-only game.

Anyway I'm going to mull over things for another day or so before I decide whether to rework a handful of features to try and get things working on the 'Reach' profile, or if I just abandon DX9 support all together, which would be painful now but would matter little within the next couple of years as DX9 becomes a distant memory. Luckily my engine is open-source and not making me any money, so I don't have to worry about angering any potential paying customers by telling them that if they use the engine that their game can't run on DX9 hardware.

Annoyance factor (9 out of 10)!

Read more here about the 'Reach' vs 'HiDef' changes in XNA.