Why always-on DRM is bad, and companies using it should feel bad

Always-on DRM means that a video game requires a constant Internet connection to be able to play. The reason companies use it is that it is currently the best means of preventing piracy. By placing a large amount of the game's logic on the servers, they can insure that pirates cannot simply crack the game overnight, anyone that wanted to crack the game would have to write their own version of the server's logic and build it into the client, and on top of that they would have to strip the existing client of all of the calls to the server for that logic.

Ok, but why is this bad?

- The company can patch your single player game at any time. You just wanted to sit down and play for 15-20 minutes? Sorry, you'll need to download this patch real quick. And "real quick" completely depends on how bogged-down their patching servers are. Besides that fact, they're patching your single player game, let that sink in for a minute. What if you already liked the game exactly how it is? Too bad, you have to accept the patch to play. Anytime there are patches there is also a potential for new bugs, I've seen games get much worse after a patch.

- You have to have an Internet connection. For some people this truly isn't an issue, but others may not have a decent Internet connection, or they may have bandwidth caps. What if you want to play the game while on a trip, or in the airplane? Tough cookies.

- You cannot play when their servers are down. Even if your Internet connection is reliable, the EA servers aren't, and when they're not working properly you may not be able to play at all. You will have a game that you paid for, and you cannot play a single-player experience because of some server issues 800 miles away. Or maybe you can play, but your city that is saved on their servers becomes corrupted, and you have to roll it back. Someday the servers will be shut down, and when that day comes you'll no longer be able to play Sim City 5, you'll never get to play it again. You can go back and play Sim City 1 still, but not 5. Some people play old games and aren't comfortable with their $60 being for a temporary experience with a game.

- You cannot play if your account gets banned. EA is notorious for banning players accounts due to stuff like a forum post that they don't approve of. I understand EA potentially banning them from the forum, but they also ban the player's Origin account entirely, which blocks them from playing any games on Origin. I'm not sure how this is even legal, but I'm betting EA covered their asses in the EULA for stuff like this.

- Servers cannot handle the same computational load per player. There's a lot of logic going on at any given time in a city, and the more logic the server wants to handle the more CPU power EA's servers are going to need. EA will only have a finite amount of CPU power to give to each player, so they'll either need more servers, which will cost them money, or they'll need to scale down how much CPU power the game needs. Why is this bad for you? This is the reason why SimCity 5 does not allow large cities, the computations that the server would have to do would become exponentially larger as the cities grew, and so rather than cut down on the simulation level to allow for a larger city, they chose to cut down on the city size itself. Had the game been running on your CPU instead of theirs this likely would not have been as much of an issue.

- Finite content due to server space. Because the plots of land are server-side, they take up space on the servers. If they allowed every player to deform their own terrain, they would have unique plots of land on a per-player basis, which would take up more space and could potentially cause issues with the simulation on their end. So because the game runs on their servers, you cannot have deformable terrain. SimCity 2000, which ran on DOS 5.x before Windows 95 was even released, almost 20 years ago, will allow you to deform terrain, but SimCity 5, which runs on a computer in your home that is several hundred times more powerful, does not.

- You cannot save your city locally. Some people like to save their city, make a copy and then burn it to the ground, or watch Godzilla tear it down. Then they have a copy where they can reload the city and all is well again, and good fun was had. But you cannot save versions of your cities on the cloud in the new SimCity. Also, the servers are having all kinds of issues with losing progress on people's cities, if those people could have save copies locally this wouldn't really be an issue.

I understand why companies use DRM, but there is a point when they take it too far and it negatively affects the experience for the customer. In this case EA screwed up large parts of a great game franchise in order to save a few bucks. I'm curious how much it will even save them in the long run as they have to maintain and run servers 24/7 during the life of the game.

Here's how I would've implemented the DRM. Single-player mode would be entirely offline, so you have no required connection to play the game. Single-player cities can never communicate with cities your friends have made, if you want that you have to play online. If you play online you get the exact experience that EA created with the current game, all the connectivity with your friends' cities, or the cities of complete strangers. This means the customer gets the best of both worlds. We're talking about a better game experience, and the publisher has denied the customer this experience because they want to make an extra buck.

My experience with SimCity 5 so far:

• Didn't purchase the game on day 1 because I knew it would be a disaster. And it was indeed a disaster, most people couldn't play.
• Purchased the game on day 2. Installed the game. After the game installed it ran an update and said the game was ready.
• I run the game's launcher and it then downloads another update. After this update completes it says to relaunch the game.
• I click the relaunch button and the launcher crashes. I run the game's launcher again, and it downloads another update. Again it says to relaunch the launcher.
• I once again click the relaunch button, and the launcher crashes again.
• I run the launcher again and the game finally says that I can play my single-player game that I have a physical copy of.
• All of the servers for North America are full, so I connect to a European server.
• I wanted to join with a friend's private region, however SimCity cannot see any of my friends from Origin, and they couldn't see me either, so I cannot play the game with my friends.
• I decided to try making my own region. When attempting this I got an error saying I could not create my own region right now. Tried this every 5 minutes for the next two hours.
• Finally it let me create a region, and now I have to pick a plot of land within the region to create a city. I chose my plot of land and now I get an error saying I cannot create a city right now, and that I should try again later. Tried this every 5 minutes for another two hours.
• I finally gave up after a total of 6 hours with the game. I spent those 6 hours entirely within the game's menus.
• I wake up the next morning and try again. This time when the game starts up it takes me straight to a tutorial mode, I cannot skip this tutorial.
• The tutorial loads up a small city, and then the city sits there doing nothing, I cannot click on any of the UI for the city and I have no prompts telling me what I should do. Hitting escape does nothing to get me out of the tutorial. I do have some UI at the top of my screen that I can use to go back to the main menu or to quit. I go back to the main menu, which then sends me back into the tutorial, which still does not work. So I quit again.
• 2 hours later I try again. I start up the game, and this time the tutorial is gone. I check my friend's list, and that is still empty. I try creating my own city again, and I'm back to not being able to create my own region.

In summary, I can't play multi-player, I can't play single player, and it's day 3 since the game was released. I'd consider this to be a failure of a game launch. If I cannot play the game on day 5 I will be demanding a refund from EA. And I'm not likely to purchase any of their games again anytime soon, I cannot support a company that puts a little bit of money above the importance of their customer. There are plenty of companies out there doing just fine without always-online DRM, so until it is problem-free it really shouldn't be used.

Simple terrain smoothing

After creating or importing terrain you may find it to be too steep or jagged. Part of this depends on how your terrain system is written, but it also depends on the resolution of the terrain and terrain you're importing.

Let's image the grid below signifies sections of our terrain, with the numbers representing elevation. We can see there are two peaks that are 20 meters tall, surrounded immediately by flat terrain that is 0 meters tall. So we basically have two very steep and pointy peaks that are 20 meters high.



To smooth the section represented here in blue, we total up all of the heights in the red squares (which gives us 20), and we divide by the number of red squares (there are 8 squares, so 20 / 8 = 2.5), and we average that result and the height of the blue square together. (20 + 2.5) / 2 = 11.25. We now have our new smoothed value, and we replace the blue square with that value.

Here we can see the results after smoothing out just the one square in the example above.

Here's what we get when we run the algorithm against every square in the grid.

Here's the algorithm:

public void SmoothTerrain(int Passes)
{
   float[,] newHeightData;

   while (Passes > 0)
   {
       Passes--;

       // Note: MapWidth and MapHeight should be equal and power-of-two values 
       newHeightData = new float[MapWidth, MapHeight];

       for (int x = 0; x < MapWidth; x++)
       {
          for (int y = 0; y < MapHeight; y++)
          {
              int adjacentSections = 0;
              float sectionsTotal = 0.0f;

              if ((x - 1) > 0) // Check to left
              {
                 sectionsTotal += HeightData[x - 1, y];
                 adjacentSections++;

                 if ((y - 1) > 0) // Check up and to the left
                 {
                    sectionsTotal += HeightData[x - 1, y - 1];
                    adjacentSections++;
                 }

                 if ((y + 1) < MapHeight) // Check down and to the left
                 {
                    sectionsTotal += HeightData[x - 1, y + 1];
                    adjacentSections++;
                 }
              }

              if ((x + 1) < MapWidth) // Check to right
              {
                 sectionsTotal += HeightData[x + 1, y];
                 adjacentSections++;

                 if ((y - 1) > 0) // Check up and to the right
                 {
                     sectionsTotal += HeightData[x + 1, y - 1];
                     adjacentSections++;
                 }

                 if ((y + 1) < MapHeight) // Check down and to the right
                 {
                     sectionsTotal += HeightData[x + 1, y + 1];
                     adjacentSections++;
                 }
              }

              if ((y - 1) > 0) // Check above
              {
                 sectionsTotal += HeightData[x, y - 1];
                 adjacentSections++;
              }

              if ((y + 1) < MapHeight) // Check below
              {
                 sectionsTotal += HeightData[x, y + 1];
                 adjacentSections++;
              }

              newHeightData[x, y] = (HeightData[x, y] + (sectionsTotal / adjacentSections)) * 0.5f;
           }
       }

      // Overwrite the HeightData info with our new smoothed info
      for (int x = 0; x < MapWidth; x++)
      {
          for (int y = 0; y < MapHeight; y++)
          {
              HeightData[x, y] = newHeightData[x, y];
          }
      }
   }
}

Here's some purposely jagged terrain, without any smoothing

Terrain after one smoothing pass

After five smoothing passes

After 50 smoothing passes. 20 passes would probably have been fine

The bounce-pad hack in LEGO Universe

The bounce-pads (also known as "bouncers") in LEGO Universe were a means of travel, when you stepped on them they sent you to a specific location in the map, usually somewhere nearby that you could not otherwise reach. Bounce-pads were one of the first gameplay elements created during the production of the game (almost 3 years before the game released), and had no real issues during the entire production run or alpha/beta testing, so we were all somewhat surprised to run into a huge bug on the very first day the game opened up to the public.

The whole company gathered in the gym around some big screen TVs so we could watch the game go live for the first time. We watched a few people get in the game and play for a short time and then we all scattered back to our desks so we could login and play the game ourselves. No sooner than had I created my first character I had a person from the live service team at my desk describing to me a serious bug. Apparently there were a large number of players that were stuck in the very first area of the game called "The Venture Explorer", a small spaceship that served as an introduction level to teach players them the basics. About two thirds of the way through the map there is a spot where you must quickbuild your first LEGO model, a bounce-pad, and afterwards you step on it and it bounces you to the next NPC to give you your next mission. It seemed that about 1 in 100 players would build the bounce-pad but could not step on it to get bounced, so they had no way to get to NPC to get the next mission. 

We had some in-game GMs talking with some of the players having this problem, and apparently they could see the bounce-pad but they couldn't step on it. The part of the code responsible for bouncing the player was the bounce-pad code that would set the player's velocity when the physics system told it that the player collided with the bounce-pad. Somewhere in that flow something was broken, and I wanted to find out what it was. We had nobody in the office that was experiencing this bug, none of our testers had seen it at any point during production either. Because we couldn't reproduce this in-house my next thought was to see what information I could get from the clients that were seeing the bug. As a gameplay programmer I didn't really know the details of what kinds of reporting and tools the live team had setup with the game to be able to get me information about this problem. As it turns out, there was almost nothing.

The client version of the game was setup to generate log messages for any errors, and there's a good chance the log file might tell me if something was amiss, like maybe the physics for the bounce-pad was failing to load, or if something was going wrong in the collision check. Sadly, the live team never got around to setting up a way for the clients to be able to send us their logs, or for a GM to be able to send a message to the client's program and have it return the logs to us. At the time they said something about possible legal reasons for us not getting information about them sent automatically to us, which seemed a bit ridiculous since the information didn't contain any account information besides the name of their in-game character, and a 32-bit account ID, which even if it got into the wrong hands is worthless. Anyway, getting any information from the client was impossible at the time.

Server log files were one thing I did have access to, unfortunately this was a client issue since not everyone was seeing it, and since the functionality to make the player bounce was entirely client-side, with only some server-side monitoring to check for cheating.

My next idea was to use some in-game tools that I had written during development, that created huge amounts of data about any object in the world. I was hoping I could use this tool to see if there might be any issues with the bounce-pad itself that the tool could find. The tool would analyze thousands of points of data on an object (at run-time) and check for any inconsistencies and report them back. This tool was not built into the version of the client that players used, so I would need to build an internal version of the client and log-in with it, additionally the server would not return the requested information unless you were using a GM account, for security reasons. It took awhile but finally I was given a temporary GM account to be able to analyze the problem, and still we were able to find nothing, mostly because the client information about the object was based on my client, which was working fine.

After about a day of sifting through logs and using tools to try and find any issues I could find nothing, and in the mean time GMs were having to sit in the game and teleport these players to the NPC so they could get their missions. The pressure was on to find a solution, but the only information I had to go on was that a small percentage of players were seeing a problem, and because this appeared to be a client-side problem and there was no system to get client logs back to me, there was nothing I could put in the game to get me any information about the problem. The reason this problem was on me was because I had written the bounce-pad system, the system that now appeared to be broken.

I enlisted the help of a couple fellow gameplay programmers to try and see what we could do to reproduce the problem in-house. We tried removing the physics asset from the computer to see how the game would react, and when starting the game the patching system would see the missing asset and simply download it again. There were code paths that could be hit if a physics file failed to load, so we put in some code to force the physics asset to fail to load, and in that case the game put in a fallback physics shape (a 1x1x1 cube), and even though that wasn't the proper shape it was still enough the player could touch it and the system would respond and bounce them, so that was a no-go. We checked to see if maybe the collision could be succeeding and somehow the bounce-pad code was failing to translate that into a bounce, but we couldn't see any point of failure, or a way to force it to fail.

So here we are with a problem we cannot produce, and a system we can't seem to make fail, and no way to get any information from the players that were seeing the problem. Leaving the bug as-is was unacceptable, as it would mean a lot of lost customers or the expense of GMs permanently stationed near the broken bounce-pad to teleport players. So the solution, was a hack.

During early development of LEGO Universe, almost all gameplay was entirely server-based. Things like attacking, picking up power-ups on the ground, and using bounce-pads were done entirely on the server and then the server would inform the client of the event. This was very secure but it resulted in laggy gameplay, which didn't work well for an action game like LEGO Universe. Along with other systems the bounce-pads were made so that the client-side object did the bouncing of the player, and simply told the server what it had done so that the server could check for any possible cheating or hacking. So, remembering that it used to work on the server years ago, and that the bounce-pads were still properly loading on the server, the solution presented itself. I put in code on the server so that if the player stood on a bounce-pad for more than half of a second and did not get a message from the client's bounce-pad that they've bounced, that it would assume the bounce-pad on the client was broken and bounce the player from the server. The result was that for those 1 in 100 players seeing the problem, that one bounce-pad on the first level would feel a little bit laggy but it would work. We also setup some server-side logging for any time the server-side bounce-pad needed to take over, and we found that we were only ever seeing the logging for that one bounce-pad in the first level of the game. For some reason that we never tracked down, it was only ever that one asset that exhibited this problem in the game, there was never another problem related to the physics for an asset not properly loading. 

We did make the assumption that the physics were likely failing somehow on those clients, because the only way we weren't able to bounce on the client was if a physics collision message was never sent to the bouncer, so I do feel some comfort in that the system I wrote may not have been the problem, it just affected my system. Even though as a team we take responsibility for the entire product, rather than saying "this is my code, that is your code", it's still feels good when you've written a system that works well, so you never want to see it break down and fail. It remains the biggest hack that I've ever made to a released product, but I don't feel bad about it, I feel like I made the best of the situation with what I was given, and in the end the players never knew the difference.