TERA Online dev tracker
All devs
Viewing dev posts for topic: what-eme-is-doing-to-reduce-lag-in-tera
SomaHeal wrote: »
Can any staff membrer update us with informations about the constant lag problems?

Denommenator is working on an update, but he's extremely busy at the moment. It's important to him (and us), however.
Thank you everyone for reporting these lag issues to us and for helping us to investigate the ongoing problems. I posted an update to the thread with some information on this week's hotfixes.
sraz wrote: »
Pondering this really makes me think there is some sort of back end process or cross server operation that gets hung up waiting on something and just causes delays in game, which would be indeed hard to isolate since logging would not report any issues since the operation does eventually go through, however players actually see a 1-4 second delay in game which we generalize to "LAG." Obviously narrowing this down would actually require ongoing resource monitoring to match up with player reports. However since the lag rarely lasts more than a few seconds, without precise time codes it's really hard to match player reports up with actual server processes.

We have been looking at those possibilities as well. We've adjusted daily back up times and have been monitoring event logs on all the servers to see if there is a process running at particular times that could contribute to lag.

Avi wrote: »
Hi GM! Another person here who is glad the issue is being investigated. The lag has been consistent for a few patches now. I am curious as to if you guys have investigated how other games are running their servers to avoid this issue? Tera has a rather small player base and I have a feeling a lot of people would be okay with downtime if the servers could be consistent. Well, mainly just instance servers. What has changed since maybe SCHMs initial patch to now? I don't think we had as much lag back then if I can remember correctly and I am doubtful that it is the increase in player base. Lag has sadly been the main reason I haven't been able to enjoy playing the game and really look forward to it getting fixed!

We do talk to Bluehole about how TERA is hosted in other regions and members of our team have worked on large scale games from Arenanet, Microsoft, Daybreak, and other studios worldwide.

There are almost weekly changes to TERA servers and services so pinpointing an exact change that could affect lag is challenging. One of the reasons we are asking for more information and diagnostic reports from the community is to help us both identify what areas are most prone to lag and to find better ways of identifying signs of when a server may need to be taken down for emergency maintenance.
Geletron wrote: »
DYJGAM94YD wrote: »
Hi. I'm a south american player, brasilian. Sorry for my bad english. Have you considered a server in miami? Many games have servers in miami for us. I think this can be a improvement in our pings, and give us some chances for competitive pvp. No one suffer with lag spikes like us..

Oi como vai. We have looked at hosting TERA servers in different places. Wonderholm was hosted in a different data center, though we had to add some additional constraints to make it work properly.

There are some infrastructure challenges that make it difficult to host TERA in multiple locations. TERA servers like to run in the same location as the services they rely on. One side affect of that restriction is that dungeon and battleground matches may be limited so that you could only play with people from the same server. That in turn could make some of the end game content less fun.

European here, from Denmark to be specific. Have you considered hosting TERA in a different location altogether ?
Given everything you say you've already attempted, I can't help but think that it seems your datacenter is having issues with its network either internally or externally.
On average, ignoring today's worse than usual lagspikes, I tend to get about 140-150ms ping on the Fey Forest server which, while playable, isn't great compared to other US based games I've played where I might normally see about 80-120ms. Of course there's the "but you're not in America" card to pull, but from what I gather it seems that even a large part of the NA playerbase is having the same kind of ping as I am, which to me is a bit worrying to say the least.

We have considered hosting TERA in different data centers and/or different physical locations. Moving everything, however is a monumental task. It took about a year of planning and months of work to upgrade all the existing hardware while minimizing down time. The Wonderholm server taught us a lot about what it would take to build and move a single world server to a new data center and we are going to use what we learned from that experiment as we plan on how to host TERA moving forward.

Our biggest concern at the moment is finding root causes of lag spikes people have been experiencing on different servers. It makes the play experience bad no matter what your normal ping is.
DYJGAM94YD wrote: »
Hi. I'm a south american player, brasilian. Sorry for my bad english. Have you considered a server in miami? Many games have servers in miami for us. I think this can be a improvement in our pings, and give us some chances for competitive pvp. No one suffer with lag spikes like us..

Oi como vai. We have looked at hosting TERA servers in different places. Wonderholm was hosted in a different data center, though we had to add some additional constraints to make it work properly.

There are some infrastructure challenges that make it difficult to host TERA in multiple locations. TERA servers like to run in the same location as the services they rely on. One side affect of that restriction is that dungeon and battleground matches may be limited so that you could only play with people from the same server. That in turn could make some of the end game content less fun.
By any chance is the load balancing you implemented dynamically switching players between carriers? i seem to be getting some pretty bad lag spikes (like someone disconnected my modem for a second) and my ping changes as if i'm being swapped from zayo to savvis between them. This makes for a MISERABLE play experience since it swaps between them during dungeons with about a 2 second disconnect while it does it.

It is my understanding that the load balancing is not dynamically switching players between ISPs during a play session but I will need to double check with our operations team to make sure I'm not giving you bad information. Some of the lag from today was due to unusual spikes in CPU usage on different servers at different times. We don't know the cause of the spikes, but we are aware of them and we are trying to determine the cause.
Teekz wrote: »
> @LHMG4YDWYJ said:
> No other game has this issue. Its just you guys. I dont mean to sound rude but its the truth. Ive never had a server lag in BDO,Wow,Sun,Cabal and those games have 5x-100000x more players

Agreed there are honestly only so many excuses you guys can keep throwing out.

I know I'm my line of work my boss would not let this unproffesionalism fly.

Fix your [filtered]

Please try to stay constructive, guys. The goal of the thread is to outline the steps we're taking to prevent lag, not make excuses for not fixing it. Lag is a constant challenge for any online game, it takes time and effort to address.
We are aware of the problem. The operations team is either restarting AV now or will be doing so soon.
Hi everyone,

I wanted to come to the forums and provide some answers to common questions players have about lag and what can be done to reduce it.

Q: Why do you always tell us that the problem is probably caused by my ISP?
A: We consider checking your internet settings as the first step in troubleshooting, kind of like how all IT people ask you if you have tried turning it off and on again. However, although this is the first thing we ask as a troubleshooting step, and a common source of lag in TERA, we do realize not all lag-related problems are caused by ISPs and the routing to our servers.

Some lag can occur when there is disruption or delay in data transfer between your computer and our servers in our Chicago data center. Network problems range from trying to connect over a weak Wi-Fi connection to a router in Chicago is having trouble and causing data loss for players in a specific region. It can also be multiple issues at once, compounding to make for a terrible play experience so we ask that you check your network settings first as a way of eliminating even one of the potential causes of lag.

One of the tools we provide in the launcher is the EME Diagnostic tool. It runs a few utilities to get traceroute and basic hardware information. We ask players that run into lag to please run the diagnostic tool and send the resulting log file to diag@enmasse.com so we can see the results. The diagnostic file is in .txt format so if you’re curious, you can open it in Notepad to see what information is sent to us.

Q: Why aren’t you doing anything about the lag?
A: We have been trying several things to reduce lag, and we’re working to do a better job informing players what we’re doing to address the issue. In the past year we have:
• Updated the TERA server hardware (increased CPU, increased RAM, etc)
• Experimented with load balancing between two different ISPs from the TERA datacenter
• Worked with our ISPs and data center representatives to find ways to improve network performance
• Set up an experimental server to test how TERA performs in a cloud-based environment
• Set up new monitoring software to notify us when server hardware performance decreases

Bluehole is also concerned about lag reports and they are working with us to identify causes of lag and possible solutions. This is a high priority item for both Bluehole and En Masse that we actively look at nearly every day.

Q: Why do you rely on us to report lag to you? Isn’t that your job?
A: We rely on customer reports in addition to our monitoring tools to pinpoint the possible lag causes. Our operations team is notified when we see a server crash, when there is an unexpected drop in the number of concurrent players, when any server resource is over-utilized, such as CPU, RAM, Disk, network, etc., or when any game process uses too many resources or crashes. They also monitor all network connectivity into and out of the TERA data center for utilization, dropped packets, etc.

Our tools, however, do not give us metrics for the lag problems observed in the TERA client, such as rubber banding, or when it appears that a dungeon instance freezes for a brief period. We can see it happen occasionally in-game while playing, but identifying the source is more difficult than just experiencing a lag spike. That is one of the main challenges in monitoring for and troubleshooting the causes of lag.

The most useful information for us when submitting a lag report is:
• The exact time you experience a major lag spike
• What you were doing in game when you experienced a lag spike
• What character class you are playing
• What server you are playing from
• Some information on how you connect to the TERA server (what country or region in North America and wi-fi, Ethernet, and ISP).

Please submit this information along with the EME diagnostic report to the diag@enmasse.com email address. While we may not be able to immediately fix what is causing your lag, this information helps us narrow down the root causes and helps us identify potential problems.

Q: What do you do when you get a lag report?
A: We usually do the following when investigating lag:
• We try to narrow down where and when the lag spikes are occurring. TERA runs on a large network of servers (I can’t remember the exact count, but it is more than 30) so narrowing down which servers are experiencing problems is very important. People who are hanging out in Velika on TR are on a different set of servers than players from TR doing a dungeon run which is why we ask for more information on what a character is doing when the lag spikes occur.
• We look at system performance for the servers where most of the reports are coming from and check for unusual spikes in resource use. What we are looking for at this stage are correlations between server resource utilization alerts that the operations team already receives and player lag reports.
• We look at network traffic into and out of the datacenter. We are checking for unusual spikes or dips in overall network traffic across multiple ISPs, as well as looking for reports of packet loss.
• We check the databases of the affected servers to see if there is any database performance problems or bad data that could affect server performance
• We look at related platform systems that could indirectly affect server performance, such as the online store and account servers.

When a server crashes we send a notification to En Masse and Bluehole employees, we generate a crash dump from the server, and we share all the information we have with the Bluehole technical team.

Q: Why don’t we get an official response from En Masse?
A: This is an area that we can improve on. Right now, we delay talking to our players until we have more information on what caused lag or a server outage. One reason why we wait is because saying “We are investigating,” doesn’t feel like a satisfactory answer and usually only leads to more questions. Not every investigation yields a solution, and we know most players just want the issue fixed. We know that players feel we have been too silent about our efforts so far, so we will try to provide more updates to the community moving forward.

We are aware there are lag reports in the following areas in game:
1. High level dungeons
2. Spikes that affect all open world and instance players form that world
3. Average ping/jitter that are different by server, despite being hosted at the same location

Q: What can I do on my side?
A: There are some common things that can cause network performance problems that you can check:
• Make sure that you are not playing over Wi-Fi if you are having lag problems.
• Make sure that you or other people in your house are not doing bandwidth intensive things such as streaming video, downloading large files, etc.
• Try playing at different times to avoid primetime internet traffic from your ISP (usually 9 PM – 11 PM in your local time zone).

Please remember that network lag is not the same as problems with your framerate. BHS is aware of problems in this area as well, but this is not an issue related to our server infrastructure.
You can contact Tera Online dev tracker at contact@teradevtracker.com - Privacy policy - Tera Online dev tracker is not affiliated with Tera Online or En Masse entertainment.