About the XSquawkBox/VATSIM X-Plane Simulation Rate Enforcement

Introduction 

I’ve written this article to explain a bit about the simulation rate enforcement decision, how it came to be, and the goals and theory of operation of the actual mechanism as implemented in XSquawkBox and other clients that use the XSquawkBox detection algorithm. 

I’m also going to briefly touch on the alternate plans I considered, and where possible, why I didn’t follow through with them.

Disclaimer 

This article is written from my perspective as the lead developer of XSquawkBox.  It is not a statement as to official VATSIM policy or decisions.   Statements should be interpreted as I am speaking for myself as the XSquawkBox lead, or for the XSquawkBox project collectively. 

Simulation Rate Enforcement? Isn’t it a performance/frame-rate limit? 

This is a very common misconception that’s been fuelled by the language used in the XSB beta, X-Pilot and by VATSIM. This probably falls onto me as we (XSquawkBox) used the language pertaining to frame-rate first.

Let’s clear up the definitions involved first so we’re all on the same page:

  • Frame Rate: The number of X-Plane physics and graphics updates per second. The rendering rate and physics rate are interlocked, so we have a single unified figure in X-Plane. This is purely internal to X-Plane.
  • Frame Period: The reciprocal of the frame rate – where the rate measures how many times something happens in a given time interval, the period is the time interval taken for a single event – in this case, the duration a single physics & graphics update requires.
  • Wall-clock Time: The time as measured on an accurate independent device such as a watch or simulation independent computer-based clock.
  • Simulation Time: The idea of time as maintained within the simulator itself. May not match the wall-clock time.
  • Simulation Rate: the ratio of simulation time to wall-clock time. The simulation rate is 1 when the flow of time in the simulation matches the flow of time outside the simulator
  • Real-time: Where rate of change in simulation time matches the rate of change in wall-clock time – that is for every second of wall-clock time that passes, a second of simulation time passes. (i.e: a Simulation rate of 1)

At the end of the day, the factor we care about the most is the X-Plane simulation rate. When you’re on VATSIM, we expect all participating simulators to run in real-time (i.e: a simulation rate of 1x, or 1:1 depending on how you want to represent it). For X-Plane 10 and 11 to run in real-time, it has a unique and strict operational requirement that it is running at a minimum of 20 frames per second to do that. (Ed-2020-01-16 – see note about accepted exceptions below).

If you do not meet the 20 frames per second minimum, X-Plane 10 and 11 transparently reduces your simulation rate so it is producing 20 updates per second of simulation time, irrespective as to slowly it’s going in real-time.

One of the traps here is because the time period computed per frame is still quite small (its capped to 50ms), and the frame rate is consistent, when an X-Plane installation is exhibiting time dilation, motion still appears smooth and control response feels natural, but your world progress rates, roll rates and climb/descent rates, if measured against a wall-clock, are lower than they should be and are lower than indicated.  Running at 15 frames per second is enough to produce a 1/4th slow-down of speeds and rates.  This is a significant contrast to most other simulators where the slow frame-rate results in large jerky movements and typically increased difficulty of control.

X-Plane 11 does have a warning dialogue that pops up if your simulation rate has been forcibly reduced below real-time due to poor performance, but many users dismiss it without reading it. From memory, Initially X-Plane 10 offered no warning at all other than a description of the feature in the manual.

Because the behaviour existed in a way that permitted users to experience it without warning (remember, we were still on X-Plane 10, not 11), I was strongly (and informally) encouraged to include a harm minimisation feature in XSquawkBox in 2016. 

Theory of the Current Enforcement Approach 

The current enforcement approach is one of non-punitive disconnection – that is, the client ensures you meet the requirement, but if you do not meet the real-time requirement, it voluntarily disconnects you before you can cause too much disruption to the network or other users, giving you a chance to correct the issue before you reconnect. 

The theory of this approach is oriented around the following factors: 

  • Users can’t dismiss and ignore being disconnected like they can with the X-Plane 11 warning dialogue. 
  • We can make it clear to the user that something is wrong and needs their action to fix.
  • By doing things this way, you get a chance to correct your client before somebody feels the need to invoke the Code of Conduct clause B9 (relating to use of simulation rate adjustment without permission) against you.

What constitutes a failure to meet the simulation-rate requirement?

We can’t easily measure the simulation rate – X-Plane doesn’t expose it in any useful form.  There is a data-ref that is supposed to expose it, but it is explicitly disclaimed as always reading 1.0 in X-Plane 9 and newer in the official data-ref references, which renders it useless for our purposes.

Because we know what triggers the time dilation feature, we do the next best thing instead – we can extract the frame period, that is, the time between frames. We know that the  simulation rate is reduced when the frame reduces below 20 frames per second as that is the precondition for it set out explicitly by Laminar Research. 20 frames per second means we have a inter-frame period limit of 50 milliseconds – that we can measure using data available from the simulator.

As a direct consequence, the main group of users we need to apply action against are users who demonstrate a sustained operation with the frame period above 50ms, which has the effective outcome of a frame-rate below 20 frames per second

Having been the XSquawkBox maintainer/developer for 4+ years, I’m well aware of some of X-Plane’s behaviours and quirks – I know it drops frames randomly and that’s the case we need to cater for the most – if a user is fine except for the occasional drop below 20fps because of the stuttering issues X-Plane has, that’s forgivable and should not act against those cases on the stutter alone.

That said, we need to be able to tell the difference between a short-lived stutter and a persistent problem

The XSquawkBox Time Dilation Detector 

The XSB Time Dilation Detector (that is literally what we called it internally inside XSquawkBox) works by watching the frame period every frame and keeping a few tallies and applying a few allowances. 

The basic principle is that we count the slow time into a “slow-time pool”, and if the slow-time pool exceeds our maximum permitted total (presently 30 seconds), we disconnect the user as they’re demonstrating a persistent problem. To make sure this remains fair, if the frame-rate is consistently good for a certain period (presently 60 seconds), we can treat any other failures previously in the session as transient, and we reset the slow-time pool back to 0. 

It is the running down and resetting of the slow-time remaining pool that prompts the warning messages that XSquawkBox displays. You will get a warning once 1 second of the slow-time pool has been consumed to make sure you are aware that we believe there is a problem. You will get a regular warning as the pool winds down (currently at every 10s of pool expended), and then finally, when it is exhausted, the disconnect will occur and you’ll get the final notification. Similarly, if the conditions to reset of the slow-time pool occur, you will be given notice so you’re aware that everything is OK again. 

Now, to deal with some of the other glitches, we have a few accommodations to try to avoid false warnings and prevent winding down the “slow-time pool” unfairly. 

The first accommodation is we don’t start timing until a fair warm-up period has completed. I’m aware that the first few moments connected to the network is hard on the simulator – it must deal with demand loading any nearby aircraft all at once, as well as a constant stream of remote traffic updates. We simply ignore this interval and don’t start taking measurements until after a fair amount of time have passed. 

The second accommodation we have made is that we look for consecutive slow or fast frames before changing our classification of if the time is slow or fast – this coupled with the fact that we never count a fast frame against the slow-pool (even if we classified the interval as being slow), means that we only wind-down the slow-time pool on sustained periods of time-dilation.  We also require that the number of consecutive frames is enough to make it clear that it’s a problem.

Because our approach to identifying time dilation is extremely conservative, I’m reasonably confident that the XSquawkBox approach is robust and that I’ve taken reasonable measures to keep it fair. 

Monitoring the Time Dilation Detector 

Starting with the next release of XSquawkBox 2.0, there will be data-refs available that provide the slow-time pool remaining as a percentage of the total pool size, and a flag to indicate if the client thinks time dilation is happening or not. This should allow for more nuanced and useful collection of data where there are doubts about our approach to this issue, as well as providing user customised avenues for early warning. 

I recommend that others using the XSB time dilation detector adopt a similar approach and make such data available. 

Previous Remedies and Warning

There’s a huge misconception that this is a new issue, that it’s restricted to X-Plane 11, and that we provided no warning about its implementation.  

  • The issue has been around since before 2016. I say this with certainty as I made a post about this on the VATSIM forums on 1 March 2016 warning users to ensure their frame-rates were adequate and gave forward warning at that time of future monitoring features related to frame-rate.
  • The time dilation behaviour was introduced in X-Plane 10 – earlier versions used an automatic fogging method to limit view distance to ensure the simulator performed adequately, and it is my understanding that that was met with heavy consumer resistance. 

Since my awareness in 2016, I have tried to educate users where possible and provide public warnings about this behaviour. There has not been sufficient impact to prevent it from happening.

The XSB time dilation disconnect feature was added to our code-base for the aborted 1.4 release – if we had released 1.4, this likely would have happened earlier. 

As it is, XSquawkBox has had the time dilation detection feature for the entire 2.0 beta family to date – a good 2 months before the VATSIM announcement – I have discussed the feature when asked about it in forums. For the most part, the feedback I have received on this feature has been rational and reasonable up until the VATSIM announcement took place.

Alternative Remedies 

Alternative Remedies have been considered and were dismissed. I’ll outline these below. 

Doing Nothing 

This simply wasn’t acceptable – as the problem is transparent or easily ignored by users, some degree of education as to how it creates problems for other users was necessary.  Ignoring it just lets the problem get worse.

Adjusting Data sent to ATC on the fly 

A suggested remedy was to recalculate the GS and TAS based on the slow simulation rate and report that to ATC. 

This was dismissed as unreasonable – whilst the technical aspects of this remedy are entirely feasible, it introduces a problem that I viewed as being significantly worse – ATC starts working with speeds that the pilot is not aware of and not consciously aware of. 

This creates a number of new problems that’d also need solving: 

  • A pilot client user would need to know what their effective TAS/GS are, which, fundamentally, requires user education. We already know from past efforts in this issue that user education will not consistently work. 
  • If a pilot client user is not aware of the time dilation and time dilation adjusted speeds, then they may believe they are already complying with ATC orders. This usually results in user and controller aggravation. 
  • A pilot client user may not be able to comply with ATC orders in this situation as it may require flight beyond Vne/Mne to reach the ATC instructed target. Also, it may also require flight at speeds that prevents a reasonable landing. 

Because the above issues creates a need for additional, unrealistic, processes to make this outcome workable, it wasn’t worth considering further. 

Adjusting the Ground Rate (aka Auto-speed) 

I’ve commented on the auto-speed plugin before. It’s not a solution. 

Whilst it fixes the TAS/GS, you end up with muted roll and pitch rates as the simulation is not running in real time and won’t let you produce as much delta-roll or delta-pitch in the same distance as you should be able to produce. This can make the aircraft harder to control and so we just don’t support it. 

It also won’t fix the in-aircraft chronographs getting out of step, or any other thing that is simulation time-linked. 

Integrated Real-time Adjustment of X-Plane Rendering Options 

XSquawkBox will not implement this option, but that’s not to say somebody else can’t. 

XSquawkBox won’t implement it because this makes the client too dependent upon specific versions of X-Plane – all the current performance auto-tuning plugins/scripts rely on “Art Controls” – adjustments in X-Plane which are not guaranteed to remain stable.  As XSquawkBox has a slow release cycle (intentionally), we cannot include features that would require that level of continuous updating. 

As you can get this behaviour by installing 3jfps wizard or other auto-tuning plugins, we considered that to be adequate to cover this possibility.  Moreover, as you can solve the problem manually though judicious tuning, we do not mandate that users install the auto-tuning plugins. 

Ghosting of Time-Dilated Traffic 

There are fundamental problems with this in the VATSIM context: 

  • Without the ability to effectively remove you completely from the world, you’re still going to interfere with traffic when you’re slow.
  • It’s not possible to selectively switch a client from Pilot to OBS (and vice-versa) to prevent it from interacting with the world in real-time with the current VATSIM architecture.  To do this requires disconnecting and renegotiating the session.
  • If you are a pilot client, you must provide position updates and exist in the world, otherwise everybody will think you’re disconnected anyway, at which point you’re just wasting network resources. 
  • This hides the problem from the user which means they’re more likely to keep on letting it be a problem. 

The four above were more than sufficient to dismiss this approach as being a far too complicated version of what we’re doing now with the automated disconnect, with no real gains on top of it, but a whole lot more support burden for the developers.

Finally, and this is not a factor in this approach’s dismissal (the four above are enough), but just my opinion as the XSquawkBox lead, I really don’t understand the point of this approach – if you’re not actively participating in and influencing the network, why do you even need to be connected?

Open to Reasonable Feedback 

We (XSquawkBox) had a problem that needed fixing, and so we fixed it.

If you want to have a genuine dialogue about alternatives we haven’t considered in depth, I’ll generally be receptive to it. Such discussions can be initiated in the VATSIM support forum for XSquawkBox or in the X-Plane User Discussions forum.

Please keep in mind that this is not an invitation to deny the problem or try to convince me to remove the enforcement feature or that it shouldn’t apply to you in your allegedly marginal case (everybody who has tried the marginal case argument have demonstrated otherwise to me so far) – 4+ years of stewardship have confirmed that we need this feature – productive conversations about how we can make it better or fairer only please.

I am personally aware of a few things that I need to fix in XSB before the stable release with regards to this feature, but they only apply to OBS and external flight models at the time of writing (I had overlooked these until I got reminded of XView’s existence recently). Bugs are a possibility, but I respond better to good data than to words given the anticipated, and now well demonstrated, backlash from users who are still in denial about the problem.

VATSIM’s Stance 

I can’t speak for VATSIM in this matter. 

XSquawkBox implemented the feature before the announcement, and as such, we’re very much a (pardon the pun) pilot case for the feature. 

I have tried to influence VATSIM’s language around this to prevent them from hemming us, the client developers, into a specific implementation so we can try to keep the actual implementation fair and focused on the actual problem, and, as such, change it should a better approach present itself.

Update – 2020-01-16

It was questioned why we don’t use GS or time skew to detect this.

First of all, when I said we only care that the simulator is running in real-time, this is a slight simplification of the truth – we only care that the simulator is running in real-time unless explicitly commanded to run at a different rate. The reason for this is operation under time acceleration or slow-down can be performed per the VATSIM Code of Conduct B9, but a key aspect of that is that explicit permission is required. I assert that you cannot get permission for something that you cannot directly control or are actively aware of. We’re also not interested in permitting operations under time dilation with permission because it’s largely unpredictable in actual slow-down, and any such features to permit it would be abused in much the same way the X-Plane 11 FPS warning is ignored.

Specifically for ground speed – the solution is non-trivial because it requires capturing data from frame to frame, independently calculating the new outcome using the right geodesic model, and deciding if the error is enough that we’ve “skewed”, but more than that, such detection will be triggered where explicit permission has been sought to run with ground speed acceleration, or similar. More than that, whilst the original proposer suggested this could be done on long intervals rather than using a per-frame sampling method, it really can’t as we can’t directly correlate a start position to a final position without data from every frame in-between. Ultimately, this is a lot more work for not a lot of benefit, but a much larger case of potential errors in the detection.

With respect to time skew, it’s fairy normal for pilots on the network to fudge their simulator clock to get the time of day they want, even whilst connected, and such events will skew the continuity of the simulator clock, tripping any such detector (I’m fairly certain we can’t tell the difference between a user clock adjustment and the passage of time). We have do not want to accidentally catch adjustment of time whilst connected to the network when we’re just interested in time dilation events.