Guide Optimizing Paper - Remove lag spikes, Fix tps & Improve performance!

Discussion in 'Paper Discussion' started by frash23, Mar 28, 2016.

  1. frash23

    frash23 Active Member

    Joined:
    Feb 18, 2016
    Messages:
    133
    Likes Received:
    53
    Last updated 2016-04-05

    Paper has a lot of performance improvements disabled by default, as they often alter the server so it no longer replicates vanilla behavior. This guide is supposed to help you tweak the settings for optimal performance while understanding what you are actually changing. If you do not understand a setting I mention here, don't change it. If you are curious, reply to the thread (make sure to @tag me) and ask me what it does. If you are still having issues with performance, take timings (2-5 minutes) and reply here, or make a new thread.

    I'll list all the config values you can consider tuning to improve performance while doing my best to explain what the setting does and when you do/don't want to change it.

    The guide is currently written for 1.9. Most options appear in 1.8 as well. If they don't, just go to the next option.



    1. paper.yml

    optimize-explosions
    Default: false
    Suggested: true

    mob-spawner-tick-rate
    Default: 1
    Suggested: 20


    disable-chest-cat-detection

    Default: false
    Suggested: true


    use-hopper-check

    Default: false
    Suggested: true


    container-update-tick-rate

    Default: 1
    Suggested: 3 - 4


    queue-light-updates

    Default: false
    Suggested: true



    2. spigot.yml

    late-bind
    Default: false
    Suggested: true


    mob-spawn-range

    Default: 4
    Suggested: 3


    entity-activation-range

    Default: animals:32, monsters:32, misc:16
    Suggested: animals:6, monsters:16, misc:2


    hopper-transfer, hopper-check, hopper-amount

    Default: transfer:8, check:8, amount:1
    Suggested: transfer:24, check:24, amount:3


    max-entity-collisions

    Default: 8
    Suggested: 1 - 2


    merge-radius

    Default: item:2.5, exp:3.0
    Suggested: item:3.5, exp:6.0


    view-distance

    Default: 10
    Suggested: 3 - 5



    3. bukkit.yml

    spawn-limits
    Default: monsters:70, animals:15, water-animals:5, ambient:15
    Suggested: monsters:50, animals:10, water-animals:3, ambient:4


    chunk-gc

    Default: period-in-ticks:600, load-threshold:0
    Suggested: period-in-ticks:300, load-threshold:300


    ticks-per.monster-spawns

    Default: 1
    Suggested: 2 - 5



    4. server.properties

    network-compression-threshold
    Default: 256
    Suggested (standalone server): 512
    Suggested (behind local Waterfall/BungeeCord): -1



    5. taco.yml

    taco.yml is the configuration file generated by TacoSpigot
    TacoSpigot :taco: is @Techcable's fork of Paper with some additional patches that sometimes are still in earlier testing.
    Taco frequently merges changes into Paper. The settings listed here may already be in paper.yml! If one of these settings seem like a major benefit to your server, check if paper.yml has this setting already before switching to TacoSpigot.

    armor-stand.optimize-movement

    Default: false
    Suggested: true


    grass-ignores-light

    Default: false
    Suggested: true


    hopper.fire-InventoryMoveItemEvent

    Default: true
    Suggested: false


    hopper.push-based

    Default: true
    Suggested: true



    6. Java & Launch flags

    Always make sure you're on the latest version of Java! (Currently 8)
    Some features are only present in Java 8, as well as being more optimized.
    If you for some reason are stuck on Java 7, you will need to add the -XX:MaxPermSize=128M flag to your launch script.

    Some people will tell you that launch flags are placebo and you should stick to just -Xmx.
    This is for a large part true. However, the garbage collector can play a large role in performance and memory usage. The best way to tweak the garbage collector to our needs is using flags.
    @Aikar has put a lot of time and effort into researching and testing different garbage collection setups, and recommends the following flags:
    Code:
    java -Xmx8G -Xms8G -XX:+AlwaysPreTouch -XX:+DisableExplicitGC -XX:+UseG1GC -XX:+UnlockExperimentalVMOptions -XX:MaxGCPauseMillis=45 -XX:TargetSurvivorRatio=90 -XX:G1NewSizePercent=50 -XX:G1MaxNewSizePercent=80 -XX:InitiatingHeapOccupancyPercent=10 -XX:G1MixedGCLiveThresholdPercent=50 -XX:+AggressiveOpts -jar paperclip.jar
    Flags are explained in @Aikar's article here.

    Large pages can also improve memory performance a lot:
    Code:
    -XX:LargePageSizeInBytes=2M -XX:+UseLargePages -XX:+UseLargePagesInMetaspace
    However, this requires configuration of your Linux system and can potentially break it. You should only attempt setting up large pages if you understand what they are and how to set them up.
    @Aikar's recommendation on finding the right values to set:
    Note: -XX:+UseLargePagesInMetaspace is Java 8 only!



    You can also consider a plugin like ClearLagg if entities are your problem and none of the above settings are working for you. I do not recommend using the automatic entity clearing provided by ClearLagg. Disable them when you get the plugin. You're interested in features like mob-range and chunk-entity-limiter.



    And you're done! If you are still experiencing issues, feel free to take timings (2 - 5 minutes) and post them here, or create a new thread.
     
    #1 frash23, Mar 28, 2016
    Last edited: May 2, 2016
    • Like Like x 12
    • Informative Informative x 10
    • Friendly Friendly x 1
  2. Akelov

    Akelov New Member

    Joined:
    Mar 28, 2016
    Messages:
    10
    Likes Received:
    1
  3. Frozon

    Frozon Member

    Joined:
    Feb 16, 2016
    Messages:
    81
    Likes Received:
    7
    Awesome guide! Thanks @frash23 for your contribution to the community <3
     
    • Agree Agree x 1
  4. frash23

    frash23 Active Member

    Joined:
    Feb 18, 2016
    Messages:
    133
    Likes Received:
    53
    The redstone patch is now in Paper [​IMG]
    Using Aikar's improved version as well.
     
    • Like Like x 2
  5. tuanjr

    tuanjr Member

    Joined:
    Feb 19, 2016
    Messages:
    44
    Likes Received:
    2
    i want to like you 1000x :D
     
  6. Aikar

    Aikar git am FixMinecraft.patch
    Paper Core Developer

    Joined:
    Feb 2, 2016
    Messages:
    38
    Likes Received:
    26
    @frash23
    fire-physics-event-for-redstone
    Default: false
    Suggested: true


    Is wrong, suggested is false. true fires the events

    I also would not suggest LOWERING netty threads. The only thing lowering can do is to reduce waste of having idle threads sitting around, which isn't likely to happen unless you're running a single core system which you really shouldn't be.

    At 4 threads where most systems will be quad core, ensures adequate distribution.

    Netty threads are also in charge of compression, so I actually raised mine to 6 on my 6 core system to improve efficiency. I've seen cases where large chunk sending hogged the threads time, causing chat delay and such. Giving more ensures packets deliver faster, up until a point that you have wasted threads not doing anything, but at most that just wastes memory.
     
    • Useful Useful x 1
  7. ediTv2

    ediTv2 New Member

    Joined:
    Feb 18, 2016
    Messages:
    16
    Likes Received:
    0
    In your opinion instead of basing netty threads on core count, would justifying it based on threads work aswell? For example a 4 core processor with 8 threads, but utilizing let's say 6 or more threads.
     
  8. Techcable

    Techcable Benevolent Sith Dictator
    Administrator Waterfall Core Developer Paper Developer

    Joined:
    Feb 1, 2016
    Messages:
    282
    Likes Received:
    124
    Nice guide, I think its even better than the wiki,
    @frash23
    TacoSpigot is used in production on servers I manage now.
    It was rather unstable in the past, but now I'm testing things before pushing live :p
     
    #8 Techcable, Mar 30, 2016
    Last edited: Mar 30, 2016
    • Agree Agree x 1
  9. Aikar

    Aikar git am FixMinecraft.patch
    Paper Core Developer

    Joined:
    Feb 2, 2016
    Messages:
    38
    Likes Received:
    26
    It can be. Think about in the past, we had 2 thread (and really old past, 3 threads) per client. 6 Def will not hurt you, and will probably help in some ways over the default 4, but with every thread, you lose out on that efficiency perfection at low activity times -- but comparing 4 to 6... it's not going to matter.
     
    • Informative Informative x 1
  10. frash23

    frash23 Active Member

    Joined:
    Feb 18, 2016
    Messages:
    133
    Likes Received:
    53
    Updated the TacoSpigot description :taco:
    Corrected, thanks.
    Makes sense. Should I recommend a value corresponding to the amount of cores on the target system? What about hyper-threading?
     
  11. Aikar

    Aikar git am FixMinecraft.patch
    Paper Core Developer

    Joined:
    Feb 2, 2016
    Messages:
    38
    Likes Received:
    26
    Probably best to just leave that setting alone and just let people who know what they are doing change it.
     
  12. Aikar

    Aikar git am FixMinecraft.patch
    Paper Core Developer

    Joined:
    Feb 2, 2016
    Messages:
    38
    Likes Received:
    26
    After many weeks of studying the JVM, Flags, and testing various combinations, I have come up with these flags as the most ideal combination to use, backed with SCIENCE.

    I strongly suggest these flags to start your server (Which BTW: You really should be using Paper instead of Spigot: https://paper.emc.gs - Paper is a drop in replacement for Spigot and all plugins should still work - but with paper, please do not ask for support in #spigot IRC channel - ask in #paper )

    Use these flags exactly (only modify the Xmx and Xms) for max memory of 10GB and LOWER. These flags work and scale accordingly to any size of memory, even 500MB)

    These flags help keep your server running CONSISTENT without any large spikes. CPU may be slightly higher, but your server will be overall more reliable and stable TPS.

    If you are running with 10GB or less memory for MC, you should not adjust these parameters. (I use 10GB myself)

    If you for sure need more than 10GB (Hopefully you are 150+ player server) use these changes:

      • -XX:G1MaxNewSizePercent=60
      • -XX:G1NewSizePercent=35
      • -XX:InitiatingHeapOccupancyPercent=15
    Explanation of flags:
    1. -Xms matching -Xmx - Why: You should never run your server with the case that -Xmx can run the system completely out of memory. Your server should always be expected to use the entire -Xmx! You should then ensure the OS has extra memory on top of that Xmx for non MC/OS level things. Therefore, you should never run MC with -Xmx settings you can't support if java uses it all.

      Now, that means if -Xms is lower than -Xmx - YOU HAVE UNUSED MEMORY! Unused memory is wasted memory. G1 (and probably even CMS to a certain threshhold, but I'm only stating what I'm sure about) operates better with the more memory its given. G1 adaptively chooses how much memory to give to each region to optimize pause time. If you have more memory than it needs to reach an optimal pause time, G1 will simply push that extra into the old generation and it will not hurt you (This may not be the case for CMS, but is the case for G1)

      The fundamental idea of improving GC behavior is to ensure short lived objects die young and never get promoted. With the more memory G1 has, the better assurance you will get that objects are not getting prematurely promoted to the old generation.

      G1 Operates differently than previous collectors and is able to handle larger heaps more efficiently. If it does not need the memory given to it, it will not use it. The entire engine operates differently and does not suffer from too large of heaps.
    2. UnlockExperimentalVMOptions - needed for some of the others specified
    3. TargetSurvivorRatio: I'm sure your all use to seeing this one suggested. Good news! It's actually a good flag to use :D This setting controls how much of the Survivor space is ABLE to be used before promotion. If survivor gets too full, stuff starts promoting to Old Gen. The reason behind this is to be able to handle memory allocation spikes.

      However, MC allocation rate for most part is pretty steady (steadily high.....), and when its steady its safe to raise this value to avoid premature promotions.
    4. G1NewSize Percent: These are the important ones. In CMS and other Generations, tweaking the New Generation results in FIXED SIZE New Gen and usually is done through explicit size setting with -Xmn.

      With G1, things are better! You now can specify percentages of an overall desired range for the new generation.

      With these settings, we tell G1 to not use its default 5% for new gen, and instead give it 50% at least!

      Minecraft has an extremely high a memory allocation rate, ranging to at least 800 Megabytes a second on a 30 player server! And this is mostly short lived objects (BlockPosition)

      now, this means MC REALLY needs more focus on New Generation to be able to even support this allocation rate. If your new gen is too small, you will be running new gen collections 1-2+ times per second!!!

      This is bad! You will have so many pauses that TPS has risk of suffering, and Spigot might be unable to catch up TPS with the cost of GC's.

      Then combine the fact that objects will now promote faster, resulting in your Old Gen growing faster.... This is bad and needs to be avoided.

      Given more NewGen, we are able to slow down the intervals of Young Gen collections, resulting in more time for short lived objects to die young and overall more effecient GC behavior.

      if you run with larger heaps (15GB+), you may want to lower the minimum to say 30%, but don't go lower than 30%. This will let G1 have more power in its own assumptions.
    5. InitiatingHeapOccupancyPercent/G1MixedGCLiveThresholdPercent: Controls when to include Mixed GC's in the Young GC collection, keeping OldGen tidy without doing a normal OldGen GC collection.

      On larger heaps(10GB+), you can raise InitiatingHeap to around 20 to reduce CPU usage, but I wouldn't go higher than that. And you also need to REDUCE the Maximum New Percentage to around 60. If you use 80% NewGen Max, you must keep this at 10 then. It doesn't hurt to leave it at 10, but "effeciency wise" you can improve it to 20 if you reduce your new gen size, but if you start seeing Old Gen GC, lower it back.
    6. AlwaysPreTouch: AlwaysPreTouch gets the memory setup and reserved at process start ensuring it is contiguous, improving the efficiency of it more.

    Also for Large Pages - IT's even more important to use -Xms = -Xmx! Large Pages needs to have all of the memory specified for it or you could end up without the gains. That memory CAN NOT be used by the OS anyways, so let something use it!
    Additionally use these flags (Metaspace is Java8 Only, don't use it for Java7):
    Code:
     -XX:+UseLargePagesInMetaspace
     
    #12 Aikar, Mar 30, 2016
    Last edited: Jun 4, 2017
    • Informative Informative x 4
    • Like Like x 2
    • Useful Useful x 1
  13. ediTv2

    ediTv2 New Member

    Joined:
    Feb 18, 2016
    Messages:
    16
    Likes Received:
    0
    I apologize if I missed something in the post, but why only implement AlwaysPreTouch when LargePages is being used versus implementing without LP as mentioned in older threads?

    Just out of my own curiosity ^
     
  14. Z750

    Z750 Taco King
    Administrator Paper Core Developer Spigot Staff

    Joined:
    Feb 2, 2016
    Messages:
    92
    Likes Received:
    94
    Paper does forcibly disable this. The setting is completely ignored.
     
    • Informative Informative x 1
  15. Aikar

    Aikar git am FixMinecraft.patch
    Paper Core Developer

    Joined:
    Feb 2, 2016
    Messages:
    38
    Likes Received:
    26
    I thought it only applied to large pages, but now i see it also works for normal heap.

    So I guess we should do it for all setups.
     
  16. tuanjr

    tuanjr Member

    Joined:
    Feb 19, 2016
    Messages:
    44
    Likes Received:
    2
    @frash23
    -when i'm using waterfall with setting network-compression-threshold: -1 -> player can't join!
    - if use paperclip .19 with setting fire-physics-event-for-redstone it's mean dont need this plugin?: https://www.spigotmc.org/resources/redstone-clock-preventer-2.6678/ ?
    - i can't find setting ticks-per.monster-spawns in bukkit.yml
    is it?
    ticks-per:
    animal-spawns: 400
    monster-spawns: 20
    autosave: 180000
     
  17. frash23

    frash23 Active Member

    Joined:
    Feb 18, 2016
    Messages:
    133
    Likes Received:
    53
    Updated :taco:
    Your proxy's network-compression-threshold should be 256 - 1024.
    BungeeCord (and thus waterfall) does not support negative values here - set it to something ridiculously high like 66000 to effectively disable it.
    There is no good reason to disable it for your proxy though.
    Depends on what kind of redstone lag you're getting. I'd suggest trying without the plugin first, then installing the plugin if the fire-physics-event-for-redstone doesn't solve your performance issue.
    It is, yes. That's how yaml works.
     
    • Like Like x 1
  18. Schneidertm

    Schneidertm New Member

    Joined:
    Mar 14, 2016
    Messages:
    4
    Likes Received:
    0
    Very useful guide. thanks a lot!
     
  19. Martyn

    Martyn New Member

    Joined:
    Feb 11, 2016
    Messages:
    17
    Likes Received:
    1
    I think one of those -XX:G1MaxNewSizePercent should be -XX:G1NewSizePercent. Also, do you have any recommendations for bungeecord/waterfall flags, or should we not care about that (up to a certain player count)?
     
  20. frash23

    frash23 Active Member

    Joined:
    Feb 18, 2016
    Messages:
    133
    Likes Received:
    53
    Aikar mentions in his Tuning bungeecord thread that he recommends using the same flags, but with the addition of -Dio.netty.recycler.maxCapacity.default=20000.

    If you're using a smaller heap (i.e. < 2GB), using -XX:G1HeapRegionSize=4M should make sure the G1 GC doesn't use a region size too small to stay efficient. Waterfall usually doesn't need more than a gigabyte, so it's a reasonable flag to include :)

    EDIT: If you are using Waterefall, these flags are not needed!
     
    #20 frash23, Mar 31, 2016
    Last edited: Apr 3, 2016
    • Useful Useful x 1

Share This Page