Thread: The Chad Way To Do Player Update

Results 1 to 9 of 9
  1. #1 The Chad Way To Do Player Update 
    WVWVWVWVWVWVWVW

    _jordan's Avatar
    Join Date
    Nov 2012
    Posts
    3,058
    Thanks given
    126
    Thanks received
    1,874
    Rep Power
    5000
    Hello,

    Today I will teach you how you can improve your player update design to improve performance and allocations, without the use of dynamically growing buffers for writing your info. Enjoy

    1. The Usual Design
    Before we can talk about the improvements, we must first talk about the "usual" design servers implement for player updating.
    Consider the following code... (Keep in mind this is just a rough sketch of code)

    Your game loop code.
    Code:
    val players = world.players
    
    for (player in players) {
        player.sendPacket(PlayerInfoPacket())
    }
    Some sort of class that holds these
    Code:
    class Viewport {
       val highDefinitions = IntArray(MAX_PLAYERS)
       val lowDefinitions = IntArray(MAX_PLAYERS)
       val players = arrayOfNulls<Player?>(MAX_PLAYERS)
    
       var highDefinitionsCount = 0
            private set
        var lowDefinitionsCount = 0
            private set
    }
    Your PlayerInfoPacket builder
    Code:
    fun buildPacket(packet: PlayerInfoPacket, buffer: Buffer) {
          val blocksBuffer = createSomeDynamicBuffer()
          buffer.accessBits()
          syncHighDefinition(buffer, blocksBuffer)
          syncHighDefinition(buffer, blocksBuffer)
          syncLowDefinition(buffer, blocksBuffer)
          syncLowDefinition(buffer, blocksBuffer)
          buffer.accessBytes()
          buffer.putBytes(blocksBuffer.flip())
    }
    
    fun syncHighDefinition(buffer: Buffer, blocksBuffer: Buffer) {
        var skipCount = 0
        for (index in viewport.highDefinitionCount) {
               val playerIndex = viewport.highDefinitions[index]
               val player = viewport.players[playerIndex]
               // Some logic here
               if (updatingPlayerInHigh) {
                    var mask = 0
                    val pendingBlocks = player.pendingBlocks()
                    for (block in pendingBlocks) {
                           mask = mask or block.mask // etc
                           blocksBuffer.put(block.build(blocksBuffer, player))
                    }
               }
               //etc
        }
    }
    
    fun syncLowDefinition(buffer: Buffer, blocksBuffer: Buffer) {
        var skipCount = 0
        for (index in viewport.lowDefinitionCount) {
               val playerIndex = viewport.lowDefinitions[index]
               val player = world.players[playerIndex]
               // Some logic here
               if (addingPlayerToHigh) {
                    var mask = 0
                    val pendingBlocks = player.pendingBlocks()
                    for (block in pendingBlocks) {
                           mask = mask or block.mask // etc
                           blocksBuffer.put(block.build(blocksBuffer, player))
                    }
               }
               //etc
        }
    }
    Again this is just a rough outline of the normal process servers use for sending their PlayerInfoPacket and building update blocks and such correctly.

    2. The Problems
    - One major problem with this usual design is the amount of raw data actually being built specifically for the blocks. Imagine there are 2000 players local to each other and that all of them have appearance block pending this tick. With this design, this would cause the appearance block to be built 4,000,000 individual times. This is because 2000 players are being looped for 2000 other players in and building the appearance block each time. Yes this implementation works but this drastically reduces overall game cycle performance. This can be improved by building the blocks 2000 times (for each player) and referencing this during the PlayerInfo packet builder and just sending the bytes that we built beforehand.

    - Another problem is that the majority of the PlayerInfo packet is written using bits instead of full bytes. Additionally the bytes from the blocks are written at the end of the bits process. And usually the update blocks are written to a dynamic growing buffer and then the dynamic buffer is written at the end. This can be improved by being able to accurately calculate how many bytes a update block will take to build and be able to allocate the correct number of bytes each time, instead of relying on a dynamic buffer and flipping it afterwards.

    - Another problem specifically for the syncLowDefinition process is that this is actually only used for persisted blocks that were previously built just being sent back to players that are being added for high definition players. Instead of rebuilding appearance block again, we can persist a players appearance block between game cycles and just write the persisted block bytes that were already built instead of rebuilding it again. This is also used for a couple other blocks, and really can be for any of the blocks that you want to persist.

    3. The Solutions
    Lets go back to the code given by step 1 and improve it.

    First we can setup our update blocks to be able to get the size of each block. For example:
    Appearance block in general kind of sucks because of RuneScape reversing the bytes and such. If they were normal human beings this can be better.
    Code:
    class PlayerAppearanceBlockBuilder : RenderBlockBuilder<Appearance>(
        index = 1,
        mask = 0x4
    ) {
        private val builders = arrayOf(
            HeadBuilder(),
            BackBuilder(),
            NeckBuilder(),
            MainHandBuilder(),
            TorsoBuilder(),
            OffHandBuilder(),
            ArmsBuilder(),
            LegsBuilder(),
            HairBuilder(),
            HandsBuilder(),
            FeetBuilder(),
            JawBuilder()
        )
    
        override fun build(buffer: RSByteBuffer, render: Appearance) {
            val block = RSByteBuffer(ByteBuffer.allocate(size(render) - 1)).apply {
                writeByte(render.gender.id)
                writeByte(render.skullIcon)
                writeByte(render.headIcon)
                if (render.transform != -1) {
                    writeTransmog(render)
                } else {
                    writeIdentityKit(render)
                }
                writeColors(render.bodyPartColors)
                writeAnimations(render)
                writeStringCp1252NullTerminated(render.displayName)
                writeByte(126)
                writeShort(0)
                writeByte(0) // Hidden
                writeShort(0)
                repeat(3) { writeStringCp1252NullTerminated("") }
                writeByte(0)
            }
            buffer.writeByteAdd(block.position())
            buffer.writeBytesReversedAdd(block.array())
        }
    
        private fun RSByteBuffer.writeAnimations(render: Appearance) = if (render.transform != -1) {
            // TODO NPC transmog
        } else {
            render.renderSequences.forEach { writeShort(it) }
        }
    
        private fun RSByteBuffer.writeTransmog(render: Appearance) {
            writeShort(65535)
            writeShort(render.transform)
        }
    
        private fun RSByteBuffer.writeIdentityKit(render: Appearance) {
            // If a builder is not of an appearance body part, we send 0 as default
            // but this will be overwritten with their equipped item if applicable.
            builders.forEach { it.build(this, render.gender, render.bodyParts.getOrNull(it.appearanceIndex) ?: 0) }
        }
    
        private fun RSByteBuffer.writeColors(colors: IntArray) {
            colors.forEach { writeByte(it) }
        }
    
        override fun size(render: Appearance): Int {
            val gender = 1
            val skull = 1
            val headIcon = 1
            val identity = if (render.transform != -1) {
                4
            } else {
                val arm = 2
                val cape = 1
                val foot = 2
                val hair = 2
                val hand = 2
                val head = 1
                val jaw = if (render.isMale()) 2 else 1
                val leg = 2
                val neck = 1
                val shield = 1
                val torso = 2
                val weapon = 1
                arm + cape + foot + hair + hand + head + jaw + leg + neck + shield + torso + weapon
            }
            val colors = 5
            val animations = if (render.transform != -1) 0 else 14
            val displayName = render.displayName.length + 1
            val combatLevel = 1
            val unknown1 = 2
            val hidden = 1
            val unknown2 = 2
            val strings = 3 * ("".length + 1)
            val unknown3 = 1
            return 1 + gender + skull + headIcon + identity + colors + animations + displayName + combatLevel + unknown1 + hidden + unknown2 + strings + unknown3
        }
    }
    Code:
    class OverHeadTextBlockBuilder : RenderBlockBuilder<OverHeadText>(
        mask = 0x1,
        index = 7
    ) {
        override fun build(buffer: RSByteBuffer, render: OverHeadText) {
            buffer.writeStringCp1252NullTerminated(render.text)
        }
    
        override fun size(render: OverHeadText): Int = render.text.length + 1
    }
    With this now we can pre build the blocks for players.
    Code:
    abstract class UpdateBlocks<A : Actor> {
        abstract fun buildPendingUpdatesBlocks(actor: A)
    
        abstract fun clear()
    
        fun Array<out RenderBlock<*>?>.calculateMask(comparator: Int): Int = fold(0) { mask, block ->
            if (block == null) mask else mask or block.builder.mask
        }.let { if (it > 0xFF) it or comparator else it }
    
        fun Array<out RenderBlock<*>?>.calculateSize(mask: Int): Int = fold(0) { size, block ->
            if (block == null) return@fold size
            when (block) {
                is LowDefinitionRenderBlock -> size + block.bytes.size
                is HighDefinitionRenderBlock -> size + block.builder.size(block.renderType)
                else -> throw AssertionError("Block is not in correct instance.")
            }
        }.let { if (mask > 0xFF) it + 2 else it + 1 }
    
        fun RSByteBuffer.writeMask(mask: Int) {
            if (mask > 0xff) writeShortLittleEndian(mask) else writeByte(mask)
        }
    }
    With this class, we keep the highDefinitionUpdates and lowDefinitionUpdates array throughout the entire lifecycle of the server.
    Code:
    class PlayerUpdateBlocks(
        val highDefinitionUpdates: Array<ByteArray?> = arrayOfNulls<ByteArray?>(World.MAX_PLAYERS),
        val lowDefinitionUpdates: Array<ByteArray?> = arrayOfNulls<ByteArray?>(World.MAX_PLAYERS)
    ) : UpdateBlocks<Player>() {
        override fun buildPendingUpdatesBlocks(actor: Player) {
            if (actor.renderer.hasHighDefinitionUpdate()) {
                highDefinitionUpdates[actor.index] = actor.renderer.highDefinitionRenderBlocks.buildHighDefinitionUpdates(actor)
            }
            // Low definitions are always built here for persisted blocks from previous game cycles. i.e Appearance.
            lowDefinitionUpdates[actor.index] = actor.renderer.lowDefinitionRenderBlocks.buildLowDefinitionUpdates()
        }
    
        override fun clear() {
            lowDefinitionUpdates.fill(null)
            highDefinitionUpdates.fill(null)
        }
    
        private fun Array<HighDefinitionRenderBlock<*>?>.buildHighDefinitionUpdates(player: Player): ByteArray {
            val mask = calculateMask(0x40)
            val size = calculateSize(mask)
            return RSByteBuffer(ByteBuffer.allocate(size)).also {
                it.writeMask(mask)
                for (block in this) {
                    if (block == null) continue
                    val start = it.position()
                    block.builder.build(it, block.renderType)
                    val end = it.position()
                    player.renderer.setLowDefinitionRenderingBlock(block, it.array().sliceArray(start until end))
                }
            }.array()
        }
    
        private fun Array<LowDefinitionRenderBlock<*>?>.buildLowDefinitionUpdates(): ByteArray {
            val mask = calculateMask(0x40)
            val size = calculateSize(mask)
            return RSByteBuffer(ByteBuffer.allocate(size)).also {
                it.writeMask(mask)
                for (block in this) {
                    if (block == null) continue
                    it.writeBytes(block.bytes)
                }
            }.array()
        }
    }
    Back to our game loop, now we can do something like this
    Code:
    class WorldSyncTask(
        val world: World,
        private val playerUpdateBlocks: PlayerUpdateBlocks
    ) : SyncTask {
    
        override fun sync(tick: Int) {
            val players = world.players
    
            for (player in players) {
                 playerUpdateBlocks.buildPendingUpdateBlocks(player)
            }
            for (player in players) {
                 player.sendPacket(PlayerInfoPacket(playerUpdateBlocks))
            }
            playerUpdateBlocks.clear()
        }
    }
    Now we need to add a couple more properties to the "Viewport" class or whatever you call it.
    These will be used for tracking the indexes of players for writing their blocks after the PlayerInfo packet finishes bits.
    Code:
    class Viewport {
       val highDefinitions = IntArray(MAX_PLAYERS)
       val lowDefinitions = IntArray(MAX_PLAYERS)
       val players = arrayOfNulls<Player?>(MAX_PLAYERS)
       val highDefinitionUpdates = ArrayList<Int>() // New
       val lowDefinitionUpdates = ArrayList<Int>() // New
    
       var highDefinitionsCount = 0
            private set
        var lowDefinitionsCount = 0
            private set
    }
    And now back to the PlayerInfoPacket builder.
    Code:
    fun buildPacket(packet: PlayerInfoPacket, buffer: Buffer) {
          buffer.accessBits()
          syncHighDefinition(buffer)
          syncHighDefinition(buffer)
          syncLowDefinition(buffer)
          syncLowDefinition(buffer)
          buffer.accessBytes()
          for (index in viewport.highDefinitionUpdates) {
              updateBlocks.highDefinitionUpdates[index]?.let { buffer.writeBytes(it) }
          }
          for (index in viewport.lowDefinitionUpdates) {
              updateBlocks.lowDefinitionUpdates[index]?.let { buffer.writeBytes(it) }
          }
          viewport.highDefinitionUpdates.clear()
          viewport.lowDefinitionUpdates.clear()
    }
    
    fun syncHighDefinition(buffer: Buffer) {
        var skipCount = 0
        for (index in viewport.highDefinitionCount) {
               val playerIndex = viewport.highDefinitions[index]
               val player = viewport.players[playerIndex]
               // Some logic here
               if (updatingPlayerInHigh) {
                    viewport.highDefinitionUpdates += player.index
               }
               //etc
        }
    }
    
    fun syncLowDefinition(buffer: Buffer) {
        var skipCount = 0
        for (index in viewport.lowDefinitionCount) {
               val playerIndex = viewport.lowDefinitions[index]
               val player = world.players[playerIndex]
               // Some logic here
               if (addingPlayerToHigh) {
                    viewport.lowDefinitionUpdates += player.index
               }
               //etc
        }
    }
    And at the end of your tick you can choose what low definition blocks to persist and such like this.
    So the high definition blocks are always cleared. We persist the low definition ones between game cycles so that these bytes do not have to be rebuilt again unless the player actually updates their appearance or something like that.
    Code:
        fun clearUpdates() {
            // Clear out the pending high definition blocks.
            highDefinitionRenderBlocks.fill(null)
            // Clear out the pending low definition blocks.
            for (index in lowDefinitionRenderBlocks.indices) {
                val renderType = lowDefinitionRenderBlocks[index]?.renderType ?: continue
                // Persist these render types.
                if (renderType is Appearance || renderType is MovementSpeed || renderType is FaceAngle || renderType is FaceActor) continue
                lowDefinitionRenderBlocks[index] = null
            }
        }
    4. The Result
    - Blocks are no longer built 4,000,000 times in my previous example mentioned above in step 2. Blocks are only built for each player one time and distributed to the low definition blocks by slicing the high definition bytes. This is the biggest improvement for game cycle performance.
    - When writing blocks to the packet, we only reference the built blocks we already made and they are indexed by the player index in the world. So we can just merely put indexes into the Viewport and write the bytes after the bits of the packet by referencing those properties.
    - We an properly allocate the buffer used for writing blocks bytes by defining the size of each block properly. This removes the use of unnecessary dynamic buffers in general.
    - When a player is being added from low definition to high definition, we do not have to build the blocks again and like mentioned before, we just reference our already built blocks.

    I suck at explaining stuff but if you want to look for yourself you can look at GitHub - runetopic/osrs-server: OSRS emulator written in kotlin. for the full implementation. The entire server only uses ByteBuffer. Even if you don't fully understand it, you are aware of the problems that most servers have with the "usual design" of this process. However these improvements will drastically improve the performance of your server, especially when it comes to high populations. This implementation can even be further improved but this is going in the right direction.

    I do not have any performance numbers for comparison but I can tell you on my server the game cycles are like 30ms. This is with 2000 players logged in, routing and updating blocks each tick, all next to each other. It does more logic than what is shown on this thread.

    5. Sources
    GitHub - runetopic/osrs-server: OSRS emulator written in kotlin.
    PlayerInfoPacketBuilder.kt
    UpdateBlocks.kt
    RenderBlockBuilder.kt
    PlayerUpdateBlocks.kt
    PlayerAppearanceBlockBuilder.kt
    Viewport.kt
    ActorRenderer.kt
    RSByteBuffer.kt
    Last edited by _jordan; 04-08-2023 at 09:42 PM.
    Attached image
    Attached image
    Reply With Quote  
     


  2. #2  
    Registered Member
    Tyluur's Avatar
    Join Date
    Jun 2010
    Age
    27
    Posts
    5,111
    Thanks given
    1,823
    Thanks received
    1,773
    Rep Power
    2637
    I know this has been a long time coming.
    Your contributions are always appreciated.

    Thanks, Jordan.

    Quote Originally Posted by blakeman8192 View Post
    Keep trying. Quitting is the only true failure.
    Spoiler for skrrrrr:

    Attached image
    Reply With Quote  
     

  3. #3  
    Discord Johnyblob22#7757


    Join Date
    Mar 2016
    Posts
    1,386
    Thanks given
    366
    Thanks received
    575
    Rep Power
    5000
    Giga Sigma playerupdating
    Attached image
    Reply With Quote  
     

  4. #4  
    Renown Programmer
    Greg's Avatar
    Join Date
    Jun 2010
    Posts
    1,182
    Thanks given
    260
    Thanks received
    1,025
    Rep Power
    2046
    Quote Originally Posted by _jordan View Post
    - One major problem with this usual design is the amount of raw data actually being built specifically for the blocks. Imagine there are 2000 players local to each other and that all of them have appearance block pending this tick. With this design, this would cause the appearance block to be built 4,000,000 individual times. This is because 2000 players are being looped for 2000 other players in and building the appearance block each time. Yes this implementation works but this drastically reduces overall game cycle performance. This can be improved by building the blocks 2000 times (for each player) and referencing this during the PlayerInfo packet builder and just sending the bytes that we built beforehand.

    - Another problem is that the majority of the PlayerInfo packet is written using bits instead of full bytes. Additionally the bytes from the blocks are written at the end of the bits process. And usually the update blocks are written to a dynamic growing buffer and then the dynamic buffer is written at the end. This can be improved by being able to accurately calculate how many bytes a update block will take to build and be able to allocate the correct number of bytes each time, instead of relying on a dynamic buffer and flipping it afterwards.

    - Another problem specifically for the syncLowDefinition process is that this is actually only used for persisted blocks that were previously built just being sent back to players that are being added for high definition players. Instead of rebuilding appearance block again, we can persist a players appearance block between game cycles and just write the persisted block bytes that were already built instead of rebuilding it again. This is also used for a couple other blocks, and really can be for any of the blocks that you want to persist.

    Reusing pre-encoded update blocks won't work when the update contains hits as the values encoded change depending on if you're the hit source or not.

    If you're calculating the encoded size of the packet beforehand (a requirement if you're using ktor for example) you can skip having a buffer all together.

    Same with the appearance block, you don't need to write it to a buffer and flip it, you can just encode all the values in reverse order with little endianness for the values > 1 byte.
    Attached imageAttached image
    Reply With Quote  
     

  5. #5  
    WVWVWVWVWVWVWVW

    _jordan's Avatar
    Join Date
    Nov 2012
    Posts
    3,058
    Thanks given
    126
    Thanks received
    1,874
    Rep Power
    5000
    Quote Originally Posted by Greg View Post
    Reusing pre-encoded update blocks won't work when the update contains hits as the values encoded change depending on if you're the hit source or not.
    Yes that is basically the only drawback. Any block where it needs to compare against the observing player. Kris has a solution for this but honestly I think it can be better where we can figure something else out while keeping this idea of pre building blocks beforehand. Thanks for being someone who knows what this thread talks about and knowing that edge case.

    If you're calculating the encoded size of the packet beforehand (a requirement if you're using ktor for example) you can skip having a buffer all together.
    For us the overall buffer is the buffer that is written to the write channel, this buffer has a pool of 40,000 bytes that we write all of our packets directly to. Those buffer allocations shown are specifically for building the players blocks with the “mask” and such.

    Same with the appearance block, you don't need to write it to a buffer and flip it, you can just encode all the values in reverse order with little endianness for the values > 1 byte.
    Yeah I know but it’s so dumb lol. Only because this is for an osrs with the intention of keeping it up to date, so one revision they reverse it and another it’s not. If this was like 667 I would just not have them reversed so I wouldn’t need the helper buffer inside of the appearance block. But yeah just overall kind of dumb that they change this all of the time
    Attached image
    Attached image
    Reply With Quote  
     

  6. #6  
    Registered Member

    Join Date
    Sep 2009
    Posts
    1,919
    Thanks given
    481
    Thanks received
    1,687
    Rep Power
    1262
    Quote Originally Posted by _jordan View Post
    Yes that is basically the only drawback. Any block where it needs to compare against the observing player. Kris has a solution for this but honestly I think it can be better where we can figure something else out while keeping this idea of pre building blocks beforehand. Thanks for being someone who knows what this thread talks about and knowing that edge case.


    For us the overall buffer is the buffer that is written to the write channel, this buffer has a pool of 40,000 bytes that we write all of our packets directly to. Those buffer allocations shown are specifically for building the players blocks with the “mask” and such.


    Yeah I know but it’s so dumb lol. Only because this is for an osrs with the intention of keeping it up to date, so one revision they reverse it and another it’s not. If this was like 667 I would just not have them reversed so I wouldn’t need the helper buffer inside of the appearance block. But yeah just overall kind of dumb that they change this all of the time
    For hits could you not cache the only two perspectives there are? IE: My hits vs yours?
    Attached image
    Reply With Quote  
     

  7. #7  
    Registered Member
    thing1's Avatar
    Join Date
    Aug 2008
    Posts
    2,111
    Thanks given
    131
    Thanks received
    1,099
    Rep Power
    2402
    Thank you for your contribution
    It's refreshing to see RSPS getting back to actually refining dated ways we do our engine work.
    Reply With Quote  
     

  8. #8  
    Member The Chad Way To Do Player Update Market Banned


    Luke132's Avatar
    Join Date
    Dec 2007
    Age
    35
    Posts
    12,560
    Thanks given
    199
    Thanks received
    7,109
    Rep Power
    5000
    giphy.gif

    the bloody hell is this jordan

    Attached imageAttached image
    Reply With Quote  
     

  9. #9  
    WVWVWVWVWVWVWVW

    _jordan's Avatar
    Join Date
    Nov 2012
    Posts
    3,058
    Thanks given
    126
    Thanks received
    1,874
    Rep Power
    5000
    Quote Originally Posted by Luke132 View Post
    giphy.gif

    the bloody hell is this jordan
    tldr: faster solution to doing player updating server implementation. (500IQ)
    Attached image
    Attached image
    Reply With Quote  
     


Thread Information
Users Browsing this Thread

There are currently 1 users browsing this thread. (0 members and 1 guests)


User Tag List

Similar Threads

  1. [718+] The right way to do target.Lock();
    By strikecarl in forum Snippets
    Replies: 4
    Last Post: 12-11-2014, 07:57 PM
  2. What is the proper way to do construction?
    By Poesy700 in forum Help
    Replies: 2
    Last Post: 03-18-2014, 12:24 PM
  3. What's the right way to do this? THREADS
    By muppet head in forum Help
    Replies: 6
    Last Post: 11-25-2012, 01:49 AM
  4. whats the best way to get players on ur server?
    By blood eater in forum RS2 Server
    Replies: 2
    Last Post: 01-13-2011, 09:25 PM
  5. The Best Way To Do Crafting
    By Mikey` in forum Help
    Replies: 1
    Last Post: 08-23-2010, 05:14 AM
Posting Permissions
  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •