conversion error on index_from_pointer

[velocity]

So it seems like the NOT_RESULT Is the same as SAME_ARGS

DICT_NOT_IN: 60670 times
DICT2_NOT_IN: 12134 times
CONVERSION_ERROR: 36402 times
TOTAL_FUNCTION: 60670 times
NOT_RESULT: 12134 times
SAME_ARGS: 12134 times

Alright I will change it to ULONG as well. There are still conversion errors tho, does this mean its actually the same.. as pre, performance wise?

[L'In20Cible]

does this mean its actually the same.. as pre, performance wise?

It means you process at least ~20% less calls than you previously did, which is still a good start I'd say. Though, it really doesn't seem to be the right approach overall for what your goal actually is.

[velocity]

Though, it really doesn't seem to be the right approach overall for what your goal actually is.

What can I do instead

[L'In20Cible]

Though, it really doesn't seem to be the right approach overall for what your goal actually is.

What can I do instead

Well, I guess it depend of which entity but surely you can hook more targeted than that function. Do you have a map (preferably for CS:S) that have such entity?

[velocity]

If you want to try something, this is a good map: https://gamebanana.com/mods/127520 for it, or https://gamebanana.com/mods/127438. I'm not sure how you want to make it more targeted, but please note it is not just player-entity collision, it's also player-player and player-flashbang collisions, etc. This goes for SetTransmit as well.

I also want to point out that there are no issues with this in sourcemod, you are welcome to try out the plugin I linked and see for yourself and compare lag or other stuff if you are up for it. Like in Sourcepython everything works smoothly until ~6 players and this is after super careful considerations for optimization and smart tricks to make it as fast as possible (that I'm capable of doing), then server var starts going crazy. Comparably in sourcemod, you can have 20 players or more, no problem, with much less respect for optimizations. Obviously it totally depends on the use case, and what you are trying to do, but usually stuff like this is common for me.

PS: I remember a long time ago, I made a post about performance problems here as I couldn't understand why my plugin after converting from ES to SourcePython lagged on my server, it was literally 1:1 conversion, yes it was HORRIBLY optimized, but it still ran without lagging with ES compared to SP. I know that these general statements do not help to improve SP, I just want to point out, these are some of the frustrations that one might have while coding in SP, but SP has come a long way since then, I think.

Hopefully, the lag is less present now after the optimizations we have discussed, I haven't tested its significance yet, but if you can come up with a more targeted solution that would be epic.

Edit: I think this was the old post: viewtopic.php?f=10&t=1902&p=12225&hilit=performance+problems&sid=a4b47dc0218f095c9670364173062d09#p12225 (...not to start an off-topic conversation)

[L'In20Cible]

Here's a quick attempt at something: https://github.com/Source-Python-Dev-Te ... collisions

Example:

Syntax: Select all

from listeners import OnEntityCollision

@OnEntityCollision
def on_entity_collision(entity, other):
    """Called when two entities are about to collide."""
    if not entity.is_player() or not other.is_player():
        return

    return entity.team_index != other.team_index


@OnEntityCollision
def on_entity_collision(entity, other):
    """Called when two entities are about to collide."""
    return not (entity.is_player() and
        other.classname in ('prop_door_rotating', 'func_breakable')
    )

Test builds (CS:S/CS:GO/Windows): https://drive.google.com/drive/folders/ ... JkVPWFe3i1

[Jezza]

Here's a quick attempt at something: https://github.com/Source-Python-Dev-Te ... llisions.h

Does this disable collision? It doesn't seem to do anything. (CS:GO/Linux&Windows)

[L'In20Cible]

Here's a quick attempt at something: https://github.com/Source-Python-Dev-Te ... llisions.h

Does this disable collision? It doesn't seem to do anything. (CS:GO/Linux&Windows)

I only tested on CS:S, and built on CS:GO, so it is possible there is some differential factors to take into consideration. Though, it doesn't override VPhysics collisions, but that example above is disabling teammates, rotating doors, and breakables collision for players. The overall idea is that we only have a single hook with 2 callbacks (pre and post) and use direct delegation to test entities rather than a noisy dynamic hook in between.

[velocity]

Nice, can't wait to try out, can you make a build for Linux/CS:GO?

[L'In20Cible]

Nice, can't wait to try out, can you make a build for Linux/CS:GO?

Jezza can probably provide you with a build. If that was for CS:S, I could fire up my VM and build it because it is up-to-date, but I'm too lazy to maintain my CS:GO (or any other games for that matter) environment which is constantly outdated and/or have expired token.

[Jezza]

Nice, can't wait to try out, can you make a build for Linux/CS:GO?

Currently, it does not work with CS:GO. You need to wait for an update from L'In20Cible.

[Jezza]

What you really need for your PassServerEntityFilter is a lookup table.

Syntax: Select all

# Python Imports
#   Collections
from collections import defaultdict

# Source.Python Imports
#   Entities
from entities.helpers import index_from_inthandle
#   Listeners
from listeners import OnEntityDeleted
from listeners import OnNetworkedEntitySpawned
#   Memory
from memory import Convention
from memory import DataType
from memory import find_binary
from memory.hooks import PreHook
#   Players
from players.entity import Player


# =============================================================================
# >> CLASSES
# =============================================================================
class PairPassFilter(dict):
    _entity_ids = defaultdict(list)

    def add_filter(self, touch_entity, pass_entity, filter=True):
        touch_address = touch_entity.pointer.address
        pass_address = pass_entity.pointer.address

        id = touch_address ^ pass_address
        if id not in self:
            self._entity_ids[touch_address].append(id)
            self._entity_ids[pass_address].append(id)

        self[id] = filter

    def remove_filter(self, touch_entity, pass_entity):
        touch_address = touch_entity.pointer.address
        pass_address = pass_entity.pointer.address

        id = touch_address ^ pass_address
        del self[id]

        self._entity_ids[touch_address].remove(id)
        self._entity_ids[pass_address].remove(id)

    def remove_entity(self, base_entity):
        ids = self._entity_ids.pop(base_entity.pointer.address, None)
        if ids is not None:
            for id in ids:
                pass_filter.pop(id, None)


class PairPlayer(Player):

    def __init__(self, index, caching=True):
        """Initializes the object."""
        super().__init__(index, caching)

        self.pair_player = None

    def add_pass_filter(self, base_entity, filter=True):
        pass_filter.add_filter(self, base_entity, filter)

    def remove_pass_filter(self, base_entity):
        pass_filter.remove_filter(self, base_entity)

    def make_pair(self, other_index):
        pair_player = self.pair_player
        if pair_player is not None:
            self.remove_pass_filter(pair_player)

        self.pair_player = PairPlayer(other_index)
        self.pair_player.pair_player = self
        self.add_pass_filter(self.pair_player)

    def set_pair_filter(self, base_entity):
        self.add_pass_filter(base_entity)
        pair_player = self.pair_player
        if pair_player is not None:
            pair_player.add_pass_filter(base_entity)


pass_filter = PairPassFilter()


# =============================================================================
# >> CALLBACKS
# =============================================================================
@OnEntityDeleted
def on_entity_deleted(base_entity):
    pass_filter.remove_entity(base_entity)


@OnNetworkedEntitySpawned
def on_networked_entity_spawned(entity):
    try:
        pair_player = PairPlayer(index_from_inthandle(entity.owner_handle))
    except (ValueError, OverflowError):
        return

    pair_player.set_pair_filter(entity)


server = find_binary("server", srv_check=False)

PassServerEntityFilter = server[b"\x55\xB8\x01\x00\x00\x00\x89\xE5\x83\xEC\x38\x89\x5D\xF4"].make_function(
    Convention.CDECL, (
        DataType.ULONG,
        DataType.ULONG,
    ),
    DataType.BOOL,
)

@PreHook(PassServerEntityFilter)
def pre_pass_server_entity_filter(args):
    touch_entity = args[0]
    pass_entity = args[1]

    if not pass_entity or touch_entity == pass_entity:
        return

    filter = pass_filter.get(touch_entity^pass_entity, None)
    if filter is not None:
        return filter

    return False

Syntax: Select all

PairPlayer(1).make_pair(2)

will enable collisions on player1 and player2.

You can also use

Syntax: Select all

pair_player.set_pair_filter(entity)

to apply the entity's collision to the pair players.
In this demo, the entities owned by the pair are automatically added to the path filter table.

As a personal opinion, I would suggest implementing such a lookup table on the C++ side, but I honestly don't know if this will happen due to disagreements like the one in #332.

[velocity]

What you really need for your PassServerEntityFilter is a lookup table.

Yes definitely would help, but I dont see how this any different from what was already suggested by L'In20Cible?

[Jezza]

What you really need for your PassServerEntityFilter is a lookup table.

Yes definitely would help, but I dont see how this any different from what was already suggested by L'In20Cible?

That will depend on the results of your tests, because we do not know the performance status of your server.

If L'In20Cible's on_entity_collision can work in CS:GO, then perhaps we can reduce the number of calls to 1/4, but if it still doesn't improve the performance using such a lookup table on the Python side, then I guess the only way is to implement the lookup table on the C++ side. However, it is not up to me to decide that.

[velocity]

Yes with the on_entity_collision, you are right, but in his previous posts on the first page of this thread, he already suggested a lookup table, that is what I was referring to. Good idea with this line tho

Syntax: Select all

if not pass_entity # aka if not args[1]

as I didn't know you could do boolean checks for this on "invalid addresses". We also reached the conclusion previously that a PostHook would be more efficient than a PreHook in this case.

Originally my post was how to handle conversion error without try/catch and that line actually concludes that.. Don't know why it wasn't mentioned, right away, but I guess the problem is more deep-rooted than this as it doesn't fix the underlying issue with python performance. I think most stuff should be implemented on the C++ side ( or at minimum demanding functions ), I don't even want to think about the overhead that comes with Boost Python with general mixing of loops between C++/Python.

[Jezza]

Yes with the on_entity_collision, you are right, but in his previous posts on the first page of this thread, he already suggested a lookup table, that is what I was referring to. Good idea with this line tho

Syntax: Select all

if not pass_entity # aka if not args[1]

as I didn't know you could do boolean checks for this on "invalid addresses". We also reached the conclusion previously that a PostHook would be more efficient than a PreHook in this case.

You may be able to gain 7% to 13% by using PostHook, but that is not the point. What's important is that I (and probably L'In20Cible) thinks that using a lookup table can solve the performance issue with your PassServerEntityFilter. In fact, I did not feel any lag in 32bot+1player, de_dust2. And since everyone's system is different, I can't predict your performance. Please try the code first and report back how much it affects performance.

I also did a quick measurement of the amount of PassServerEntityFilter calls (32bots + 1player), and it seems that on average the PassServerEntityFilter is called 250 times per tick.

Code: Select all

tick: 30000
func_call: 7591605
prefalse: 3740712
postfalse: 3466442

func_call/tick: 253.0535
prefalse/tick: 124.6904
postfalse/tick: 115.54806666666667

This is not a small number, but if you are just processing lookup tables, it should not become a problem.

Syntax: Select all

from ctypes import memmove
from ctypes import c_void_p
from time import perf_counter

from memory import Convention
from memory import DataType
from memory import alloc
from memory.hooks import PreHook

no_op = b"\x90\x90\x90\x90\x90\x90\xC3"
func_caller = alloc(len(no_op), auto_dealloc=False)
func_caller.unprotect(len(no_op))

memmove(c_void_p(func_caller.address), no_op, len(no_op))

func_caller_function = func_caller.make_function(
    Convention.CDECL, (
        DataType.ULONG,
        DataType.ULONG,
    ),
    DataType.BOOL,
)

@PreHook(func_caller_function)
def pre_func_caller_function(args):
    touch_entity = args[0]
    pass_entity = args[1]

    if not pass_entity or touch_entity == pass_entity:
        return False

    filter = pass_filter.get(touch_entity^pass_entity, None)
    if filter is not None:
        return filter

    return False

pass_filter[1^2] = True

s = perf_counter()
for i in range(125):
    func_caller_function(1, 2)
    func_caller_function(1, 1)
e = perf_counter()
print("time", e-s)

Code: Select all

time 0.0005662888288497925

Originally my post was how to handle conversion error without try/catch and that line actually concludes that.. Don't know why it wasn't mentioned, right away, but I guess the problem is more deep-rooted than this as it doesn't fix the underlying issue with python performance. I think most stuff should be implemented on the C++ side ( or at minimum demanding functions ), I don't even want to think about the overhead that comes with Boost Python with general mixing of loops between C++/Python.

That's because it's unclear what you want to do with the PassServerEntityFilter. For example, the reason this is fast is because it's practically just doing a lookup. But if you want to use Entity, Player, etc. to do other processing, please provide that code.

And about try/catch, I don't know much about the entities passed to the PassServerEntityFilter, so I can't say for sure, but if the PassServerEntityFilter doesn't receive non-networked entities, then just check

Syntax: Select all

if not pass_entity

with a DataType.POINTER and you won't get the Conversion Error.

[L'In20Cible]

Latest commits works for me on CS:S and CS:GO, and also add support for CollisionHash. For example:

Syntax: Select all

from entities.collisions import CollisionHash
from events import Event
from players.entity import Player

collisions = CollisionHash()

@Event('player_say')
def player_say(game_event):
    player = Player.from_userid(game_event['userid'])
    entity = player.view_entity
    if entity is None:
        return
    if collisions.has_pair(player, entity):
        collisions.remove_pair(player, entity)
    else:
        collisions.add_pair(player, entity)

[velocity]

That's because it's unclear what you want to do with the PassServerEntityFilter. For example, the reason this is fast is because it's practically just doing a lookup. But if you want to use Entity, Player, etc. to do other processing, please provide that code.

I'm not doing any other processing, other than making checks on who should collide with who, just like in SourceMod script. I have learned from past mistakes, that game-oriented attributes like player.dead could overflow the var. I'm doing less than the script, because right now I'm trying to isolate the lag, so it's just two EntityDictionary lookups for address1 and address2. What I might have failed to mention, but is evident, is that speeding up functions like this helps compensate the additional processing of SetTransmit, which is also required to make it work fully, which can also be seen in the .sp. On top of this, not to mention OnPlayerRunCmd for 'other server stuff'. So, I think it is only an advantage that the hook of these functions is as optimized as possible.

I also did a quick measurement of the amount of PassServerEntityFilter calls (32bots + 1player), and it seems that on average the PassServerEntityFilter is called 250 times per tick.

I have just implemented the new optimizations L'In20Cible combined with urs (Address -> Entity) etc... So I cannot confirm what I'm about to say: I tested with 20 bots and var didn't seem that off, but I have my suspicion regarding the actual impact of bots based solely on their movement. For example, if you stand still the PassServerEntityFilter almost doesn't fire at all, after 15 seconds it's like 100 calls, but if you move around the calls are much more frequent. On top of that, if two players are 'grinding' each other, or when an entity is colliding with another entity is where the function gets an insane amount of calls. So I would like to test out with real players, as well, before concluding (just hard to get 20 players xD).

Latest commit works for me on CS:S and CS:GO, and also add support for CollisionHash.

Nice, what is CollisionHash? Is CollisionHash Jezzas py implementation in c++ for faster look up when using OnEntityCollision?

I really want to try these things out! I have a small request, if possible, can your branch be merged with Cookstars branch implementation of(don't know if that's you Jezza) SetTransmit. So I can try them combined and see if the server can handle it? I know there are disagreements with the implementation of SetTransmit, but either solution is better than hooking SetTransmit with EntityCondition, at the moment. Also a build for Linux/CS:GO.

This thread got longer than expected, and I appreciate the time and effort, so I would like to take a moment and thank you guys: L'In20Cible, Jezza, and Ayuto for your suggestions and general insights for speed optimizations etc.

On the occasion of this, I want to shine some light on this post again: viewtopic.php?f=9&t=2036&p=12854&hilit=donation&sid=7857ff58a2aedd0f39f20ee65a3d50c8#p12854

[Jezza]

Latest commit works for me on CS:S and CS:GO, and also add support for CollisionHash.

Excellent work!

However, there are two problems, one is that the global CollisionHash instance is not being destroyed properly when the plugin is unloaded. I'm not sure why, but it seems to be a problem with gc.collect. This applies to other objects as well, but it becomes a problem if the CollisionHash instance continues to remain with collision pair. (Tested on CS:GO/Linux)

The second, which is more of a feature extension than a problem, is that CollisionHash does not have a collision direction. Although it is not necessary for the problem that velocity needs, PassServerEntityFilter is divided into touching and passing, and it can decide collision on either side, but CollisionHash cannot.

Syntax: Select all

@OnEntityCollision
def on_entity_collision(entity, other):
    if not entity.is_player() or not other.is_player():
        return

    if entity.team_index == 3 and other.team_index == 2:
        return False

For example, this script allows only CT players to pass through other T players, but CollisionHash is not able to achieve this.

[L'In20Cible]

but I have my suspicion regarding the actual impact of bots based solely on their movement

Bots are indeed extremely noisy because they scan for targets, paths, etc. I've added some exclusions to OnEntityCollision to filter out contents we don't really care about in the context of collisions (visibility, attacks, audition, etc.).

On CS:S with 32 bots for 10,000 ticks:

Syntax: Select all

Before:
    Total calls: 1896297
    Average calls per frame: 189.6297

After:
    Total calls: 28858
    Average calls per frame: 2.8858

On CS:GO with 20 bots for 10,000 ticks:

Syntax: Select all

Before:
    Total calls: 2791137
    Average calls per frame: 279.1137

After:
    Total calls: 36250
    Average calls per frame: 3.625

Nice, what is CollisionHash? Is CollisionHash Jezzas py implementation in c++ for faster look up when using OnEntityCollision?

It is a wrapped IPhysicsObjectPairHash. The physics engine uses that to filter collisions through collision events. That class is independent of OnEntityCollision, and can be used independently. Just fill your pairs in and they won't collide, or you can use the listener if you have dynamic conditions to test for.

However, there are two problems, one is that the global CollisionHash instance is not being destroyed properly when the plugin is unloaded. I'm not sure why, but it seems to be a problem with gc.collect. This applies to other objects as well, but it becomes a problem if the CollisionHash instance continues to remain with collision pair. (Tested on CS:GO/Linux)

I was not able to reproduce:

Syntax: Select all

# ../testing/testing.py

from entities.collisions import CollisionHash
from listeners import OnEntityCollision

collisions = CollisionHash()

@OnEntityCollision
def on_entity_collision(entity, other):
    ...

Syntax: Select all

sp plugin load testing;sp plugin unload testing
[SP] Loading plugin 'testing'...
CCollisionHash::CCollisionHash
CCollisionManager::Initialize
CEntityCollisionListenerManager::Initialize
[SP] Successfully loaded plugin 'testing'.
[SP] Unloading plugin 'testing'...
CEntityCollisionListenerManager::Finalize
CCollisionHash::~CCollisionHash
CCollisionManager::Finalize
[SP] Successfully unloaded plugin 'testing'.

But I went ahead and added AutoUnload support to it just in case.

The second, which is more of a feature extension than a problem, is that CollisionHash does not have a collision direction. Although it is not necessary for the problem that velocity needs, PassServerEntityFilter is divided into touching and passing, and it can decide collision on either side, but CollisionHash cannot.

Perhaps we could add a separate class for that, but as far as the current class is concerned, I think we should leave it as is because it is consistent with the engine. Adding a separate class seems easy enough if we use a shared abstract base class.