Page 1 of 1

Threaded Menu crash

Posted: Sun Feb 07, 2021 5:24 am
by decompile
Hey,

I'm currently having a random server crash on sending a threaded menu which I cant replicate. It just happens randomly, mostly it works and then suddenly it crashes.

Syntax: Select all

loading_state = itertools.cycle(['   ', '.  ', '.. ', '...'])
def build_loading_menu(menu, index):
menu.clear()
menu.append('Loading' + next(loading_state))
loading_menu = SimpleMenu(build_callback=build_loading_menu)


def _send_threaded_menu(index, callback, args, kwargs, set_page):
print('DEBUG 1')

# Returns a menu class which is ready to be sent to a player
menu = callback(*args, **kwargs)

print('DEBUG 2')

# New menu has been loaded - We can close the old loading menu
loading_menu.close(index)

print('DEBUG 3')

# Send new menu
menu.send(index)

print('DEBUG 4')

# If set_page is defined - Set player page on new menu
if set_page:
print('DEBUG 5')

menu.set_player_page(index, set_page)

print('DEBUG 6')


def send_threaded_menu(index, callback, args=(), kwargs=None, set_page=None):
kwargs = kwargs if kwargs is not None else dict()

# Send loading menu
loading_menu.send(index)

# Add function to WorkerThread
ThreadedClass.function_call(_send_threaded_menu, (index, callback, args, kwargs, set_page))


Code: Select all

DEBUG 1
DEBUG 2
Could not find steamerrorreporter binary. Any minidumps will be uploaded in-processUserMessageBegin:  New message started before matching call to EndMessage.
 
L 02/07/2021 - 02:02:13: Engine error: UserMessageBegin:  New message started before matching call to EndMessage.
 
Segmentation fault (core dumped)
Add "-debug" to the ./srcds_run command line to generate a debug.log to help with solving this problem


The ThreadedClass has the base of a WorkerThread from here:
viewtopic.php?p=13740#p13740

I also uncommented the loading_menu parts, where I got the same message, but for sending the actual menu.

Syntax: Select all

loading_state = itertools.cycle(['   ', '.  ', '.. ', '...'])
def build_loading_menu(menu, index):
menu.clear()
menu.append('Loading' + next(loading_state))
loading_menu = SimpleMenu(build_callback=build_loading_menu)


def _send_threaded_menu(index, callback, args, kwargs, set_page):
print('DEBUG 1')

# Returns a menu class which is ready to be sent to a player
menu = callback(*args, **kwargs)

print('DEBUG 2')

# New menu has been loaded - We can close the old loading menu
# loading_menu.close(index)

print('DEBUG 3')

# Send new menu
menu.send(index)

print('DEBUG 4')

# If set_page is defined - Set player page on new menu
if set_page:
print('DEBUG 5')

menu.set_player_page(index, set_page)

print('DEBUG 6')


def send_threaded_menu(index, callback, args=(), kwargs=None, set_page=None):
kwargs = kwargs if kwargs is not None else dict()

# Send loading menu
# loading_menu.send(index)

# Add function to WorkerThread
ThreadedClass.function_call(_send_threaded_menu, (index, callback, args, kwargs, set_page))


Code: Select all

DEBUG 1
DEBUG 2
DEBUG 3
Could not find steamerrorreporter binary. Any minidumps will be uploaded in-processUserMessageBegin:  New message started before matching call to EndMessage.
 
L 02/07/2021 - 04:10:34: Engine error: UserMessageBegin:  New message started before matching call to EndMessage.
 
Segmentation fault (core dumped)

Re: Threaded Menu crash

Posted: Sun Feb 07, 2021 6:25 am
by Ayuto
Please post the full test code to reproduce the crash.

Re: Threaded Menu crash

Posted: Sun Feb 07, 2021 7:08 am
by decompile
As stated in the second line, I have no way to reproduce this code. I cant explain why it happens.

Re: Threaded Menu crash

Posted: Sun Feb 07, 2021 8:22 am
by Ayuto
Well, I have an assumption, but I would need to see the full test code (especially your Thread class). I guess it doesn't execute the menu.send() line atomically. So, a user message is started, but then the thread is being switched and another one is started.

https://en.m.wikipedia.org/wiki/Context_switch

Re: Threaded Menu crash

Posted: Sun Feb 07, 2021 3:00 pm
by Ayuto
Additionally, to your PM, this might help understanding the problem:

Syntax: Select all

# Only an empty CS:S server is required for this test/explanation plugin.
# No bots; no players. Just load this test plugin with test1 uncommented.
# Then comment out test1, uncomment test2 and reload. Continue like that with test3.

from engines.server import engine_server
from memory import get_virtual_function
from memory import make_object
from bitbuffers import BitBufferWrite
from filters.recipients import RecipientFilter

# Import this snippet: https: //pastebin.com/raw/wbHxTt6d
from test6.worker import WorkerThread

# virtual bf_write *UserMessageBegin( IRecipientFilter *filter, int msg_type ) = 0;
UserMessageBegin = get_virtual_function(engine_server, 'UserMessageBegin')

# virtual void MessageEnd( void ) = 0;
MessageEnd = get_virtual_function(engine_server, 'MessageEnd')

recipients = RecipientFilter()

def test1(arg):
print(arg, 'START')
# Simply send a user message
buffer = make_object(
BitBufferWrite,
UserMessageBegin(engine_server, recipients, 0))

buffer.write_byte(0) # Required for user message 0

# This will cause the thread do be busy and might cause a context switch.
# Alternativly, time.sleep()
y = 0
for x in range(100000):
y += x % 2

MessageEnd(engine_server)

print(arg, 'END')

def test2():
buffer = UserMessageBegin(engine_server, recipients, 0)

# Crashes here, because a second user message is started before the first
# one is finished
buffer = UserMessageBegin(engine_server, recipients, 0)

def test3():
worker1 = WorkerThread()
worker2 = WorkerThread()

for x in range(10):
worker1.add_job(test1, ['worker1'])
worker2.add_job(test1, ['worker2'])

# There is a very high chance that you crash. Even with only one worker
# thread, because the main thread also continously starts user messages.
# Context switching causes the function calls to test1() to stop somewhere
# during the execution. It can be before line 1, in line 1, after line 1,
# but before line 2. Basically, anywhere in that function, because it's not
# atomic/thread safe.
worker1.start()
worker2.start()


# TEST 1 - Demonstrate a working user message. You server won't crash and the
# plugin will load fine.
#test1('main thread')

# TEST 2 - Demonstrate two calls to UserMessageBegin without finishing the
# first one. This will crash your server.
#test2()

# TEST 3 - Demonstrate a high chance of crashing. You might need to reload the
# plugin multiple times if you are lucky.
#test3()

TEST1

Code: Select all

main thread START
main thread END

TEST3

Code: Select all

worker1 START
worker2 START

--> Crash due to context switch from worker1 to worker2

Re: Threaded Menu crash

Posted: Sun Feb 07, 2021 7:59 pm
by L'In20Cible
Using a reentrant lock to send the menu would likely fix the context switch, however, the main unlocked thread (aka the game) could still send a message at any time during game-play likely causing the exact same crash if a worker is currently executing regardless of its locking state. Threading bitbuf messages is risky, and I don't think there is an easy solution -- if any at all because we have no control over the original (or external) calls.

Re: Threaded Menu crash

Posted: Mon Feb 08, 2021 12:41 pm
by Ayuto
Yeah, I thought about a mutex as well, but discarded the idea, because the best solution is to simply send the user message in the main thread. A generic solution might be to pre-hook UserMessageBegin and set the lock and post-hook MessageEnd and unlock.

Re: Threaded Menu crash

Posted: Sat Feb 13, 2021 12:28 pm
by decompile
Hi,

sorry for the late response, I haven't found the time to reply yet.

My crashing issue has been resolved, after Ayuto checking out the functions I was using. The reason for the crash was, that I was building the menu in a seperate thread, which allowed the main thread & seperate thread to collide with each other while building bitbuf messages. As the engine error said, starting a new bitbuff message before the other one was closed. To resolve this issue, I just had to pass the data to the main thread, and build the menu from there.

Thank you guys for your help.