Game crashed after equipping items

reddy36996

Hi,
In our live games, we are seeing lot of crashes and we have been able to pinpoint the issue. It seems to be related to spine.
I have attached the list of devices where similar crashes have been observed on Google play console.

Only happening on android not iOS. We didn't had any of the devices except for Moto E5 Plus (attached to this post) where we were able to reproduce the crash.

We were able to further isolate the bug in Spine Examples scene named "Mix and Match Equip" provided in the package.
I have edited "EquipsVisualsComponentExample.cs" script to repack skin into single texture. Not able to reproduce crash without making this change. Here's the link of updated unitypackage with my changes.

To reproduce, make android build with above package and scene, then keep switching gun and goggles until it crashes.
Also attaching crash report from the Moto E5 device.Spine3.8testCrashreport.txt

Spine Versions Tried : 3.8, 3.7, 3.6 (crash happening on all these versions)
Unity versions Tried : 2018.4.12f1, 2019.3.1f1
Device Architecture: Mono, IL2CPP

Harald

We are sorry for the trouble and thanks for reporting!

The problem here was that the calls to GetRepackedSkin() create new Texture and Material instances, which require calls to Destroy(packedTexture); just like when calling new Texture2D();. If this is omitted, the resources are not freed (or not freed quickly enough) so that after many calls, an out of memory exception is raised on your device.

In your case, please make sure to Destroy() the output packed Texture and Material instances - either every time before they are replaced, or grouped before they get too many if you want to delay the GC load.
You may also have a look at Resources.UnloadUnusedAssets which may be interesting as well.

We have added a documentation section to the respective methods mentioning this requirement and also added the Destroy() code sections to the example scene scripts.
Updated 3.8 unitypackage can be downloaded here as usual:
Spine Unity Download

For reference: the issue has been tracked under this ticket:
https://github.com/EsotericSoftware/spine-runtimes/issues/1681

reddy36996

Hi Harald,

The problem here was that the calls to GetRepackedSkin() create new Texture and Material instances, which require calls to Destroy(packedTexture);

I also came to same conclusion but what doesn't make sense is it happens only with few devices. This issue is not reproducible in any of the other devices we have which is not mentioned in the screenshot. If it's a memory issue, it should happen across all devices as long as we keep repeating calls to GetRepackedSkin()

In your case, please make sure to Destroy() the output packed Texture and Material instances - either every time before they are replaced, or grouped before they get too many if you want to delay the GC load.

Spine is already providing that functionality in AtlasUtilities.ClearCache(), we tried calling it every time GetRepackedSkin() gets called, but that didn't had any effect. Adding Resources.UnloadUnusedAssets makes some difference, as Destroy() doesn't release memory until that asset is unloaded from memory. With this change, sample project is not crashing anymore but it had no effect in our project.
Also we can't use Resources.UnloadUnusedAssets in our project even if it had worked bcoz Whenever it runs on main thread, game gets laggy, specially when we do this calls very frequently. Overall user experience will decrease.

I appreciate if we can get some alternate solution for this.

Harald

reddy36996 đã viết
If it's a memory issue, it should happen across all devices as long as we keep repeating calls to GetRepackedSkin()

I would at least suspect that the issue could be memory related, as the log file that you posted lists "<gsl_memory_alloc_pure:2203>: GSL MEM ERROR: kgsl_sharedmem_alloc ioctl failed" 124 times out of 237 lines.

Regarding that it should happen on all devices: I would not assume this, what if Unity's asset cleanup kicks in regularly before it runs out of memory on some devices, and too late on others?

reddy36996 đã viết
Spine is already providing that functionality in AtlasUtilities.ClearCache(), we tried calling it every time

No, your assumption is incorrect. AtlasUtilities.ClearCache() clears cached regions that are produced by AtlasRegion.ToTexture() calls, but not repacked atlas textures. They are not cached in Spine structures but left in responsibility of the user.

reddy36996 đã viết
Adding Resources.UnloadUnusedAssets makes some difference, as Destroy() doesn't release memory until that asset is unloaded from memory.

What exactly do you mean by until that asset is unloaded from memory.? If you mean unloading via Resources.UnloadAsset, this is not intended for a resource created with new.
Anyway, you should of course not hold any references to it any longer, but I assume that you don't.

The basic pattern in Unity is (or was) that every new Texture2D() call should be balanced by a matching Destroy() call. At least it was like this many years ago, now documentation is for some reason not listing it any more. Perhaps clearing any references to it and waiting for GC / Unity's asset cleanup is the recommended way now?

reddy36996 đã viết
Also we can't use Resources.UnloadUnusedAssets in our project even if it had worked bcoz Whenever it runs on main thread, game gets laggy, specially when we do this calls very frequently.

This is why I recommended the Destroy(); call.
Perhaps this is related to a Unity bug with your used version in combination with some devices? At least this seems to have happend in the past:
https://answers.unity.com/questions/893772/texture2d-generation-causing-memory-leak-system-ou.html

reddy36996 đã viết
With this change, sample project is not crashing anymore but it had no effect in our project.

Does the sample scene stop crashing when you are calling Destroy() only and not Resources.UnloadUnusedAssets?

Are you perhaps having some other calls like new Texture() that are not balanced with coresponding Destroy() / cleanup code? Where exactly does your own project crash, what is the scenario there? Did you analyse memory usage of the device that was crashing?

One thing I forgot to ask:
Which version of the spine-unity runtime (name of the unitypackage) are you using?

reddy36996

No, your assumption is incorrect. AtlasUtilities.ClearCache() clears cached regions that are produced by AtlasRegion.ToTexture() calls, but not repacked atlas textures. They are not cached in Spine structures but left in responsibility of the user.

I see. I'm assuming these repacked atlas textures are created inside AtlasUtilities class. Can you point out exact method and line number where it is being created. Based on that, i'll try to destroy them frequently during the session along with calls to Resources.UnloadUnusedAssets() and see if that makes any difference.

The basic pattern in Unity is (or was) that every new Texture2D() call should be balanced by a matching Destroy() call. At least it was like this many years ago, now documentation is for some reason not listing it any more. Perhaps clearing any references to it and waiting for GC / Unity's asset cleanup is the recommended way now?

Right now, how it works is GC.Collect frees any mono memory that is allocated and not being referenced anymore i.e. mostly objects or any collections. Whereas Resources.UnloadUnusedAssets() frees any memory occupied by unity objects like prefabs, textures, audio, etc. GC.Collect is internally called within Resources.UnloadUnusedAssets().

This is why I recommended the Destroy(); call.

Destroying any unity object doesn't release memory allocated to that object until we call Resources.UnloadUnusedAssets().
Note: Memory is released even without calling it if we switch to another scene as Resources.UnloadUnusedAssets() is called internally during scene unloading.

Perhaps this is related to a Unity bug with your used version in combination with some devices? At least this seems to have happend in the past:

I don't think we have any memory leak. We have profiled thoroughly in our devices and memory stays constant after reaching some value e.g. In one of the devices, our game's memory reaches till 1.2gb from 0.5gb. After that it remains fixed at that value.Just keeps increasing/decreasing few MBs every frame.

Does the sample scene stop crashing when you are calling Destroy() only and not Resources.UnloadUnusedAssets?

No, it still crashes.

Are you perhaps having some other calls like new Texture() that are not balanced with coresponding Destroy() / cleanup code? Where exactly does your own project crash, what is the scenario there? Did you analyse memory usage of the device that was crashing?

We are calling new Texture() at few places but they are just one time calls. I don't think they are related to crash as crash doesn't happen if we disable skin repacking in spine. To explain the scenario in our project would be bit lengthy, it would be better if we can fix crash in sample project itself. Then we will apply the same solution into our project.

Harald

I see. I'm assuming these repacked atlas textures are created inside AtlasUtilities class. Can you point out exact method and line number where it is being created

Yes, it is called inside the GetRepackedSkin() method, in this line here:
spine-runtimes/AtlasUtilities.cs at 3.8

Right now, how it works is GC.Collect frees any mono memory that is allocated and not being referenced anymore i.e. mostly objects or any collections. Whereas Resources.UnloadUnusedAssets() frees any memory occupied by unity objects like prefabs, textures, audio, etc. GC.Collect is internally called within Resources.UnloadUnusedAssets().

What would be the interesting part is whether Destroy() on e.g. Texture2D leads to freeing the internal resources as well (GPU memory part of a Texture2D, etc), since normal GC.Collect never led to freeing it alone. However, you are mentioning further down the text that this is not the case currently:

Destroying any unity object doesn't release memory allocated to that object until we call Resources.UnloadUnusedAssets().

No, it still crashes.

Ok, I thought that it was already no longer crashing with the added Destroy() call alone.

All in all I think the best solution will then be to add an additional GetRepackedSkin() overload providing an inout ref Texture argument to allow reuse of the same Texture2D across multiple calls to GetRepackedSkin().