i have been working on a project in the renpy engine and have come across it not saving all of my variables. specifically having problems with lists of dictionaries where the dictionaries are all procedurally generated and added to the list. in python this means procedurally generating the dictionary then appending a copy to the list such that its not a pointer to to the individual dictionary used to generate an item,but rather an actual dictionary object. i understand renpy is picky about what it saves. does anyone have any pointers for data storage/ structures when using renpy, such that the renpy save and load functions will save all of my data for a given game.
I’ve never used RenPy, but I’ve gathered much experience with Python in general. So, I don’t know how open or restrictive RenPy is, but I hope this will help.
My first instinct was to use the pickle module, which comes with every default installation of Python. It’s purpose is to allow saving Python objects to disc and loading them again from there. It also has two methods, .dumps and .loads, which instead of using files just transform their objects to and from strings (or bytes, to be more specific). I don’t know why RenPy wouldn’t use pickle or similar things from the get-go, but it would be worth a try to just convert the data you want to save using pickle and then let RenPy save the result instead.
Alternatively, the very same could be done with the JSON module, althought this one will produce quite a bit more data to store.
Renpy is fairly open, you can basically do anything you want in python in the background. I currently am using the json module to save the specific lists renpy is not saving but this will make for terrible user experience because they have to remember to first click my save button before using renpy’s save feature along with first load the renpy save then load using my button to get all data saved and loaded.
The issue is with how Renpy handles pickling and more specifically how it marks data as dirty for saving. Complex objects like dictionaries if I remember are stored as pointers and when Renpy looks for data to mark as dirty it does a shallow comparison so it checks against the dictionaries and lists memory address (which does not change) instead of the values. This is why more complex objects can be missed when Renpy saves.
As stated on their saving page
"Note that it's the change to the variables that matters – changes to fields in objects will not cause those objects to be saved."
I am sure you have seen this page already but it does hold some good info
https://www.renpy.org/doc/html/save_load_rollback.html
I know there are workarounds for this but I have had issues getting them to work in the past though in all fairness I did not look far into them. I think one workaround that was suggested was a deep copy into a new object as this would change the memory address or redefine the entire object each change instead of changing properties.
One workaround may be persistent data where you have more control over what gets saved but these are shared across all plays so you have to have a way to partition out the persistent data correctly to prevent interference across plays.
https://www.renpy.org/doc/html/persistent.html
1 Like
I was aware of most of what you have outlined, the solution to this problem is what i am looking for, which is why i asked about good data structers and storage methods for renpy. Thanks for outlining and succinctly stating what the problem is so that others may better understand the problem. From what i have observed any variable or object assigned a value using another variable inherently makes its value a pointer at which point renpy dose not save that variable(i am way more familiar with c++ but i believe this is an inherent trait of python, my understanding of python is that most variables are pointers rather than the object itself which is why you don’t specify and object as a pointer unlike in c++ where if you want a pointer you have to specifically create it). So if i initialize a list with 1,2,3 it will save it right up to the moment i append the list with the value of another variable which = 4. at which point it ignores it. same with lists of dictionaries, the initialized list of 2 dummy dictionaries is saved but once i clear the list and proceduraly generate dictionaries and append them to the list its not saved. i would know how to write the code to this problem in c++ to avoid this becuse i can state that these are lists of objects themselves and not lists of pointers to objects, but im not aware of how to do this in python.
If RenPy looks for changed in the variables, it should be enough to recreate the top level, in this case the list, in order to get it saved. Once you have filled your list with the generated dictionaries, try simply adding the line myList = myList[:] (with whatever name you chose for that variable) to create a new list object with the same contents. Maybe that fixes things.
that would not work for the reason grotlover2 stated, the new list would be a pointer to the old list meaning it would be an un-changing memory address, to get python to make a new object you would have to do newlist = oldlist.copy() which still results in newlist being a pointer to the duplicate list in memory, also an un-changing memory address.
1 Like
The [:] part was for making a copy. It’s the slicing syntax, usually used to get parts of a list, but here using the default values, which result in getting the full list back.
And if you use the explicit .copy() method, it should be exactly the same. If you assign that new copy to your variable, RenPy should detect the change and save things properly.
No matter which of these two variants you use, according to what I’ve gathered here and by looking at the RenPy documentation, it should be enough to save the contents of your list. Have you tried these things already?
Edit: I think I’ll be back out of this convo. There seems to be much more arcane magic involved than I anticipated. Before I’ll make a fool out of myself, I’ll let those discuss that actually used the system before.
2 Likes
Ya python like most modern languages has kind of taken that control away which is a shame as I love that level of control in C/C++. That being said though even if you could Renpy uses a special cache to store variables and values so even if you could it may not work.
I have one fix for this I did in the past but may not work for you as it was a direct engine modification.
Outside of that I have 3 suggestions that may work but can not detail any of the above till I get off work.
1 Like
Ok so i got home from work and was able to try what you said before and it does work, sorry i was not clearly thinking. Thank you very much for this. Thinking back i made a wrong assumption before that we would have a new variable assigned the copy’s memory address, but since were reassigning a new memory address to the same variable it does work because the renpy sees the value as changing(since the memory address is changing).
Just curious because i am not familiar with python will the old list that get deleted or will this lead to a memory leak?(im probably going to find out during play testing)
1 Like
It will get garbage collected. Python handles that all for you.
1 Like