Differences

This shows you the differences between two versions of the page.

--- pcb:preferences_subsystem [2020/07/26 14:18]
cparker
+++ pcb:preferences_subsystem [2021/02/28 19:56] (current)
cparker [Development Blog]
@@ Line 7: / Line 7: @@
   * The system should be able to interact with specified preferences files. \\ This allows subsystems to create their own if they want. It will also allow the user to specify such a file at startup.
   * The system should manage the reading and writing of the files transparently. \\ We don’t want developers to have to think too hard about how this works.
-  * The system should not put constraints on what can be contained by the preference. \\ It should be allowed to be anything. Ultimately it’s a string when it’s stored and read, but it should be convertible into any data type.
+  * The system should not put constraints on what can be contained by the preference. \\ It should be allowed to be anything; any data type.
-  * The files should be human readable. \\ There’s an interesting question here about locales…
   * The system should be partitionable such that a subsystem may keep track of their own preferences without having to interact with the preferences of any other subsystem.
-=== Implementation Overview ===
+Specifically for the pcb system preferences...:
-Preferences will be stored in a file as key/value pairs. An algorithm will read the file into a dictionary type structure, and then for every key it will look up that key in a "preferences registry". The preferences registry will be a list of structures that include things like the name of the preference, some help text, default value, etc. and a pointer to a function that can be used to interpret the value string associated with the key. The function does its thing, and sets the destination variable with its value.
+  * The files should be human readable. \\ There’s an interesting question here about locales…
+  * This should be able to handle the command line arguments as well.
+  * I'd like to enforce that keys are unique, as this makes things easier. So, one subsystem shouldn't use the same key as another or the system.
-To save preferences this works in just the reverse. There is also a "writer" function associated with preferences in the registry, and this is called to convert the data back into a string. We generate a series of key/value pairs that are then written to a text file.
+=== Architecture ===
+The general problem is that we'd like to be able to store data, and then restore it to the software. The system should be able to handle any kind of data, and it should be able to accommodate different storage formats. A "back-end" will be the part of the code that is responsible for storing data in a particular format. Each back-end should be familiar with a common set of data-types, including integers, floating points, and strings (bytes). More complex data-types will be stored as blocks of bytes, and it is expected that the implementor of the data-type will implement a converter function to do this.
-The key/value pair structures can be returned to the calling function to be retained for future inquiry. For example, maybe a plug-in chooses to store a preference in the main pcb preferences file. When pcb is initialized, if the plug-in isn't loaded yet, the preference will still be read and retained so that when the plug-in does load, the read value will be available to it for conversion.
+The expectation is that subsystems are going to have their own structures to store their preferences, for example the Settings structure in global.h. So this system will generate descriptions of the preference data only. It will not actually retain the preference values, but rather read them and write them to the other locations in memory where the rest of the code expects to find them.
-A python pseudo-code might look something like this:
+That means that our work here is essentially going to be to define framework the describes the preference metadata so that we can save and load it in a programmatic fashion. To that end, we're going to define data structures that don't hold the preference data itself, but describe it and have a pointer to the actual data so that we can retrieve and store it. Preference metadata consists of things like name, help text, and data type.
-<code python>
-def load_preferences(filename, registry):
-  file_data = read_pref_file(filename)
-  for key, value in file_data:
-    if key in registry:
-      registry[key].reader(key, value, registry[key].ptr)
-  return file_data
-def save_preferences(filename, registry):
-  file_data = []
-  for key in registry:
-    file_data.append([key, registry[key].writer(key, registry[key].ptr)])
-  write_pref_file(file_data)
-  return len(file_data)
-</code>
-(why is it always so much easier in python?)
-=== Data Structures ===
+There are a number of standard data types, including int, double, and string, and we'll also include a forth data type "pointer" as essentially an "other". This will allow the preference objects to be more complex structures.
-We're going to draw heavily on the work that's already been done with the preferences (HIDAttributes) in designing the data structures for this system. The first data structure is what's populated when a preference file is read:
-<code c>
+Ideally, we'll just be able to create a structure of metadata objects and say "save all of these to a file", or "load all of the data from that file according to these specs". To that end, we'll need a structure that we can iterate over.
-typedef struct
-{
-  /*! key */
-  char * key;
-  /*! value */
-  char * value;
-} PreferenceFileItem;
-</code>
-Data will be read from the file, and populate an array (object_list?) of these objects. This array will then be looped over looking for matching keys in a second, sorted, array structure containing the next data structure. The second data structure describes a preference item.
+=== Process ===
+One of the considerations here is separating the storage and application so that the methods can change. If we decide we'd rather store preferences in an SQL database, we can do that without having to rewrite the whole system. We only have to write functions to load and save the key values pairs in the database.
+I'm not too worried about the speed of these routines, so, it doesn't matter much if the lists are sorted. Reading, and writing preferences doesn't happen very often.
+We're going to go with a two step process. The read and write functions will take a pointer to a list of PreferenceItems. The readers and writers can interact explicitly with the data they are reading/writing and there's no need to build an intermediate list. This better supports the option for alternate storage formats.
+  * load_preferences(file, preflist)
+   - open the preference store
+   - for each item in the store
+   - Look up the preference in the preflist
+   - get a reader: check for a reader in the structure and use it if it's there. Otherwise lookup the default reader for the preference type.
+   - apply the reader to set the preference
+  * save_preferences(preflist, file)
+I'd also like this process to work for command line arguments so that any preference stored in a file can also be specified on the command line.
+=== Data Structures ===
+We're going to draw heavily on the work that's already been done with the preferences (HIDAttributes) in designing the data structures for this system.
+== PreferenceDefinition ==
+The primary data structure describes a preference item.
 <code c>
 /*!
- * \brief PreferenceItem data structure
+ * \brief PreferenceDefinition data structure
  */
 typedef struct
@@ Line 63: / Line 58: @@
   /*! Human readable name of the preference */
   char * name;
+  /*! preference type identifier (enum) */
+  int type;
   /*! Text that describes what the preference influences */
   char * help_text;
@@ Line 70: / Line 67: @@
   void (*writer)(char * index_str, void * ptr);
   /*! data pointer passed to reader and writer */
+  PreferenceValue * ptr;
+  /*! a value that can be used to initialize the preference; type defined as per above. */
+  PreferenceValue default_val;
+} PreferenceDefinition;
+</code>
+There will be a number of "reader"/"writer" functions implemented by default to handle common data types like floats, ints, coords, etc. The data pointer will be passed to the reader and writer functions, and could contain anything. But it probably is a pointer to the variable to be set by the converted value from the PreferenceItem. Using function pointers like this allows for a lot of flexibility in how subsystems can use this code. The reader functions, for example, may also be responsible for notifying a GUI that the preference has been updated.
+Note: the default_val should initialize the value to something that is SAFE, i.e. something that will never, ever cause PCB to crash.
+== Preference Values ==
+My initial thought was to make this a void pointer so that it could be anything. The current PCB pref stuff uses a union structure for this instead, and I'm wondering if that would be easier to work with. The union structure could have a "pointer" item in it that would allow for more complicated items to be allocated. This would keep the val with the structure in memory, which would be nice.
+The realization that I'm having comes out of writing an object list copy function for a preference definition. If the data payload is a custom user defined object, then the copy function won't know how to make a copy of it. Normally the way it works is that when an object is added to a list, the list makes it's own copy. That way it doesn't have to worry about the user changing, or unallocating, the data. I've been thinking for a while that I need a version of the list that assumes ownership of the data when it's added.
+One alternative might be to register preference types... and require a copy function to be defined for each type. This might be okay if I provide defaults for all the common types, but could be ornery if I require the user to do it.
+Another alternative might just be to provide a "pointer" type and say that if you're going to use a more complex structure, then the module that uses it has to provide it and make sure that it says allocated.
+For now I'm going to go with the last option, because it does seem reasonable, and it also seems like the least work for now. Additionally, I think I'm going to go with the union idea, as that gets rid of having to cast a ton of void pointers.
+Note that in the PreferenceDefinition structure, the val is actually a pointer. This has to be because this is actually a pointer to somewhere else in memory where the actual value of the preference is stored. Because it's a union, it effectively allows it to be basically a pointer to anything, and I can interpret it as such.
+== PreferenceValue ==
+Based on the previous discussion, we're now going to define a union type to handle these values. (This used to be PreferenceItem and held key, value, and type triples).
+<code c>
+typedef union
+{
+  /*! integer value */
+  int ival;
+  /*! double value */
+  double dval;
+  /*! string pointer */
+  char * sptr;
+  /*! void pointer */
   void * ptr;
-  /*! a string that can be used to initialize the preference */
+} PreferenceValue;
-  char * default_str;
-} PreferenceItem;
 </code>
-There will be a number of "reader"/"writer" functions implemented by default to handle common data types like floats, ints, coords, etc. The data pointer will be passed to the reader and writer functions, and could contain anything. But it probably is a pointer to the variable to be set by the converted value from the PreferenceFileItem. Using function pointers like this allows for a lot of flexibility in how subsystems can use this code. The reader functions, for example, may also be responsible for notifying a GUI that the preference has been updated.
+== Lists ==
+Presently, the intent is to use object_list for all of the lists.
+=== Implementation Overview ===
+Preferences will be stored in a file as key/value pairs. An algorithm will read the file into a dictionary type structure, and then for every key it will look up that key in a "preferences registry". The preferences registry will be a list of structures that include things like the name of the preference, some help text, default value, etc. and a pointer to a function that can be used to interpret the value string associated with the key. The function does its thing, and sets the destination variable with its value.
+To save preferences this works in just the reverse. There is also a "writer" function associated with preferences in the registry, and this is called to convert the data back into a string. We generate a series of key/value pairs that are then written to a text file.
+The key/value pair structures can be returned to the calling function to be retained for future inquiry. For example, maybe a plug-in chooses to store a preference in the main pcb preferences file. When pcb is initialized, if the plug-in isn't loaded yet, the preference will still be read and retained so that when the plug-in does load, the read value will be available to it for conversion.
+A python pseudo-code might look something like this:
+<code python>
+def load_preferences(filename, registry):
+  file_data = read_pref_file(filename)
+  for key, value in file_data:
+    if key in registry:
+      registry[key].reader(key, value, registry[key].ptr)
+  return file_data
+def save_preferences(filename, registry):
+  file_data = []
+  for key in registry:
+    file_data.append([key, registry[key].writer(key, registry[key].ptr)])
+  write_pref_file(file_data)
+  return len(file_data)
+</code>
+(why is it always so much easier in python?)
-Note: the default_str should initialize the value to something that is SAFE, i.e. something that will never, ever cause PCB to crash.
 === Functions ===
-There are a number of key functions here: load_preferences, save_preferences, read_pref_file, write_pref_file, apply_preferences, collect_preferences, readers, and writers. Note that the file IO is being deliberately kept separate so that the actual file format is independent. Also, this allows data from a single file to be scanned more than once against different preference registries. We saw some pseudo-code for the load_preferences and save_preferences above (although it should have used apply_preferences and collect_preferences instead of spelling it out). So let's think about some of the others, again in pseudo-code (python...).
+There are a number of key functions here: load_preferences, save_preferences, readers, and writers. Note that the file IO is being deliberately kept separate so that the actual file format is independent. Also, this allows data from a single file to be scanned more than once against different preference registries. We saw some pseudo-code for the load_preferences and save_preferences above (although it should have used apply_preferences and collect_preferences instead of spelling it out). So let's think about some of the others, again in pseudo-code (python...).
 <code python>
-def read_pref_file(filename):
+def load_preferences(filename, pref_list):
   with open(filename, r) as f:
     f.seek(-1) # seek to the end
@@ Line 218: / Line 276: @@
 === Questions ===
- * If a user loads a new preferences file, does it need to notify anything that this happened? \\ Presently, with preferences, values are updated immediately. This means that all preferences need to be such that changing them at any given moment doesn’t lead to disaster. However, this model also provides the flexibility that, if there isn’t such a preference, it can specify its own handler function which could take care of any of the necessary tasks to enact the change.
+  * If a user loads a new preferences file, does it need to notify anything that this happened? \\ Presently, with preferences, values are updated immediately. This means that all preferences need to be such that changing them at any given moment doesn’t lead to disaster. However, this model also provides the flexibility that, if there isn’t such a preference, it can specify its own handler function which could take care of any of the necessary tasks to enact the change.
- * Should I try to use the glib class structure and create an actual manager object
+  * Should I try to use the glib class structure and create an actual manager object
+  * When collecting preferences, do we collect everything, or only things that aren't the same as the default?
+  * I've made an implicit assumption here that the preference keys are strings. Maybe I shouldn't? They could just be integers...
+==== Old stuff ====
+== Original Idea ==
+Originally, I was going to do it this way... but this is just extra work. So, I'm ditching the intermediate step.
+I'm breaking the preferences process into four steps:
+  - Read the preferences file and create a list of key value pairs (preference items) \\ pilist = ppm_read_pref_file(fname); \\ This returns a list of PreferenceItem objects that contains the keys and values read out of the specified file.
+  - Apply preferences to the PCB data structures \\ ppm_apply_preferences(pdlist, pilist); \\ This iterates over the PreferenceItems in pilist. For each item it looks up the key in a preference definition list, pdlist, and calls the function associated with that preference definition.
+  - Collect preferences from PCB data structures \\ pilist = ppm_collect_preferences(pdlist); \\ This goes through the preferences definitions in pdlist and creates a set of key value pairs, pilist, that can be stored in the text file.
+  - Write the preference data to a file. \\ ppm_write_pref_file(pilist); \\ Store the preferences in a file.
+== New Idea ==
+===== Development Blog =====
+I'm starting this because I'm having trouble remembering where I am and what I've done between development sessions.
+At this point, I've written data converters for ints, doubles, and strings. I've also written tests for them, so, I know they're working the way I think they should.
+== Tasks ==
+  * Read function
+= 20201108 =
+Today I managed to get a the function for writing preference files written, and a simple test that uses it.
+One note: when I initially wrote the test, I did something like this (abbreviated):
+<code c>
+int intval;
+char charvals[] = "Preference string";
+PreferenceDefinition intpref = {.key = "integer", .val = &intval};
+PreferenceDefinition strpref = {.key = "string", .val = &charvals};
+</code>
+However, this doesn't work. The val field of the strpref ended up equaling the address of charvals, whereas I needed it to be a double pointer. This makes me wonder what I ought to do here. If preferences are stored in a struct like
+<code c>
+typedef struct {
+  char * charvals;
+} Pref;
+</code>
+Then I have a pointer to point to. However, if the preferences are stored as "global" variables, then I just get the address of the string, and I don't have a container to point to. I wonder if I need to have some sort of accommodation for that?
+= 202201115 =
+I'm working today on the function that reads the files. I've looked at the current file format and it is pretty much "key = value". So, I think I'm going to allow that format also. That way I don't have to deal with trying to figure out how to get everyone's preferences converted to the new format. This means that I have to implement non-white-space delimiters, but I don't think that will be too hard. It also provides the developers with another option for delimiters.
+= 20201227 =
+Worked on the apply function. It works now.
+I also found a bunch more functions related to attribute handling in hid/common/hidinit.[c,h], and I'm starting to have a better sense of how the current systems work together. There's a "node" list, and each subsystem registers a node that is effectively an array of HID_Attributes. There are functions for parsing the command line arguments, reading and writing files, etc, all similar in function to the code that I've just implemented...
+I need to think a little bit on how to handle these things hierarchically. How do subsystems inherit preferences from above? Can they back-propagate preferences? Can they share across-subsystems?
+Thinking "out-loud":
+When a subsystem wants to inherit something, it should be familiar with what it's parent's attributes are and just grab those attributes directly. It should only store locally things that are specific to the subsystem.
+But what if we passed a command-line parameter for a subsystem? The core won't know what to do with it because it's not in the core's list. We want that to be accessible to the subsystem, however. Maybe this is a good argument for the intermediate key/value pair structure. Perhaps we treat command-line arguments differently, and just save those vectors globally so that subsystems can inspect them at their leisure.
+= 20210214 =
+In spite of keeping this blog for the purpose of remembering where I am... I've completely forgotten. I recall that I had started failing tests, and so I was rebuilding the system in a new branch to try to figure out where things went wrong. I guess I'm going to start getting back into this by bringing one class at a time into the new branch.
+Starting from master, I've added the PreferenceValue, PreferencePair, and preference cstring converters to the new base. This includes all of the tests, which pass.
+The next steps are to bring back in the apply functions, the parsing functions, and the file IO functions.
+One further note: support for PreferenceString isn't yet complete. There aren't converters for it, and it's not integrated with the reading and writing. The purpose of this data type is to allow arbitrary data to be stored in the files. Handling this type will require adding support for things like escape characters so that newlines and things can be included in the data. Right now I know that there's no arbitrary data that needs to be stored, so, I'm going to proceed without fully completing this, but, I probably need to add more remarks in the code about it.
+= 20210228 =
+As per usual, since I didn't have things completely laid out before-hand, I've made a bit of a mistake. I added a type enum to the PreferenceValue objects, however, these objects were supposed to point to the value in the preference structure, not actually contain the value themselves. It's probably still useful to have this type defined, but, not quite as useful as I had thought it would be.
+Now I've also created a PreferenceAlias type that is just a union of the different pref options. This will serve the purpose I had originally intended for PreferenceValue.
+The PreferenceDefinition type has been brought back into the fold, and all the tests now pass. This was a little tricky because I had to change a bunch of return values to PreferenceValue from char * in order to keep things more generic.
+All of the existing code has now been brought back into the codebase, and everything is passing its tests!
+Next steps are to figure out how to make this system handle command line arguments.

gEDA Project Wiki

User Tools

Site Tools

Differences

Page Tools