Topics

LST File Load Order and error situations with .MOD

Tom Parker
 

Okay, I get it now... and while I agree with Eric that the INCLUDE/EXCLUDE thing is a different topic, let me step back for a moment and ask a similar question of what we *want* to have happen.

Let me specifically address one thing first:

--- In pcgen_developers@..., "FerretDave" <ferret.griffin@...> wrote:
Two main points in question -
[snip A to deal with it later]
B) homebew a.PCC defines something, homebrew b.pcc modifies that
thing. If I include a.PCC and b.PCC in c.PCC (which is loaded in
GUI), can I ensure that a.PCC is loaded before b.PCC?
This should be unambiguous. Regardless of file load order, the order of loading is ALWAYS: BASE, COPY, MOD. So even if b.PCC is loaded before a.PCC, ALL of the base objects in BOTH a.PCC and b.PCC will be loaded before any COPY statement in a.PCC and b.PCC are processed. Only after all COPY statements are processed will any MOD statements be processed.

So if you assume b.PCC is loaded first, effectively you have:
b.PCC base objects
a.PCC base objects
b.PCC COPY
a.PCC COPY
b.PCC MOD
a.PCC MOD

You can swap A and B, or insert a C, or whatever, and you will find the base objects will always be loaded before the MODs.

Here's the trick:

--- In pcgen_developers@..., "FerretDave" <ferret.griffin@...> wrote:
A) RSRD defined something, then .MOD'd it in another file, I couldnt
then .MOD it in my homebrew - that was an old issue.
So let me define the specific scenario as this:
a.PCC:
MyAbility <> MULT:YES <> CHOOSE:SKILL|TYPE=Boring
b.PCC:
MyAbility.MOD <> CHOOSE:SKILL|TYPE=Boring|TYPE=Constructive
c.PCC:
MyAbility.MOD <> CHOOSE:SKILL|TYPE=Boring|TYPE=Detailed

Load a.PCC, b.PCC and c.PCC via abc.PCC (so that RANK does not have an impact). What happens (in other words, which MOD "wins")?

In general, for any of the tokens that "overwrite", this type of problem can happen with multiple MODS. I hadn't considered that in my original note.

Similar to the INCLUDE/EXCLUDE discussion, I want to ask: Should this be allowed? Should the data above throw an error due to potentially conflicting MODs?

Let's alter the example a bit to define conflicting:
d.PCC:
MyAbility <> MULT:YES <> CHOOSE:SKILL|TYPE=Boring <> DESC:Does Stuff
e.PCC:
MyAbility.MOD <> CHOOSE:SKILL|TYPE=Boring|TYPE=Constructive
f.PCC:
MyAbility.MOD <> DESC:Woot!

This "DEF" set of MODs is what we might call "innocent", in that the items modified are not overlapping. So in theory, both MODs can be applied in any order and the resulting object will be the same. Therefore, I think we should allow this type of modification without any error regardless of what we do for "ABC".

This really gets back to what data integrity we want. For example, the CHOOSE system currently complains if you do this:
CHOOSE:STAT|TYPE=Strength|TYPE=Dexterity|TYPE=Intelligence|TYPE=Strength

Note the TYPE=Strength twice.

It complains specifically because the inclusion of Strength twice is highly likely to mean the user made a data error. Some might see this as "logically not a problem" (since it's all an OR and putting something in twice is no big deal), but the likelihood that this is a problem scenario indicates (to me anyway) that the situation should be highlighted. I'll note for the record this really did happen, and really did catch an error not long ago (see the main list if I recall)

So the question on example ABC is, does the two MODs that are potentially conflicting mean something more should be done to highlight that to the user? The reason it might be so is we probably need to ensure that when both sources are loaded, the data shouldn't do:
CHOOSE:SKILL|TYPE=Boring|TYPE=Constructive|TYPE=Detailed

... so should we protect against that, and if so, how?

TP.

dave_the_ferret
 

Greetings,

Thanks, it's good to know that its always BASE,COPY,MOD, with MODs applied in order of inclusion in the PCC. It's bits like that that I'm asking to be confirmed in the docs.

With your examples of conflicting .MODs, in my view they are all legit methods, certainly the ABC example I can see being used legally and so I wouldn't want that to warn me.

(as an example, one source defines a feat, another source alters the prerequisites of that feat, and then a third source updates the feat description - forgotten realms is full of things like that)

However your final point:
CHOOSE:SKILL|TYPE=Boring|TYPE=Constructive|TYPE=Detailed
Shouldn't happen, as the *intent* would be to replace the CHOOSE completely, rather than add to it... or I'd need to specifically .CLEAR that original choose (? can you clear a choose?).

I *do* fully agree with it warning if there are duplicate options in that list though.

Cheers
D

--- In pcgen_experimental@..., "thpr" <thpr@...> wrote:




Okay, I get it now... and while I agree with Eric that the INCLUDE/EXCLUDE thing is a different topic, let me step back for a moment and ask a similar question of what we *want* to have happen.

Let me specifically address one thing first:

--- In pcgen_developers@..., "FerretDave" <ferret.griffin@> wrote:
Two main points in question -
[snip A to deal with it later]
B) homebew a.PCC defines something, homebrew b.pcc modifies that
thing. If I include a.PCC and b.PCC in c.PCC (which is loaded in
GUI), can I ensure that a.PCC is loaded before b.PCC?
This should be unambiguous. Regardless of file load order, the order of loading is ALWAYS: BASE, COPY, MOD. So even if b.PCC is loaded before a.PCC, ALL of the base objects in BOTH a.PCC and b.PCC will be loaded before any COPY statement in a.PCC and b.PCC are processed. Only after all COPY statements are processed will any MOD statements be processed.

So if you assume b.PCC is loaded first, effectively you have:
b.PCC base objects
a.PCC base objects
b.PCC COPY
a.PCC COPY
b.PCC MOD
a.PCC MOD

You can swap A and B, or insert a C, or whatever, and you will find the base objects will always be loaded before the MODs.

Here's the trick:

--- In pcgen_developers@..., "FerretDave" <ferret.griffin@> wrote:
A) RSRD defined something, then .MOD'd it in another file, I couldnt
then .MOD it in my homebrew - that was an old issue.
So let me define the specific scenario as this:
a.PCC:
MyAbility <> MULT:YES <> CHOOSE:SKILL|TYPE=Boring
b.PCC:
MyAbility.MOD <> CHOOSE:SKILL|TYPE=Boring|TYPE=Constructive
c.PCC:
MyAbility.MOD <> CHOOSE:SKILL|TYPE=Boring|TYPE=Detailed

Load a.PCC, b.PCC and c.PCC via abc.PCC (so that RANK does not have an impact). What happens (in other words, which MOD "wins")?

In general, for any of the tokens that "overwrite", this type of problem can happen with multiple MODS. I hadn't considered that in my original note.

Similar to the INCLUDE/EXCLUDE discussion, I want to ask: Should this be allowed? Should the data above throw an error due to potentially conflicting MODs?

Let's alter the example a bit to define conflicting:
d.PCC:
MyAbility <> MULT:YES <> CHOOSE:SKILL|TYPE=Boring <> DESC:Does Stuff
e.PCC:
MyAbility.MOD <> CHOOSE:SKILL|TYPE=Boring|TYPE=Constructive
f.PCC:
MyAbility.MOD <> DESC:Woot!

This "DEF" set of MODs is what we might call "innocent", in that the items modified are not overlapping. So in theory, both MODs can be applied in any order and the resulting object will be the same. Therefore, I think we should allow this type of modification without any error regardless of what we do for "ABC".

This really gets back to what data integrity we want. For example, the CHOOSE system currently complains if you do this:
CHOOSE:STAT|TYPE=Strength|TYPE=Dexterity|TYPE=Intelligence|TYPE=Strength

Note the TYPE=Strength twice.

It complains specifically because the inclusion of Strength twice is highly likely to mean the user made a data error. Some might see this as "logically not a problem" (since it's all an OR and putting something in twice is no big deal), but the likelihood that this is a problem scenario indicates (to me anyway) that the situation should be highlighted. I'll note for the record this really did happen, and really did catch an error not long ago (see the main list if I recall)

So the question on example ABC is, does the two MODs that are potentially conflicting mean something more should be done to highlight that to the user? The reason it might be so is we probably need to ensure that when both sources are loaded, the data shouldn't do:
CHOOSE:SKILL|TYPE=Boring|TYPE=Constructive|TYPE=Detailed

... so should we protect against that, and if so, how?

TP.

Tom Parker
 

--- In pcgen_experimental@..., "FerretDave" <ferret.griffin@...> wrote:
With your examples of conflicting .MODs, in my view they are all
legit methods, certainly the ABC example I can see being used
legally and so I wouldn't want that to warn me.
Can and should are different things... see below.

(as an example, one source defines a feat, another source alters
the prerequisites of that feat, and then a third source updates the
feat description - forgotten realms is full of things like that)
My caution with your position starts here. This example is actually DEF, not ABC, because PRExxx and DESC are different, and thus innocent overlaps. I agree this should always be legal. The ABC example is specifically overwriting the exact same token... so the key question is whether the SAME token should allow overwriting by two MODs.

However your final point:
CHOOSE:SKILL|TYPE=Boring|TYPE=Constructive|TYPE=Detailed
Shouldn't happen, as the *intent* would be to replace the CHOOSE
completely, rather than add to it... or I'd need to specifically
.CLEAR that original choose (? can you clear a choose?).
.CLEAR is not possible on CHOOSE. Changing a MULT:YES Ability to a MULT:NO one (the only reason you'd need .CLEAR) is just not something that seems reasonable (changes WAY too many things under the covers).

However, I sense you missed the reason for that point. The problem here is: What intent? The load system today will do an overwrite on the data as you say the intent should be... and CHOOSE is defined as an overwrite, so literally that is correct behavior... BUT... who's intent are we talking about here? Yours? Mine? Publishers? Seems to me the last of those should trump the others.

Let's recap ABC to bring it into this note:
a.PCC:
MyAbility <> MULT:YES <> CHOOSE:SKILL|TYPE=Boring
b.PCC:
MyAbility.MOD <> CHOOSE:SKILL|TYPE=Boring|TYPE=Constructive
c.PCC:
MyAbility.MOD <> CHOOSE:SKILL|TYPE=Boring|TYPE=Detailed

Now, let's get more specific on the example for ABC:

Assume A is SRD, B is Publisher DD and C is publisher EE.

Assume DD and EE never comprehended each other's work (very likely!)

Assume Andrew coded B and you coded C, neither owning or using the other source, and not necessarily comprehending each other's work (likely?)

Assume Constructive things appear in B (with a described change to some ability in A) and Detailed things in C (also with a described change to the same ability in A).

What if I - as a user - want to use SRD, DD and EE all loaded at the same time? Should I have to do additional data work?

It seems to me that would be the combined intent of the two publishers would be for the CHOOSE to be exactly as I specified:
CHOOSE:SKILL|TYPE=Boring|TYPE=Constructive|TYPE=Detailed

The question is: How do I get that CHOOSE... it's not in any data.

With the MOD structure shown in the ABC example, neither dataset would comprehend the other dataset, and the result is... never valid without custom work. In my opinion, that "never" situation and requiring custom work by the user shouldn't be the default case... it should at least be a warning if not an error. So the question is: How should we handle this?

One method is to create another dataset, call it BC, that must be loaded if both B and C are used. ICK. We have no way to control / enforce that.

Another alternative is to put the full CHOOSE back in the SRD, but then you will get warnings in the SRD that there are no Skills of TYPE=Constructed or TYPE=Detailed. Also a bad option, IMHO because the SRD should be pure to what is in the SRD.

Yet another alternative would be to make a replacement for CHOOSE that does .CLEAR and actually adds to the list rather than overwrites. (This is hugely disruptive)

I would assert the best-practice alternative data structure is to do:
CHOOSE:SKILL|TYPE=SkillTypeForThisAbility

... then NEVER require a MOD of the Ability, just including the TYPE on the appropriate skills. (with potentially a HIDETYPE in the PCCs).

Note that this solution never has to deal with multiple conflicting MODs, never requires comprehending an order of operations issue, and guarantees that cross-publisher mixing will not produce results that people have to fix by hand.*

(*Note that even if a skill has two TYPE tokens with MODs to add the new types [e.g. if a Skill is both Constructive and Detailed], that is an "innocent" overlap since TYPE is an additive token, not overwriting... the warning/error issue is focused on overwriting tokens)

I know we've had similar situations with conflicting data already and used variables or other methods like the TYPE trick above to overcome the overlap. So we obviously have ways to accomplish it - at least in some cases - why shouldn't we be automatically detecting those issues? (and in cases where we can't work around it, define the solution for how we do that)

So let me rephrase the challenge / counter-example I'm looking for (to say this isn't an error... or at least a warning):
Why should we allow two overwriting MODs from different sources such that they always produce a legal result, and there is not a non-ambiguous data structure that is a preferable solution?

TP.

dave_the_ferret
 

comments inline

--- In pcgen_experimental@..., "thpr" <thpr@...> wrote:







--- In pcgen_experimental@..., "FerretDave" <ferret.griffin@> wrote:
With your examples of conflicting .MODs, in my view they are all
legit methods, certainly the ABC example I can see being used
legally and so I wouldn't want that to warn me.
Can and should are different things... see below.

(as an example, one source defines a feat, another source alters
the prerequisites of that feat, and then a third source updates the
feat description - forgotten realms is full of things like that)
My caution with your position starts here. This example is actually DEF, not ABC, because PRExxx and DESC are different, and thus innocent overlaps. I agree this should always be legal. The ABC example is specifically overwriting the exact same token... so the key question is whether the SAME token should allow overwriting by two MODs.
Good point, I'd skipped over that without thinking it through fully.
So if two things are loaded, and both mod the same token...
As it's potentially legit (I know about the other mod and want to replace it), then I specifically do *not* want an error - I don't want my gaming group to be concerned when they load my homebrew.

However, there's no way for the code to know that I know...

So, if there's a concern that it could be done unintentionally, and we want to (by default) warn about possible mod conflicts - we need a way to suppress that - how about we have a flag to 'dont warn about this specific duplicate' (specified on the lst row to apply *only* to that specific row)?

Or, more preferably, we have a programatical way to ensure that both MODS are applied, in turn, and the order is fixed (by RANK or order in PCC) so that only the second one is applied).

My preference is that the last MOD applied takes precedence, and overrides any previous mods - if I cant clear a token, I need to be able to ensure its set explicitly to only what I want it to be.


As an overriding feature then - if I select & load two (potentially conflicting sources) in the GUI source selection, then I get warnings about the conflicts. However, if I create a single PCC that includes both lst files [possibly via a sub-pcc], and I select and load *that* via the GUI source selection, then the warnings are automatically suppresed?

Or perhaps, more in keeping, something in the PCC akin to the 'FORWARDREF' option to suppress warnings for specifically listed items?


However your final point:
CHOOSE:SKILL|TYPE=Boring|TYPE=Constructive|TYPE=Detailed
Shouldn't happen, as the *intent* would be to replace the CHOOSE
completely, rather than add to it... or I'd need to specifically
.CLEAR that original choose (? can you clear a choose?).
.CLEAR is not possible on CHOOSE. Changing a MULT:YES Ability to a MULT:NO one (the only reason you'd need .CLEAR) is just not something that seems reasonable (changes WAY too many things under the covers).
I can give an example otherwise - a (hidden) feat gives a choice of feat A or feat B, and I want to .mod it to give a choice of feat B or Feat C *instead* - removing the choice of feat A.

However, I sense you missed the reason for that point. The problem here is: What intent? The load system today will do an overwrite on the data as you say the intent should be... and CHOOSE is defined as an overwrite, so literally that is correct behavior... BUT... who's intent are we talking about here? Yours? Mine? Publishers? Seems to me the last of those should trump the others.
I'd disagree, Publishers should be first, rather than last, the whole point of homebrew is that I'm modifying the rules as defined by the publisher, so my homebrew (with a higher numbered rank) should be loaded last and be able to override anything defined by the publisher.


Let's recap ABC to bring it into this note:
a.PCC:
MyAbility <> MULT:YES <> CHOOSE:SKILL|TYPE=Boring
b.PCC:
MyAbility.MOD <> CHOOSE:SKILL|TYPE=Boring|TYPE=Constructive
c.PCC:
MyAbility.MOD <> CHOOSE:SKILL|TYPE=Boring|TYPE=Detailed

Now, let's get more specific on the example for ABC:

Assume A is SRD, B is Publisher DD and C is publisher EE.

Assume DD and EE never comprehended each other's work (very likely!)
Agreed! :-)

Assume Andrew coded B and you coded C, neither owning or using the other source, and not necessarily comprehending each other's work (likely?)
Quite possible... however... I'm focusing on homebrew, so I can be fairly sure of what sources I'm using together. Though I do see your point for official sources, though in the main I suspect that there's not that much .mod going on in the official sources?

Assume Constructive things appear in B (with a described change to some ability in A) and Detailed things in C (also with a described change to the same ability in A).

What if I - as a user - want to use SRD, DD and EE all loaded at the same time? Should I have to do additional data work?
No, official sources as provided by PCGen, should all work without caveats.
It seems to me that would be the combined intent of the two publishers would be for the CHOOSE to be exactly as I specified:
CHOOSE:SKILL|TYPE=Boring|TYPE=Constructive|TYPE=Detailed

The question is: How do I get that CHOOSE... it's not in any data.
Though I can confirm that that is the opposite of my homebrew intent (I want to force choices of my homebrew options, rather than the standard options, not add to the list)

With the MOD structure shown in the ABC example, neither dataset would comprehend the other dataset, and the result is... never valid without custom work. In my opinion, that "never" situation and requiring custom work by the user shouldn't be the default case... it should at least be a warning if not an error. So the question is: How should we handle this?

One method is to create another dataset, call it BC, that must be loaded if both B and C are used. ICK. We have no way to control / enforce that.
I do (partially) recall a conversation about pathfinder where people may or may not be using one source, and thus there was consideration for a source to have two pcc's - along the lines of 'pathfinder with apg and bestiary' and 'pathfinder with bestiary but without apg' to fudge a workaround, not sure if that ever got implemented though.
That can be controlled via PRECAMPAIGN.




Another alternative is to put the full CHOOSE back in the SRD, but then you will get warnings in the SRD that there are no Skills of TYPE=Constructed or TYPE=Detailed. Also a bad option, IMHO because the SRD should be pure to what is in the SRD.
I think this is where abilities are coming in, and Drew (at least) is making inroads to allow more of the 'standard' stuff be defined by variables, so SRD stuff can be turned off. While this works for Rage/Turn etc, I'm not sure it copes with the level we're talking about... or are we just trying to convolute things too much here? Though I see your point about making a 'clean' solution that will always work in future too.

Yet another alternative would be to make a replacement for CHOOSE that does .CLEAR and actually adds to the list rather than overwrites. (This is hugely disruptive)

I would assert the best-practice alternative data structure is to do:
CHOOSE:SKILL|TYPE=SkillTypeForThisAbility

... then NEVER require a MOD of the Ability, just including the TYPE on the appropriate skills. (with potentially a HIDETYPE in the PCCs).
If the RSRD is updated to handle things in that way, then it probably breaks all homebrew out there so far, but certainly does fix this going forward.
That's probably asking a bit too much at the moment though.



Note that this solution never has to deal with multiple conflicting MODs, never requires comprehending an order of operations issue, and guarantees that cross-publisher mixing will not produce results that people have to fix by hand.*

(*Note that even if a skill has two TYPE tokens with MODs to add the new types [e.g. if a Skill is both Constructive and Detailed], that is an "innocent" overlap since TYPE is an additive token, not overwriting... the warning/error issue is focused on overwriting tokens)

I know we've had similar situations with conflicting data already and used variables or other methods like the TYPE trick above to overcome the overlap. So we obviously have ways to accomplish it - at least in some cases - why shouldn't we be automatically detecting those issues? (and in cases where we can't work around it, define the solution for how we do that)

So let me rephrase the challenge / counter-example I'm looking for (to say this isn't an error... or at least a warning):
Why should we allow two overwriting MODs from different sources such that they always produce a legal result, and there is not a non-ambiguous data structure that is a preferable solution?
Put like that, I dont think I can think of a reason.
If there's a better way of doing it, then yes, give a warning, and suggest the alternative.

I think we both have examples of where the confliction can be considered valid and others where it can be considered illegal, and so I suspect that just some method of indicating whether its an expected conflict or not is the cleanest way forward.


Cheers
D
TP.