Uh oh!
There was an error while loading. Please reload this page.
Sharing cache between adaptive learners and equivalence oracles #136
Replies: 2 comments 14 replies
Dear @stateMachinist, recently the ADT learner has been switched to
|
That's exactly what I had in mind. I think you don't need the |
Alright, thank you. Merging the counters seems like a good idea to me. There's another thing I've noticed while building my benchmark: When I replaced the regular MealyCacheOracle with a SymbolQueryCache for non-adaptive learners, I would sometimes get slightly different performance results. Then I noticed that processQueries in MealyCacheOracle sorts queries descending by length to exploit prefix-closedness (see here). The SymbolQueryCache did not do this (see here). I think this is a bug, because there is no necessity for this difference in performance and it potentially introduces bias when comparing adaptive and non-adaptive learners. Again, I found this issue in version 0.17. I haven't tested it with the latest development version yet, but from a quick glance at the code, it seems to me that this issue may still be present. Furthermore, I am mentioning this because it may also affect other specialized oracles/caches that I haven't looked into. |
This is not a bug but a disadvantage of adaptive queries. The normal Mealy queries are preset, meaning that you know all the input symbols that will be posed to the oracle beforehand so that you can sort them intelligently. With adaptive queries this is not the case. You would have to already provide system outputs in order to obtain the next input symbols of the query which makes sorting them ahead-of-time impossible. |
I think there was a misunderstanding. I understand that there are natural limitations with adaptive queries. However, the SymbolQueryCache was also capable of processing preset queries via processQueries, but did not employ sorting for these. And since it could be used as a cache for preset queries, I would also expect it to have this generic optimization for preset queries. But since there seems to be a stronger separation between adaptive and preset views on the SUL now, I am not sure if it is still possible to run into such a scenario. |
Ah, I see. You are right, the previous implementation did not support reordering. I think this specific use-case was never really considered (although it sounds fairly reasonable). The If you want to optimize the query order for the EQ part, you probably need to set up a second (duplicate) query cache for the membership oracle view (which internally delegates to the SUL cache then). This should also work for the current version with adaptive queries. However, note that you need to specify a batch size for the EQ oracle, since you can't really re-order singleton test queries. It might be worthwhile to think about extracting this feature into a separate query filter. This way you could save on the duplicate memory consumption and it may also be used for de-duplicating some functionality in the current Mealy/Moore caches. |
I've taken another look at the new class hierarchy. In my example, it is the SULAdaptiveOracle wrapping the cache. I saw that SULAdaptiveOracle implements SingleAdaptiveMembershipOracle, which also has processQueries, but does not perform query sorting. Wouldn't the simplest solution be to always perform query sorting in the default implementation of processQueries for Mealy machine oracles? This way, oracle setup would remain the same and you would never lose performance by accident if you forgot to plug in that sorting filter. |
That's a good example of how reordering impacts learning performance. However, I do not quite see how that classifies as 'masking' of algorithmic properties. From the perspective of the oracle: If it has no obligations as to preserving a specific order, why should it not choose the most efficient way of processing the queries? From the perspective of the learner: If the learner wants its queries to be processed in a specific order, why should it delegate them to a callee that does by contract not guarantee preservation of order? I think part of the issue might be that we use different definitions of the term 'algorithm' in this context. I think that, strictly speaking, when you do batch processing, part of the learning algorithm is outsourced into the oracle, and so you may lose control over some algorithmic details of the learner (as the MAT framework draws the line between learner and teacher at the level of individual queries). However, I do not see any issues with this, as long as the contract of the API clearly communicates that. The implications of allowing parallelization of queries are even more significant for the learner, since that may rob it of determinism. Personally, I don't understand what's the use of batch processing if it doesn't employ any optimizations in its implementation. I think if the API clearly states contractual conditions, implementers of learning algorithms can freely decide which details of their algorithm they are willing to delegate. One advantage of this would be saving redundant code: If many algorithms perform batch processing, why should all of them reimplement such a generic optimization like sorting? On the other hand, I may be overlooking some relevant use cases, and having an optional filter for reordering would certainly be more flexible. In the end, it is about trading flexibility vs. complexity of use. I only wanted to provide another perspective, I don't mean to meddle with your design decisions. |
I agree with all of your points and they are the reason why the interfaces are designed the way they are (at least I like to think so -- I wasn't around in the early days). For whatever reason, I was under the impression that the link between the learner and the oracle needed to be "pure", i.e. learner <-> oracle <-> everything else. However, this doesn't need to be the case (and your argumentation really helped me by wondering "have we been doing this wrong all the years?"). You can easily construct valid chains like learner <-> counter <-> oracle <-> counter <-> ..., so you can still extract the "raw" learner performance even if your oracle does all kinds of optimizations. So yeah, it was really just a brainfart on my end. Which leads me to the question: why stop there? With your reasoning, you could also argue that oracles should be caching by default since it is another low-hanging fruit of optimization. One reason to keep it separate could be the increased memory requirements. I wonder whether there are similar "pitfalls" for re-ordering (maybe an expensive Another point to consider is that your use-case (of Mealy machines) may be a little bit special because you use an even lower-level abstraction in the form of In my opinion, oracles (as the interface that users have to provide to access their systems) should be as simple as possible (just answer queries) and all kinds of optimizations should be opt-in. However, I also agree with you that the amount of options may be overwhelming for new users. Maybe a specific utility class (akin to the current Experiment) that just does all of the optimizations at once would be helpful? I'm with you on the idea that query ordering should become a first-class citizen rather than being buried in a cache implementation. I'm just trying to find a balance between flexibility and simplicity. |
I think the most important thing is internal consistency. The old version was not internally consistent, because it would reorder in one implementation and not in another, although neither of them explicitly declared their behavior in this regard (see my previous comment here). Your objection is well-reasoned: When deciding which optimizations to include by default within pre-implemented algorithmic subcomponents, where do you draw the line? I have no good answer to this. But I understand your desire to make the line clean, i.e. rather use no optimizations by default instead of some subset selected by unclear criteria. Providing a pre-configured environment for experiments might be a reasonable solution, but I don't have an opinion on the Experiment class. Personally, I've never used it, as I found it easier to cook up my own setup for benchmarking learners and aggregating results. Another (potentially complementary) idea could be to control such optimizations via constructor flags and have the default constructors set those flags to false. This way, the class hierarchy would not become any more complex than it already is, and the user would have a good chance of becoming aware of available optimizations during object instantiation. |
Yes, I agree.
Well, it is more or less just a glorified while-loop since a lot of the features need to be configured externally (caching, statistics, parallelism, etc.). Maybe it would make sense to refactor this class into more of a builder where you could configure It could be a little bit of a challenge to implement this cleanly because while there are already a lot of abstractions available, you need to select specific implementations for, e.g., Mealy vs. Moore caches, etc. However, this sounds promising enough to invest some time into it. I think it makes sense to extract two feature requests from this discussion:
What do you think? |
I don't know how complicated this approach would get in terms of implementation when considering all the possibilities in configuring learning setups. But in general, I think having some kind of builder would be a good idea. Users could directly see which components are part of the learning setup. This should also enable easy parallelization? However, I don't know how much of a priority this feature is. I think it's nice to have, but I don't know if it is worth the effort and the complexity it introduces to maintenance. Though it might not even need to be very configurable. Maybe it is enough to cover common scenarios. And an interface that is less general, but easier to understand, may be more useful than something more complex. About extracting query reordering: From a design perspective, wrapping it into its own filter sounds very clean. But how will users know that this optimization is available? I think there's a risk that users just don't notice such features if they are required to search through the LearnLib packages to stumble upon them. Regarding DFAs: I would argue that prefix-closed DFAs can be simulated by a Mealy or Moore machine with binary output. Might save a lot of redundant code. |
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
In LearnLib 0.17, there was a SULSymbolQueryOracle which extended MembershipOracle and it was possible to share a cache between an ADTLearner and an equivalence oracle by using SymbolQueryCache. I just saw that this cache does not exist anymore. And instead of SULSymbolQueryOracle, there is now a SULAdaptiveOracle which does not extend MembershipOracle. I am confused by these changes. Is it still possible to have cache sharing for adaptive learners without resorting to custom implementations?
All reactions