Make USEDECL emit include tags?

[metaturso]

I have been using EDEP for the past week with the Semantic version built in Emacs 24.5 and more recently with CEDET 2.0 because of its greater parser debugging capabilities.

Your grammar has only had a few hiccups in a couple of files that used var instead of public, besides that it has been churning code all day, every day, like nothing.

Anyway, the first thing that caught my eye about EDEP is the lack of include tags :)

As I was playing about with SemanticDB, I've immediately noticed that I could get EDEP to emit includes with the simple change of USEDECL to INCLUDE-TAG

use_declaration
 : namespace_name (INCLUDE-TAG $1 nil)
 | namespace_name T_AS T_STRING (INCLUDE-TAG $1 nil :alias $3)
 | T_NS_SEPARATOR namespace_name (INCLUDE-TAG $2 nil)
 | T_NS_SEPARATOR namespace_name T_AS T_STRING (INCLUDE-TAG $2 nil :alias $4)
 ;

You clearly know what you are doing, so I figure you know already about the macro INCLUDE-TAG.

I would like to know what's your rationale for not using the built-in macro for it cannot be as simple a fix like I suggest above.

As far as I understand PHPTags is there to reproduce some of the SemanticDB functionality without the need to give SemanticDB knowledge about the project files through EDE. Perhaps this is why you opted for a custom macro?

Real good work @jorissteyn. I'm really enjoying using EDEP thank you very much for it.

P.S.: this is very much related to php-mode issue #235, I will be replying to that thread next weekend.

[jorissteyn]

Thanks for the questions and PRs of course!

I did initially use built-in macros for use statements. When grouped use statements were introduced, that made it a bit more complicated because there are now several points in the syntax where the name and type can be set, so that's where the custom macros come in play. You're right those should still work if you replace the USEDECL macro with INCLUDE-TAG. But that will break the namespace resolution routines in current edep. Does emitting include-tags give you an immediate benefit I am not aware of?

Anyway, you're on the right track asking these questions, how we deal with this is fundamental to our 'ideal' semantic integration. I don't know if I can say something coherent about this right now.

The equivalent to c's include directives would be require/include statements. Use statements in PHP are similar to using directives in c++, and use-as statements behave similar to namespace aliasses in c++ (namespace X = Y;). So the problem we're solving isn't unique. The c bovine parser in semantic does not emit include tags for these statements, but custom tags ('using and 'type :kind 'alias) and handles them in the c.el support routines. Making wisent-php emit tags similar to the c parser is easy and enabling the c support routines for semantic-php (see comment in semantic-php.el above define-child-mode) will make that work pretty well (tried that), but for current edep@master it doesn't matter since it's such a hack anyway (in semantic-analyze-current-context). So we just emit a custom tag structure with the info we need, more or less how semantic c does it.

Using include tags for anything other than require/include makes little sense, one reason being that use statements are not inherited in another file. Other reasons are use-statements don't always represent file names (use Symfony as SF; new SF\Console;) and they're optional so you would only solve half the cases. So the short answer to your question: what semantic by default does with include tags doesn't suit PHPs use statements very well.

PHPTags is there to reproduce some of the SemanticDB functionality without the need to give SemanticDB knowledge about the project files through EDE

No not really. A semanticdb phptags backend is very well in line with semantics design, similar to the cscope, gnu global or javap backends. The current problem in edep is that it takes a shortcut by resolving a fully qualified name and querying that directly in the typecache, basically bypassing lookup in current scope / other buffers. But if you'd hook up a 'composer' backend for semanticdb that would definately work. There is no magic in EDE that will do that for us; (I might be wrong here!) EDE is used to manage locations of source files and configure a project, not necessarily looking up tags given a name. Looking up a file name in a c include using EDE is easy, but translating that to PHP... it's not just a matter of implementing some API (i.e. EDE).

Stevens work in wisent-php.el uses the EDE component but that functionality isn't specific to EDE, it actually works similar to the clean-analyze-context hack but without semanticdb. Maybe this is a missing component: a semanticdb backend that uses the ede-composer (ede-php-root) configuration / routines to find a tag.

I'm currently experimenting in the branch clean-analyze-context on improving how we work with typecache/scope/semanticdb. There isn't much to see there yet. The idea is to never resolve the fully qualified name "manually" (for lack of a better word) but make it work just by having everything in the scope object correctly. That means that if you open some buffers containing PHP source files, you get all the functionality (completion, jump) without any custom semanticdb backend. Namespaces without braces were the biggest hurdle there, which work nicely now. If that works out, the phptags and composer semanticdb backends would truely be optional additions, and semantic-php (parser+support routines) would be very useful standalone (which would be awesome when merged in CEDET).

If any of my above rambling sparked some ideas or more questions don't hestitate to ask. I'll try to answer succinct next time :). I also like having issues for questions and discussions, so don't hold back!

[stevenremot]

Hey @jorissteyn,

The c bovine parser in semantic does not emit include tags for these statements, but custom tags ('using and 'type :kind 'alias) and handles them in the c.el support routines.

Actually Eric Ludlam told me the 'using tag is handled directly in the semantic internals. I tried to use it and this simplified my code a bit, but I could not make it work when there with the syntax use ... as ...;, when there was an alias part. I did not really figured why...

Stevens work in wisent-php.el uses the EDE component but that functionality isn't specific to EDE, it actually works similar to the clean-analyze-context hack but without semanticdb. Maybe this is a missing component: a semanticdb backend that uses the ede-composer (ede-php-root) configuration / routines to find a tag.

Actually I am writing a semanticdb backend for ede-php-root ! This in the branch ede-php-root-separation of my cedet repository. I took your code as an example, so there are some similarities ;-) . I planned to tell you when it is fully functionnal enough to put it in a separate repository, so that we can make sure our two backends does not conflict each other.

The idea is to never resolve the fully qualified name "manually" (for lack of a better word) but make it work just by having everything in the scope object correctly.

That's a good idea, and if it can help I think what you want to do is close to what I did on my repository. I override semantic-get-local-variables to add arguments, $this, self, parent and function local variables (but my method for importing function local variables is bad), then I override semantic-ctxt-scoped-types to dynamically include tags in the scope.

[stevenremot]

Hey,

just an update to tell you I finished writing the semanticdb backend for ede-php-root! I extracted all the autoloading facility in a new project: https://github.com/stevenremot/ede-php-autoload

It is just a minimalistic backend that uses the current ede-php-autoload-project (new name for ede-php-root-project) to find tags for semanticdb-find-tags-by-name.

It will be compatible with your code as long as you don't enforce the phptags-backend anywhere and let semantic decide what to use by itself.

[jorissteyn]

Thanks for the updates and sorry for my late reply.

I've tried the EDE component and it works very well, nice work! I tried it earlier this week and found a few bugs related to automatically configuring the ede project from a composer.json. But you seem to have fixed them all in the commits you made yesterday :) I can now ia-fast-jump to any class in Symfony using just ede-php-autoload.

I have a minor improvement to make it support dev dependencies, I'll do a PR for that soon. Another thing I found cumbersome is that you have to specify an arbitrary file (:file) in order to configure a root directory. Is that something EDE requires or can we change it?

I'm still working on splitting EDEP into separate components. Eventually, both ede-php-autoload and phptags.el should drop the mode-local override of semanticdb-typecache-find so they both optionally work together with semantic-php. Before then, they won't work together and it wouldn't make sense anyway.

See you around!

[stevenremot]

I'm glad it works nicely now! Did you use it with your wisent-php?

The :filekeyword is required by EDE, I don't think I can get rid of it, sadly.

I got this mode-local overrive from your code, and when I copied it the reason of its presence was really clear, but now I can't remember why this is there. Could you recall me what problem was solved by this override please ?

[jorissteyn]

Did you use it with your wisent-php?

Tried it with both.

what problem was solved by this override

Well that incidentally has to do with include tags, so we're back on-topic :)

Semantic has two search modes: a regular search will find tags in the current buffer and its included tags. And a deep ("brutish") search finds tags in all databases as configured by the find-throttle and semanticdb-project-system-databases.

When analyzing context and scope (semantic-analyze-*, semantic-ia) and semantic needs to lookup a tag, it does a regular search. For completion (semantic-complete-jump, ...) semantic does deep searches. This works well for C but since we can't reliably emit include tags we need to do more than the regular search in almost all cases. By falling back to our custom database in semanticdb-typecache-find we get that behaviour.

It don't think it's the most appropriate place to do this, especially when having multiple custom databases, but the typecache gets searched every time semantic does a regular search, so it's convenient.

I plan on removing the override and simply always do a deep search. Like this:

(simplified current implementation)

(define-mode-local-override semanticdb-find-translate-path php-mode (path brutish)
 (if brutish
 (semanticdb-find-translate-path-brutish-default path)
 (semanticdb-find-translate-path-includes-default path)))

(what I plan to change it in)

(define-mode-local-override semanticdb-find-translate-path php-mode (path brutish)
 (semanticdb-find-translate-path-brutish-default path))

[jorissteyn]

I wasn't planning on making EDEP work with ede-php-autoload, since phptags is so hard-wired and another backend besides phptags makes little sense. But I did it anyway. I wanted to experiment with how multiple/pluggable backends would work in semantic so I created the branch multiple-db-backends, see commit message if interested. It works great at least in the ways I've tried it, had to ignore a test failure though.

I won't merge the branch in or make any further commits. I want to focus on splitting semantic-php and phptags from EDEP, making it obsolete.

For the record: some of what I wrote in my previous message is incorrect. I got rid of both the find-translate-path hack and the semanticdb-typecache-find hack. Semantic does use system databases for non-deep searches, so no problems there. The whole db stuff in semantic isn't very straightforward.

[stevenremot]

Thank you for your experiment @jorissteyn , I removed semanticdb-typecache-find as it is useless!

Speaking about future plans, I nearly stopped working on my CEDET fork. Not that I don't want to work on it anymore, but I would like to avoid spending time on the same features you might be implementing in your semantic-php implementation. I think it would be nice to coordinate so that we can work together on the semantic support for PHP, after you finished splitting semantic-php and phptags. What do you think about it?

It may be a bit out of place to talk to much about organisation on this issue. Do you want me to open another one for this?

[metaturso]

👍 @stevenremot. If possible we should discuss this as a separate topic.

Also, apologies for not getting back to you on this matter, @jorissteyn. FWIW I have spent a couple of intense weekends trying to get have semantic find and scan lost includes, and found little success there too.