CDT/Archive/Flexible Project Structure

Note: The contents of this page refer to design documents reflecting work in progress of Eclipse CDT features and for some features that have never been implemented. Therefore they may not refer to the current state of CDT.

Overview

The fact that Eclipse projects need to map so closely to the file system is very limiting to many CDT users. They often have existing directory structures that contain their code which don't map easily to Eclipse projects. As such we'd like to provide the flexibility to add files and directories from anywhere into their projects and have all existing Eclipse functionality work properly with them (and that's the hard part). Another problem is that often times there is too much content under the project itself. This causes unnecessary overhead in things like refreshing the workspace. It also requires CDT to allow users to set source folders, or folders that CDT should be interested for source files. A main reason behind this is so we don't spend unnecessary time indexing files that the user isn't interested in. So to summarize, we just need to make project content more flexible.

Issues

This section will (attempt) to describe how Eclipse projects currently work (currently meaning 3.3 and the little I've seen of 3.4 to this point), and the issues that C/C++ developers might have.

The platform creates a .project file wherever the project creator tells it to (new project wizard for example). We'll refer to this location as the root project directory. The content of the project is determined by the file system content under the root project directory. Note that this can be anywhere in the file system, not just in the workspace. So if you have an existing code base somewhere, you'd want to make the project root directory include your code base so the files you're interesting show up in views like the Project Explorer. If any files you're interested don't fall under the root directory, you'll have a much more difficult time working with those files. They won't be displayed in the project views, won't be indexed, won't be tied to your project settings (include paths, symbols, etc.).

There are several problems with this approach. Because the project content mirrors the file system, you'd have to find the shortest common path in the file system that contains all of the files you're interested in. Typically this isn't that hard to do, but in severe cases it could be the root of a drive! Here are some examples:

C:\MyProjects\CommonCode\ C:\MyProjects\ProjectA\ C:\MyProjects\ProjectB\ C:\MyProjects\ProjectC\ C:\MyProjects\ProjectD\

Assuming ProjectA uses some of the common code shared across the different projects, you'd want the root project directory to be C:\MyProjects\. But now you have ProjectB, C, D, etc. all be under ProjectA's root.

The project location issue gets more problematic when you need to work on multiple projects at the same time. When you create ProjectB, the "natural" project root is C:\MyProjects\ as well. But you can't create it there because the .project file (and .cproject file, etc) from ProjectA are already created in that physical directory. See https://bugs.eclipse.org/bugs/show_bug.cgi?id=78438 for more info. Your only choices are:

1) Make the root C:\ - I think you can understand why this is not a good choice  :) 2) Make the root C:\MyProjects\ProjectB - now you're missing all of the common code under Project B

One might ask at this point why you don't just have one Eclipse project for all of these projects rooted at C:\MyProjects\. There are several reasons for this. Each of these projects A-D were divided in the first place for a reason. They each have their own build artifacts and project properties like include paths and symbols. What would Eclipse's builder actions like build and clean do? C/C++ projects just don't fit into that model.

Another counter argument might be that C:\MyProjects\CommonCode should be made its own project and each of the other projects be made to depend on it. This patter is a very valid one but is not appropriate in all cases. Today, a dependent project must create a library of some sort to be included into ProjectA. If the project is preexisting or is used by multiple IDE/development flows, this may not be feasible. Also the pattern of "Poor mans library", where a copy of the source is included by reference into multiple projects is very simple to understand and can be found in many education and tutorial situations. This pattern keeps the focus on the code and not the build process.

Another issue with .project files is that some users will want to share these in version control along with their code base, while others will not want them to "contaminate" their code base. Many teams have developers using different IDE's, and most other IDE's allow their project files to be created anywhere in the file system. This allows the user to decide if they're needed in version control or not. With Eclipse, it must reside in the code base. It should be noted that most version control systems have "ignore" capabilities by file extension - so this would seem to not be a big issue. But we've run into at least one Eclipse plugin from a version control provider that simply deletes files it's not aware of. Things don't work really well when the .project file is deleted out from under you!  :)

Background

The Eclipse platform tried to address some of the issues with project content flexibility by introducing linked resources. In many cases this was just what the doctor ordered. But there are several things about linked resources that aren't so good.

1) This is by far the most serious of the issues, and the main reason we feel that linked resources cannot be used at all. The team support simply doesn't work with linked resources. Right now it flat out ignores them. We could potentially fix this in the CVS team support, but what about all of the third party plugins out there? It's hard to ship and IDE without team support, and users just use a wide variety of version control systems. I don't think linked resources can be taken seriously as an option for project flexibility until at least all of the major version control suppliers support them in their plugins.

The above limitation is not an issue if the linked resources are links to resources already under VC in another project and if the .project file (which records linked resources) is under VC in this project. However this does not solve the general case of a flexible directory structure for the project itself.

2) You can't make file system changes with links. For example, if you want to rename a file or folder, or move a file around, you can't do this with linked resources. It only changes the link itself, not the underlying resource.

3) Creating new resources in a project with links is confusing at best. Let's say you have a project with a linked folder and file at the root. If you create a new file or folder at the root, it is created in the workspace, not where the other folder/file are in the file system. But if you create a new file under the linked folder, it gets created where you'd expect.

4) Linked resource paths maybe stored in the .project file as absolute paths. This makes it very difficult to share .project files even when not using team support within Eclipse. This issue can be resolved by using the existing Path variables facility. However this requires each user to set these varibales appropriately for their machine/workspace. A better facility to import these settings into a workspace is needed.

Possible Solutions

There are several possible solutions that have been suggested:

1) Linked resources

It's not clear if this can work or not - mainly due to the team support problems mentioned above. Even if we could, it doesn't do anything for the .project file location problem. It does for those that don't want the project files checked into version control. But for those that do, they'll have the same problems they do now.

2) EFS

We could potentially implement a EFS file system that allows you to add and exclude files and directories from anywhere using URLs. This file system then presents a logical hierarchy to the IResource tree and handles the parent/child mapping. The issue we run into immediately is that there is a lot of code in the CDT (and elsewhere?) that assumes that the parent/child mapping in the IResource tree matches the physical file system and cheats by not asking EFS for that information but going directly to java.io.File.

There also may be a lot of code that uses IResource#getLocation rather than IResource#getLocationURI, which would at least appear to be a problem. Files are actually first class citizen resources in this model at least (not linked that is). So in theory version control would work. We could potentially used the new "hidden" resource attribute for hiding resources even when using EFS. This is available in Eclipse 3.4 and would appear to be all we need to be able to hide files/folders under a project that we're not interested in.

Preliminary investigation leads me to believe that EFS is not a workaround for the .project file location issue. In theory it sounds possible, but there's no way to determine if the file being requested in FileStore#getChild is a temp "project" file, or some file being created by the user. A crude filename test could be done I suppose, but there are many more of the temp project type files created. For example, CDT creates a .cproject file, and any other plugins could be creating their content in the .project file location. We'd want all of these types of files to be created in some arbitrary location, and then the content of the project to be fetched from natural project root.

3) Allow relative paths in .project and allow a specified root path.

If .project allowed folders to be defines such as ../MyCommonCode and also allowed the base folder to be set to some relative path, much of this problem would go away. Resource filters now appear to have general include lists as well as exclude lists and this facility could be used to get only the portion a parent directory needed. This also allows the eclipse specific files to be in a eclipse subdirectory of the root of the project or in a peer directory to the real project directory. This allows eclipse support to ride along with other IDE support or to be grafted on to an existing project that would never tolerate eclipse specific files (think GCC or the Linux kernel).

This change would almost certainly not be limited to CDT and would impact the base platform. Although C & C++ user may be the most particular about directory structure, the issues here are really caused by the assumptions of the platform code itself and ultimately should be solved at that level. I would imagine there are a number of PHP, JavaScript, Ruby, etc projects that could also benefit from this support.

Other Considerations

Assuming we can find a way to make projects more flexible, this section lists some possible things to think about from a CDT perspective.

Should we do away with source paths? If everything under the project is indeed of interest, then do we still need source paths? Perhaps you may still only want to index a subset of the project. Or for managed builder, you only want to build the content of certain folders. Are these the same set, or should these be controlled separately?

What type of UI would we need to hide/unhide resources?

To improve the process of setting path variables per workspace: Settings could be imported from user/machine/site settings when a new workspace is created or when a project is added to a workspace. Alternatively, a product could override the "default" scope for preferences to consult user/machine/site definitions from preference files, environment variables, or registry settings.

Comments

Discussion pages rarely get looked at so I have created this section.

2011/09/04: wmills

This page has not been updated for a few releases. It would be great if an expert could update it with current status. From my POV I believe the issues discussed here to still be a problem. I have added some edits above but they could do with review.

I am no expert. I have been an occasional & frustrated user of eclipse for ~ 5 years. About 50% of my frustration comes from the limitations on resource handling. I have read much more of the plug-in developers guide that the normal C users and played with RCP a bit.

The issue with linked resources using absolute paths can be resolved by using the existing Path variable facility. I have added a suggestion to make relative paths legal in the platform resource code itself.