Jump to content
Mikael Kalms

Replication of projects with Xlinks

Recommended Posts

Hi,

I am just starting to experiment with Xlinks. We have a main project and would like to have some of our external libraries in separate repositories, and then pull these libraries into the main project tree by creating a couple of Xlinks.

 

I do not understand how to collaborate with other people that run local servers when the main repository includes Xlinks. The combination of replication + Xlinks is what confuses me.

 

Example:

1. I create the repo MainApplication@local, set up a workspace on my workstation, and make a few checkins to it.

2. I create the repo ExternalLibrary@local, set up a workspace on my workstation, and make a few checkins to it.

3. I create an Xlink (relative, read+write) from MainApplication/External/ExternalLibrary to ExternalLibrary@local and checkin this Xlink.

4. I create two repos with matching names in Cloud, create a sync view (or two separate sync views), and replicate the contents of both repos to cloud.

 

So far so good. However, how does my colleague begin to work on this project? This used to be our workflow:

5. My colleague will browse the repos on Cloud, choose MainApplication@<org>@Cloud, and chooses to create a SyncView + create a local repo.

6. My colleague pulls changesets in the SyncView.

7. My colleague creates a workspace for MainApplication@local on his workstation, and then performs "Update workspace".

 

This will fail at step 7. My colleague will get the message

"/External/ExternalLibrary

Error: The specified repository couldn't be found:

ExternalLibrary"

 

Question 1: How should my colleague perform setup differently when he begins to work on the project for the first time?

Question 2: What if I add new changesets to ExternalLibrary@<org>@Cloud, and update the Xlink in MainApplication@<org>@Cloud, how should my colleague perform "get latest" if he doesn't know that I have modified ExternalLibrary?

Question 3: What if I introduce an additional Xlink to MainApplication@<org>@Cloud -- from /External/ExternalLibrary2 to ExternalLibrary2@<org>@Cloud -- how should my colleague perform "get latest" if he doesn't know about the new library dependency?

Share this post


Link to post
Share on other sites

Hello,

Let me provide some useful links where you can read some details about configuring a sync replication view:

http://blog.plasticscm.com/2011/08/this-is-how-i-use-synch-view.html

https://www.plasticscm.com/documentation/gui/plastic-scm-version-control-gui-guide.shtml#Chapter23:TheSynchronizationview

 

1. If you are using different repos (Xlinked), your colleague will need to create a sync view for both of them. He will need to create a local " ExternalLibrary" and " MainApplication" repos. If he only syncs the "MainApplication" repo when running an update, he won't be able to download the Xlinked content.

It's interesting that you create the Xlinks with the "relative server" flag:

https://www.plasticscm.com/documentation/xlinks/plastic-scm-version-control-xlinks-guide.shtml#Chapter5:RelativeXlinks

"When Xlinks are replicated, the target they point to is still the original one. So, say that you are replicating a repository with an xlink component1 pointing to changeset number 6 on repository MyLibrary on server mainserver.location1.com:8087. The destination of your replica is on another Plastic SCM server at otherserver.location2.com:8087. When you replicate the top-level repository, the Xlink in your replicated repository will still point to mainserver. If it is reachable from the location2.com network, the data for MyLibrary repository will be downloaded from location1.com.

Most of the time, you'll want to replicate the MyLibrary repository to location2.com as well, and perform any changes in the local replica.

To tell the Xlink that you want to use the MyLibrary repository found in your default server (that of the top-level repository), you need to create the Xlink with a relative server."

 

2. When your colleague syncs the "MainApplication" repo, if the Xlink is poiting to a changeset your colleague doesn't have replicated, he won't be able to see this content. We will also need to sync the changeset of the "ExternalLibrary" library.

 

3. He will be able to download the reachable content but he will get an error message alerting that an Xlinked repo is not found. He will need to sync the "ExternalLibrary2" if he wants to download the Xlinked content.

 

Best regards,

Carlos.

Share this post


Link to post
Share on other sites

Thanks for the confirmation. My interpretation of this is: Plastic lacks some key features for doing distributed development on repositories with relative Xlinks.

The solutions above, requiring each colleague to maintain a local Sync View on his machine, are less than ideal. They require me to maintain a written sync view specification in a text document outside of Plastic and I need to notify all my colleagues every time I change it.

 

The core operation that I miss is 'replicate all that I need in order to work on project X'. Today the process is:

1. replicate primary project repo

2. Attempt to update workspace

3. If there are any errors related to relative Xlinks, replicate those repositories as well, and goto 2

... And repeat until there are no errors.

That is not a great process.

Plastic has access to all the information necessary to do all this in one single step, server->server, without needing to update any workspace.

(the operation I miss is similar to, but not identical to, 'git pull' including submodules) 

Share this post


Link to post
Share on other sites

After a good night's sleep, I have a better summary of my concern:

Plastic's 'update/switch workspace' mechanisms understand and handle Xlinks automatically, both absolute and relative. However, Plastic's replication & sync view operations ignore these. The latter is a deficiency which makes distributed development clunkier than it should be. 

Share this post


Link to post
Share on other sites

Hi Mikael,

 

Well, yes, this is something that could be obviously improved, it can always be, but it works quite well as it is (you can compare it with git submodules... which simply are a pain to use).

 

We are not doing server-side Xlink expansion during replica. We could, certainly. But then it means you have to create the local repo if needed, then walk each replicated changeset tree looking for xlinks, then launch new replicas. Of course it can be done, we simply find the sync view a much simpler way to deal with all that.

Suppose you pull a branch and you get 3 new changesets. Each of the changesets can have a modification on the Xlink pointing to different changesets in an xlinked repo. Should we pull all intermediate changesets found? Only the latest? It can be a pain to pull all the intermediate because maybe you don't need them. Should we add flags to let the user choose? (and overcomplicate the operation even more?). When you pull this way, you wouldn't only require "pull" permission, you'd also need create repo (just in case), and pull permission in other repos. Should the operation fail if the xlinks can't be pulled? Be the main pull reverted? Or simply keep it and notify the error? (probably this last option is good enough).

 

Well, in short, yes, we could implement some sort of expand-xlinks option in replica, and it would be useful.

But, saying that "Plastic lacks some key features for doing distributed development on repositories with relative Xlinks", well, I don't agree with that. The sync view is super useful, lets you see what's coming before you push (even lets you diff).

Yes, I agree that in case you are constantly adding/removing Xlinks then it can be a pain. To be honest, in my experience I'm more used to see repos that, once setup, tend to be pretty stable in terms of xlink configuration.

 

So, in short:

* We must add some sort of expand-xlinks option. Won't be straightforward, though.

* I think the sync view can serve you quite well. Of course expansion would be more comfortable.

* With the raise of monorepos, I'm not a big fan of Xlinks anymore. I tend to think more and more on putting everything on a bigger repo, which makes everything simpler, and then we have options like "replica without data" (I wrote a blogpost about it last year). In fact, the plan is to go (finally!) for writable plastic-drive (probably with a cooler name) so downloads happen on demand... but that's just sharing some of the future things we hope to work on this year :)

Thanks for your feedback and support!

 

pablo

 

 

 

Share this post


Link to post
Share on other sites

Thanks for sharing your thoughts Pablo.

I understand that it is difficult to design a simple but effective "xlink expansion during replication" operation. Lots of factors to take into consideration, as you have outlined above.

As for "Plastic lacks some key features for doing distributed development on repositories with relative Xlinks" - pardon if I was being a bit abrasive. I was so used to the Plastic and its GUI "just working", that I was surprised to find that in this particular scenario things aren't also magically working. "Key features" is too strong a wording.

 

If I get to be armchair UX designer for a moment, then:

- Automatically expanding Xlinks sounds excellent, at least at a first thought

- Being able to store sync views on the server, and colleagues being able to use a shared server-side-stored sync view, would get rid of the problem of "I added an xlink and therefore all my colleagues need to edit their sync views" because then I can edit the shared sync at roughly the same time that I change the set of Xlinks -- however, I haven't thought this through and so I don't know if it solves the real-world problem well


I've thought a bit about monorepos.

On one hand, they make a bunch of things easier when it comes to collaborating across projects within an organization. On the other hand, they do not help when working across organizations (using opensource software or publishing opensource software); the replication will then become between a subfolder of the monorepo, and an entire repository on GitHub, and we do not have tooling for such replication. We'll absolutely keep an eye on how Plastic develops in this respect. (The nodata replication feature is very nice btw.)

 

Since we don't have tooling for "replicate from subfolder in project, to a repository in GitHub" and vice versa, we have to go with one Plastic repo for each GitHub repo, and then Xlinks from our main project to the opensource repos ... right? or is there a simple way to do such Plastic subfolder<->Git repo replication today? ... and we will always use absolute Xlinks (to our Cloud org) to avoid problems with developers not being sure which repos they need to replicate before updating workspace on a project.

 

Thanks for your time; we have a better understanding of the available options now.

Share this post


Link to post
Share on other sites

Hi Michael,

I had a Board meeting today, so I'm late answering.

Thanks again for the feedback and suggestions.

UX: totally right, I was already convinced, but now I see it even more important. We must expand Xlinks. The only concern here is where to put it. We have a tight roadmap (I must blog about it to share it with everyone).

syncview.conf => we can make it shared and take advantage of Plastic global config. I'm checking whether it is doable.

 

Monorepos: you are right about collaborating with open source or external projects. No other way to do it at this point.

We have one big feature in mind: subtree branches. Like branch only a certain part of the tree (with merge restrictions in order to solve it consistently - and server-side merge is part of the picture). These branches could be replicated... and then we have a solution better than Xlinks to solve the "partial repo sharing" problem. We've been thinking on this for quite a while, we are eager to put our hands on it.

 

Always great to chat with you! Thanks!

pablo

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

×