Look at factoring out all the conditionals on use_relative_paths. (Since the ZSyncer itself has persistent configuration, this might require moving much of its functionality into a new class that does the actual syncing and which would be instantiated on the fly.)
Need to audit all remote methods and ensure that they consistently trap the right errors, return meaningful response codes, etc. And check the local callers for consistency too. I think it’s kind of a hodge-podge now.
Also clean up the UI to more consistently use StatusMsg and response streaming. Probably some refactoring to do there.
Our separate logfile should be reimplemented using the python logging module. (It would also be nice if there were a config option to mix these in to zope’s standard event log.)
It would be good to refactor the tests so that:
These should be moved to the Sourceforge collector.
If you sync across multiple timezones, everything should “just work”. There was a problem prior to CMF 1.5.1 in that the folder_contents view defined by CMFDefault uses the Date() method of each object, which displayed the DublinCore modification time according to the timezone in which the object originated, but does not display the timezone. In CMF 1.5.1 this was fixed to convert the date into the local timezone before display.
The “status” display can easily be wrong. We rely on bobobase_modification_time for objects that don’t provide DublinCore timestamps. This is not good. Among other things, it means that you frequently get “out of date” for no good reason. Be suspicious of the status. Again, we recommend that you keep the clocks on your servers in sync. This will greatly reduce the likelihood of false status reports.
“status” display for FilesystemDirectoryViews is meaningless. In debug mode, zope updates their bobobase_modification_time periodically.
Big syncs: Transferring a lot of stuff fails because ZSyncer does everything in memory. E.g. if I sync a folder containing ~70 MB of stuff, the destination server’s ram useage shoots up by 300 MB!!! Note that import of large .zexp’s seems to succeed in cases when zsyncer fails.
This may be somewhat mitigated since we got rid of XML-RPC; should test it again.
Things are also somewhat better since we switched to using a tempfile instead of StringIO when syncing something larger than Config.upload_threshold_kbytes. Still haven’t done this on the client side for uploads
Further possible optimization: for pull only: Use a separate thread (or process?) to build the pickle to a temp file, and send it back in chunks with a content-range with an empty total size. Or - probably better - _p_jar.exportFile() can write to a TemporaryFile; we could use a zpublisher IStreamIterator implementation, maybe the default one would just work.
For push, there doesn’t seem to be anything like content-range for POST. So I probably have to start uploading after building the whole file. (One possibility to investigate: connection: keep-alive might help? or not.) But I can still be nice to the client by using httplib’s connection.send(chunk) in a loop and never have to load the whole thing into memory. (But this has to be implemented per each back-end.) OR - suggested by chris mcdonough - use a PUT request instead, which afaik means the httplib client code doesn’t have to fiddle w/ the request much at all - just hand it a file and it behaves sensibly. (The server is another question... apparently it currently makes multiple copies of the tempfile, see http://mail.zope.org/pipermail/zope-dev/2005-April/024616.html)
Switching syncers while working is clunky. You either have to edit the syncer, which is error-prone, or go out of the current syncer and into another one. Instead of having a bunch of syncers configured for different destination servers, maybe have one syncer configured with multiple destinations, and in Folders.dtml allow the user to toggle which of the configured destinations are used for comparisons and syncing.
Currently with multiple destination servers, only the first one is used for comparisons. This is not very informative.
Quick solution: If multiple destination servers, allow user to toggle which is used for comparison (without changing the syncer config)
More extensive solution: provide a combined comparison mode in which each object’s status is shown on all destination servers. The status could then be any of:
ok
missing
out-of-date
ok/missing
ok/out-of-date
missing/out-of-date
ok/missing/out-of-date
alternatively, have a status column for each destination. Or maybe rows within each object’s row? This a lot of information - especially with some of the other recent enhancements like showing the mod. time for the destination - now we have to decide WHICH destination to show, or how to show all of them, or whether to show the latest only, etc. Hard to keep the UI from getting cluttered with all this info. The next item might help wit this.
status column could contain links to more detailed information, e.g. in the multi-destination case, show the usual comparison for each server in turn. This could also be used to do more expensive comparisons as described below.
Config.py should be replaced by extensions to zope.conf. Look into how that’s done.
Log dir should be configurable in the config file.
Comparison enhancements
Motivation:
Goals:
Comparison code should be pluggable somehow. Default comparison should be enhanced as follows:
any difference in DC Metadata -> not equal.
any difference in Properties -> not equal.
any difference in security settings (permission/role mapping, local roles) -> not equal.
for ObjectManagers: comparison should continue to be atomic rather than recursive by default.
Recursive means that if foo/bar/bat/baz is considered different, then foo, bar, and bat would be considered different even if there are no other differences. Atomic means that we ignore foo._objects. Atomic is both easier and cheaper. Also, recursive would be confusing since you don’t know how much of the tree you need to sync to fix the problem.
A better solution to recursive comparison is already implemented in the UI.
Ideas for additional comparisons:
There are many potential comparisons but many are likely to be too expensive for routine recursive use. this should be investigated. One option would be to give the user control over which comparisons are performed.
improve timestamp comparison: I don’t think this is possible for non-CMF objects, for the reason given below.
this was my strategy:
dest_time = remote call to DateTime()
local_time = DateTime()
fudge_factor = dest_time - local_time
when comparing times:
if dest_modtime <= source_modtime + fudge_factor:
status is OK
PROBLEM: if the clocks on the two systems have changed since dest_modtime or source_modtime, this doesn’t work at all. And for hypothetical two-way syncs, it’s useless even if the clocks are right.
Somebody on zope@zope.org suggested that zsyncer should forcefully set the mod time of the source & destination objects so that they match. This is not possible since bobobase_modification_time is database-level and zope can’t control it.
CMF content is easier since it has real metadata: we can ignore bobobase_modification_time and use the DublinCore modification date as our timestamp. When comparing, we’re OK only if they are identical. This should work very reliably. DONE!
add a new status (“Changed”? “Newer”?) if the timestamps look ok but other comparisons indicate a difference. (should this be red, orange, or other color?)
compare size - easy and cheap, will catch many differences, but ObjectManagers and many other objects don’t support it or might have a different interface for getting it. (get_size vs getSize, maybe others)
compare metatype: this always works. unfortunately it’s unlikely that we have 2 different types, but it does happen. so this is probably only rarely useful as a way to avoid more expensive comparisons.
compare all DublinCore metadata if available. Again, mostly useful as a short-circuit; even so, comparing DC modification time should prevent the need for this.
simple comparison of the object’s “content”, whatever that is - e.g. body of a DTML Method. Not useful for all types. Potentially too expensive on the network and not useable for all types - would need to be toggleable. Should only be done after a meta_type check.
recursive sync: a new (toggleable) sync mode, in which zsyncer tries to walk down the tree and automagically do the right thing depending on status: sync only missing or out-of-date things (don’t try to descend into missing folders, just sync them). This could leave some out-of-sync objects if the comparisons are wrong, but it could also be a big win when syncing a few scattered changes in a huge folder tree. Result should report only the items that were actually synced.
There should always be a confirmation page in this mode, that shows you what would be synced and allows deselecting items.
Hmm, this is not really necessary if I just fix the filtering in recursive comparison mode.
Aspect-based syncing: Currently, sync is all-or-nothing. What if you have a 200 MB file and change its permissions, or its properties, or its CMF metadata? You have to sync the whole thing. Worse, what if you have a huge portal_skins folder, with some stuff you don’t want to sync yet, but there are changes to the skin paths (properties) that you need to sync ASAP? Another use case: Access Rule settings on a folder.
This is not a YAGNI - I have needed it for months :-)
one problem is making this extensible to handle more features, e.g. custom products. maybe just register the attributes that are considered “content”.
The new _callRemote() method is intended to be useful here.
meta_type) as unsyncable. On the source, this would disallow syncing it; on the destination, this would disallow manage_deleteRemote. (OK, that’s really 2 flags.) The UI should tell who marked it, when, & why. The rationale is similar to the “immutable” bit in unix. The syncer itself could store these as an OOBTree whose items are path: flag. This could get stale if objects move. Or, set properties on objects, like zsyncer_unsyncable (boolean); but that might annoy people, and requires the object to implement PropertyManager. TANSTAAFL.