Removing files from a source system after a transfer
Post any question you may have in regards to GoAnywhere Director and let our talented support staff and other users assist you.
1 post
Page 1 of 1
- Posts: 1
- Joined: Mon Dec 12, 2011 12:17 pm
I am trying to port some existing scripted file transfers over to Go Anywhere Director (ver 3.6.9).
One particular requirement we have is to remove files from the source system if and only if they have been delivered to the destination.
Given that multiple files are fetched if they match a wildcard the temptation would be to simply use a delete within a file transfer to the source system using the same wildcard as was used to transfer those files.
However this approach has a critical race problem in that if new files that match the wildcard have turned up on the source system in the time between the initial fetch and the delete then the wildcard will match them too and they would be deleted without ever being transferred!
What is needed is for only the explicit files that have been fetched to be included in the list to be deleted. There is such a variable but the problem is that it consists of the full pathnames of the files as they are sitting on the Go Anywhere Server.
Is there available an option to save an explicit list of the source filenames that any wildcard matched? This would be the most flexible and safe approach.
Alternatively is there a way of processing the list of the full paths of the files as they are stored on the Go Anywhere server, so that 'basename' type operations could turn them back into relative path filenames that could be deleted from the source directory. This is not as foolproof as it does not cope with situations where you may want to collect files from more than one directory in the same transfer.
As a side issue to this I am rather disappointed that the sftp and scp 'mget' only seems to function with the simple wildcards rather than the much more powerful regular expressions. This failing means that I am forced to fetch each pattern individually whereas a scp from the command line would have been able to use shell pattern matching to do things like: scp host:/etc/{passwd,group} /tmp
Even if the target system is not Unix based it would be possible to allow arbitrarily fancy regular expressions to be used if a file list was done first from the stated base directory and then the regular expressions applied to that.
One particular requirement we have is to remove files from the source system if and only if they have been delivered to the destination.
Given that multiple files are fetched if they match a wildcard the temptation would be to simply use a delete within a file transfer to the source system using the same wildcard as was used to transfer those files.
However this approach has a critical race problem in that if new files that match the wildcard have turned up on the source system in the time between the initial fetch and the delete then the wildcard will match them too and they would be deleted without ever being transferred!
What is needed is for only the explicit files that have been fetched to be included in the list to be deleted. There is such a variable but the problem is that it consists of the full pathnames of the files as they are sitting on the Go Anywhere Server.
Is there available an option to save an explicit list of the source filenames that any wildcard matched? This would be the most flexible and safe approach.
Alternatively is there a way of processing the list of the full paths of the files as they are stored on the Go Anywhere server, so that 'basename' type operations could turn them back into relative path filenames that could be deleted from the source directory. This is not as foolproof as it does not cope with situations where you may want to collect files from more than one directory in the same transfer.
As a side issue to this I am rather disappointed that the sftp and scp 'mget' only seems to function with the simple wildcards rather than the much more powerful regular expressions. This failing means that I am forced to fetch each pattern individually whereas a scp from the command line would have been able to use shell pattern matching to do things like: scp host:/etc/{passwd,group} /tmp
Even if the target system is not Unix based it would be possible to allow arbitrarily fancy regular expressions to be used if a file list was done first from the stated base directory and then the regular expressions applied to that.
1 post
Page 1 of 1
