Unix to Dos file format translation

Post any question you may have in regards to GoAnywhere Director and let our talented support staff and other users assist you.
4 posts Page 1 of 1

Trinity_Chris

Posts: 14
Joined: Wed May 21, 2014 9:00 am

Post by Trinity_Chris » Tue Aug 05, 2014 2:11 pm
Hi there,

Is there an easy way to do Unix to Dos translation (replace LF with CRLF) ?

I'm SCPing a file from a server and I cant see an option to change the format as part of the transfer as you can with FTP

Thanks
Chris

Support_Rick

Support Specialist
Posts: 590
Joined: Tue Jul 17, 2012 2:12 pm
Location: Phoenix, AZ

Post by Support_Rick » Tue Aug 05, 2014 3:12 pm
Chris,

If performing the transfer in Binary doesn't retain the formatting you want, there is a freeware program called Unix2Dos (Just Google that and you'll see it).

Upon successful upload of a file, you can call a local command (Unix2Dos) and pass the name of the file just received and it will translate all the LF to CRLF for you.
Rick Elliott
Lead Solutions Consultant
(402) 944.4242
(800) 949-4696

Jonathan

Posts: 18
Joined: Wed Jan 02, 2013 5:27 am
Location: Mapledurham, UK

Post by Jonathan » Wed Aug 06, 2014 3:40 am
Hi Guys,

I have actually done some work around this in the past using Director, Python, Unix2Dos, AWK, Perl and sed to find the quickest approach and using the Director Search and Replace task is just as quick as any other methods.

For all of my tests I used a 51MB CSV file that contained 2,823,001 lines,

Director

In Director you can just use a Search and Replace task
Code: Select all
<project name="convertlf" mainModule="Main" version="2.0">

       <module name="Main">

              <searchAndReplace inputFile="/root/testFiles/test.csv" outputFile="/root/testFiles/converted.csv" searchFor="\n" replaceWith="\r\n" version="1.0" disabled="false" />

       </module>

</project>
Code: Select all
18/10/13 12:22:47.0492 PM     INFO      Executing task 'searchAndReplace 1.0'
18/10/13 12:22:48.0865 PM     INFO      1 file(s) were scanned for '\n' and a total of 2,823,000 replacements were made with '\r\n'
18/10/13 12:22:48.0865 PM     INFO      Finished task 'searchAndReplace 1.0'
Unix2Dos

You can call Unix2Dos as a native command and all you need to pass is the input file but for my test I did not run it in Director.
Code: Select all
root@ga:~/testFiles# time unix2dos test.csv
unix2dos: converting file test.csv to DOS format ...

real    0m1.065s
user    0m0.932s
sys     0m0.112s

Python
Code: Select all
data = open("test.csv", "rb").read()
newdata = data.replace("\n","\r\n")
if newdata != data:
    f = open("test.csv", "wb")
    f.write(newdata)
    f.close()
Code: Select all
root@ga:~/testFiles# time python cnvrt.py

real    0m0.203s
user    0m0.064s
sys     0m0.136s

Perl
Code: Select all
root@ga:~/testFiles# time perl -p -e 's/\n/\r\n/' < test.csv > converted.csv

real    0m3.055s
user    0m2.868s
sys     0m0.136s

These 2 were more for fun than anything else and only apply if you are running on a Unix / Linux system

AWK
Code: Select all
root@ga:~/testFiles# time awk 'sub("$", "\r")' test.csv > converted.csv

real    0m0.631s
user    0m0.436s
sys     0m0.192s
sed
Code: Select all
root@ga:~/testFiles# time sed 's/$'"/`echo \\\r`/" test.csv > converted.csv

real    0m3.436s
user    0m3.312s
sys     0m0.096s

atarrant

Posts: 10
Joined: Tue Mar 27, 2012 7:59 pm

Post by atarrant » Tue Mar 29, 2016 11:22 am
Just wanted to point out that the Director Search and Replace option mentioned above does not validate the current format of the file like unix2dos does. If you use the unix2dos command on a dos file it checks the current format and you get "/r/n" as desired. However with the Search and Replace solution mentioned above you would end up with "/r/r/n" which is probably not what you want.
4 posts Page 1 of 1