Page 1 of 1

Integrating GAMFT with CEPH storage cluster

Posted: Fri Feb 15, 2019 11:11 am
by Support_Josh
Question:
Can GoAnywhere MFT integrate with CEPH cluster storage? We have a local CEPH storage and would like to use it instead of an S3 resource. CEPH is becoming more and more popular as an alternative to expensive Amazon buckets or on-premise storage solutions.

Answer:
I can confirm that GoAnywhere does work correctly/as designed with CEPH storage (Amazon compatible) buckets. Although we do not setup the initial CEPH cluster, we can integrate it with MFT and use an Amazon S3 Bucket resource to connect to the CEPH cluster.

To create an S3 Bucket from CEPH:
# s3cmd -c /root/.s3cfg mb s3://FTP
WARNING: Could not refresh role
Bucket 's3://FTP/' created
Note: You can also use “—configure” option if you don’t already have a configuration file. This file contains the access keys and the information that defines the bucket.

To upload a file to your bucket:
# s3cmd put anaconda-ks.cfg s3://FTP
WARNING: Could not refresh role
upload: 'anaconda-ks.cfg' -> 's3://FTP/anaconda-ks.cfg'  [1 of 1]
1245 of 1245   100% in    1s   682.61 B/s  done
-OR- 
# s3cmd put /tmp/sensu_client_loaded_files s3://FTP
WARNING: Could not refresh role
upload: '/tmp/sensu_client_loaded_files' -> 's3://FTP/sensu_client_loaded_files'  [1 of 1]
29 of 29   100% in    0s     2.32 kB/s  done

To list the files in your bucket:
# s3cmd -c /root/.s3cfg ls s3://FTP
WARNING: Could not refresh role
2017-11-27 19:59      1245   s3://FTP/anaconda-ks.cfg
2017-11-27 20:00        29   s3://FTP/sensu_client_loaded_files

Your CEPH server should be running a RADOS gateway (A kind of webserver) and all necessary firewall ports should be open:
# lsof -i :7480
COMMAND  PID USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
radosgw 5131 ceph   27u  IPv4  43714      0t0  TCP *:7480 (LISTEN)

Once you have a working bucket and gateway, copy your keys from the CEPH configuration file into a GoAnywhere S3 Bucket resource.
CEPH S3 Resource.png
CEPH S3 Resource.png (25 KiB) Viewed 5136 times
CEPH S3 Resource2.png
CEPH S3 Resource2.png (12.41 KiB) Viewed 5136 times
Note: RADOSGW does not support HTTPS so some people use a TLS proxy. We recommend bypassing it until you have things working.

Once the resource is defined, you may use the resource within any S3 task as a normal S3 resource.

The connection outlined above was made was using the Bucket-Owner which is seen as bad practice for security reasons. We recommend to create sub-users or IAM users and then copy their keys into the resource.