FileSender Cloud Backends
FileSender 2.0 can store the uploaded file data in the cloud. Support is currently being developed for storing data in Azure blobs and Amazon S3 is another possible cloud target.
It was reported that performance with the S3 backend (and maybe others) can be greatly increased by bumping up upload_chunk_size and related values. For example using 40mb chunks:
$config['upload_chunk_size'] = 1024 * 1024 * 40;
$config['download_chunk_size'] = $config['upload_chunk_size'];
$config['upload_crypted_chunk_size'] = $config['upload_chunk_size'] + $config['upload_crypted_chunk_padding_size'];
Increasing upload_chunk_size also requires you to verify PHP’s memory limit is equal or bigger than about 5x upload_chunk_size.
Azure
At the moment a new container is created for each file and files are uploaded to multiple blobs which are upload_chunk_size bytes in size. This is the CloudAzure storage_type. If other styles of upload/download are desired there might be other Azure storage types created for FileSender to allow easy choice of schema.
Setup
Azure support uses the PHP bindings. These bindings can be installed through composer into the subdirectory of optional-dependencies which is already setup to help you. FileSender is setup to look into the optional-dependencies/azure directory for these supporting files if you enable Azure in your configuration.
cd optional-dependencies/azure
.. download composer.phar and check it ...
.. see https://getcomposer.org/download/ ...
php composer.phar install
Azurite: a local Azure server
You may wish to test against the free and open source Azurite server clone. This can be useful not only for development and CI but also for many other reasons. Having a local Azure server can let you test if FileSender is working with your PHP libraries and allowing upload and downloads etc without having to hit the network. If a bug can be replicated using Azurite then it allows developers to replicate the issue without knowing about your exact cloud setup.
While you can use the -g “global” option as suggested on the azurite site, another option is to just create a new directory and download azurite as a normal system user into that directory. The code will be installed into a subdirectory node_modules.
$ npm install azurite
$ mkdir data
$ cd node_modules/.bin
$ cat start.sh
#!/bin/bash
export http_proxy=
export https_proxy=
unset http_proxy
unset https_proxy
cd ./node_modules/.bin
./azurite -l ../../data
$ chmod +x start.sh
$ ./start.sh
Configuration
The main configuration keys for Azure are the connection string to use to connect and maybe other settings to use to store your data. You might like to store the first in the optional config-passwords.php file to keep it separate from your config.php file and reduce the risk of accidentally sharing sensitive information.
The config-passwords.php has the same format as config.php but allows you to keep passwords (databases, clouds, etc) out of the main config.php file.
You might like to also have a switch in your config-passwords.php to allow local testing without opening or editing the config-passwords.php file. One option here is to setup the test config in config-passwords.php. Since the test azure server is for local use only and on 127.0.0.1 you can also put that into your main config.php and only include the sensitive ‘real’ server setup in config-passwords.php. The config-passwords.php file is loaded after config.php so you can set a personal setting in config.php to switch to a test connection string in config-passwords.php. Note that you have to use Config::get() to access your sysadmin_setting_testcloud setting in config-passwords.php.
This has the added plus of not having to open config-passwords.php to
see the local only test connection settings. Note that there should be
no closing ?>
in either the config.php or config-passwords.php
files.
# cat config.php
... since this is a local only test server, security is less interesting
$config['cloud_azure_connection_string'] = 'DefaultEndpointsProtocol=http;AccountName=devstoreaccount1;AccountKey=Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw==;BlobEndpoint=http://127.0.0.1:10000/devstoreaccount1;';
$config['cloud_azure_other_settings'] = 'something public is ok';
$config['sysadmin_setting_testcloud'] = true;
# cat config-passwords.php
... the less you have to see this file the better ...
<?php
if( !Config::get('sysadmin_setting_testcloud') ) {
$config['cloud_azure_connection_string'] = 'DefaultEndpointsProtocol=http;AccountName=devstoreaccount1;AccountKey=Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw==;BlobEndpoint=http://myrealazure-provider:10000/devstoreaccount1;';
}
Running a test
To help check if your setup can connect to a cloud server and do something of use a small test script has been created. This script will create the container if it does not exist already and write a blob to the server before reading the data back again.
It is recommended to run this test against an Azurite server first and then turn on your real credentials to run it against your real cloud provider.
cd scripts/cloud/azure
php test-azure-simpleblob.php
Use as a storage provider in FileSender
Now that you know that your configuration and software setup can talk to your cloud provider you can switch over your storage_type to start storing the file content in the cloud.
$config['storage_type'] = 'CloudAzure';
Amazon S3
Setup
Amazon S3 support uses the PHP bindings. These bindings can be installed through composer into the subdirectory of optional-dependencies which is already prepared to help you. FileSender is setup to look into the optional-dependencies/s3 directory for these supporting files if you enable S3 in your configuration.
cd optional-dependencies/s3
.. download composer.phar and check it ...
.. see https://getcomposer.org/download/ ...
php composer.phar install
For more information on installing these bindings see Amazon’s page.
Zenko CloudServer: your local S3 server
Zenko CloudServer is an open source S3 server implementation. This can be useful for testing and to see that the FileSender code is working as you expect. Installation of CloudServer is fairly simple and is also shown below.
git clone https://github.com/scality/S3.git
cd ./S3
npm install
npm start
Configuration
The main configuration keys for S3 are the connection string to use to connect and maybe other settings to use to store your data. You might like to store the first in the optional config-passwords.php file to keep it separate from your config.php file and reduce the risk of accidentally sharing sensitive information.
The config-passwords.php has the same format as config.php but allows you to keep passwords (databases, clouds, etc) out of the main config.php file. If the config-passwords.php file exists then it is loaded after config.php.
See the configuration section for Azure above for information about how you might setup the config.php and config-passwords.php so that you can avoid opening the later file.
$config['cloud_s3_endpoint'] = 'http://localhost:8000';
$config['cloud_s3_key'] = 'accessKey1';
$config['cloud_s3_secret'] = 'verySecretKey1';
// optional, Ensure that the bucket exists if you want to use a
// single bucket.
// $config['cloud_s3_bucket'] = 'filesender';
// optional, if you wish to have a bucket-a-day configure these options
// please note that the buckets are created/deleted via cronjob, so make
// sure you are running cron.php regularly!
// after turning this on, create the daily buckets with
// php scripts/task/S3bucketmaintenance.php --verbose
// $config['cloud_s3_use_daily_bucket'] = true;
// $config['cloud_s3_bucket_prefix'] = "FileSenderDaily-";
Running a test
To help check if your setup can connect to a cloud server and do something of use a small test script has been created. This script will create the bucket if it does not exist already and write a 30 byte test data packet to the server before reading the data back again.
It is recommended to run this test against a local CloudServer server first and then turn on your real credentials to run it against your real cloud provider.
cd scripts/cloud/s3
php test-s3-simple.php
Use as a storage provider in FileSender
Now that you know that your configuration and software setup can talk to your cloud provider you can switch over your storage_type to start storing the file content in the cloud.
$config['storage_type'] = 'CloudS3';