Those of you making use of documents and other file attachments in Sugar might find it of help to know that said attachments can be stored on Amazon S3 storage in place of the default upload directory. Support for S3 is built-in functionality and can be enabled via some configuration changes.
More intriguing, however, is the fact that this functionality leverages one of those gems I reference and you can take advantage of it as well. The ability to upload attachments to an S3 bucket is accomplished through the extension of the UploadStream class found in Sugar 6.6 and higher.
But if S3 support does not appeal to you, perhaps you might be interested in a slightly different extension of UploadStream.
One of the main problems of the default behavior for attachments in Sugar is that all files end up in the same folder (i.e. <sugar>/upload). Over time, the contents of this folder can become rather difficult to manage. To simplify things, we could organize its contents based on the creation date of the database record associated with the file. For example, MyFile.doc, added to Sugar on April 27, 2015 would be stored as <sugar>/upload/2015/04/27/<SomeGUID>. This approach makes the functionality much more scalable and is easily implemented.
Here is the code that allows us to do that:
More intriguing, however, is the fact that this functionality leverages one of those gems I reference and you can take advantage of it as well. The ability to upload attachments to an S3 bucket is accomplished through the extension of the UploadStream class found in Sugar 6.6 and higher.
But if S3 support does not appeal to you, perhaps you might be interested in a slightly different extension of UploadStream.
One of the main problems of the default behavior for attachments in Sugar is that all files end up in the same folder (i.e. <sugar>/upload). Over time, the contents of this folder can become rather difficult to manage. To simplify things, we could organize its contents based on the creation date of the database record associated with the file. For example, MyFile.doc, added to Sugar on April 27, 2015 would be stored as <sugar>/upload/2015/04/27/<SomeGUID>. This approach makes the functionality much more scalable and is easily implemented.
Here is the code that allows us to do that:
This comment has been removed by the author.
ReplyDeleteBeen trying to do this for ages and just saw you mentioned SugarCRM v6.6 which I assume this means this cannot be done on CE? I believe CE only goes up to v6.5. If i'm correct, do you have any idea if it;s possible to do similar on CE?
ReplyDeleteIt is not possible in an upgrade safe manner within CE because the necessary classes that provide this capability are only available in 6.6 and higher.
DeleteHi Angel. Great to meet you at partner summit. We're still looking for a good solution for module loader. Making exception in class for module loader does not work well. You run into issues where module loader is trying to find files which don't exist - particularly if you have a cron to empty upload folder regularly - which is required to keep it small & rely on s3 for actual file storage. Any thoughts ? John
ReplyDeleteHi John,
DeleteLikewise, nice meeting you at the summit.
Help me understand why Module Loader would have a problem with the exception. I am not sure I grasp why that would be the case. If you add an exception that tells the system to go to directory /blah/blah for Module Loader requests, but /yay/yay for all others, why would it have an issue at that point? I am not sure I follow.
BTW, I spoke to the customer I mentioned to you that might be interested in the Alfresco link. They definitely are keen to give it a look. Is there a place that I can direct them to that has more details on the project? Thanks!
Hi Angel,
ReplyDeleteI am trying this and it works well from incoming email attachments but it's not working for documents uploaded into my custom module (built from a documents template back in 6.x)
It appears the file is being saved in upload/2015/11/12/tmp/. but it's not moved to upload/2015/11/12/. as I would expect. Any thoughts?
thank you!
FrancescaS
I recently saw similar behavior while in the midst of demonstrating the ability to extend this class.
DeleteI need to debug the code a bit as I don't recall that happening before. I am not sure what I am might have changed in between.
Sorry about that.
This comment has been removed by the author.
DeleteCould the problem be that the final_move function in include/upload_file.php has upload dir and temp dir hardcoded? There are a few other places in this script where the dir name is hardcoded.
DeletePotentially.
DeleteI ran a quick diff to compare upload_file.php from 7.6.1 with that from the version I used when I originally developed the example (6.7) and it does look like there was a change introduced into the final_move() method since then.
This comment has been removed by the author.
ReplyDeleteWhen creating a Note from the interface the final_move function in upload_file.php is called with param $temporary = 1
ReplyDeletethe final move should move from /tmp/xyzxyz to upload/2015/11/13/GUID instead it moves to upload/2015/11/13//2015/tmp/SomeID because of that flag.
So the change between 6.7 and 7.6 may be in how Notes are saved rather than upload_file.php (I don't have a copy of 6.7 to compare to)
More digging in this rabbit hole...
final_move is called with param $temporary = 1 even without your customization, yet it works. Now I'm totally puzzled.
ReplyDeleteBack in the rabbit hole tomorrow....
This comment has been removed by the author.
ReplyDeleteNo luck, and still trying.
ReplyDeleteFrom my reverse-engineering it seems to me the file is uploaded to /tmp/xyzxyz so that it can be checked for size/type before we even attempt to create the bean.
The bean is then saved and the file is moved by the UploadFile::final_move in upload_file.php to upload/2015/12/15//2015/tmp/SomeID
Without the directory split customization the file is then moved from this tmp directory to its final resting place upload/2015/12/15//2015/BeanID and the tmp file is removed
It is this last step that does not appear to happen when the custom code above is included but I'm stuck in my reverse engineering to determine where in the code this last move is taking place.
I really need to get this working....
Solved. I can send you an extensive email with the details.
ReplyDeleteIn summary:
ReplyDeletegetSubDir() uses
$GLOBALS['_REQUEST']['id'];
and
$GLOBALS['_REQUEST']['type'];
which are not set in 7.6.1
and
clients/base/api/ModuleApi.php function "moveTemporaryFiles" does not use UploadStream to determine the destination of the final file.
Verified problem persists in 7.7.2
ReplyDeleteclients/base/api/ModuleApi.php
modules/Mailer/AttachmentPeer.php
still not using UploadStream rather use the following to retrieve upload dir filepath:
SugarConfig::getInstance()->get('upload_dir', 'upload')
Hi Francesca Shiekh, could you send us an email with the details?
DeleteWe couldn't replicate this config in our system, same problem with the dir /tmp/BeanID
Thanks in advance