Many customers have noticed that some of the Google services that are integrated into SmarterMail have stopped working in previous versions of SmarterMail and the current production-ready BETA we have available for download. Some customers have also received emails from Google regarding upcoming changes to Google's product APIs. In order to reduce some of the confusion around these issues, we wanted to update customers on what is happening with Google.
Within the last year, Google has been aggressively pushing changes to their APIs. However, they haven't been very open about how restrictive these changes would be or how they would break many of the products and services utilizing Google's existing APIs. In addition, their guidelines (and there are pages and pages discussing the various API guidelines and restrictions) are general and not terribly clear as to what's a "permitted application type", plus their "restricted" scopes are overly general regarding Gmail and Drive data. Then there are workarounds that allow you to use some data in a very specific way to get around the need for any assessments, but these workarounds severely limit how the services are used. It's all very confusing and circuitous, granting permissions in some cases but removing it for a slight variation of permitted access.
What IS clear, however, is that Google is moving away from providing simple access to your data to being overwhelmingly protective in an almost "we want to make it hard for you to leave" kind of way. These changes don't simply touch the areas SmarterMail integrated, but all Google services. In our case, areas like email migration, OAuth for account verification, Google Drive integration and the use of IMAP retrieval and SMTP accounts are all affected (or will be) and countless others are as well. So, while you may think these issues are SmarterMail's, they're not: this is all on Google.
When we first built SmarterMail, we did so with the aim to make it as easy as possible for people to migrate to it. Therefore, we built in migration utilities for a number of different email services, including Gmail. Then, we built a further integration with Google Drive (and other cloud storage providers) to make it easy for customers to share files by linking to them within the body of an email. The processes for both the migration to SmarterMail and the Google Drive integration were straightforward, logical and relatively easy to implement, while still requiring customers to authenticate and approve the use of their Google products.
With their upcoming API changes, however, Google has designated specific data types and products as having data that is more sensitive than others. These "scopes" (specifically, ANY Gmail data and any Google Drive data) may require additional steps from Google to verify, and then authorize, the use of that data.
One of these new steps is a security assessment that covers things like external network penetration testing, application penetration testing, deployment reviews, and policy and procedure reviews. All of this carries a rather hefty price tag, ranging from $15,000 to $75,000.00. (To read more, see Google's OAuth API Verification FAQ, particularly the Security Assessment area.) Unfortunately, what Google doesn't make clear, or even seem to care about, is how companies like you, who run your own on-premises solutions, could afford or even perform such an assessment.
As a result of all this confusion, we're taking the drastic step of removing Google Migration and Google Drive integrations from SmarterMail, beginning with our next release, as it no longer works based on the changes Google has made.
What we should all be concerned about is how cloud services in general might be approaching access to one's OWN data, moving forward. With companies like Google walling up their services, accessing the data stored within those services, especially through third-party products, gets more and more difficult. That means limiting access to your own data via any other solutions other than what's provided to you by the particular cloud service provider. And if Google is forcing this change, others are sure to follow. If nothing else, at least for those of us who use on-premises software, this change in direction should re-enforce to us that we've made the right choice.
Now, that's not to say all is lost, at least as far as migrating your Gmail messages, contacts, calendar items, etc. is concerned. We have been digging into Google's data archive and download features, which were implemented as part of GDPR. (Thank goodness the EU put this in place! Otherwise we all could be in a world of hurt.) But to make the point about accessing your own data, when you go to actually download your information from Google, they present you with a screen that almost tries to dissuade you from actually downloading it. It's not a helpful list of what you can do with your data, it's a list of what NOT to do and how you can lose information you've downloaded if you're not careful.
If you DO decide to download your data from Google, a possible approach would be for us to build an import tool into webmail to parse that Google archive. This would essentially replace the migration utility for Gmail previously built into SmarterMail.
For the time being, keep your eyes on this blog post as we'll be updating it once more information becomes available.