The Issue
Suppose, for example, that we have a service (a microservice of course), which needs to manage files in a repository, such as a user service in which we must upload and store an avatar (or the user’s photo).
Let’s assume that our service is hosted by a cloud service provider, and it uses the provider’s file service (S3 on AWS, for instance).
Our service must ensure then, among other things, that S3 correctly saves the image, and later that it deletes it if desired.
We could blindly trust that these processes will be executed correctly and ignore any issues. As for uploading files there is no major problem, in the event of a fault we can return an exception and that’s it. We could even do the same for deletion: trust that it will work and voila, job done.
Fallback Job, the solution
How? Very simply:
- In our service we will store a flag which will confirm whether the provider’s file has been deleted correctly.
- When we execute the deletion in a normal workflow, we will wait for the provider’s response. In the event that they notify us that everything has gone OK, we will then save the flag as “DELETED”
- We will provide a PRIVATE endpoint, which will allow us to delete the records that are “NOT DELETED”
- Our Fallback Job will run every X time calling this private endpoint to delete the marked file
Code
As always, the complete and working code can be found at:
Project structure
The structure is quite similar to that of our previous articles. The only thing that has been added here is the deletingFallback folder which is an Adapter, which will allow us to abstract the file provider implementation as well as implement a mock or stub for unit testing.
Endpoints / Controllers
Function that controls the requests of the DELETE endpoint.
Main and Dependency Injection
In the file “cmd/server/main.go” we find the initialization of all the services, the log tool, the repository, the FileServer Adapter, the services, and finally the Gin framework.
Delete Service
In the file “pkg/deleting/service.go” we find the logic needed to delete users. The first thing we do is try to delete the image associated with the user and, if we do not succeed, what we do first is register the error and mark the user as “to delete”. If an error has not occurred, we delete it directly from the repository/database.
Deletion Service of users who were marked
In the file “pkg/deletingFallback/ service.go” we can find the RemoveUsersFallback method which should obtain the users that have been marked, and then for each one we will try to delete the image from the file server. If this fails we skip the iteration and move on to the next user.
Fallback Job
In the execution, all we do is validate that the execution has been successful and our service will have ended.
Main
Server Validation Running
In this method, all we do is ping the server and see if it is live (StatusCode = 200)
Fallback Service Call
This method simply creates an HTTP Delete call to the specified endpoint and verifies that the response is correct, otherwise, it will throw up a “Panic” error which will stop the execution of the service.
Tips
If the Fallback job fails, it means that there is a problem with the main service, therefore the service will keep restarting and eventually the INFRA team will detect it and send us a notification.
We must set out a resource “/health“, “/ping“, which we can consult to see if our service is working correctly.
We should always require special authorization for the Fallback job service, and depending on our architecture or design we may need to have it in a separate API.
A considerable improvement to this pattern would be to use message queues, and perhaps in a future article we will look at how.
About the Author
Mauricio Bergallo is a Systems Engineer with extensive knowledge in a variety of programming languages. Great experience and understanding with all aspects of the software development life cycle.
