Category Archives: Development

Git Basics: Pull Requests

By Agustin Aliaga – Developer at Santex

Git is a powerful version-control system designed to make software development collaboration easy. It can be used for personal (single contributor) repositories, but it really stands out in projects where multiple developers modify the same codebase every day. Multiple branching models can be adopted. However, in most cases, a Pull Request (PR) will be created when a bug fix or a new feature is ready to be merged into the main branch. The PR itself is an attempt to merge specific changes from one branch to another. Let’s explore the Pull Request workflow in detail.

PR Creation

Whenever a PR is created, the following good practices should be considered:

  • Adding a helpful description to the PR that answers the following questions: What is the requirement or bug? Is there a link to the issue (e.g. on JIRA)? How does your code fix it? Is there anything else the reviewer should take into consideration?
  • Making small, consistent and logical PRs: We want our PR to be merged as soon as possible. Imagine how hard it would be to review hundreds of file changes at the same time. If this happens, it is likely that the reviewer won’t even have the time to do it properly. So try to keep it simple and concise.
  • Making sure the PR’s metadata has been set. For example, by assigning the reviewer (to trigger a notification), setting the assignees, adding a label (if needed), etc.
  • Configuring the repo’s branching security settings to keep developers away from self-merging their PRs or pushing to a shared base branch. I recommend enforcing Github’s branching security settings. When a large team of devs is collaborating in the same repo, accidents may happen. A single miss click in a “merge” button can cause terrible headaches. Protecting important branches is something I’d advise most of the times.
  • Avoiding submissions that include unresolved conflicts. Fixing those conflicts shouldn’t be a reviewer’s responsibility. PR creators should dedicate time to solve them either by using Github’s conflict resolution tool (when it’s simple enough) or by using your favorite diff tool locally. You can achieve this by pulling the base branch and merging it into your feature branch. After you pushed it to the remote repo, the PR will automatically update.

Automated tests

If a Continuous Integration system (such as Jenkins, Travis, or CircleCI) is set up, a bunch of hooks can be configured to run unit tests on PR creation. This will allow the team to detect and catch bugs rapidly, as well as prevent conflictive code from being merged. This is a long topic that requires its own blog post, so I’ll just move on to the following stages.

Code Review

After everything related to the creation of the PR is dealt with and the CI tests passed, it is time for a code review. Projects tend to have tight deadlines, which sometimes makes code reviews seem like a waste of time by other team members or external agents. Ironically, code revisions actually help to increase productivity and reduce rework because we avoid bad practices in our code and share knowledge between contributors.

Some benefits of implementing code reviews are:

  • Less experienced devs get to learn from their mistakes.
  • Experienced developers consolidate their knowledge as they teach others.
  • A high-quality codebase is ensured.

Humility: a soft skill that matters

Sometimes, as work starts to accumulate and things get tense, some engineers tend to become less aware of their attitude towards their peers. It is always important to avoid egocentric behavior, listen to our co-workers, and moderate our communication when reviewing other people’s work. If we write a PR review in a disrespectful/arrogant manner, we could be damaging the team’s confidence and the work environment.

The “reviewer” role

Ideally, teams should implement peer reviews. This means that anyone should have the required experience and skills to review other’s code. In practice, collaborators regularly have different levels of experience on both the technology used for the project and the codebase itself, including its set-up, architecture, code styling, deployment, etc. In this case, experienced developers should be conducting the reviews and newer team members should be included progressively as they get comfortable with the project.

Merging the PR

After the PR was approved, it’s time to merge it. We have a couple of options to do so. Let’s explore them:

  • Create a Merge Commit (default): This option will merge all the commits from the feature branch plus a new merge commit. This is the safest way to perform the merge, since it is a “non-destructive” operation. The downside of using this option is that since it creates a merge commit to tying together the history of both branches, it pollutes your history tree with multiple “irrelevant commits”.
  • Squash and Merge: This option will “squash” all the commits into a single one. If the PR includes a lot of commits, this could be the way to go. It will make your history tree much cleaner and easier to read in your base branch. Nevertheless, you will lose granularity due to the bigger size of the resulting commit.
  • Rebase and Merge: The “rebase” operation is another way to combine commits from two different branches. This will put your feature branch commits on top of your base branch’s latest commit, and effectively rewrite the commit history. After this, it will perform a fast-forward merge, making it look like all the work was done on the base branch. This is extremely dangerous when the rewritten branch is public and shared with other team members. Generally, the rule of thumb is to keep rebasing operations for private-only branches.

References

 

Code documentation good practices

By Francisco Verastegui – Santex’ Technical Board Member

Code documentation is an important practice of the development process and it’s worth the effort in the long term as the application gets bigger and more complex, letting us save time and minimize the learning curve to understand the functionality of the API, libraries and applications. Here we explain 4 practices that we hope you embrace as part of your development process.

  1. Document your APIs in a simple and concise way

Libraries and APIs are made to be used by people that might not have time to read the code or just might not have access to it, so documentation should reflect your code objectives in a simple (easy to understand) and concise (focusing on the important facts) way.

  1. Keep your documentation code up-to-date

Update your documentation each time you change your code – especially if business rules are affected. Software evolves over time, and so does the code. Therefore it’s important not to start documenting too early in the stages of the code because you might be forced to change it a lot of times.

  1. Focus on the ‘Why’ not the ‘How’

The main idea of this principle is: “Your code documentation should explain the ‘Why’ and your code the ‘How’”.

Good source code can be self-explanatory, but we should give it meaning. So we shouldn’t repeat the how. The following examples explain the same method with different code documentation approaches. The examples are in Java, but we are able to apply these concepts to any other programming language as well.

Example 1

In this case, the code documentation (JavaDoc) just explains the ‘How.’ The context isn’t clear, and neither are the business rules that are the reason of the creation of the method. Basically, the documentation is providing the same information that we could get reading the code.

Example 2

In this example, the method’s JavaDoc focuses on the ‘Why,’ explaining the context and the business rules that support it. It is also important to explain the business reason behind an exception that the method might throw.

Detailed explanation

“When we are editing a recurring series”: This is the context – whether to include it or not will depend on if it is a business-related method or just an ‘isolated’ method like the ones we can find in a utility class (reused by different parts of our code).

“we have to enforce the rule that recurring {@link Order}s can’t exceed a period of more than 24 hours”: This is the main part providing the ‘Why’ because it explains a business rule and the main reason for creating the method. A method can explain, or be supported by, more than one business rule.

“If the remove TIME portion is less than the install TIME portion, then it is safe to assume that the remove date has rolled onto the next day (e.g. June 1st 7PM -TO- June 2nd 3AM, is still a 24 hour period)”: Business rule considerations are important to have a good understanding of the method behavior. To include it or not will depend on the complexity and conditions of the rule we are trying to code.

@throws OrderEditException

– if the order was already deleted by a different user: Explanation of the reason (Why) the method is throwing a specific type of exception. It is recommended to do this for any application business exception.

It is important to realize that it is perfectly possible to understand the meaning and the business implications of the method just by reading the code documentation. This is a key concept for APIs that are public and designed to be reused throughout different projects and applications.

  1. Don’t document trivial code

Avoid documenting getter or setter method (unless it does something business-related), so remove it from your IDE’s auto-generated code template. Avoid documenting simple procedures perfectly explained in reading the code. For example:

As you can, see doing this only makes code harder to read.

Good committing

By Jose Torres – iOS Developer at Santex

People in software development know there are a ton of difficulties a team faces when working on software projects. That’s why best practices were introduced as a way of preventing  common problems that software teams eventually face. Best practices cover many aspects, from the environment setup to the way we write code and produce deliverables. Unfortunately, one aspect usually forgotten or underestimated is the handle of the repository. This usually happens when the team is small – if you are the only member on the team, you are prone to get disorganized. This can dangerously hide other potential issues usually faced when the team grows and you find yourself in complicated situations or flows. As the team increases its size, a lack of good practices will reveal itself and problems will surface.

As engineers, we should understand the importance of doing things right. It shouldn’t matter that our current context shields us from facing complex situations. We have to be prepared for when the context changes and things start to scale. It happens often that engineers have this feeling of “things are working fine,” so therefore the context is “that should mean we are doing things well,right?” Not facing issues in the moment is not a guarantee that things are being done right.

There are many areas of good practices surrounding Software Configuration Management. Let’s focus on some of the best practices when committing code.

When committing code, we are introducing new code changes to our code base. In distributed version control systems, this is done first in our local repository and then the code is applied (“pushed”) to the central repository. There are multiple approaches on how to handle the branching model of a repository, but we will focus only about the committing. It is basically a general rule that a commit in code should reflect an atomic change. This means that it can only be applied entirely, never partially. And, of course, it should not break the build. All build routines and tests should run successfully.

When committing code, we also have to include a commit message. This message is meant to give more information about the code changes that have been introduced. Such a message will last through eternity, and it is our duty to use it wisely.

Image 1: Sample git log

Limit your commit first line (title) up to 50 characters

This is one of the most common suggestions, but it is still forgotten. The first line is treated as the title of the commit. This means it should be brief and concise. It must be no longer than 50 characters, and ideally should not be too short either. This characters length may not be a hard limit, but having it in mind ensures that the author thinks for a moment on how to concisely describe the commit. This helps to get quick contextual information when navigating through the git history and ensures that all clients will display the commit info properly.

Consider a blank line before the description

In order for Git to differentiate the title from the body of a commit, a blank line must be added. Of course, this is only required if we want to add a description. Whether a description is needed or not is up to the author. However, it is a good practice to include a description that adds an explanation of the changes introduced and also some contextual information, like a related ticket number or link.

Include a link related to the issue on Bitbucket/GitHub/JIRA/etc

This is related to the previous suggestion. It is important to have a reference to the source of the request, so we can have more context of the change overall. This should not be part of the title of the commit, but rather part of the description with the full link. If it is a bug fix, requirement, enhancement, etc., most of these are tracked with software development tools that assign them a unique identifier which makes it easier to track development. On one hand, it helps each commit in the repository to have a full reference. On the other hand, it gives visibility to external tools about when, by whom and in which branches the changes are applied.

Write the message using present tense/imperative mode

Finally, the golden rule of committing: all the writing must be done using an imperative mode like [FIX] and not using past tense like [FIXED].

Define a template

Having all of the previous considerations in mind, the team should define a template. After discussing what should be included in the ticket as the description and syntax of the titles, your team will have a clear idea of how they should commit code. Here is a sample I use for my current project:

Image 2: Sample of a git commit structure

All of these good practices will be useful in many situations that software teams often face.

Help during the review process

When doing code reviews, we have to check all of the new code that is introduced. Having a clean commit history with descriptive titles and useful messages helps to understand the reason behind the changes. Also, as the reviewer will have access to the description with a link to an external tool, he/she is able to understand the full context of the change.

Help when handling merge/rebase operations

This is my favorite. Every time we have to deal with rebase operations it is extremely useful to have a clean commit history. This helps to make sure you are moving the right code changes from one branch to another. Without this information, it will be almost impossible to do without having assistance from the original author, which as we know may be impossible sometimes.

Help when writing release notes

At the moment of preparing the release notes, a clean commit history is useful so we can see all changes that are part of our release branch (or any branch we are using to prepare a release).

Help during maintenance

Finally, one of the most important reasons for having a clean commit history is so that we can understand changes that were introduced a long time ago. Often as software engineers, we are in a situation of not understanding why a specific change was made or why a piece of code was originally introduced. A well written commit will bring together all of this information, and will give important insight for the new authors to avoid having side effects when performing a change.

Image 3: Sample of a rebase operation for moving code between branches

There are really a ton of recommendations and conventions for good committing. These were just some of them! These are the ones I think are the most basic and need-to-know, and are based on the official Git Man page1.

Let me know if you have any others you would like to add or suggest.

Links:

– [1] https://mirrors.edge.kernel.org/pub/software/scm/git/docs/git-commit.html#_discussion

Music that Moves You

Maria Alejandra Diaz tells us about her experience playing the cello in her homeland of Venezuela, and the hopes she has for the future in her new home, Peru.

What made you decide to study music? Was it a personal decision or something your parents made you do?

Honestly, ever since I was little I’ve been very interested in the fine arts. First it was ballet and contemporary dance. Later I took up painting. Now that I’m an adult, I’ve entered the world of what is known in Venezuela – and internationally – as  “El Sistema,” which was founded by maestro Jose Antonio Abreu. His main objective has been to broaden the horizons – culturally and musically – of the country and get the youth interested in music.

My parents were always the first ones to support me in any initiative I may have had surrounding my curiosity for the fine arts. They supported me in developing other skills not just related to academia, although the decision was always mine to pursue such activities. They encouraged me to stick with such activities and not just abandon them further down the road.

Why the cello and not something else? Did you ever consider pursuing other instruments as well?

At first, I was most interested in the violin. My dad took me to see the Children’s Orchestra of Tachira, which is the province of Venezuela where I’m from, so that I could get a better idea of what instruments caught my attention. From that moment on, I decided I wanted to learn to play the violin.

Later, I auditioned for the Youth Orchestra Foundation “Luis Gilberto Mendoza” which was the base from which El Sistema operated. They evaluated aptitude for rhythm and solfege which is the fundamental basis for every musician. They loan out instruments and let you use their facilities until you can raise enough money to buy your own instrument.

When they asked me which instrument I wanted to play, they told me that there were no violins available to loan out, and they only had cellos and trumpets available. Of course, the trumpet was never to my liking, with the mouthpieces and parting of the lips and everything, and therefore I stayed with the cello, which was the best decision I could have made. It is an instrument with a unique sound and accompaniment and is – according to expert opinions – the sound that most resembles the human voice.

Are there musicians in your family?

My great-grandfather, on my mom’s side, had an orchestra called “Filo Rodríguez y Su Orquesta” in the 1940s. He was a trumpeter and the director. One of his sister’s was also an opera singer in a circus in Italy during the same time. My mother played piano, my father participated in the Christmas masses in choirs and playing, and my sister is an opera singer (Mezzo-soprano).

How old were you when you performed at your first recital? Where was it?

They usually recommend that you start studying music at an early age, not only because it’s common knowledge that children pick up new skills more easily, but above all because of the muscle memory that comes with studying string instruments (violin, viola, cello, bass). Starting early helps your fingers develop accordingly. For example, for the case of the violin, it is best that players have thin, not so large fingers. In my case, I started at 14 years old, and I had my debut 3 months after starting at the Pre-Children’s Orchestra in San Cristóbal – Táchira State.  This was, of course, very ironic because of my age, but it was the group that It started at the same time that I entered “the system” and it was for ages 8 to 12. There was already a Children’s Orchestra, so we could not call ourselves the same thing or play the same pieces as them because of the level of difficulty .

Then I moved quickly to the Children’s Orchestra and then to the Youth Orchestra. It was in the latter that I had the opportunity to play as a guest on 2 occasions with the Simón Bolívar Symphony Orchestra of Táchira. A year and a half later, I formally joined that orchestra.

With the Simón Bolívar Orchestra, I had the opportunity to be a soloist and play with musicians of international renown – such as violist Frank Di Polo, pianist Arnaldo Pizzolante, and guitarist Alirio Díaz, among others.

Who are some musicians (past or present) that you admire?

Among the musicians that I admire most are Jacqueline du Pré and Yo-Yo Ma. With rock, I like Apocalyptica, which is a symphonic metal band formed by 4 cellists graduated from a classical music academy called Sibelius. In the beginning, they covered Metallica songs which then made them famous. Also, 2cellos is a duo of Croatian cellists who make versions of songs by Michael Jackson, Guns and Roses, Jimi Hendrix, and more. It’s really fun to listen to them.

How much time do you dedicate to practicing cello?

In Venezuela, I devoted every afternoon to studying and practicing cello. The Foundation had an academic center where they teach classes in rhythm, theory and solfege, harmony and then classes related to each specific instrument taught by a professional – usually members of the Simón Bolívar Orchestra. All of these classes were distributed throughout the week.

In addition to the classes, there were also rehearsals with the Orchestra, which were everyday, Monday through Friday from 6pm to 9 or 10 at night, depending on the difficulty of the work and the time needed to rehearse. If they were complicated pieces to execute, each leader who was a kind of representative of the group of musicians for a particular instrument, would host additional workshops to go in depth and see the synchronization, harmony, solos, and tuning.

It really was a full-time commitment. But when you like what you do, you’re not aware of how much time it takes.

Do you have plans for the future with regard to music?

Since I moved to Peru, I have not been able to continue with music because when I left the country, I didn’t know where I was going to settle down. Also because of economic issues, I had to sell my cello in Venezuela to help my mother with medical treatment. I’ve been checking with friends who still belong to El Sistema to find out if there are organizations like the ones there which can help me with loaning a cello while I save to buy my own, and also where I could resume my classes. In the near future, I hope to belong to another orchestra and continue enjoying this most beautiful form of art.

The impact of Generation Y on employee, manager and owner roles

by Walter Abrigo, Managing Director of  Santex.

The uncertainties of today’s world

When we look at the news from our own countries and the rest of the world, we can say without a doubt that “stability” no longer exists. Today we are living in a volatile, uncertain, complex and ambiguous world. The only certainty we have is that there are no certainties.

Since there aren’t, let’s think about the Panama Papers. How many high-level people were completely unauthorized? Or, for example, who would have thought that Iran would have a predominant role in the problem of Syrian refugees when it was once the world’s number one enemy and nobody sat down to talk to that country? Nor did they talk about the presence of a Latin American Pope among other contemporary events.

The skills of the workforce in the upcoming years

From the point of view of hard skills, by 2030, 600 million new jobs will have been created in the world. Jobs established as a result of the Technology Revolution, characterized by the use of megadata (IoT), 3D printers and robots in the manufacturing process.

In terms of soft skills, by 2020 in Latin America, 50% of the workforce will be millennials (born between 1984 and 1991) and their values will now be the values of the world, including:

  • Significant and transcendent lives
  • Happiness not being the result of a specific job
  • Care for the environment
  • Fewer sexual taboos
  • Extreme transparency
  • Want of  instant feedback

The skills of companies in the next years

Given so much uncertainty and the speed of change, there are three critical characteristics to develop in order to survive in the world of work in the upcoming years:

  • Adaptability: Facing a changing world, adjusting and accommodating (subsistence)
  • Agility: Being flexible and elastic in order to redesign processes to meet changing requirements without losing order
  • Alignment: Being attentive to permanently meet the needs of all stakeholders, always considering the long-term

Given these characteristics, it will be of overriding importance that organizations:

  • Incorporate technology and permanently redesign their processes
  • Promote collaborative work and networking (both externally and internally within the organization)
  • Encourage self-supervision with a goal-oriented culture

The characteristics of the employees, managers and organizations of the future

These new features mean that traditional practices lose their validity and if employees, managers, and organizations intend to surf these new waves, collaborative technologies will be the central nervous system of the future of work regardless of the role we play. Here are some characteristics to keep in mind:

Employee:

  • Flexible work environment
  • Customized work
  • Shared information
  • Can become a leader
  • Leverages new ways to communicate and collaborate
  • Changes from being knowledgeable to wanting to learn new skills
  • Learns and teaches at will

Manager:

  • Needs to be a leader
  • Supports  from front and center
  • Understands technology
  • Leads by example
  • Embraces vulnerability
  • Believes in sharing and collective intelligence
  • Is a fire starter
  • Recognition in real time in terms of feedback and commitment
  • Aware of personal limits
  • Adapts to the employee of the future

Organization:

  • Has small teams of employees distributed globally
  • Intrapreneurship
  • Connected workforce
  • Always operates as a small company
  • Focuses on what you want, not what you need
  • Adapts quickly to change
  • Creative ecosystems
  • Runs in the cloud
  • Women have senior management roles
  • More horizontal and with decentralized decision-making
  • Telling stories
  • Democratization of  learning and teaching
  • Moving from profit to prosperity

What we are doing at Santex to adapt and align ourselves

Stepping in line with the above, governance at Santex responds to a holacratic system and management is Agile.

Holacracy is a self-management practice for running purpose-driven, responsive companies.

Specifically, these are the actions we develop to try  to be adaptable, agile and aligned.

  • Creation of Agile Management Teams that intervene in the 3 core processes of the company
  • Implementation of an aggressive Incentive Policy that distributes up to 30% of profits among employees
  • Creation of a Cultural Advisory Board composed of collaborators whose objective is to help promote the values of the company: Participation, Courage, Passion, and Trust.
  • Creation of a Technical Advisory Board composed of collaborators whose objective is to outline the company’s technological future.

Sources

7 Habits of Highly Effective People

By Jose Meyer – Project Manager at Santex

Want to give up smoking? Lose weight? Maybe start a course or embark on a career you always wanted to start? These are some of the promises that we usually make, for example at the beginning of a new year.

We have renewed hopes, enthusiasm, joy! But usually it’s not enough, and many of these desires remain unresolved. It is said that only 8% of the people meet or achieve these self-imposed goals or resolutions made at the beginning of the year (or at any other time).

One of the causes of this can be attributed to the lack of habits that allow us to do what is needed, or on the contrary, to quit the habits that don’t allow us to move forward.

A habit is made up of 3 parts:

  • The stimulus or signal we receive to do something.
  • The routine or activities we do.
  • The reward we receive for having completed those activities.

Identifying these components is a great first step in being able to start doing or stop doing whatever we want.

In the book “7 Habits of Highly Effective People” the author, Stephen Covey, summarizes years of study and research on the habits that we need to incorporate to improve our efficiency in both professional and personal life.

Habit 1: BE PROACTIVE

Complaining, blaming the government, blaming our parents or other people about our fate is a characteristic of REACTIVE people. These people live their lives according to the circumstances that surround them, or given certain conditions.

On the other hand, PROACTIVE people are conditioned by their own decisions. They take responsibility for their actions and accept the consequences of what they have decided to do.

A reaction occurs when a stimulus is received. Human beings have the FREEDOM to choose their answers. Proactive people put that freedom to use.

Making an analogy with the IT world, the materialization of this habit would be the recognition of ourselves as the PROGRAMMERS of our lives.

Habit 2: START with the GOAL in MIND

The creation process is made up of 2 parts; all things are created 2 times. The first creation is the mental creation, then comes the materialization – the physical creation. Habit 2 is about mental creation, visualization.

This habit refers to the creative process of imagining what we want to be, where we want to go. Imagine what the result will be of the activities we want to carry out.

Following the analogy from the IT world, this habit refers to designing or creating the PROGRAM of our life.

Habit 3: FIRST things FIRST

When someone asks us what the important things in life are, most would say health, family, friends, and work matter the most. But then in the practice of our daily routines, we do things that do not go according with what we think or want it to be.

For example: We say that health is important, but we do not dedicate time to exercising, and/or we continue to consume unhealthy “foods”. We say that family and friends are important to us. However, we do not always dedicate the time necessary to preserve, understand, or make the relationship grow. In the work environment we often do not dedicate the time needed to renew ourselves and continue learning in order to keep ourselves “up to date”.

The third habit refers to defining priorities in all the orders of our lives and in all the roles that we as human beings must fulfill (parents, spouses, children, brothers, professionals, friends, neighbors, etc).

The relationship between “The Urgent” and “The Important” is raised with this principle. We should make time for the things that are “Important and Not Urgent”. These are the things we plan based on what we want to develop in our lives, and as a result should dedicate less time to the “Not Urgent – Not Important” or “Not Important – But Urgent”.

To end the IT analogy, this habit refers to EXECUTING the designed program. It has to be executed based on the priorities we have designed.

Habit 4: Think WIN – WIN

This habit is about seeking mutual benefits. We should share profits, knowledge, recognition, etc., especially in those relationships that we are interested in maintaining over time.

“Think Win-Win” is not an easy task. In general, we are prepared for the type of relationship Win – Lose. If the other wins, that means that we lose. We should stop believing that.

Thinking win-win requires the extra effort and creativity to look for mutually beneficial alternatives and solutions.

Habit 5: Search to UNDERSTAND First, BEFORE Being UNDERSTOOD

Most people listen to respond, but we do not listen to understand, and that is a big issue.

There are different levels of listening – each one depending on the degree of attention we pay to what the other person is telling us. In order to understand, we should develop what is known as “Empathic Listening” which consists of putting all of our attention and intention into trying to understand what the other person need to express to us.

For example, in the case of the development of a new product, the listening would be before manufacturing it in order to understand the needs of our future consumers. In essence, it is to get out of our own way of thinking and into the shoes of the other person.

It is similar to the fact that to be able to provide a prescription, a doctor must diagnose the patient. This means he/she must understand what is happening to us.

This habit responds to a deep and perhaps unconscious human longing to feel understood.

Habit 6: SYNERGY

This habit is based on creative cooperation. Synergy suggests that through the collaboration of a group of people, we can obtain greater and better results than working individually. “Everything is greater than the sum of its parts.”


Something very interesting about this habit is that it raises the differences between people. Not only should these differences be accepted, but also VALUED. Strength is obtained from differences and not similarities.

Two people with 10 years of work experience working together at the same time, sharing knowledge, have in fact 20 years of experience. There may be 5 primary colors, but by combining them we can obtain many more colors.

Synergy requires developing common objectives and a sense of mission: “Neither your idea nor mine, but an improved common idea”.

Habit 7: SHARPEN THE SAW

This principle refers to the renewal and continuous improvement of the body, mind, and spirit, social relationships (friends, family, neighbors, etc) and emotional relationships (relationships with ourselves). This well-applied habit generates or allows the development of the other 6 habits. Basically, it states that we must keep in mind the objective of continuously improving.

Although the concept of sharpening the saw refers to the integral improvement of ourselves, we can take an example of what happens in the world of software development. It is well known that one language or framework can become obsolete in a couple of years, and if programmers do not update themselves, they also become obsolete. This leaves them with less employability in the labor market.

CONTINUITY OF MATURATY 

During the development of habits, the author speaks of “Private Victory” and “Public Victory.”

The first three habits that refer to the private victory are habits that are achieved individually: be proactive; start with the goal in mind and prioritize.

The following three habits refer to the Public Victory: Think Win-Win – seek to understand before being understood and to synergize are habits that involve the relationships with other people.

To achieve the Public Victory, the author indicates that private victory should be achieved first. With the private victory we achieve the “Independence” that basically refers to being responsible for ourselves, our actions, and their consequences.

With the public victory we achieve the “interdependence”, needed to achieve, improve,  and grow through the relationship with other people.

With both victories, we leave aside the “dependence” of other people who rule over the state in which our life is located.

Reference:

  • The 7 Habits of Highly Effective People,  Stephen Covey, 1989

Deep learning and vision, from simple manipulation to image classification: Part 2

Introduction:

After we revisited some basic concepts related to computer visión in our previous post, it is time to move forward and explore more sophisticated algorithms that will recognize either a dog or cat in a given image.

Through this post, we will work with the Dogs vs Cats problem from Kaggle and its data, which can be found here. You’ll need to register with Kaggle in order to download the train and test data.

After you register and download the data, we’ll perform an exploratory analysis and then build, train and evaluate a convolutional neural network for binomial classification. The model will output 0 or 1 for the cases when it determines that the image contains a dog or cat respectively.

[Step 1] Data exploration:

As stated before, data exploration is -most of the time- the first step before we even try to come with preliminary experiments. By just looking at the files in each of the files train.zip and test1.zip we’ve downloaded, we can spot the following details:

Table 1: Initial dataset observations

As our test set is not labeled, it will not be possible for us to use it for getting performance metrics. The files will, therefore, be only used to generate the final submission file for the Kaggle judge.

Another important observation we can make by opening some of the images in the test and train sets is that they seem to be different in size and aspect ratio. In order to confirm this, we’ll randomly plot and compare some of them.

Snippet 1: Randomly plot images from the train set

train_path = "data/train"
images = glob.glob(os.path.join(train_path, "*.jpg"))

plt.figure(figsize=(16, 8))
for index in range(6):
    plt.subplot(2, 3, index+1)
    img_index=int(np.random.uniform(0, 24999))
    plt.imshow(plt.imread(images[img_index]))
    plt.title(images[img_index])

Figure 1: Sample images from the training set

As we run the above script several times, we observe that our intuition was right: images differ from each other in size and aspect ratio. Normalization seems to be needed but immediately several questions arise: What would be the size we will use for resizing and normalizing all the images so they can later be used to train our model? Wouldn’t the new size need to be determined so it works for both larger and smaller images? Finally, what proportion of images are small, medium or large?

To address those questions, we prepare the following script to get the distribution over height and width (in 100-pixel ranges) for each image in the train set:

Snippet 2: Distribution oversize in the training set

max_w=0
max_h=0
min_w=2048
min_h=2048

arr_h=[0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
arr_w=[0, 0, 0, 0, 0, 0, 0, 0, 0, 0]

for img_index in range(len(images)):
    img=Image.open(images[img_index]).size
    img_w=img[0]
    img_h=img[1]
    
    arr_w[int(img_w / 100)-1] += 1
    arr_h[int(img_h / 100)-1] += 1

    if img_w > max_w: max_w = img_w
    elif img_w < min_w: min_w = img_w if img_h > max_h: max_h = img_h
    elif img_h < min_h: min_h = img_h

print("Max Width: %i - Min Width: %i \nMax Height: %i - Min Height: %i" % (max_w, min_w, max_h, min_h))

If we plot the arr_w and arr_h vectors containing the number of images with width and height ranging from 0 to 1,000 pixels (in 100-pixel intervals), we observe that the majority of them are smaller than 400 x 400 pixel.

Figure 2: Height and width distributions

We can now come up with a strategy for resizing and padding our images as the only preprocessing task we’ll do before training our convolutional neural network. The resizeImg and padImg functions will maintain the original aspect ratio for each image while padding if necessary for images with different aspect ratios:

Snippet 3: Resizing and padding functions

resize_default=64

def resizeImg(image):
    
    img_w=image.size[0]
    img_h=image.size[1]
    slot = int(img_w/100) +1 if img_w > img_h else int(img_h/100) +1 
    
    if slot!=0:
        if img_w >= img_h:
            img = image.resize((resize_default,int(resize_default*img_h/img_w)), Image.ANTIALIAS)
        else:
            img = image.resize((int(resize_default*img_w/img_h),resize_default), Image.ANTIALIAS)
        
    return img;

def padImg(image):
    
    img_w=image.size[0]
    img_h=image.size[1]
    
    if img_w > resize_default or img_h > resize_default:
        if img_w >= img_h:
            new_size = (img_w, img_w)
        else:
            new_size = (img_h, img_h)
    else:
        new_size = (resize_default, resize_default)
        
    img = Image.new("RGB", new_size)
    img.paste(image, (int((new_size[0]-img_w)/2),int((new_size[1]-img_h)/2)))
        
    return img;


#testImage = Image.open(images[int(np.random.uniform(0, 24999))])
testImage = Image.open(images[468])
resized = resizeImg(testImage)
padded = padImg(resized)

plt.figure(figsize=(12, 8))
plt.subplot(1, 3, 1)
plt.imshow(testImage)
plt.title("Original")
plt.subplot(1, 3, 2)
plt.imshow(resized)
plt.title("Resized")
plt.subplot(1, 3, 3)
plt.imshow(padded)
plt.title("Padded")

Calling both functions will have the following output:

Figure 3: Padding and resizing of images

All images will be resized to 64×64 pixels and if padded vertically or horizontally if necessary. We can batch process all images as a preliminary step or include the functions right before we provide the samples to the trainer when fitting the model.

[Step 2] Building the convolutional neural network:

Up to this point, we’re familiar with convolutions for image processing. We’ve also explored the data we have available and decided that padding and resizing are needed in order to provide our model a normalized input pattern. The 64×64 pixel image equals to 4,096 features (input neurons) we need to fit into a 2-class classifier. It means that for every 64×64 pixel image we feed into the convolutional network, it’ll try to predict whether the input data belong to the classes cat or dog.

In addition to the two functions we’ve already seen for resizing and padding, we’ll need some other ones before we train the network. The get_label and getXYBatch functions shown in Snippet 4 are explained below:

Get_label: as we’ll get an output vector for every input pattern (or image), it will have a 2-element vector shape. There are only two possible values for the resulting vector: [0, 1] and [1, 0]. The first one will count as “cat” whereas the second one as “dog” in terms of the result the network is predicting.

getXYBatch: given our computer don’t have infinite memory, allocating all the 25,000 images for training is just not possible. We’ll resize and pad batches of 60-to-500 images and then feed the trainer with them in the training steps.

Snippet 4: get_label and getXYBatch functions

# extract labels
# positives = [1, 0], negatives = [0, 1]
def get_label(path):
    if path.split('/')[-1:][0].startswith('cat'): 
        return np.array([1, 0])
    else:
        return np.array([0, 1])

def getXYBatch(X_input, Y_input, batch_size):   
    X_array = np.array(padImg(resizeImg(Image.open(X_input[0])))).reshape([-1]) / 255
    Y_array = Y_input[0]

    choice = np.random.choice(range(len(X_input)), batch_size, replace=False)
    for item in choice:
        tmpimg = np.array(padImg(resizeImg(Image.open(X_input[item])))).reshape([-1]) / 255
        X_array = np.vstack((X_array, tmpimg))
        Y_array = np.vstack((Y_array,Y_input[item]))

    X_array = X_array[1:]
    Y_array = Y_array[1:]
    
    X_array = X_array.reshape([-1,resize_default,resize_default,3])
    
    return X_array, Y_array;

Now we split the train set into two parts for actual training but also for validation. We’ll use a 10% of the training images to measure how well the model is performing after, let say, 100 iterations. The following code will do it for us:

Snippet 5: Splitting the training set

train_path = "data/train"
images = glob.glob(os.path.join(train_path, "*.jpg"))
random.shuffle(images)

# extract pixels
data_images = images
        
data_labels = np.array([get_label(p) for p in images])
data_labels_out = np.argmax(data_labels, 1)

print("Positive samples: %i\nNegative samples: %i \n" % (len(data_labels_out)-np.count_nonzero(data_labels_out)
                                                      , np.count_nonzero(data_labels_out)))
#Split Data Sets
X_train, X_test, y_train, y_test = train_test_split(data_images, data_labels, test_size=0.2)
y_train_out = np.argmax(y_train, 1)
y_test_out = np.argmax(y_test, 1)

Finally, before jumping into the model’s code itself (assuming we’re excited about it). We’ll define some convenience functions to simplify the layers construction:

dropout: turn off hidden neurons given a probability (only in the training phase).
weight_variable: variables for the neurons’ weights.
bias_variable: variables for the neurons’ biases.
conv2d: convolution between the input and weights, with strides 1 and padding ‘SAME’
max_pool_2x2: max pooling operation, keeps only the maximum elements after each convolutional layer.

Snippet 6: Common tensorflow methods

def dropout(x, prob, train_phase):
    return tf.cond(train_phase, 
                   lambda: tf.nn.dropout(x, prob),
                   lambda: x)

def weight_variable(shape):
  return tf.Variable(tf.truncated_normal(shape, stddev=0.1))

def bias_variable(shape):
  return tf.Variable(tf.constant(0.1, shape=shape))

def conv2d(x, W):
  return tf.nn.conv2d(x, W, strides=[1, 1, 1, 1], padding='SAME')

def max_pool_2x2(x):
  return tf.nn.max_pool(x, ksize=[1, 2, 2, 1],strides=[1, 2, 2, 1], padding='SAME')

Now let’s build the layers of the network. Our model will have an input layer followed by convolution and max-pooling layers. In the last part of the network architecture, we will flatten the feature maps and have a fully connected layer. A representation of the model is shown in Figure 4.

Figure 4: Neural Network Architecture

We define two x and y variables for the 64×64 pixel images. As they use the RGB schema (3 channels), the final shape for the input layer will be 64x64x3.

Snippet 7: Network implementation

sess = tf.InteractiveSession()

# tf Graph Input
x = tf.placeholder(tf.float32, [None,64,64,3]) 
y = tf.placeholder(tf.float32, [None, 2])

# dropout placeholder
keep_prob = tf.placeholder(tf.float32)

# train flag placeholder
train_phase = tf.placeholder(tf.bool) # For Batch Normalization

# Set model weights
W1 = weight_variable([3, 3, 3, 32])
b1 = bias_variable([32])

W2 = weight_variable([3, 3, 32, 64])
b2 = bias_variable([64])

W3 = weight_variable([3, 3, 64, 64])
b3 = bias_variable([64])

W4 = weight_variable([16 * 16 * 64, 512])
b4 = bias_variable([512])

W5 = weight_variable([512, 2])
b5 = bias_variable([2])

# hidden layers
conv1 = tf.nn.relu(conv2d(x, W1) + b1)
maxp1 = max_pool_2x2(conv1)

conv2 = tf.nn.relu(conv2d(maxp1, W2) + b2)
#maxp2 = max_pool_2x2(conv2)

conv3 = tf.nn.relu(conv2d(conv2, W3) + b3)
maxp3 = max_pool_2x2(conv3)

# fully connected
maxp3_flat = tf.reshape(maxp3, [-1, 16 * 16 * 64])

full1 = tf.nn.relu(tf.matmul(maxp3_flat, W4) + b4)
drop1 = tf.nn.dropout(full1, keep_prob)

#output
output = tf.matmul(drop1, W5) + b5
softmax=tf.nn.softmax(output)

loss = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(logits=output, labels=y))

all_variables = tf.trainable_variables() 

As describing each function and method used will be tedious and make this post super long, feel free to browse in the Tensor Flow official documentation those you are interested in: https://www.tensorflow.org/api_docs/

You may also want to revisit some concepts related to earning and optimization such as Loss Functions, Stochastic Gradient Descent and Cross Entropy.

[Step 3] Training time:

Now we just need to define some hyperparameters and let the trainer fit the model to our training data. We’ll display the model accuracy after every 50 steps. Running the snippet below will show the training progress as shown in Figure 5.

Snippet 8: Trainer

# Hyper-parameters
training_steps = 2000
batch_size = 500
display_step = 100

# Mini-batch Gradient Descent
training_accuracy = []
training_loss     = []

for i in range(training_steps):
    
    X,Y = getXYBatch(X_train,y_train,batch_size)
    
    batch_accuracy, batch_loss, _ = sess.run([accuracy, loss, train_step],
                                             feed_dict={x:X, y:Y, train_phase: True, keep_prob: 1.0})
    training_accuracy.append(batch_accuracy)
    training_loss.append(batch_loss)
    # Displaying info
    if (i+1)%display_step == 0 or i == 0:
        print("Step %05d: accuracy=%.4f\tloss=%.6f\tlearning rate=%.6f" %
              (i+1, batch_accuracy, batch_loss, learning_rate.eval()))

save_path = saver.save(sess, "./saved/model2K .ckpt")
print("Model saved in file: %s" % save_path)      
        
plt.figure(figsize=(10,4))
plot_titles = ["Training accuracy", "Training Loss"]
for i, plot_data in enumerate([training_accuracy, training_loss]):
    plt.subplot(1, 2, i+1)
    plt.plot(plot_data)
    plt.title(plot_titles[i])

Figure 5: Progress while training

We can also plot the accuracy and loss at each training step. In an ideal scenario, the accuracy will increment over time whereas the loss will decrease.

Figure 6: Training accuracy and Loss

[Step 4] Using the model with different images:

Our final test consists of using the model with a completely new image that the model hasn’t seen before. We can browse for cats or dog on the internet and pass the images to the classifier using the following code:

Snippet 9: Using the model

y=tf.nn.softmax(output)
test_img = Image.open(X_test[int(np.random.uniform(0, len(X_test)))])

input_array = np.array(padImg(resizeImg(test_img))).reshape([-1]) / 255
input_array = input_array.reshape([-1,64,64,3])

softmax=tf.nn.softmax(output)
prediction = sess.run(tf.argmax(y, 1), feed_dict={x: input_array, train_phase: False, keep_prob: 1.0})
print("Predicted: " + ("Cat" if prediction[0]==0 else "Dog"))
test_img

Figure 7: Model output with an unseen image:

Hopefully, the model will predict accurately the class (cat or dog) for each image we input. However, there are several other techniques we can use from this point in order to make a more precise model.

Discussion:

In this post, we’ve built a convolutional neural network model capable of classifying images based on if they contain a cat or dog. While we didn’t revisit all the term and concepts required to fully understand what we coded and why it’s a good starting point to see how these techniques can be used in real-life scenarios. Have you ever seen a captcha asking you to click on images containing, let’s say, cars in order to verify you are not a bot? Can you think of other possible use cases for this type of binary classification?

Find the full code of this post at: https://github.com/gariem/samples/tree/master/meetup/santex-machinelearning/cats-dogs

Deep learning and vision: From simple manipulation to image classification – Part 1

Introduction:

When the MANIAC I computer defeated a human in a chess-like game for the first time in 1956, it created a fork in history and the idea of machines being able to complete tasks by replicating how the human brain works started to gain hope. Those, however, were tough times to even achieve discrete performance in other tasks due to the lack of data and computing power available.

Since then, a series of the so-called “AI Winters” took place one right after another and  the dream of computers performing at similar levels to humans almost had vanished. It was only until the beginning of 2005 that AI started to regain attention with Deep Learning being a singular force propelling its growth.

Today, companies are pouring billions of dollars into AI development and intelligent machines continue to take place in real-world activities every day.

In this post series, we will review basic concepts about image manipulation, convolutional neural networks, and Deep Learning. We will then dive deeper into the computer vision field and train a Convolutional Neural Network to recognize cats and dogs in arbitrary images, all of this using the Python programming language, TensorFlow, and several other convenience packages. If you are new to tensorflow, you can browse through its site and examples at https://www.tensorflow.org

[Getting ready] Setting up our environment

Through this tutorial we will make use the python programming language as well as several other packages and tools including Anaconda (conda) as our environment manager. Follow these instructions to get our environment ready:

  1. Download and install Anaconda. It is available for free download at https://www.anaconda.com/download/
  2. Once you’ve Anaconda installed, you will need to create a conda environment and add some packages to it. Let’s call it “deeplearning”. The following commands will complete the task:
  • Download and install Anaconda. Make sure it is registered in your PATH environment variable.
  • Update Anaconda packages: conda update --all
  • Create an Anaconda environment: conda create -n deeplearning python=3.6 jupyter scikit-learn scikit-image
  • Activate the environment: source activate deeplearning
  • Update setuptools: pip install --upgrade -I setuptools
  • Find the right tfBinaryURL for you in this url and run: pip install --upgrade tfBinaryURL

Now your environment should be ready. Let’s test it by running the Jupyter Notebook and executing a simple hello world command:

  • Open a terminal and run: jupyter notebook
  • In the browser window opened by the Jupyter server, create a new python 3 notebook using the buttons at the top right section of the page.
  • In the new notebook input box, enter print("hello world") and then press the shift and enter keys to execute the command. You should see the “hello world” message printed right after the command input box.
  • You can also give a name to your notebook by editing the default “untitled” name in the top left section.

Figure 1: Notebook with “hello world” message

[Warm up] Image processing: Filters and Convolutions

As we will be training and using a “convolutional” neural network, it is a good idea to understand why are those called that way. So before we build our CNN model, we will recap some concepts needed in image processing.

A convolution is basically a mathematical operation of two functions having the third one as the result. Convolutions are applied in several fields including image processing and computer vision.

In the field of image processing, a convolution matrix is used for image manipulation like blurring, sharpening or edge detection. The original image is treated as a matrix with values from 0 to 255 according to the color intensity in each pixel. For grayscale images, this matrix will have only two dimensions WxH (Width x Height). However, for color images using the RGB scheme there will be a third dimension and therefore the matrix will become a structure with shape WxHx3 (Width x Height x 3 RGB Channels).

Image manipulation and convolutions in practice

Despite the formal definition and all maths behind convolutions for image processing, we can understand it as a relatively simple operation similar but not equal to matrix multiplication. Let’s see two classic examples with a grayscale image:

Image Equalization:
Before we start with convolutions, let’s warm up doing some basic image manipulation by equalizing a grayscale image.

The image on the left has been acquired with a sensor (camera or telescope) and suffers from over exposition. Therefore, it looks like there is too much lightness in the whole image. We need to enhance the image up to the point where it looks like the one in the right side below:

Figure 2: Galaxy image before and after equalization

Performing exploration of the data we have is a recommended practice in almost any discipline. As we are given an image, calculating and visualizing its histogram seems to be a common task to start with. To obtain our grayscale image histogram we just need to count how many pixels do have an intensity of 0, 1, 2 and so for up to 255, where 0 is a total black pixel and 255 a complete white one. Let’s code it:

# read the image in grayscale mode “L”
img_matrix = imageio.imread('images/galaxia.jpg')
img = Image.fromarray(img_matrix, 'L')


# count pixels at each value in the 0-255 range
rows, cols = img_matrix.shape
histogram = np.zeros(256)
for x in range(0, cols):
   for y in range(0, rows):
     histogram[img_matrix[y, x]] = histogram[img_matrix[y, x]] + 1


# plot the histogram using pyplot
plt.figure(figsize=(14, 4))
ax = plt.subplot(1, 2, 1)
ax.bar(range(256), histogram)
plt.title("Image histogram")

After running the script above, we can see the histogram with a notorious deviation to the right. The total of the pixels have values higher than 100. It is our goal to make the histogram look more distributed or, more properly, equalize it.

Figure 3: Image histograms before and after equalization

The snippet below shows a simple algorithm to achieve the histogram equalization:

histogram_eq = histogram / (rows * cols)
accum = np.zeros(256)
accum[0] = histogram_eq[0]
for i in range(1, 255):
   accum[i] = accum[i - 1] + histogram_eq[i]
image_new = np.zeros((rows, cols), dtype=np.uint8)
for x in range(rows):
   for y in range(cols):
     image_new[x, y] = np.abs(np.floor(255 * accum[img_matrix[x, y]]))

Now, to visualize the new equalized image, we just need to convert the image_new array back to a grayscale image. Try it yourself!.

Want a challenge? You can try to apply a similar algorithm to equalize the following colorful image. While the principle is similar, one cannot just compute and equalize the three RGB channels independently.

Figure 4: Overexposed colorful image

Test yourself and show us what approach you used to address this challenge. We’d love to see your code and discuss your solution!

Image convolutions:

What happened with our previous exercise is that we did a linear transformation on an image by simply modifying how distributed the gray tones were. However, that’s not all we can do with an image. Another modification we can do is to replace each pixel with the mean value of its neighbors which is what we call median filter. A median filter is a non-linear filter often used to reduce noise in images. You can read more about the median filter in this article in Wikipedia.

As we previously stated, a convolution is an operation between two functions to obtain a third one. In the image processing domain, the first function will be our original image and the second one a convolution matrix (also called Kernel or Filter matrix) with shape NxN where N is an even number frequently having values of 3, 5 or 7.

The animation below shows how we compute an output matrix as a result of performing a convolution with an input image matrix and a 3×3 kernel:

Animation 1: Convolution with a 3×3 Kernel

To obtain the modified image (in the shape of a numerical matrix containing grayscale values), we start by taking a subsection of the input image with the same shape as our Kernel, then we perform element-wise multiplications between our input sample and kernel. Finally, we add the 9 products and divide the result by the sum of all values in the kernel. The initial 320 value for our output matrix is, therefore, the result of the following operation:

But, why do we have a top row and first column with zero values? The answer is that, in order to be able to perform the element-wise operation described before for the input elements at the border of our input matrix, we need to pad the original image with as many rows and columns as the size of our kernel matrix minus one, divided by two. So in our example, our image will be padded with 1 column and 1 row because our kernel matrix size is 3: (3-1)/2

An important element is also introduced here and must be remembered for later: the strides. As we will perform the operation in each pixel, we need to follow some approach to visiting all the pixel in the original image such as a sliding window. The stride will determine how many pixels we move right and bottom in each step. Most of the time the strides are 1 for both horizontal and vertical displacements.

You may also be wondering how does one determine the shape the convolution kernel will have? There are several well-known kernels you can use for different purposes. Below you can see some of them and the result they produce when applied:

Now let’s code it. Try yourself this simple function for padding an image and performing a convolution with any kernel matrix:

def filter_simple(source, kernel, mask_rows, mask_cols):
   padding_rows = int((mask_rows - 1) / 2)
   padding_cols = int((mask_cols - 1) / 2)
   rows, cols = source.shape
   padded = np.zeros((rows + 2 * padding_rows, cols + 2 * padding_cols), dtype=np.uint8)
   padded[padding_rows:padding_rows + rows, padding_cols:padding_cols + cols] = source
   result = np.zeros((rows + 2 * padding_rows, cols + 2 * padding_cols), dtype=np.uint8)
   for i in range(padding_rows, padding_rows + rows - 1):
     for j in range(padding_cols, padding_cols + cols - 1):
       aux = padded[i - padding_rows:i + padding_rows + 1, j - padding_cols: j + padding_cols + 1]
       out_value = 0
       for x in range(mask_rows):
         for y in range(mask_cols):
           out_value = out_value + (aux[x,y] * kernel[x,y]) / np.sum(kernel)
       result[i, j] = out_value
  result = result[padding_rows:padding_rows + rows, padding_cols: padding_cols + cols]
  return result

Now you can call the function it as simple as:

# define our kernel
kernel_blur = np.matrix([[1,1,1],[1,1,1],[1,1,1]])
# kernel_edge = np.matrix([[-1,-1,-1],[-1,9,-1],[-1,-1,-1]])
image_new = filter_simple(img_matrix,kernel_blur, 3, 3)

Figure 5: Simple edge detection with convolution filters

There are a couple of places where you can also improve the code. Have you also noticed that the execution is not so fast? Do you think you can improve it? Motivate yourself and read about optimization for convolution operations. Again, we’d love to see your code and discuss any question you may have.

Most of image processing and numeric libraries in languages such as python also offer ready-to-use optimized functions for 2D and 3D convolutions. Check out this example in the SciPy documentation!

[Finally] Discussion

Convolutions are a key concept to understand before we move to Convolutional Neural Networks as the kernel, strides, and other parameters have a lot of importance when dealing with them. In the next post, we will see how a neural network can learn its own kernels instead of using predefined ones. The ability to perform transformations such as noise reduction and edge detection that filters add is probably one of the most important reasons for CNN to become so popular and accurate.

References:

  1. Bengio, Y. (2016). Machines Who Learn. Scientific American, 314(6), 46-51. doi:10.1038/scientificamerican0616-46
  2. https://www.tensorflow.org/
  3. https://en.wikipedia.org/wiki/Kernel_(image_processing)

 

Finding and fixing security vulnerabilities

By Hernan Amaya – Java Developer at Santex

One of the main pillars of the Information Technology industry is security. Can you imagine what software would be like without security? Everyone could access privileged information everywhere, causing potentially precarious situations. It’s obvious that everybody who owns software wants to be secure against cyber attacks. Developers are always concerned about designing and implementing software that is protected. Yet, no matter what tools or knowledge are at our disposal, no one can be certain that his or her development is 100% secure. That is why once a certain software is stable, it is good to determine whether or not it has any security vulnerabilities. Luckily, nowadays it is possible to use security vulnerability scanners for this purpose.

Our main goal is to share how to use different security scanners. We will find and explain how to fix security vulnerabilities in a web application. Every tool exposed is automatic and straightforward and uses default configurations. We will scan demo.testfire.net. This web application is available for commercial use and free tools. To conclude, we will highlight all the vulnerabilities found in this web application.

AppScan

IBM developed this commercial tool. It runs on Windows and has a trial version that allows you to  scan the web application located at demo.testfire.net.

After installing and running the software in your local environment, select the following options: File -> New -> Regular Scan for accessing the wizard.

AppScan offers three possible exploration methods. The first option is for scanning web applications. The second option is for REST APIs. The third option is for SOAP web services.

Leave AppScan option selected and press the Next button.

Enter https://demo.testfire.net and click Next.

Leave the recorded option selected and click Next.

Scroll and select ‘Complete’ to let AppScan do an integral scan.

Leave ‘Start a full automatic scan’ selected and press Finish. A pop-up will appear asking whether to save the project or not. Click ‘Yes’ to save it or ‘No’ to continue without saving it. The scan will start. Wait until it finishes. You may receive suggestions to change configuration options for improving the scan.

Press ‘Apply Recommendations’ and wait for the new scan phases to finish.

Click on ‘Issues’ to view the results of the analysis.

Open VAS

This tool is Open Source and free. Several operating systems such as Windows and Linux support it. After installing Open VAS follow these instructions:

First, access to the URL where Open VAS is running.

Click the purple icon and then select “Task Wizard”.

Enter demo.testfire.net and press “Start Scan”.

Wait for the task to complete. You can access the results by clicking on the progress bar whenever you want. When it is complete, you will be able to see all the results.

Vega

Vega is a free and Open Source web security scanner. It is a web security testing platform   to test web applications. It runs on Linux and Windows.

After you have installed Vega follow these instructions.

Press Scan -> Start a New Scan. The wizard will start. Enter demo.testfire.net in the Scan Target URI and click Next.

Leave the default modules and click Next. You can examine them and add the ones you consider to be necessary, but adding new modules will require more analysis time.

Leave cookies and authentication identity blank and click Next.

Leave the parameters that are already on the exclusion list. Add the ones you consider important to exclude. Click Finish.

Wait until the scanning process ends. It shouldn’t take long.

Zed Attack Proxy

This tool is free and Open Source as well. It runs on several Operating Systems such as Linux and Windows. After installing it, follow these instructions.

In Quick Start enter https://demo.testfire.net and press start.

This is a summary of the results:

 

 

Conclusion

Security is crucial in software development. Having tools for finding vulnerabilities and suggesting how to fix them is a wonderful benefit.

In this analysis, we scanned demo.testfire.net using several tools. AppScan is a commercial tool that has a desktop and web application version. The technical difference between them is the way you interact with them, and that the latter one’s result is a PDF file. The commercial difference is that with the web application, you are able to pay for each individual scan, while with the desktop application, you have to pay for an entire year.

To sum up, we have discovered that nowadays the most powerful security scanner available is IBM’s AppScan. However, using several free, Open Source scanners is an excellent alternative. Consequently, combining scanners such as Open VAS, Vega and ZAP can be powerful as well.

The VR Experience

By Hernan Senki, Front-End Developer at Santex

These past few days, I’ve been busy visiting the Egyptian pyramids, exploring the depths of the oceans, jumping out of parachutes, shooting off rockets into space, and driving in car races. I’ve been immersed in all these experiences and more, all without leaving my desk, thanks to the Samsung Gear VR powered by Oculus virtual reality headset that Santex lent to try out.

Gear VREven though research for virtual reality, or “VR”, has been going on since the ‘90’s, it’s only in recent years that the mass production of devices has allowed us to experience VR in our households. Companies like Oculus (recently acquired by Facebook), HTC, Sony, and other tech giants are investing large amounts of resources into development because it is, undoubtedly, ‘the wave of the future.’

After updating the Galaxy S7, I put it in the Gear VR, put on my headphones, and was ready to give it a try. I was expecting something like an alpha version experience with lots of errors, but what I got was rather surprising. From Oculus Home, I chose to visit the Egyptian pyramids. Later, I went by the Sydney Opera House and to tell the truth, these videos, images, and experiences feel 100% real. There’s even sound involved that allows you to really travel and feel the experience without leaving your seat.

Tired from traveling, I thought I’d try launching a rocket. In contrast from the earlier experiences, this was neither video nor photo, but rather a very realistic animation which gave the sensation of being immersed in a videogame, minus the aspect of interaction. After sitting behind the controls of the rocket, I then transitioned into being stuck in a bed which appeared to be in a kid’s bedroom – dark and illuminated only lightning bolts entering through the window, creating a terrifying setting where everything was transformed into a nightmare-like experience that I couldn’t escape. The surprises around every turn were shocking and enough to make one jump or scream! I would not recommend it to users with cardiac problems or those who are scared easily.

One of the most highly anticipated experiences for the average user is the virtual reality videogames, which, unfortunately, I didn’t get to try because it required an Oculus control that doesn’t come with the Gear VR package. It allows you to move around, shoot, point, and have greater interaction with the stories. I was always skeptical of these VR videogames, thinking that nothing is more comfortable than a traditional Joystick and a comfy couch, but after trying these other VR experiences, I could probably be persuaded to give it a shot. I would like to try something like Battlefield or Call of Duty and feel fully immersed in the game.

Other Uses

Virtual reality isn’t just fun and games – it can also be applied to different industries like:

Education: It’s a valuable instructive tool. Imagine if instead of reading about places and looking through photos, you could be transported to those places and actually feel surrounded by them. Those are the advances that VR is making possible.Gear VR

Training: VR is currently being used to train a variety of professionals including doctors, automobile drivers, pilots, soldiers and troops, and more. The application of VR in these fields allows trainees to gain valuable hours of practice time while reducing physical risks and costs.

Health: The medical field has not only been using VR to train doctors and surgeons by simulating the operating room, but also to train psychologists and psychiatrists who have reported success cases in treating phobias for patients, such as fear of heights, social phobias, and more.

What Can Be Expected

Virtual Reality is the future, and that future is getting closer and closer. What we are currently witnessing are the first steps. There’s still a lot of room for improvement with both the hardware and software. The applications for VR are currently rather limited in amount and variety, creating an important space for software development companies and audiovisual content generators to work. Without a doubt, the entertainment industry will soon be the tipping point for VR and 360. Soon people won’t be buying from the web, we’ll be going places virtually to select the products we want. E-commerce will become VR-commerce, and it will be the same with concerts, conventions, gatherings, trainings, and more.

It’s a technology that shouldn’t be ignored. It will allow us to travel wherever we want, to live unique experiences that weren’t ever possible without VR. We can interact with family and friends as if they were right in front of us – all in 360º.

Describing what VR is and how it feels is difficult because everyone thinks and reacts differently. The best I can do is to recommend trying the experience and arriving at your own conclusions.