Using scripts to improve your workflow

Development these days is very complicated. You have to contend with many types of technologies. Source code management. Maven or Gradle build scripts. Testing of all sorts. Deployment to various destinations. The permutations are endless. CI/CD (continuous integration & continuous deployment) tools like Jenkins can handle several of these challenges. But sometimes all you need is a simple script to do the job.

For example, on I’ve written different versions of the same script. They all sit on my desktop and I just click on the one that I need when I need it. I may have one that executes Maven to performs unit tests on the whole suite. Or one that only executes tests on a subset. I have also written a script that builds and deploy to a JBoss server without doing the unit tests first. This can save a lot of time when you’ve already executed your tests, fixed code, test, fix code in endless iterations. You don’t want to run those tests all over again. So script can be handy for all kinds of tasks.

Apache Spark

I’m currently learning Apache Spark running in a VirtualBox VM on a Windows host. Needless to say, scripting here is indispensable. I’ve written a script to start the VM in headless mode with a static IP address. Once the VM is up the script launches an SSH session for me.

What is really priceless is the example below of a BASH script that I put together to deploy an application into the Spark Server. This script works well because I just need to double-click on the script and it runs.  What the script does is nothing miraculous except that it simplifies a convoluted process that gets performed often.

As this example. all my scripts are written in BASH and execute in Git BASH from the Windows desktop. The script below will first build the application with code that currently checked-out. Then uploads the resulting JAR file to the VM using the SCP command. Once on the server, the JAR is submitted to Spark using the SSH command to perform a remote CLI call. Finally, once that process is done, another SCP call is made to retrieve the resulting CSV file.

# Change to the project directory
cd /c/dev/projects/apache-spark

# Build the application into a JAR file
mvn clean install

# Upload the file to the VM
scp -i /c/Users/Don/.ssh/id_rsa -P 22 /c/dev/projects/apache-spark/target/jar/SparkApp.jar spark@192.168.0.50:/home/spark/deployment/SparkApp.jar

# Submit the JAR to the Spark server
ssh -i /c/Users/Don/.ssh/id_rsa -p 22 spark@192.168.0.50 '/home/spark/spark/bin/spark-submit -v --class com.sparkinaction.App --master local /home/spark/deployment/SparkApp.jar'

# Retrieve result and copy to project folder
scp -i /c/Users/Don/.ssh/id_rsa -P 22 spark@192.168.0.50:/home/spark/result/sightings.csv /c/dev/projects/apache-spark/target/spark-result/result.csv

For those that don’t work in Linux much, there are two commands that I find indispensable.

SCP – This command copies files between computers using SSH. It’s secure and allows me to log in using an SSH key.

SSH – Everyone knows SSH, but did you know you can use it to execute commands remotely. That’s what I did above to submit the Java application to the Spark server.

I’m a rebel and no one understands me

A couple of weeks ago I found this image on LinkedIn and was immediately in awe. It describes me perfectly. Yet, some managers can be afraid of these types of employees as they can be aggressive achievers. I know I’ve run into issues a few times where I’m not able to achieve my full potential for one reason or another and it’s frustrating. Continue reading “I’m a rebel and no one understands me”

Why we’ll never time travel.

Time travel is possible. Theoretically anyway. Or so I’ve read. The mathematics prove it. The theory of relativity says it so. Yet no one as of today has travelled any when. No one born before today, anyway. But what if time travel were actually possible, beyond theory. What if it was just a matter of attaining more computing power and using the right materials to build the time machine. Perhaps not today, or tomorrow, but let’s say within the next one to two hundred years it had become somewhat practical to do so. How could we prove for sure that someone had achieved it?

Everyone has their story of having met someone that they thought could have come from another time. I remember when I was younger working in a department store. a customer had asked me if we accepted cash. Another time, I received a spam email from a supposed time traveller who needed parts for his time machine. This one I was more inclined to believe was true since the traveller never asked for money, only ship parts that most probably still won’t exist until we build a time machine. Continue reading “Why we’ll never time travel.”

Artificial Intelligence in the News – April 2017

Artificial Intelligence is certainly a strange beast. I’ve read some really interesting articles on the subject recently on how creative and aggressive they can be. Below are the more interesting ones I’ve read.

AI does Knock Knock Jokes

AI Gets Aggressive when playing games

Japanese AI Writes a Novel, Nearly Wins Literary Award

Listen to New Google AI Program Talk Like a Human and Write Music

Can artificial intelligence read a tweet’s sentiment?

Learning the Basics of Machine Learning

Educating yourself on machine learning is a challenging preposition. There is so much content online that it’s easy to get fooled into reading an article that is beyond capabilities. Unfortunately, in many cases, reading and rereading this sort of content is often the only way to crack that nut open. But recently I’ve found some great resources that should hopefully be able to help the uninitiated into getting a foothold into this popular segment of IT. Continue reading “Learning the Basics of Machine Learning”

Taking on Machine Learning

About two months ago I took steps to get into the Machine Learning bandwaggon. It was tough to take that first step for many reasons. The first was the tough decision of choosing the right programming language to learn. Did I want to stick to the JVM and Java or chose another JVM language? Take up Python. Or learn something else. This article from KDnuggets made that decision much harder. Luckily, due to circumstances from a recent project, I decided to turn to Scala. And so far, I haven’t been disappointed. Continue reading “Taking on Machine Learning”

The Best Scala Programming Resources

In the last few weeks that I’ve spent learning Scala I’ve discovered several amazing online resources. So much so that it made it difficult to justify purchasing a book, even though I did buy the much-recommended Programming in Scala title. Every time I needed help in figuring out how to do something, the following websites were indispensable.

Some of these sites are no longer active, but the information they provide is still very much relevant to new and experienced Scala programmers. Continue reading “The Best Scala Programming Resources”